Evaluating ai agent performance

Abstract

A variety of metrics are described for evaluating the performance of artificial intelligence agents, e.g., in the context of user requests and generative model responses within a specific domain, such as physiological monitoring or associated health and wellness coaching, that provides a ground truth for responses to requests. These metrics may be used, e.g., to determine whether and how to deliver responses to a user, as well as for evaluating the performance of underlying generative models, agents, and so forth. In another aspect, a quality matrix may be provided for an agent that compares expected to actual behavior for different classes of user requests.

Description

EVALUATING Al AGENT PERFORMANCERELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 654,286 filed on May 31, 2024, U.S. Patent Application No. 19 / 033,945 filed on January 22, 2025, and U.S. Patent Application No. 19 / 033,966 filed on January 22, 2025, where the entire contents of each of the foregoing applications are hereby incorporated by reference.

[0002] This application is also related to U.S. Provisional Patent Application No. 63 / 515,026 filed on July 21, 2023, U.S. Provisional Patent Application No. 63 / 587,319 filed on October 2, 2023, U.S. Provisional Patent Application No. 63 / 624,921 filed on January 25, 2024, and International Pat. App. No. PCT / US24 / 38851 filed on July 19, 2024 filed on July 19, 2024. The entire content of each of the foregoing applications is hereby incorporated by reference.TECHNICAL FIELD

[0003] The present disclosure generally relates to physiological monitoring systems. Particularly, but not exclusively, the present disclosure relates to user interaction with a physiological monitoring system. Particularly, but not exclusively, the present disclosure relates to generating context specific responses to a query received from a user of a physiological monitoring system.BACKGROUND

[0004] Physiological monitoring systems can provide a user with a rich vein of physiological data and analysis, where a user can monitor metrics such as sleep performance, activity, strain, and recovery, as well as use this information to make informed decisions based on the data and / or metrics. However, with the increasing complexity of such systems, it can be difficult for a user to access and analyze data of interest.

[0005] There remains a need for improved access to complex, data-rich systems such as a continuous physiological monitoring system.SUMMARY

[0006] A variety of metrics are described for evaluating the performance of artificial intelligence agents, e.g., in the context of user requests and generative model responses within a specific domain, such as physiological monitoring or associated health and wellness coaching, that provides a ground truth for responses to requests. These metrics may be used, e.g., todetermine whether and how to deliver responses to a user, as well as for evaluating the performance of underlying generative models, agents, and so forth. In another aspect, a quality matrix may be provided for an agent that compares expected to actual behavior for different classes of user requests.

[0007] According to one aspect of the present disclosure there is provided a computer program product comprising computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, performs the steps of: obtaining a message from a user of a physiological monitoring system, the message including a query associated with the physiological monitoring system; classifying the message into one or more topics, wherein each of the one or more topics is associated with the query included in the message; mapping, using one or more mapping functions, the one or more topics to a set of context specific data, wherein each of the one or more mapping functions obtains context specific data associated with a respective topic of the one or more topics, the context specific data comprising static data, and / or dynamic data associated with the physiological monitoring system; generating a command comprising a context portion based on the set of context specific data and an instruction portion, wherein the instruction portion comprises instructions for a first large language model (LLM) to perform a rephrasing task based on the context portion; providing the command to the first LLM and subsequently obtaining a text-based response from the first LLM, wherein the text-based response comprises a natural language representation of the context portion of the command; and causing a response based on the text-based response to be output to the user.

[0008] According to another aspect of the present disclosure there is provided a method comprising: obtaining a message from a user of a physiological monitoring system, the message including a query associated with the physiological monitoring system; classifying the message into one or more topics, wherein each of the one or more topics is associated with the query included in the message; mapping, using one or more mapping functions, the one or more topics to a set of context specific data, wherein each of the one or more mapping functions obtains context specific data associated with a respective topic of the one or more topics, the context specific data comprising static data, and / or dynamic data associated with the physiological monitoring system; generating a command comprising a context portion based on the set of context specific data and an instruction portion, wherein the instruction portion comprises instructions for first a large language model (LLM) to perform a rephrasing task based on the context portion; providing the command to the first LLM and subsequently obtaining a textbased response from the first LLM, wherein the text-based response comprises a naturallanguage representation of the context portion; and causing a response based on the text-based response to be output to the user

[0009] According to a further aspect of the present disclosure there is provided a system comprising: a wearable physiological monitor including one or more sensors, a first processor configured to obtain a physiological metric for a user based on a signal from the one or more sensors, and a communications interface for coupling with a remote resource; a server coupled in a communicating relationship with the wearable physiological monitor, the server including a second processor configured by computer executable code to: obtain a message from the user, wherein the message includes a query; classify the message into one or more topics, wherein each of the one or more topics is associated with the query included in the message; map, using one or more mapping functions, the one or more topics to a set of context specific data, wherein each of the one or more mapping functions obtains context specific data associated with a respective topic of the one or more topics, the context specific data comprising static data, and / or the physiological metric for the user; generate a command comprising a context portion based on the set of context specific data and an instruction portion, wherein the instruction portion comprises instructions for a large language model (LLM) to perform a rephrasing task based on the context portion; provide the command to the LLM and subsequently obtain a response from the LLM, wherein the response comprises a natural language representation of the context portion; and a user interface configured to present the response to the user.

[0010] According to an additional aspect of the present disclosure there is provided a method comprising: obtaining a message from a user, the message including an ambiguous query; generating a structured query in a query language by providing one or more prompts to a large language model (LLM), wherein the one or more prompts comprise a predetermined instruction, the message from the user, and a schema for the query language; parsing the structured query to generate an abstract syntax tree (AST) representation of the structured query; evaluating the AST representation of the structured query to obtain context specific data related to one or more topics associated with the ambiguous query, wherein the context specific data is obtained using one or more mapping functions identified from evaluation of the AST representation of the structured query; and outputting the context specific data as a resolution to the ambiguous query.

[0011] According to another aspect of the present disclosure there is provided a computer program product comprising computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, performs the steps of obtaining a message from a user, the message including an ambiguous query;generating a structured query in a query language by providing one or more prompts to a large language model (LLM), wherein the one or more prompts comprise a predetermined instruction, the message from the user, and a schema for the query language; parsing the structured query to generate an abstract syntax tree (AST) representation of the structured query; evaluating the AST representation of the structured query to obtain context specific data related to one or more topics associated with the ambiguous query, wherein the context specific data is obtained using one or more mapping functions identified from evaluation of the AST representation of the structured query; and outputting the context specific data as a resolution to the ambiguous query.

[0012] According to a further aspect of the present disclosure there is provided a device comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the device to perform the steps of obtaining a message from a user, the message including an ambiguous query; generating a structured query in a query language by providing one or more prompts to a large language model (LLM), wherein the one or more prompts comprise a predetermined instruction, the message from the user, and a schema for the query language; parsing the structured query to generate an abstract syntax tree (AST) representation of the structured query; evaluating the AST representation of the structured query to obtain context specific data related to one or more topics associated with the ambiguous query, wherein the context specific data is obtained using one or more mapping functions identified from evaluation of the AST representation of the structured query; and outputting the context specific data as a resolution to the ambiguous query.

[0013] According to a further aspect of the present disclosure there is provided a method for dynamically optimizing load on a large language model (LLM). The method comprises identifying a prompt to be provided to an LLM for performing a task related to a physiological monitoring system; obtaining one or more load values indicative of a computational load on the LLM; generating a command comprising instructions for the LLM to generate an output according to an output length criterion, wherein the output length criterion is based on the one or more load values; and providing, to the LLM, the prompt and the command such that a subsequent output generated by the LLM satisfies the output length criterion.

[0014] According to an additional aspect of the present disclosure there is provided a computer program product comprising computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, performs the steps of identifying a prompt to be provided to an LLM for performing a task related to a physiological monitoring system; obtaining one or more load values indicative of a computational load on the LLM; generating a command comprising instructions for the LLM togenerate an output according to an output length criterion, wherein the output length criterion is based on the one or more load values; and providing, to the LLM, the prompt and the command such that a subsequent output generated by the LLM satisfies the output length criterion.

[0015] According to another aspect of the present disclosure there is provided a device comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the device to perform the steps of identifying a prompt to be provided to an LLM for performing a task related to a physiological monitoring system; obtaining one or more load values indicative of a computational load on the LLM; generating a command comprising instructions for the LLM to generate an output according to an output length criterion, wherein the output length criterion is based on the one or more load values; and providing, to the LLM, the prompt and the command such that a subsequent output generated by the LLM satisfies the output length criterion.

[0016] According to another aspect of the present disclosure there is provided a method for generating cross-component responses to user requests. The method comprises obtaining a portion of a natural language message received from a user of a physiological monitoring system, wherein the portion of the natural language message relates to a request from the user in relation to one or more operations performed by the physiological monitoring system; providing, to a large language model (LLM), a prompt operable to cause the LLM to output a code block which encodes a response to the request and is processable by one or more components of the physiological monitoring system to generate one or more component specific representations of the response, wherein the prompt comprises a predetermined instruction, the portion of the natural language message, a language schema for code within the code block, and an object schema related to the one or more operations performed by the physiological monitoring system; and obtaining the code block from the LLM.

[0017] According to a further aspect of the present disclosure there is provided a computer program product comprising computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, performs the steps of obtaining a portion of a natural language message received from a user of a physiological monitoring system, wherein the portion of the natural language message relates to a request from the user in relation to one or more operations performed by the physiological monitoring system; providing, to a large language model (LLM), a prompt operable to cause the LLM to output a code block which encodes a response to the request and is processable by one or more components of the physiological monitoring system to generate one or more component specific representations of the response, wherein the prompt comprises a predeterminedinstruction, the portion of the natural language message, a language schema for code within the code block, and an object schema related to the one or more operations performed by the physiological monitoring system; and obtaining the code block from the LLM.

[0018] According to an additional aspect of the present disclosure there is provided a device comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the device to perform the steps of obtaining a portion of a natural language message received from a user of a physiological monitoring system, wherein the portion of the natural language message relates to a request from the user in relation to one or more operations performed by the physiological monitoring system; providing, to a large language model (LLM), a prompt operable to cause the LLM to output a code block which encodes a response to the request and is processable by one or more components of the physiological monitoring system to generate one or more component specific representations of the response, wherein the prompt comprises a predetermined instruction, the portion of the natural language message, a language schema for code within the code block, and an object schema related to the one or more operations performed by the physiological monitoring system; and obtaining the code block from the LLM.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The foregoing and other objects, features, and advantages of the devices, systems, and methods described herein will be apparent from the following description of particular embodiments thereof, as illustrated in the accompanying drawings. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the devices, systems, and methods described herein. In the drawings, like reference numerals generally identify corresponding elements.

[0020] Fig. 1 shows a physiological monitoring device.

[0021] Fig. 2 illustrates a physiological monitoring system.

[0022] Fig. 3 shows a smart garment system.

[0023] Fig. 4 is a block diagram of a computing device.

[0024] Fig. 5 shows a block diagram of a system for context-driven prompt engineering.

[0025] Fig. 6 is a flow chart illustrating a method for quantifying artificial agent performance.

[0026] Fig. 7 illustrates different metrics that can be calculated.

[0027] Fig. 8 is a flow chart illustrating a method for generating ground truth questionanswer (QA) test sets related to a subject.

[0028] Fig. 9 is a flow chart illustrating a method for quantifying expected behavior of an artificial agent.DESCRIPTION

[0029] The embodiments will now be described more fully hereinafter with reference to the accompanying figures, in which preferred embodiments are shown. The foregoing may, however, be embodied in many different forms and should not be construed as limited to the illustrated embodiments set forth herein. Rather, these illustrated embodiments are provided so that this disclosure will convey the scope to those skilled in the art.

[0030] All documents mentioned herein are hereby incorporated by reference in their entirety. References to items in the singular should be understood to include items in the plural, and vice versa, unless explicitly stated otherwise or clear from the text. Grammatical conjunctions are intended to express any and all disjunctive and conjunctive combinations of conjoined clauses, sentences, words, and the like, unless otherwise stated or clear from the context. Thus, the term “or” should generally be understood to mean “and / or” and so forth.

[0031] Recitation of ranges of values herein are not intended to be limiting, referring instead individually to any and all values falling within the range, unless otherwise indicated herein, and each separate value within such a range is incorporated into the specification as if it were individually recited herein. The words “about,” “approximately” or the like, when accompanying a numerical value, are to be construed as indicating a deviation as would be appreciated by one of ordinary skill in the art to operate satisfactorily for an intended purpose. Similarly, words of approximation such as “approximately” or “substantially” when used in reference to physical characteristics, should be understood to contemplate a range of deviations that would be appreciated by one of ordinary skill in the art to operate satisfactorily for a corresponding use, function, purpose, or the like. Ranges of values and / or numeric values are provided herein as examples only, and do not constitute a limitation on the scope of the described embodiments. Where ranges of values are provided, they are also intended to include each value within the range as if set forth individually, unless expressly stated to the contrary. The use of any and all examples, or exemplary language (“e.g.,” “such as,” or the like) provided herein, is intended merely to better describe the embodiments and does not pose a limitation on the scope of the embodiments. No language in the specification should be construed as indicating any unclaimed element as essential to the practice of the embodiments.

[0032] In the following description, it is understood that terms such as “first,” “second,” “top,” “bottom,” “up,” “down,” “above,” “below,” and the like, are words of convenience and are not to be construed as limiting terms unless specifically stated to the contrary.

[0033] The term “user” as used herein, refers to any type of animal, human or nonhuman, whose physiological information may be monitored using an exemplary wearable physiological monitoring system.

[0034] The term “continuous,” as used herein in connection with heart rate data, refers to the acquisition of heart rate data at a sufficient frequency to enable detection of individual heartbeats, and also refers to the collection of heart rate data over extended periods such as an hour, a day or more (including acquisition throughout the day and night). More generally with respect to physiological signals that might be monitored by a wearable device, “continuous” or “continuously” will be understood to mean continuously at a rate and duration suitable for the intended time-based processing, and physically at an inter-periodic rate (e.g., multiple times per heartbeat, respiration, and so forth) sufficient for resolving the desired physiological characteristics such as heart rate, heart rate variability, heart rate peak detection, pulse shape, and so forth. At the same time, continuous monitoring is not intended to exclude ordinary data acquisition interruptions such as temporary displacement of monitoring hardware due to sudden movements, changes in external lighting, loss of electrical power, physical manipulation and / or adjustment by a wearer, physical displacement of monitoring hardware due to external forces, and so forth. It will also be noted that heart rate data or a monitored heart rate, in this context, may more generally refer to raw sensor data such as optical intensity signals, or processed data therefrom such as heart rate data, signal peak data, heart rate variability data, or any other physiological or digital signal suitable for recovering heart rate information as contemplated herein. Furthermore, such heart rate data may generally be captured over some historical period that can be subsequently correlated to various other data or metrics related to, e.g., sleep states, recognized exercise activities, resting heart rate, maximum heart rate, and so forth.

[0035] The term “computer-readable medium,” as used herein, refers to a non-transitory storage media such as storage hardware, storage devices, computer memory that may be accessed by a controller, a microcontroller, a microprocessor, a computational system, or the like, or any other module or component or module of a computational system to encode thereon computer-executable instructions, software programs, and / or other data. The “computer-readable medium” may be accessed by a computational system or a module of a computational system to retrieve and / or execute the computer-executable instructions or software programs encoded on the medium. The non-transitory computer-readable media may include, but are not limited to,one or more types of hardware memory, non-transitory tangible media (for example, one or more magnetic storage disks, one or more optical disks, one or more USB flash drives), virtual or physical computer system memory, physical memory hardware such as random access memory (such as, DRAM, SRAM, EDO RAM), and so forth. Although not depicted, any of the devices or components described herein may include a computer-readable medium or other memory for storing program instructions, data, and the like.

[0036] Fig. 1 shows a physiological monitoring system. The system 100 may include a wearable monitor 104 that is configured for physiological monitoring. The system 100 may also include a removable and replaceable battery 106 for recharging the wearable monitor 104. The wearable monitor 104 may include a strap 102 or other retaining system(s) for securing the wearable monitor 104 in a position on a wearer’s body for the acquisition of physiological data as described herein. For example, the strap 102 may include a slim elastic band formed of any suitable elastic material such as a rubber or a woven polymer fiber such as a woven polyester, polypropylene, nylon, spandex, and so forth. The strap 102 may be adjustable to accommodate different wrist sizes, and may include any latches, hasps, or the like to secure the wearable monitor 104 in an intended position for monitoring a physiological signal. While a wrist-worn device is depicted, it will be understood that the wearable monitor 104 may be configured for positioning in any suitable location on a user’s body, based on the sensing modality and the nature of the signal to be acquired. For example, the wearable monitor 104 may be configured for use on a wrist, an ankle, a bicep, a chest, or any other suitable location(s), and the strap 102 may be, or may include, a waistband or other elastic band or the like within an article of clothing or accessory. The wearable monitor 104 may also or instead be structurally configured for placement on or within a garment, e.g., permanently or in a removable and replaceable manner. To that end, the wearable monitor 104 may be shaped and sized for placement within a pocket, slot, and / or other housing that is coupled to or embedded within a garment. In such configurations, the pocket or other retaining arrangement on the garment may include sensing windows or the like so that the wearable monitor 104 can operate while placed for use in the garment. United States Pat. No. 11,185,292 describes non-limiting example embodiments of suitable wearable monitors 104, and is incorporated herein by reference in its entirety.

[0037] The system 100 may include any hardware components, subsystems, and the like to support various functions of the wearable monitor 104 such as data collection, processing, display, and communications with external resources. For example, the system 100 may include hardware for a heart rate monitor using, e.g., photoplethysmography, electrocardiography, or any other technique(s). The system 100 may be configured such that, when the wearable monitor104 is placed for use about a wrist (or at some other body location), the system 100 initiates acquisition of physiological data from the wearer. In some embodiments, the pulse or heart rate may be acquired optically based on a light source (such as light emitting diodes (LEDs)) and optical detectors in the wearable monitor 104. The LEDs may be positioned to direct illumination toward the user’s skin, and optical detectors such as photodiodes may be used to capture illumination intensity measurements indicative of illumination from the LEDs that is reflected and / or transmitted by the wearer’s skin. In one embodiment, the physiological monitoring system (e.g., the wearable monitor and battery) takes a form other than that shown in Fig. 1. For example, the physiological monitoring system may be arranged and configured as a ring to be worn on a finger or thumb of a user, or as a bicep band, sock, or other accessory, apparel item, or the like.

[0038] The system 100 may be configured to record other physiological and / or biomechanical parameters including, but not limited to, skin temperature (using a thermometer), galvanic skin response (using a galvanic skin response sensor), motion (using one or more multiaxes accelerometers and / or gyroscope), blood pressure, and the like, as well environmental or contextual parameters such as ambient light, ambient temperature, humidity, time of day, and so forth. For example, the wearable monitor 104 may include sensors such as accelerometers and / or gyroscopes for motion detection, sensors for environmental temperature sensing, sensors to measure electrodermal activity (EDA), sensors to measure galvanic skin response (GSR) sensing, and so forth. The system 100 may also or instead include other systems or subsystems supporting addition functions of the wearable monitor 104. For example, the system 100 may include communications systems to support, e.g., near field communications, proximity sensing, Bluetooth communications, Wi-Fi communications, cellular communications, satellite communications, and so forth. The wearable monitor 104 may also or instead include components such as a Global Positioning System (GPS), a display and / or user interface, a clock and / or timer, and so forth.

[0039] The wearable monitor 104 may include one or more sources of battery power, such as a first battery within the wearable monitor 104 and a second battery 106 that is removable from and replaceable to the wearable monitor 104 in order to recharge the battery in the wearable monitor 104. Also or instead, the system 100 may include a plurality of wearable monitors 104 (and / or other physiological monitors) that can share battery power or provide power to one another. The system 100 may perform numerous functions related to continuous monitoring, such as automatically detecting when the user is asleep, awake, exercising, and so forth, and such detections may be performed locally at the wearable monitor 104 or at a remoteservice coupled in a communicating relationship with the wearable monitor 104 and receiving data therefrom. In general, the system 100 may support continuous, independent monitoring of a physiological signal such as a heart rate, and the underlying acquired data may be stored on the wearable monitor 104 for an extended period until it can be uploaded to a remote processing resource for more computationally complex analysis.

[0040] In one aspect, the wearable monitor may be a wrist-worn photoplethysmography device.

[0041] Fig. 2 illustrates a physiological monitoring system. More specifically, Fig. 2 illustrates a physiological monitoring system 200 that may be used with any of the methods or devices described herein. In general, the system 200 may include a physiological monitor 206, a user device 220, a remote server 230 with a remote data processing resource (such as any of the processors or processing resources described herein), and one or more other resources 250, all of which may be interconnected through a data network 202.

[0042] The data network 202 may be any of the data networks described herein. For example, the data network 202 may be any network(s) or internetwork(s) suitable for communicating data and information among participants in the system 200. This may include public networks such as the Internet, private networks, telecommunications networks such as the Public Switched Telephone Network or cellular networks using third generation (e.g., 3G or IMT-200), fourth generation (e.g., LTE (E-UTRA) or WiMAX-Advanced (IEEE 802.16m)), fifth generation (e.g., 5G), and / or other technologies, as well as any of a variety of corporate area or local area networks and other switches, routers, hubs, gateways, and the like that might be used to carry data among participants in the system 200. This may also include local or short- range communications infrastructure suitable, e.g., for coupling the physiological monitor 206 to the user device 220, or otherwise supporting communicating with local resources. By way of non-limiting examples, short range communications may include Wi-Fi communications, Bluetooth communications, infrared communications, near field communications, communications with RFID tags or readers, and so forth.

[0043] The physiological monitor 206 may, in general, be any physiological monitoring device or system, such as any of the wearable monitors or other monitoring devices or systems described herein. In one aspect, the physiological monitor 206 may be a wearable physiological monitor shaped and sized to be worn on a wrist or other body location. The physiological monitor 206 may include a wearable housing 211, a network interface 212, one or more sensors 214, one or more light sources 215, a processor 216, a haptic device 217 or other user input / output hardware, a memory 218, and a strap 210 for retaining the physiological monitor206 in a desired location on a user. In one aspect, the physiological monitor 206 may be configured to acquire heart rate data and / or other physiological data from a wearer in an intermittent or substantially continuous manner. In another aspect, the physiological monitor 206 may be configured to support extended, continuous acquisition of physiological data, e.g., for several days, a week, or more.

[0044] The network interface 212 of the physiological monitor 206 may be configured to couple the physiological monitor 206 to one or more other components of the system 200 in a communicating relationship, either directly, e.g., through a cellular data connection or the like, or indirectly through a short range wireless communications channel coupling the physiological monitor 206 locally to a wireless access point, router, computer, laptop, tablet, cellular phone, or other device that can locally process data, and / or relay data from the physiological monitor 206 to the remote server 230 or other resource(s) 250 as necessary or helpful for acquiring and processing data from the physiological monitor 206.

[0045] The one or more sensors 214 may include any of the sensors described herein, or any other sensors or sub-systems suitable for physiological monitoring or supporting functions. By way of example and not limitation, the one or more sensors 214 may include one or more of a light source, an optical sensor, an accelerometer, a gyroscope, a temperature sensor, a galvanic skin response sensor, a capacitive sensor, a resistive sensor, an environmental sensor (e.g., for measuring ambient temperature, humidity, lighting, and the like), a geolocation sensor, a Global Positioning System, a proximity sensor, an RFID tag reader, and RFID tag, a temporal sensor, an electrodermal activity sensor, and the like. The one or more sensors 214 may be disposed in the wearable housing 211, or otherwise positioned and configured for physiological monitoring or other functions described herein. In one aspect, the one or more sensors 214 include a light detector configured to provide light intensity data to the processor 216 (or to the remote server 230) for calculating a heart rate and a heart rate variability. The one or more sensors 214 may also or instead include an accelerometer, gyroscope, and the like configured to provide motion data to the processor 216, e.g., for detecting activities such as a sleep state, a resting state, a waking event, exercise, and / or other user activity. In an implementation, the one or more sensors 214 may include a sensor to measure a galvanic skin response of the user. The one or more sensors 214 may also or instead include electrodes or the like for capturing electronic signals, e.g., to obtain an electrocardiogram and / or other electrically-derived physiological measurements.

[0046] The processor 216 and memory 218 may be any of the processors and memories described herein. In one aspect, the memory 218 may store physiological data obtained bymonitoring a user with the one or more sensors 214, and or any other sensor data, program data, or other data useful for operation of the physiological monitor 206 or other components of the system 200. It will be understood that, while only the memory 218 on the physiological monitor is illustrated, any other device(s) or components of the system 200 may also or instead include a memory to store program instructions, raw data, processed data, user inputs, and so forth. In one aspect, the processor 216 of the physiological monitor 206 may be configured to obtain heart rate data from the user, such as heart rate data including or based on the raw data from the sensors 214. The processor 216 may also or instead be configured to determine, or assist in a determination of, a condition of the user related to, e.g., health, fitness, strain, recovery sleep, or any of the other conditions described herein.

[0047] The one or more light sources 215 may be coupled to the wearable housing 211 and controlled by the processor 216. At least one of the light sources 215 may be directed toward the skin of a user adjacent to the wearable housing 211. Light from the light source 215, or more generally, light at one or more wavelengths of the light source 215, may be detected by one or more of the sensors 214, and processed by the processor 216 as described herein.

[0048] The system 200 may further include a remote data processing resource executing on a remote server 230. The remote data processing resource may include any of the processors and related hardware described herein, and may be configured to receive data transmitted from the memory 218 of the physiological monitor 206, and to process the data to detect or infer physiological signals of interest such as heart rate, heart rate variability, respiratory rate, pulse oxygen, blood pressure, and so forth. The remote server 230 may also or instead evaluate a condition of the user such as a recovery state, sleep state, exercise activity, exercise type, sleep quality, daily activity strain, and any other health or fitness conditions that might be detected based on such data.

[0049] The system 200 may include one or more user devices 220, which may work together with the physiological monitor 206, e.g., to provide a display, or more generally, user input / output, for user data and analysis, and / or to provide a communications bridge from the network interface 212 of the physiological monitor 206 to the data network 202 and the remote server 230. For example, physiological monitor 206 may communicate locally with a user device 220, such as a smartphone of a user, via short-range communications, e.g., Bluetooth, or the like, for the exchange of data between the physiological monitor 206 and the user device 220, and the user device 220 may in turn communicate with the remote server 230 via the data network 202 in order to forward data from the physiological monitor 206 and to receive analysis and results from the remote server 230 for presentation to the user. In one aspect, the userdevice(s) 220 may support physiological monitoring by processing or pre-processing data from the physiological monitor 206 to support extraction of heart rate or heart rate variability data from raw data obtained by the physiological monitor 206. In another aspect, computationally intensive processing may advantageously be performed at the remote server 230, which may have greater memory capabilities and processing power than the physiological monitor 206 and / or the user device 220.

[0050] The user device 220 may include any suitable computing device(s) including, without limitation, a smartphone, a desktop computer, a laptop computer, a network computer, a tablet, a mobile device, a portable digital assistant, a cellular phone, a portable media or entertainment device, or any other computing devices described herein. The user device 220 may provide a user interface 222 for access to data and analysis by a user, and / or to support user control of operation of the physiological monitor 206. The user interface 222 may be maintained by one or more applications executing locally on the user device 220, or the user interface 222 may be remotely served and presented on the user device 220, e.g., from the remote server 230 or the one or more other resources 250.

[0051] In general, the remote server 230 may include data storage, a network interface, and / or other processing circuitry. The remote server 230 may process data from the physiological monitor 206 and perform physiological and / or health monitoring / analyses or any of the other analyses described herein, (e.g., analyzing sleep, determining strain, assessing recovery, and so on), and may host a user interface for remote access to this data, e.g., from the user device 220. The remote server 230 may include a web server or other programmatic front end that facilitates web-based access by the user devices 220 or the physiological monitor 206 to the capabilities of the remote server 230 or other components of the system 200.

[0052] The system 200 may include other resources 250, such as any resources that can be usefully employed in the devices, systems, and methods as described herein. For example, these other resources 250 may include other data networks, databases, processing resources, cloud data storage, data mining tools, computational tools, data monitoring tools, algorithms, and so forth. In another aspect, the other resources 250 may include one or more administrative or programmatic interfaces for human actors such as programmers, researchers, annotators, editors, analysts, coaches, and so forth, to interact with any of the foregoing. The other resources 250 may also or instead include any other software or hardware resources that may be usefully employed in the networked applications as contemplated herein. For example, the other resources 250 may include payment processing servers or platforms used to authorize payment for access, content, or option / feature purchases. In another aspect, the other resources 250 mayinclude certificate servers or other security resources for third-party verification of identity, encryption or decryption of data, and so forth. In another aspect, the other resources 250 may include a desktop computer or the like co-located (e.g., on the same local area network with, or directly coupled to through a serial or USB cable) with a user device 220, wearable strap 210, or remote server 230. In this case, the other resources 250 may provide supplemental functions for components of the system 200 such as firmware upgrades, user interfaces, and storage and / or pre-processing of data from the physiological monitor 206 before transmission to the remote server 230.

[0053] The other resources 250 may also or instead include one or more web servers that provide web-based access to and from any of the other participants in the system 200. While depicted as a separate network entity, it will be readily appreciated that the other resources 250 (e.g., a web server) may also or instead be logically and / or physically associated with one of the other devices described herein, and may for example, include or provide a user interface 222 for web access to the remote server 230 or a database or other resource(s) to facilitate user interaction through the data network 202, e.g., from the physiological monitor 206 or the user device 220.

[0054] In another aspect, the other resources 250 may include fitness equipment or other fitness infrastructure. For example, a strength training machine may automatically record repetitions and / or added weight during repetitions, which may be wirelessly accessible by the physiological monitor 206 or some other user device 220. More generally, a gym may be configured to track user movement from machine to machine, and report activity from each machine in order to track various strength training activities in a workout. The other resources 250 may also or instead include other monitoring equipment or infrastructure. For example, the system 200 may include one or more cameras to track motion of free weights and / or the body position of the user during repetitions of a strength training activity or the like. Similarly, a user may wear, or have embedded in clothing, tracking fiducials such as visually distinguishable objects for image-based tracking, or radio beacons or the like for other tracking. In another aspect, weights may themselves be instrumented, e.g., with sensors to record and communicated detected motion, and / or beacons or the like to self-identify type, weight, and so forth, in order to facilitate automated detection and tracking of exercise activity with other connected devices.

[0055] One limitation on wearable sensors can be body placement. Devices are typically wrist-based, and may occupy a location that a user would prefer to reserve for other devices or jewelry, or that a user would prefer to leave unadorned for aesthetic or functional reasons. This location also places constraints on what measurements can be taken, and may also limit useractivities. For example, a user may be prevented from wearing boxing gloves while wearing a sensing device on their wrist. To address this issues, physiological monitors may also or instead be embedded in clothing, which may be specifically adapted for physiological monitoring with the addition of communications interfaces, power supplies, device location sensors, environmental sensors, geolocation hardware, payment processing systems, and any other components to provide infrastructure and augmentation for wearable physiological monitors. Such “smart garments” offer additional space on a user’s body for supporting monitoring hardware, and may further enable sensing techniques that cannot be achieved with single sensing devices. For example, embedding a plurality of physiological sensors or other electronic / communi cation devices in a shirt may allow electrocardiogram (ECG) based heart rate measurements to be gathered from a torso region of the wearer; wireless antennas to be placed above the upper portion of the thoracic spine to achieve desired communications signals; a contactless payment system to be embedded in a sleeve cuff for interactions with a payment terminal; and muscle oxygen saturation measurements to be gathered from muscles such as the pectoralis major, latissimus dorsi, biceps brachii, and other major muscle groups. This non- exhaustive list illustrates just some examples of technology that may be incorporated into a single garment.

[0056] Smart garments may also free up body surfaces for other devices. For example, if sensors in a wrist-worn device that provide heart rate monitoring and step counting can be instead embedded in a user’s undergarments, the user may still receive the biometric information they desire, while also being able to wear jewelry or other accessories for suitable occasions.

[0057] The present disclosure generally includes smart garment systems and techniques. It will be understood that a “smart garment” as described herein generally includes a garment that incorporates infrastructure and devices to support, augment, or complement various physiological monitoring modes. Such a garment may include a wired, local communication bus for intra-garment hardware communications, a wireless communication system for intra-garment hardware communications, a wireless communication system for extra-garment communications and so forth. The garment may also or instead include a power supply, a power management system, processing hardware, data storage, and so forth, any of which may support enriched functions for the smart garment.

[0058] Fig. 3 shows a smart garment system. In general, the system 300 may include a plurality of components — e.g., a garment 310, one or more modules 320, a controller 330, a processor 340, a memory 342, and so on — capable of communicating with one another over a data network 302. The garment 310 may be wearable by a user 301 and configured tocommunicate with a module 320 having a physiological sensor 322 that is structurally configured to sense a physiological parameter of the user 301. As discussed herein, the module 320 may be controllable by the controller 330 based at least in part on a location 316 where the module 320 is located on or within the garment 310. This position-based information may be derived from an interaction and / or communication between the module 320 and the garment 310 using various techniques. It will be understood that, while two controllers 330 are shown, the garment 310 may include a single inter-garment controller, or any number of separate controllers 330 in any number of garments 310 (e.g., one per garment, or one for all garments worn by a person, etc.), and / or controllers may be integrated into other modules 320.

[0059] For communication over the data network 302, the system 300 may include a network interface 304, which may be integrated into the garment 310, included in the controller 330, or in some other module or component of the system 300, or some combination of these. The network interface 304 may generally include any combination of hardware and software configured to wirelessly communicate data to remote resources. For example, the network interface 304 may use a local connection to a laptop, smart phone, or the like that couples, in turn, to a wide area network for accessing, e.g., web-based or other network-accessible resources. The network interface 304 may also or instead be configured to couple to a local access point such as a router or wireless access point for connecting to the data network 302. In another aspect, the network interface 304 may be a cellular communications data connection for direct, wireless connection to a cellular network or the like.

[0060] The data network 302 may generally include any communication network through which computer systems may exchange data. For example, the data network 302 may include, but is not limited to, the Internet, an intranet, a LAN (Local Area Network), a WAN (Wide Area Network), a MAN (Metropolitan Area Network), a wireless network, a cellular data network, an optical network, and the like. To exchange data via the data network 302, the system 300 and the data network 302 may use various methods, protocols, and standards including, but not limited to, token ring, Ethernet, wireless Ethernet, Bluetooth, TCP / IP, UDP, HTTP, FTP, SNMP, SMS, MMS, SS7, JSON, XML, REST, SOAP, CORBA, IIOP, RMI, DCOM and Web Services. To ensure data transfer is secure, the system 300 may transmit data via the data network 302 using a variety of security measures including, but not limited to, TSL, SSL and VPN. By way of example, some embodiments of the system 300 may be configured to stream information wirelessly to a social network, a data center, a cloud service, and so forth.

[0061] In some embodiments, data streamed from the system 300 to the data network 302 may be accessed by the user 301 (or other users) via a website. The network interface 304may thus be configured such that data collected by the system 300 is streamed wirelessly to a remote processing facility 350, database 360, and / or server 370 for processing and access by the user. In some embodiments, data may be transmitted automatically, without user interactions, for example by storing data locally and transmitting the data over available local area network resources when a local access point such as a wireless access point or a relay device (such as a laptop, tablet, or smart phone) is available. In some embodiments, the system 300 may include a cellular system or other hardware for independently accessing network resources from the garment 310 without requiring local network connectivity.

[0062] In one example, the network interface 304 may be configured to stream data using Bluetooth or Bluetooth Low Energy technology, e.g., to a nearby device such as a cell phone or tablet for forwarding to other resources on the data network 302. In another example, the network interface 304 may be configured to stream data using a cellular data service, such as via a 3G, 4G, or 5G cellular network. It will be understood that the network interface 304 may include a computing device such as a mobile phone or the like. The network interface 304 may also or instead include or be included on another component of the system 300, or some combination of these. Where battery power or communications resources can advantageously be conserved, the system 300 may preferentially use local networking resources when available, and reserve cellular communications for situations where a data storage capacity of the garment 310 is reaching capacity. Thus, for example, the garment 310 may store data locally up to some predetermined threshold for local data storage, below which data is transmitted over local networks when available. The garment 310 may also transmit data to a central resource using a cellular data network only when local storage of data exceeds the predetermined threshold.

[0063] The garment 310 may include one or more of a shirt (or other top), shorts / pants (or other bottom), an undergarment (e.g., undershirt, underwear, brassiere, and so on), a sock or other footwear, a shoe, a facemask, a hat or helmet (or other head adornment), a compression sleeve, a sweatband, kinesiology tape or elastic therapeutic tape, a glove, and the like. More generally, the garment 310 may include any type(s) of wearable clothing or adornment suitable for wearing by a user and retaining one or more sensing modules as contemplated herein.

[0064] The garment 310 may include one or more designated areas 312 for positioning a module to sense a physiological parameter of the user 301 wearing the garment 310. One or more of the designated areas 312 may be specifically tailored for receiving a module 320 therein or thereon. For example, a designated area 312 may include a pocket structurally configured to receive a module 320 therein. Also or instead, a designated area 312 may include a first fastener configured to cooperate with a second fastener disposed on a module 320. One or more of thefirst fastener and the second fastener may include at least one of a hook-and-loop fastener, a button, a clamp, a clip, a snap, a projection, and a void.

[0065] The designated areas 312 may include at least one of a torso region, a spinal region, an extremity region (e.g., one or more of an arm region such as a sleeve, and a leg region such as a pant leg), a waistband region, a cuff region, and so on. Also or instead, one or more of the designated areas 312 may include at least a region adjacent to one or more muscle groups of the user 301 — e.g., muscle groups including at least one of the pectoralis major, latissimus dorsi, biceps brachii, and so on.

[0066] By placing a pocket or the like in one of these designated areas 312, a position of a module 320 can be controlled, and where an RFID tag, sensor, or the like is used, the designated area 312 can specifically sense when a module 320 is positioned there for monitoring, and can communicate the detected location to any suitable control circuitry. In this manner, a garment 310 may facilitate the installation of modules 320 in many different, discrete locations, the placement of which can be controlled by the configuration of the garment 310, and the use of which can be automatically detected when corresponding control modules 320 are placed there for use. Also or instead, the garment 310 may facilitate the placing of the modules 320 over relatively large regions of the garment 310. For example, a garment 310 may include a relatively large region (in terms of surface area) where a module 320 can be affixed or otherwise secured, e.g., by loops, straps, buttons, sheets of hook-and-loop fasteners, and so forth.

[0067] In general, each designated area 312 may include a pocket such as any of those described above, or any other mounting fixture or combination of fixtures. Where a pocket is used, the pocket may be configured as described above to preferentially urge a module 320 within the pocket toward the user’s skin under normal pressure. Without limiting the generality of the foregoing, this may generally include an exterior layer of the pocket that is less elastic than an interior surface of the pocket so that when circumferential tension is applied (e.g., when the garment 310 is donned), the pocket preferentially urges a contact surface of the sensor inward toward the intended target surface with at least a predetermined normal force (when the garment 310 is properly sized for the user). In this respect, it will be understood that although some variation in normal force among users and garments is inevitable, typical tensions for comfortable use of properly fitted athletic wear are generally known, and adequate contact force to obtain a high quality physiological signal is generally known, and in any event readily observable in acquired data. As such, adequate circumferential tensions and resulting normal contact forces needed to promote good contact between sensing regions of the module 320 (such as LEDs, capacitive touch sensors, photodiodes, and the like) and the user’s skin may readily bedetermined, and can advantageously facilitate the use of wrist-worn sensor housings such as those described above with one of the garments 310 described herein for off-wrist monitoring if / when desired.

[0068] In one aspect, the designated areas 312 may usefully be positioned where reinforcing elastic bands are typically provided on garments, e.g., around the mid-torso for a sports bra, around the waist on shorts or underwear, or on the sleeves of a t-shirt. In one aspect, the designated areas 312 may also usefully be positioned according to the intended physiological measurement, e.g., near major arteries suitable for heart rate detection using photoplethysmography. In one aspect, the garment 310 may usefully distribute these designated areas 312 (and supporting infrastructure such as wired connectors, location identification tags, and the like) at the intersection of regions where good physiological signals can be obtained and regions where adequate normal forces for good sensor contact can be generated by clothing. For example, this may include the ankles, the waist, the mid-torso, the biceps, the wrists, the forehead, and so on.

[0069] The garment 310 may also or instead incorporate other infrastructure 315 to cooperate with a module 320. For example, the garment infrastructure 315 may include wires or the like embedded in the garment 310 to facilitate wired data or power transfer between installed modules 320 and other system components (including other modules 320). The infrastructure 315 may also or instead include integrated features for, e.g., powering modules, supporting data communications among modules, and otherwise supporting operation of the system 300. The infrastructure may also or instead include location or identification tags or hardware, a power supply for powering modules 320 or other hardware, communications infrastructure as described herein, a wired intra-garment network, or supplemental components such as a processor, a Global Positioning System (GPS), a timing device, e.g., for synchronizing signals from multiple garments, a beacon for synchronizing signals among multiple modules 320, and so forth. More generally, any hardware, software, or combination of these suitable for augmenting operation of the garment 310 and a physiological monitoring system using the garment 310 may be incorporated as infrastructure 315 into the garment 310 as contemplated herein.

[0070] The modules 320 may generally be sized and shaped for placement on or within the one or more designated areas 312 of the garment 310. For example, in certain implementations, one or more of the modules 320 may be permanently affixed on or within the garment 310. In such instances, the modules 320 may be washable. Also or instead, in certain implementations, one or more of the modules 320 may be removable and replaceable relative to the garment 310. In such instances, the modules 320 need not be washable, although a module320 may be designed to be washable and / or otherwise durable enough to withstand a prolonged period of engagement with a designated area 312 of the garment 310. A module 320 may be capable of being positioned in more than one of the designated areas 312 of the garment 310. That is, one or more of the plurality of modules 320 may be configured to sense data using a physiological sensor 322 in a plurality of designated areas 312 of the garment 310.

[0071] Removable and replaceable modules 320 may provide several advantages such as ease of garment care (e.g., washing) and power management (e.g., removal for recharging). Furthermore, removability may facilitate replacement and / or repositioning of modules within the garment 310 for different sensing activities or other reconfigurations, replacement of damaged or defective modules 320, and so forth.

[0072] A module 320 may include one or more physiological sensors 322 and a communications interface 324 programmed to transmit data from at least one of the physiological sensors 322. For example, the physiological sensors 322 may include one or more of a heart rate monitor, an oxygen monitor (e.g., a pulse oximeter), a thermometer, an accelerometer, a gyroscope, a position sensor, a Global Positioning System, a clock, a galvanic skin response (GSR) sensor, or any other electrical, acoustic, optical, or other sensor or combination of sensors and the like useful for physiological monitoring, environmental monitoring, or other monitoring as described herein. In one aspect, the physiological sensors 322 may include a conductivity sensor or the like used for electromyography, electrocardiography, electroencephalography, or other physiological sensing based on electrical signals. The data received from the physiological sensors 322 may include at least one of heart rate data, muscle oxygen saturation data, temperature data, movement data, position / location data, environmental data, temporal data, and so on.

[0073] In one aspect, a module 320 may be configured for use on multiple body locations. For example, the module 320 may be one of the wrist-worn sensors described above. The module 320 may be adapted for use with a garment 310 in various ways. In one aspect, the module 320 may have relatively smooth, continuous exterior surfaces to facilitate sliding into and out of a pocket, such as any of the pockets described herein, or any other suitable retaining structure(s). In another aspect, an LED and / or sensor region may protrude from a surface of the module 320 sufficiently to extend beyond a restraining garment material and into a contact surface of a user. The module 320 may also include hardware to facilitate such uses. For example, a module 320 may usefully incorporate a contact sensor for detecting contact with a user. However, the exposed contact surfaces of the module 320 may be different when retained by a wrist strap (or other limb strap) than when retained by a garment pocket. To facilitatemultiple retaining modes, the module 320 may usefully incorporate two or more contact sensors (such as capacitive sensors or other touch sensors, switches, or the like) at two different locations, each positioned to detect contact with a wearer in a different retaining mode. For example, a module 320 may include a capacitive sensor adjacent to an optical sensing system that contacts the user’s skin when the module 320 is retained with a wrist strap. The module 320 may also or instead optically detect contact when the capacitive sensor is covered by a garment fabric or the like that prevents direct skin contact, or a second capacitive sensor may be placed within another region exposed by the garment 310 retaining system. In another aspect, the garment 310 may include a capacitive sensor that provides a signal to the module 320, or to some other system controller or the like, when a region of the garment near the module 320 is in contact with a user’s skin.

[0074] In one aspect, the physiological sensors 322 may include a heart rate monitor or pulse sensor, e.g., where heart rate is optically detected from an artery, such as the radial artery. In one embodiment, the garment 310 may be configured such that a module 320 is positioned on a user’s wrist, where a physiological sensor 322 of the module 320 is secured over the user’s radial artery or other blood vessel. Secure connection and placement of a pulse sensor over the radial artery or other blood vessel facilitates measurement of heart rate, pulse oxygen, and the like. It will be understood that this configuration is provided by way of example only, and that other sensors, sensor positions, and monitoring techniques may also or instead be employed without departing from the scope of this disclosure.

[0075] In some embodiments, heart rate data may be acquired using an optical sensor coupled with one or more light emitting diodes (LEDs), all in contact with the user 301. To facilitate optical sensing, the garment 310 may be designed to maintain a physiological sensor 322 in secure, continual contact with the skin, and reduce interference of outside light with optical sensing by the physiological sensor 322.

[0076] Thus, certain embodiments include one or more physiological sensors 322 configured to provide continuous measurements of heart rate using photoplethysmography or the like. The physiological sensor 322 may include one or more light emitters for emitting light at one or more desired frequencies toward the user’s skin, and one or more light detectors for received light reflected from the user’s skin. The light detectors may include a photo-resistor, a phototransistor, a photodiode, and the like. A processor may process optical data from the light detector(s) to calculate a heart rate based on the measured, reflected light. The optical data may be combined with data from one or more motion sensors, e.g., accelerometers and / or gyroscopes, to minimize or eliminate noise in the heart rate signal caused by motion or otherartifacts. The physiological sensor 322 may also or instead provide at least one of continuous motion detection, environmental temperature sensing, electrodermal activity (EDA) sensing, galvanic skin response (GSR) sensing, and the like.

[0077] The system 300 may include different types of modules 320. For example, a number of different modules 320 may each provide a particular function. Thus, the garment 310 may house one or more of a temperature module, a heart rate / PPG module, a muscle oxygen saturation module, a haptic module, a wireless communication module, or combinations thereof, any of which may be integrated into a single module 320 or deployed in separate modules 320 that can communicate with one another. Some measurements such as temperature, motion, optical heart rate detection, and the like, may have preferred or fixed locations, and pockets or fixtures within the garment 310 may be adapted to receive specific types of modules 320 at specific locations within the garment 310. For example, motion may preferentially be detected at or near extremities while heart rate data may preferentially be gathered near major arteries. In another aspect, some measurements such as temperature may be measured anywhere, but may preferably be measured at a single location in order to avoid certain calibration issues that might otherwise arise through arbitrary placement.

[0078] In another aspect, the system 300 may include two or more modules 320 placed at different locations and configured to perform differential signal analysis. For example, the rate of pulse travel and the degree of attenuation in a cardiac signal may be detected using two or more modules at two or more locations, e.g., at the bicep and wrist of a user, or at other locations similarly positioned along an artery. These multiple measurements support a differential analysis that permits useful inferences about heart strength, pliability of circulatory pathways, and other aspects of the cardiovascular system that may indicate cardiac age, cardiac health, cardiac conditions, and so forth. Similarly, muscle activity detection might be measured at different locations to facilitate a differential analysis for identifying activity types, determining muscular fitness, and so forth. More generally, multiple sensors can facilitate differential analysis. To facilitate this type of analysis with greater precision, the garment infrastructure may include a beacon or clock for synchronizing signals among multiple modules, particularly where data is temporarily stored locally at each module, or where the data is transmitted to a processor from different locations wirelessly where packet loss, latency, and the like may present challenges to real time processing.

[0079] The communications interface 324 may be any as described herein, for example including any of the features of the network interface 304 described above. The communications interface 324 may be a separate device that provides the ability for the modules 320 tocommunicate with one another and / or with other components of the system 300), or there may be a central module that communicates with other modules 320 (or with another component of the system 300). It will be understood that communications may usefully be secured using any suitable encryption technology in order to ensure privacy and security of user data. This may, for example, include encryption for local (wired or wireless) communications among the modules 320 and / or controller 330 within the garment 310. This may also or instead include encryption for remote communications to a server and other remote resources. In one aspect, the garment 310 and / or controller 330 may provide a cryptographic infrastructure for securing local communications, e.g., by managing public / private key pairs for use in asymmetric encryption, authentication, digital signatures, and so forth. The keys for this infrastructure may also or instead be managed by an external, trusted third party.

[0080] The controller 330 may be configured, e.g., by computer executable code or the like, to determine a location of the module 320. This may be based on contextual measurements such as accelerometer data from the module 320, which may be analyzed by a machine learning model or the like to infer a body position. In another aspect, this may be based on other signals from the module 320. For example, signals from sensors such as photodiodes, temperature sensors, resistors, capacitors, and the like may be used alone or in combination to infer a body position. In another aspect, the location may be determined based on a proximity of a module 320 to a proximity sensor, RFID tag, or the like at or near one of the designated areas 312 of the garment 310. Based on the location, the controller 330 may adapt operation of the module 320 for location-specific operation. This may include selecting filters, processing models, physiological signal detections, and the like. It will be understood that operations of the controller 330, which may be any controller, microcontroller, microprocessor, or other processing circuitry, or the like, may be performed in cooperation with another component of the system 300 such as the processor 340 described herein, one or more of the modules 320, or another computing device. It will also be understood that the controller 330 may be located on a local component of the system 300 (e.g., on the garment 310, in a module 320, and so on) or as part of a remote processing facility 350, or some combination of these. Thus, in an aspect, a controller 330 is included in at least one of the plurality of modules 320. And, in another aspect, the controller 330 is a separate component of the garment 310, and serves to integrate functions of the various modules 320 connected thereto. The controller 330 may also or instead be remote relative to each of the plurality of modules 320, or some combination of these.

[0081] Location detection (i.e., of the modules 320 and / or physiological sensors 322) may also usefully be recorded and used in a number of ways by a human user and / or by thesystem 300. For example, a detected location may be stored, along with the corresponding garment, so that a user can retrieve a placement history and replace the module 320 to a previous location for a particular garment as desired. In another aspect, the detected location may be used by the system 300 to analyze data and make garment specific recommendations. For example, the system 300 may evaluate the quality of a signal, e.g., using any conventional metrics such as signal-to-noise ratio, or using quality metrics more specific to physiological signals such as correlation to an expected signal or pulse shape, consistency with a rate or magnitude typical for a sensor, pulse-to-pulse consistency for a particular user, or any other measure of signal quality using statics, machine learning, digital signal processing techniques, or the like. A quality metric, however derived, may be used in turn to recommend specific placements of a module 320 on a garment 310 for a user, or to recommend a particular garment 310 for the user. Thus, for example, after acquiring data over a range of garments and activities, the system 300 may generate a user-actionable recommendation such as, “It appears that when you are jogging, the most accurate heart rate signals can be obtained when you are wearing an XL shirt model number xxxxxx. You may wish to wear this shirt for active workouts, and you may wish to purchase more of this type of shirt for regular use.” As another example, the user-actionable recommendation may suggest: “It appears that one of your modules is not obtaining accurate temperature readings when located on your sleeve elastic band. You may wish to try a different location for this module, or to try a different garment.” More generally, data quality may be measured for a number of different modules at different locations in different garments during different activities, and this data may be used to generate customized recommendations for a user on a per-garment and per-location basis. These recommendations may also be tailored to specific activity types where this data is accurately recorded by the system 300, either from user input, automatic detection, or some combination of these.

[0082] The controller 330 may be configured to control one or more of (i) sensing performed by a physiological sensor 322 of the module 320 and (ii) processing by the module 320 of the data received from a physiological sensor 322. That is, in certain aspects, the combination of sensors in the module 320 may vary based on where it is intended to be located on a garment 310. In another aspect, processing of data from a module 320 may vary based on where it is located on a garment 310. In this latter aspect, a processing resource such as the controller 330 or some other local or remote processing resource coupled to the module 320 may detect the location and adapt processing of data from the module 320 based on the location. This may, for example, include a selection of different models, algorithms, or parameters for processing sensed data.

[0083] In another aspect, this may include selecting from among a variety of different activity recognition models based on the detected location. For example, a variety of different activity recognition models may be developed such as machine learning models, lookup tables, analytical models, or the like, which may be applied to accelerometer data to detect an activity type. Other motion data such as gyroscope data may also or instead be used, and activity recognition processes may also be augmented by other potentially relevant data such as data from a barometer, magnetometer, GPS system, and so forth. This may generally discriminate, e.g., between being asleep, at rest, or in motion, or this may discriminate more finely among different types of athletic activity such as walking, running, biking, swimming, playing tennis, playing squash, and so forth. While useful models may be developed for detecting activities in this manner, the nature of the detection will depend upon where the accelerometers are located on a body. Thus, a processing resource may usefully identify location first using location detection systems (such as tags, electromechanical bus connections, etc.) built into the garment 310, and then use this detected location to select a suitable model for activity recognition. This technique may similarly be applied to calibration models, physiological signals processing models, and the like, or to otherwise adapt processing of signals from a module 320 based on the location of the module 320.

[0084] Determining the location of a module 320 may include receiving a sensed location for the module 320. The sensed location may be provided by a proximity detection circuit such as a near-fi eld-communication (NFC) tag, an (active or passive) RFID tag, a capacitance sensor, a magnetic sensor, an electrical contact, a mechanical contact, and the like. Any corresponding hardware for such proximity detections may be disposed on the module 320 and the garment 310 for communication therebetween to detect location when appropriate. For example, in one aspect, an NFC tag may be disposed on or within the garment 310, and the module may include an NFC tag sensor 320 that can detect the tag and read any locationspecific information therefrom. Proximity detection may also or instead be performed using capacitively detected contact, electromagnetically detected proximity, mechanical contact, electrical coupling, and the like. In this manner, a garment 310 may provide information to an installed module 320 to inform the module 320, among other things, where the module 320 is located, or vice-versa.

[0085] Thus, communication between a module 320 and the garment 310 (or a processor of the garment 310) may be used to determine the location of a module 320 on the garment 310. Communication of location information may be enabled using active techniques, passive techniques, or a combination thereof. For example, a thin, flexible, cheap, washable NFC tagmay be sewn into the garment 310 in various locations where a module 320 may be placed. When a module 320 is placed in the garment 310, the module 320 may query an adjacent NFC tag to determine its location. Furthermore, the NFC technique or other similar techniques may provide other information to the module 320, including details about the garment 310 such as the size, whether it is a gender specific piece, the manufacturer information, model or serial number of the garment, stock keeping unit (SKU), and more. Similarly, the tag may encode a unique identifier for the garment 310 that can be used to obtain other relevant information using an online resource. The module 320 may also or instead advertise information about itself to the garment 310 so that the garment 310 can synchronize processing with other modules 320, synchronize communication among modules 320, control or condition signals from the module 320, and so forth. The module 320 can then configure itself within the context of the current garment 310 and associated modules 320, and / or to perform certain types of monitoring or data processing.

[0086] Determining the location of a module 320 may also or instead be based, at least in part, on an interpretation of the data received from a physiological sensor 322 of the module 320. By way of example, movement of a module 320 as detected by a sensor may provide information that can be used to predict a position on or within the garment 310. Also or instead, the type of data that is being received from a module 320 may indicate where the module 320 is located on the garment 310. For example, locations may produce unique signatures of acceleration, gyroscope activity, capacitive data, optical data, temperature data, and the like, depending on where the module 320 is located, and this data may be fused and analyzed in any suitable manner to obtain a location prediction.

[0087] According to the foregoing, determining the location of a module 320 may also or instead include receiving explicit input from the user 301, which may identify one of the designated areas on the garment 310, or a general area of the body (e.g., left wrist, right ankle, and so forth). Because the location of the module 320 relative to the garment 310 may be determined from an analysis of a plurality of data sources, the system 300 may include a component (e.g., the processor 340) that is configured to reconcile one or more potential sources of location of information based on expected reliability, measured quality of data, express user input, and so forth. A prediction confidence may also usefully be generated in this context, which may be used, for example, to determine whether a user should be queried for more specific location information. More generally, any of the foregoing techniques may be used along or in combination, along with a failsafe measure the requests user input when locationcannot confidently be predicted. Also or instead, a user may explicitly specify a prediction preemptively, or as an override to an automatically generated prediction.

[0088] Once determined using any of the techniques above, the location of a module 320 may be transmitted for storage and analysis to a remote processing facility 350, a database 360, or the like. That is, in addition to the module 320 using this information locally to configure itself for the location in which it is worn, the module 320 may communicate this information to other modules 320, peripherals, or the cloud. Processing this information in the cloud may help an organization determine if a module 320 has ever been installed on a garment 310, which locations are most used, and how modules 320 perform differently in different locations. These analytics may be useful for many purposes, and may, for example, be used to improve the design or use of modules 320 and garments 310, either for a population, for a user type, or for a particular user.

[0089] As stated above, the system 300 may further include a processor 340 and a memory 342. In general, the memory 342 may bear computer executable code configured to be executed by the processor 340 to perform processing of the data received from one or more modules 320. One or more of the processor 340 and the memory 342 may be located on a local component of the system 300 (e.g., the garment 310, a module 320, the controller 330, and the like) or as part of a remote processing facility 350 or the like as shown in the figure. Thus, in an aspect, one or more of the processor 340 and the memory 342 is included on at least one of the plurality of modules 320. In this manner, processing may be performed on a central module, or on each module 320 independently. In another aspect, one or more of the processor 340 and the memory 342 is remote relative to each of the plurality of modules 320. For example, processing may be performed on a connected peripheral device such as smart phone, laptop, local computer, or cloud resource.

[0090] The memory 342 may store one or more algorithms, models, and supporting data (e.g., parameters, calibration results, user selections, and so forth) and the like for transforming data received from a physiological sensor 322 of the module 320. In this manner, suitable models, algorithms, tuning parameters, and the like may be selected for use in transforming the data based on the location of the module 320 as determined by the controller 330 and / or processor 340 as described herein. By way of example, algorithms that convert data from an accelerometer in a module 320 into a count of a user’s steps may be different depending on whether the module 320 is worn on the user’s wrist or on the user’s waist band. Similarly, the intensity of an LED and corresponding sensitivity of a photodetector may be different for a PPG device placed on the wrist or the thigh. Thus, the module 320 may self-configure for a locationby controlling one or more of sensor types, sensor parameters, processing models, and so forth based on a detected location for the module 320.

[0091] Selection of an algorithm may also or instead include an analysis of one or more of the sensor data, metadata, and the like. By way of example, an algorithm may be selected at least in part based on metadata received from one of the module 320 and the garment 310. This metadata may be derived from communication between the module 320 and the garment 310 — e.g., between a tag and tag reader for exchanging information therebetween. For example, the garment 310 may include, e.g., stored in a tag such as an NFC tag or other wirelessly readable data source, garment-specific metadata that is readable by or otherwise transmittable to one or more of the plurality of modules 320, the controller 330, and the processor 340. Such garmentspecific metadata may include at least one of a type of garment 310, a size of the garment 310, garment dimensions, a gender configuration of the garment 310, a manufacturer, a model number, a serial number, a SKU, a material, fit information, and so on. In one aspect, this information may be provided with one or more of the location identification tags described herein. In another aspect, the garment 310 may include an additional tag at a suitable location (e.g., near or accessible to a processor or controller) that provides garment-specific information while other tags provide location-specific information.

[0092] The metadata may also or instead include at least one of a gender of the user 301, a weight of the user 301, a height of the user 301, an age of the user 301, metadata associated with the garment 310 (e.g., the garment size, type, material, etc.), and the like. The metadata may be derived, at least in part, from user-provided input, or otherwise from information derived from the user 301 such as a user’s account information as a participant in the system 300. By way of example, a processing algorithm may be selected depending on the material of the garment 310 as communicated by its serial or model number in an identification tag, the physiology of the user 301 as implied by the garment size, and so on. The metadata may also or instead be used to verify the authenticity of the garment 310, and otherwise control access to the garment 310 and / or modules 320 coupled to the garment 310. In one aspect, metadata (e.g., size, material) may be encoded directly into the garment metadata. In another aspect, the garment 310 may publish a unique identifier that can be used to retrieve related information from a manufacturer or other data source. This latter approach advantageously permits correlation of garment-specific data with other user-specific data such as height, weight, body composition, and so forth.

[0093] Simply knowing a priori where a module 320 is positioned may allow for the use of algorithms that have been developed to perform optimally in that particular location. This canrelieve a significant computational burden otherwise borne by the module 320 to analytically evaluate location based on available signals. Other information may also or instead be used to select an optimal algorithm. For example, based on the gender or dimensions of a garment, the algorithm may employ different models or different model parameters.

[0094] The processor 340 may be configured to assess the quality of the data received from a physiological sensor 322 of the module 320. For example, the processor 340 may be configured to provide, based on the quality of the data, a recommendation regarding at least one of the location of a module 320 and an aspect of the garment 310 (e.g., size, fit, material, and so on). For example, the processor 340 may be configured to detect when the garment does not properly fit the wearer for acquisition of physiological data, for example, by detecting when a module is moving (e.g., from accelerometer data) but data quality is poor or absent for a sensed physiological signal. In general, the garment 310 may store its own identifier and / or metadata, e.g., as described herein, or garment identification data may be stored in tags, e.g., at designated areas 312 of the garment 310. The processor 340 may be configured to use this garment identification information and / or metadata to provide a recommendation regarding a different garment 310 for the user 301, or for an adjustment to the current garment 310. For example, if a particular garment 310 seems to result in low-quality data, the user 301 could be encouraged to select an alternative size, or to make some other adjustment. Moreover, data on how many times a garment 310 is used may be gathered and used to inform business decisions, for example, which garments 310 provide the highest-quality data, and which garments 310 are most preferred by users 301.

[0095] The system 300 may further include a database 360, which may be located remotely and in communication with the system 300 via the data network 302. The database 360 may store data related to the system 300 such as any discussed herein — e.g., sensed data, processed data, transformed data, metadata, physiological signal processing models and algorithms, personal activity history, and the like. The system 300 may further include one or more servers 370 that host data, provide a user interface, process data, and so forth in order to facilitate use of the modules 320 and garments 310 as described herein.

[0096] It will be appreciated that the garment 310, modules 320, and accompanying garment infrastructure and remote networking / processing resources, may advantageously be used in combination to improve physiological monitoring and achieve modes of monitoring not previously available.

[0097] One or more of the devices and systems described herein may include circuitry for both wireless charging and wireless data transmission, e.g., where the corresponding circuitscan operate independently from one another, and where the corresponding antennae are located proximal to one another (for instance, the circuitry for wireless charging and the circuitry for wireless data transmission may include separate coils disposed substantially along the same plane, or otherwise in relative close proximity in a device or system). In such aspects, one or more measures may be taken so that a wireless data transfer process does not interfere with a wireless power transfer process, more specifically by coupling the data circuitry into the electromagnetic field for the wireless power transfer in a manner that alters the resonant frequency or otherwise destructively interferes with power transfer, thereby decreasing efficiency when charging a device. For example, a switch may be included to disable circuitry for data transmission when certain wireless charging activity is present, thereby allowing for relatively unimpeded and efficient wireless charging of a device. The switch may also be operable to enable operation of data transmission circuitry when certain wireless charging activity is not present.

[0098] Thus, for example, in the context of a physiological monitor, such as any of those described herein, the physiological monitor may include both a wireless power receiver (or similar) and a wireless data tag reader (or similar). In general, these sub-systems may conform to one or more Near Field Communication (NFC) specifications for protocols and physical architectures, or any other standards suitable for wireless power and data transmission. The power circuitry may be used, e.g., to charge a battery on the physiological monitor so that the device can be recharged without physically connecting to a power source. The data circuitry may be used, e.g., as a wireless data tag reader or the like to read data from nearby data sources such as identification tags in user apparel and the like. In general, the physiological monitor may include separate circuity (separate coils) for these wireless power and data systems, such as separate processing circuitry and / or separate antennae. The antennae may be disposed substantially along the same plane of the physiological monitor (e.g., with one coil disposed substantially inside or adjacent to the other). In one aspect, the antennae may be in parallel planes, however, it will be noted that distance tolerances for NFC standard devices are relatively small, and the physically housing for these antennae will preferably enforce an identical or substantially identical distance for both antennae in such architectures. In this context, the positions of the antennae may be as close to parallel as possible within reasonable manufacturing tolerances, or as close to parallel as possible when disposed on two different layers of a shared printed circuit board, or preferably, when disposed on a single layer of a shared printed circuit board. The physiological monitor may further include a switch (e.g., a radio frequency (RF) switch or the like) in-line with the coil for the wireless data tag reader todisable the wireless data tag reader when power is being received to mitigate any effects on the efficiency of the wireless power transfer process. In particular, the switch may be configured to open when power is being received, and may be configured to close when the physiological monitor is looking for data tag to read.

[0099] Fig. 4 is a block diagram of a computing device 400. The computing device 400 may, for example, be a device used for continuous physiological monitoring, or any other device supporting a physiological monitor in the systems and methods described herein. The device may also or instead be any of the local computing devices described herein, such as a desktop computer, laptop computer, smart phone, Intemet-of-Things (loT) device (e.g., smart home system, audio system, thermostat, connective television, exercise device, and so forth), or other local computing device that might be used by a user as a physiological monitoring device, or in combination with a physiological monitoring device.. The device may also or instead be any of the remote computing resources described herein, such as a web server, a cloud database, a file server, an application server, or any other remote resource or the like. While described as a physical device, it will be understood that the exemplary computing device 400 may also or instead be realized as a virtual computing device such as a virtual machine executing a web server or other remote resource in a cloud computing platform. In general, the device 400 may include one or more sensors 402, a battery 404, a processor 408, memory 410, a network interface 414, and a user interface 416, or virtual instances of one or more of the foregoing.

[0100] The sensors 402 may include any sensor or combination of sensors suitable for heart rate monitoring as contemplated herein, as well as sensors 402 for detecting calorie bum, position (e.g., through a Global Positioning System or the like), motion, activity and so forth. In one aspect, this may include optical sensing systems including LEDs or other light sources, along with photodiodes or other light sensors, that can be used in combination for photoplethysmography measurements of heart rate, pulse oximetry measurements, and other physiological monitoring.

[0101] The sensors 402 may also or instead include one or more sensors for activity measurement. In some embodiments, the system may include one or more multi -axes accelerometers and / or gyroscope to provide a measurement of activity. In some embodiments, the accelerometer may further be used to filter a signal from the optical sensor for measuring heart rate and to provide a more accurate measurement of the heart rate. In some embodiments, the wearable system may include a multi -axis accelerometer to measure motion and calculate distance. Motion sensors may be used, for example, to classify or categorize activity, such as walking, running, performing another sport, standing, sitting or lying down. The sensors 402may, for example, include a thermometer for monitoring the user’s body or skin temperature. In one embodiment, the sensors 402 may be used to recognize sleep based on a temperature drop, Galvanic Skin Response data, lack of movement or activity according to data collected by the accelerometer, reduced heart rate as measured by the heart rate monitor, and so forth. The body temperature, in conjunction with heart rate monitoring and motion, may be used, e.g., to interpret whether a user is sleeping or just resting, as well as how well an individual is sleeping. The body temperature, motion, and other sensed data may also be used to determine whether the user is exercising, and to categorize and / or analyze activities as described in greater detail below. In another aspect, the sensors 402 may include one or more contact sensors, such as a capacitive touch sensor or resistive touch sensor, for detecting placement of a physiological monitor for use on a user. More generally, the sensors 402 may include any sensor or combination of sensors suitable for monitoring geographic location, physiological state, exertion, movement, and so forth in any manner useful for physiological monitoring as contemplated herein.

[0102] The battery 404 may include one or more batteries configured to allow continuous wear and usage of the wearable system. In one embodiment, the wearable system may include two or more batteries, such as a removable battery that may be removed and recharged using a charger, along with an integral battery that maintains operation of the device 400 while the main battery charges. In another aspect, the battery 404 may include a wireless rechargeable battery that can be recharged using a short range or long range wireless recharging system.

[0103] The processor 408 may include any microprocessor, microcontroller, signal processor or other processor or combination of processors and other processing circuitry suitable for performing the processing steps described herein. In general, the processor 408 may be configured by computer executable code stored in the memory 410 to provide activity recognition and other physiological monitoring functions described herein.

[0104] In general the memory 410 may include one or more non-transitory computer- readable media for storing one or more computer-executable instructions or software for implementing exemplary embodiments. The non-transitory computer-readable media may include, but are not limited to, one or more types of hardware memory, non-transitory tangible media (for example, one or more magnetic storage disks, optical disks, USB flash drives), and the like. In one aspect, the memory 410 may include a computer system memory or random access memory, such as DRAM, SRAM, EDO RAM, and the like. The memory 410 may include other types of memory as well, or combinations thereof, as well as virtual instances ofmemory, e.g., where the device is a virtual device. In general, the memory 410 may store computer readable and computer-executable instructions or software for implementing methods and systems described herein. The memory 410 may also or instead store physiological data, user data, or other data useful for operation of a physiological monitor or other device described herein, such as data collected by sensors 402 during operation of the device 400.

[0105] The network interface 414 may be configured to wirelessly communicate data to a server 420, e.g., through an external network 418 such as any public network, private network, or other data network described herein, or any combination of the foregoing including, e.g., local area networks, the Internet, cellular data networks, and so forth. Where the device is a physiological monitoring device, the network interface 414 may be used, e.g., to transmit raw or processed sensor data stored on the device 400 to the server 420, as well as to receive updates, receive configuration information, and otherwise communicate with remote resources and the user to support operation of the device. More generally, the network interface 414 may include any interface configured to connect with one or more networks, for example, a Local Area Network (LAN), a Wide Area Network (WAN), the Internet, or a cellular data network through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (for example, 802.11, Tl, T3, 56kb, X.25), broadband connections (for example, ISDN, Frame Relay, ATM), wireless connections, or some combination of any or all of the above. The network interface 412 may include a built-in network adapter, network interface card, PCMCIA network card, card bus network adapter, wireless network adapter, USB network adapter, modem or any other device suitable for interfacing the computing device 400 to any type of network capable of communication and performing the operations described herein.

[0106] The user interface 416 may include any components suitable for supporting interaction with a user. This may, for example, include a keypad, display, buzzer, speaker, light emitting diodes, and any other components for receiving input from, or providing output to, a user. In one aspect, the device 400 may be configured to receive tactile input, such as by responding to sequences of taps on a surface of the device to change operating states, display information and so forth. The user interface 416 may also or instead include a graphical user interface rendered on a display for graphical user interaction with programs executing on the processor 408 and other content rendered by a physical display of device 400.

[0107] An aspect of the present disclosure is a system and method for allowing a user to intelligently query the physiological monitoring system in order to provide personalized, and readily understandable, responses. As described in detail below the system takes a user’s input query and utilizes context-driven prompt engineering in order to generate a query which issupplied to a large language model (LLM). The output is provided as a natural language output which is therefore easy to understand for the user.

[0108] Fig. 5 illustrates the components of a system 500 for context-driven prompt engineering. The system 500 in an embodiment implements one or more of the methods described herein.

[0109] The system 500 comprises an interaction module 502 communicatively coupled to a user device 504, one or more services 506, a first large language model (LLM) 508, a second LLM 510, and one or more vector databases 512. The interaction module 502 comprises a context service 514, a chat history service 516, and a response presenter service 518. The context service 514 comprises topic classifiers 520, a query agent 522, mapping functions 524, and a context reducer 526. Also shown within Fig. 5 are user data 528 and a command 530 that comprises an instruction portion 532 and a context portion 534 and optionally comprises an optimization portion 535. Fig. 5 further shows an LLM response 536, a message 538, a response 540, a load optimizer 542, and a quality control module 544.

[0110] The interaction module 502 obtains the message 538 from the user device 504 and generates the command 530 for the second LLM 510 by dynamically incorporating into the command 530 contextual data — both static and dynamic — relating a query within the message 538. The LLM response 536 is then used to provide the response 540 provided to the user.[OHl] The context service 514 receives the message 538 from the user and outputs context data that is used to generate the context portion 534 of the command 530. The topic classifiers 520 identify one or more topics related to the query within the message 538. Here, a topic may be understood as a high-level category, theme, or subject. For example, the query, “what is my optimal resting heart rate?” may relate to topics such as “heart rate”, “user”, and “optimal heart rate”. The topic classifiers 520 may utilize a hierarchy of topic classifier and / or the first LLM 508 to identify topics within the message 538. The first LLM 508 is any suitable proprietary or third party large language model. For example, the LLMs of the present disclosure may be proprietary LLMs that are internally deployed within and / or for the physiological monitoring system (e.g., as part of the other resources 250 shown in the physiological monitoring system 200 of Fig. 2). Alternatively, the LLMs may be offered as a service by a third party. In such implementations, the LLMs are accessible via one or more application programming interface (API) endpoints. Example third party services include GPT-3 provided by OpenAI, Bloom by BigScience, and Google Bard. In one embodiment, the first LLM 508 is a simpler LLM than the second LLM 510 because the first LLM 508 is only tasked with identifying topics and not tasked with the more complex rephrasing task. This enables thetopic classifiers 520 to identify the topics related to the message 538 more efficiently, by reducing the computational requirement needed, as it is a simpler model. In one embodiment, if one or more of the topics identified by the topic classifiers 520 are identified as complex, or ambiguous, topics, then the query agent 522 is used to generate a structured query from the message 538. As described in more detail below in relation to Fig. 9, the structured query provides a mechanism for resolving complex or ambiguous queries within the message 538. The mapping functions 524 comprise a set of functions configured to map from a topic to a set of context data related to the topic. Each topic identified by the topic classifiers 520 is associated with at least one of the one or more mapping functions 524. A mapping function may obtain static data or dynamic data related to a topic. Static data generally corresponds to context data that remains fixed unless manually edited, whereas dynamic data refers to context data that is user specific and dynamically changes. Examples of static data include documents or portions obtained from knowledge base articles, blog articles, online encyclopedias, user manuals, and the like. Such static data may be obtained from the one or more vector databases 512. Examples of dynamic data include user data 528 such as a user’s heart rate, strain, sleep duration, respiratory rate, personal goals, recent exercise activityjournalling data, and the like. Such user data 528 may be obtained from one or more services 506 of the physiological monitoring system. Other examples of dynamic data include local environmental data, availability of local facilities, local event listings, matching data, and / or data about likely interests based on user profile information. The context reducer 526 may aggregate the static and / or dynamic context data obtained by the mapping functions 524 to form the context portion 534 of the command. Aggregating the context data may include concatenating the context data or converting the context data to a structured form. Additionally, or alternatively, aggregating the context data may include identifying static and / or dynamic context data that has an associated score, probability, or likelihood above a predetermined threshold. Additionally, or alternatively, the context data may be filtered for relevance as part of the aggregation, or after the context data has been aggregated. The chat history service 516 may also provide as part of the context portion 534 one or more example conversations obtained from the one or more vector databases 512 based on a similarity between the message 538 and the historical messages linked to the one or more vector databases 512.

[0112] The command 530 provided to the second LLM 510 comprises the instruction portion 532 and the context portion 534 (generated by the context service 514 and, optionally, the chat history service 516). The instruction portion 532 comprises a predetermined instruction or prompt that provides the relevant instructions for the second LLM 510 to perform therequested rephrasing task. The predetermined instruction or prompt may include templates selected based on relevance and with template variables completed based on the contextual data gathered. The context portion 534 contains the contextual data or information upon which the rephrasing task is to be performed. In one embodiment, the command 530 comprises the optimization portion 535 that comprises instructions for the second LLM 510 to limit a length of the text-based response generated by the second LLM 510 according to a computational load on the second LLM 510. The computational load may be an empirical measure of the current load on the second LLM 510 or a prediction of the expected load on the second LLM 510. Advantageously, limiting the length of the output generated by the second LLM 510 based on the load on the second LLM 510 helps to manage system resources dynamically and effectively as there may be varying numbers of requests being handled by the second LLM 510 at any given time. The second LLM 510 may be any suitable proprietary or third party large language model that can provide a natural language representation of the context portion 534.

[0113] The LLM response 536 comprises the text-based natural language response to the query, or queries, contained in the message 538. The response presenter service 518 may be used to incorporate additional information, or elements, within the LLM response 536 prior to the response 540 (which is generated from the LLM response 536) being output to the user. The response presenter service 518 may replace template tokens within the LLM response 536 with elements (e.g., graphics, charts, etc.) to generate the response 540. Template tokens may be used to identify portions of the LLM response 536 that should be replaced by an image or graphic, a chart, a bullet point, a hyperlink, data related to the user, proprietary data, a document, an interactive element, and / or formatted text. As such, the response 540 provided to the user (via the user device 504) comprises a rich text response to the query, or queries, contained within the message 538.

[0114] In such systems it is desirable to know if the system is functioning as expected and to be able to monitor the performance of the system over time (e.g., through the use of a module such as the quality control module 544). The system provides an objective set of metrics by which to monitor system performance.

[0115] Fig. 6 is a flow chart illustrating a method 600 for quantifying artificial agent performance (e.g., quantifying performance of an artificial agent such as the interaction module 502 shown in Fig. 5). The method 600 may be used in cooperation with any of the devices, systems, and methods described herein. In one embodiment, the method 600 is performed by an audit or quality control module such as the quality control module 544 shown in Fig. 5. In general, the method 600 obtains two statements related to an interaction between auser and an artificial agent (e.g., two of: a query provided by the user to the artificial agent, a response or output generated by the artificial agent in response to the query, a ground truth statement related to the query, contextual statements related to the query, etc.). The two statements are compared by decomposing each statement into a set of clauses and determining a matching between the two sets of clauses which encodes similarities between the two sets. A metric is applied to the matching to determine a value which quantifies the coherence between the two statements. That is, the matching is determined from the first statement to the second statement such that the metric / value quantifies the coherence from the first statement to the second statement based on the matching. The value is then used to calculate a performance score of the artificial agent in relation to the interaction.

[0116] The method 600 advantageously allows the performance of an artificial agent to be quantified to a high degree of granularity. That is, different metrics can be used to encode different performance behaviors of the artificial agent in real-time or near real-time. The performance of the artificial agent over time can be tracked with respect to one or more metrics thereby allowing sub-optimal behaviors of the artificial agent to be identified and remediated quickly and efficiently. In conventional systems, manual monitoring of outputs is required to determine the performance of an artificial agent. The query and response volumes of most deployed systems vastly exceed the ability to check with manual oversight (e.g., 10 million users posting 10 million interactions a day leads to 100 million daily interactions to check for quality purposes). Additionally, the range of potential questions and answers are essentially infinite, which makes a sampling approach difficult and inaccurate. The method 600 addresses this bottleneck by allowing the performance of an artificial agent to be efficiently quantified with no manual oversight thereby allowing high volumes of queries and responses to be evaluated for quality purposes. Furthermore, the method 600 can be used to monitor and improve the performance of an artificial agent when changes to prompt engineering can lead to potentially unpredictable results. In addition to such online performance evaluation, the performance of an artificial agent can be evaluated offline to obtain a robust baseline measure of the artificial agent’s behavior under controlled conditions. Here, offline is to be understood as referring to evaluating the performance of the artificial agent after the interaction has taken place (e.g., after the interaction has completed). As will be described in more detail below, by quantifying different behaviors of the artificial agent, the overall performance and operation of the artificial agent can be efficiently improved.

[0117] As shown in step 602, the method 600 may include identifying a plurality of statements related to an interaction between a user and an artificial agent. In one embodiment,the user is a user of a physiological monitoring system such as that shown in Fig. 2 and the artificial agent forms a part of the physiological monitoring system (e.g., the interaction module 502 shown in Fig. 5). The interaction can take place as part of a conversation between the user and the artificial agent within an application. More generally, the interaction is any interaction involving the artificial agent. As such, the query may not necessarily originate from a user and may instead be an automatically generated query (e.g., a suggested question or the like). For example, the query can be a question that is generated by the system (e.g., by the artificial agent, an LLM, or the like) to prompt or elicit an interaction. Such automatically generated questions can be based on a conversation history of one or more users so that the questions mirror the type of questions that a user may ask. The questions can be generated by an LLM or predetermined based on one or more rules. While the foregoing is described with reference to the above described physiological monitoring system such a methodology may be used in other LLM based systems where there is a conversation between a user and an artificial agent. In particular, the methods of the foregoing disclosure are particularly useful and valuable in systems that require a high degree of confidence or accuracy in the response provided by an artificial agent and / or an LLM. For example, within a medical setting, the methods of the foregoing disclosure can be used to ensure a high degree of fidelity to curated content or to ensure a complete response is provided to a given query. Additionally, or alternatively, the methods of the foregoing disclosure can provide a user with information about how an output / response was scored thereby providing the user with explainable insights regarding the limitations to the output / response provided by an artificial agent or LLM.

[0118] The plurality of statements include at least two of: a query provided by the user to the artificial agent; an output generated by the artificial agent in response to the query; and / or one or more reference statements related to the query. As described below, different pairwise combinations of statements allow for different metrics to be calculated to quantify different aspects of the artificial agent’s performance with respect to the interaction. The query corresponds to a text-based, natural language, message provided to the artificial agent by the user (e.g., “What is HRV?”). In one embodiment, the query corresponds to the message 538 shown in Fig. 5. The output corresponds to a text-based, natural language, response generated by the artificial agent in response to the query received from the user (e.g., “HRV measures variation in heartbeats. Your HRV of 42ms suggests a readiness for optimal performance”). The artificial agent is configured to generate the output using a large language model (LLM) such as the first LLM 508 or the second LLM 510 shown in Fig. 5. In one embodiment, the output corresponds to the response 540 shown in Fig. 5 which is generated by the interaction module inresponse to the message 538. The one or more reference statements related to the query include statements, or text, which relate to the query such as context specific data or ground truth responses. The one or more reference statements can also include derived statements-e.g., statements, such as the query, which have been processed by an LLM or the like-and / or expected statements. As described in detail above, context specific data is used to generate a contextual and personalized response to the user’s query. Specifically, context specific data may be used to augment the information and knowledge available to the LLM. Context specific data may be static — e.g., statements or data obtained from articles, static files, and the like — or dynamic — e.g., statements or data which are directly related to the user or the activity of the user. A ground truth response is a reference statement corresponding to an expected response to a given query. The system (e.g., the system 500 shown in Fig. 5) can comprise a database of queries and corresponding ground truth responses. As described in more detail below, such predetermined question-answer (QA) pairs allow for controlled offline auditing or monitoring of the performance of an artificial agent. The skilled person will appreciate that online evaluation of artificial agent performance often only includes the query, the output, and the context data because ground truth statements to online user queries are unlikely to be available. Offline evaluation of artificial agent performance often further includes ground truth statements related to the query.

[0119] As shown in step 604, the method 600 may include iterating over one or more pairwise combinations of statements and performing steps 606 and 608. That is, steps 606 and 608 are performed for one or more pairwise combinations of the plurality of statements. In one embodiment, steps 606 and 608 are performed for each pairwise combination of the plurality of statements and / or all pairwise combinations of the plurality of statements. The set of all pairwise combinations of the plurality of statements include all 2-combinations generated from the plurality of statements. The set of pairwise combinations thus includes all possible pairs of statements that can be formed where each statement is paired exactly once with every other statement, without repeating any statements within a pair. For example, if there are three statements {SI, S2, S3}, then the set of pairwise combination of statements would be {(SI, S2), (SI, S3), (S2, S3)}. For ease of reference, in the following description a pairwise combination of the plurality of statements is considered to involve a first statement of the plurality of statements and a second statement of the plurality of statements.

[0120] As shown in step 606, the step 604 may include determining a matching between the first statement and the second statement. The matching encodes similarities between one or more clauses of the first statement and one or more clauses of the second statement. Here, aclause corresponds to a claim, request, or component of a statement (which may comprise multiple clauses). For example, the statement “What is my strain and my recovery?” contains two clauses: “What is my strain?”, and “What is my recovery?”, whereas the statement “What is the time?” contains only a single clause. Considering each statement as a set of clauses allows fine grained, accurate, and localized matchings to be made between statements. This is particularly advantageous for compound queries where the query should be split into separate components to consider the performance of the artificial agent at providing a response to each component in a given statement.

[0121] In general, a matching provides a unidirectional mapping from components, clauses, or portions of a first statement to components, clauses, or portions of a second statement. Each clause within the first statement can match, or map, to zero or one clauses in the second statement. Two clauses are matched, or mapped, if the similarity between the two statements exceeds a similarity criterion or threshold. For example, the clause “What is my strain?” matches the clause “Your strain is 10” because both clauses relate to the same topic (strain) but does not match the clause “You’re doing great” because the two clauses are semantically and factually distinct. A matching can be numerically represented as a list of pairs where the first element of a pair identifies a clause within the first statement and the second element of the pair identifies zero or one clauses within the second statement. For example, a first statement, SI, may comprise the set of clauses {Al, A2, A3} and a second statement, S2, may comprise the set of clauses {Bl, B2}; an example matching SI— >S2 between these two sets can be represented as {(Al, B2), (A2, -), (A3, -), (-, Bl)}. This example matching indicates that clause Al in the first statement matches clause B2 in the second statement, but there are no clauses in the second statement which match clauses A2 and A3 of the first statement (as indicated by the null, or empty, character “-”) and there are no clauses in the first statement which match clause Bl in the second statement. As is clear, matchings are bijective (i.e., the matchings from SI— >S2 are the same as the matchings from S2— >S1).

[0122] In one embodiment, the matching between the first statement and the second statement is determined by decomposing the first statement into a first set of clauses, decomposing the second statement into a second set of clauses, and then mapping (or matching) the first set of clauses to the second set of clauses in order to determine the matching, where a first clause matches a second clause if a similarity between the two clauses satisfies a similarity criterion. A statement can be decomposed into a constituent set of clauses using a large language model (LLM) such as the first LLM 508 or the second LLM 510 shown in Fig. 5 and described in more detail above. More particularly, a prompt can be provided to the LLM, and a set ofclauses for the statement can then be obtained from the LLM in response to the provided prompt. The prompt comprises the statement and a command, or instruction, which is operable to cause the LLM to extract clauses from the statement. The command may further cause the LLM to extract complete, or overlapping, clauses such that each clause forms a grammatically complete claim or clause. For example, the statement “Your strain is 10 and recovery 20%” can be decomposed into clauses (“Your strain is 10”, “recovery 20%”) or into grammatically complete clauses (“Your strain is 10”, “Your recovery is 20%”).

[0123] The step of mapping (or matching) the first set of clauses to the second set of clauses in order to determine the matching can be performed using an LLM (e.g., the first LLM 508 or the second LLM 510 shown in Fig. 5). The LLM used to perform the mapping can be the same as, or different from, the LLM used to decompose the statements into clauses as described above. A prompt for the LLM is generated by combining the first set of clauses, the second set of clauses, and a command operable to cause the LLM to determine potential matches between clauses within the first set of clauses and the second set of clauses. In one embodiment, the first set of clauses and the second set of clauses included within the prompt are numbered (enumerated) and the command causes the LLM to output a matching (or mapping) in the format “(1, 2), (2, -), (-, 1), ...” where the first element of the pair references the numbering of the first set of clauses related to a first statement and the second element of the pair references the numbering of the second set of clauses related to a second statement (with indicating no match). The matching between the first statement and the second statement can then be obtained from the LLM as the matching, or mapping, between the two sets of clauses. As described below, the matching is parsed along with the statement and the clauses to determine a performance score for the artificial agent.

[0124] As shown in step 608, the step 604 may include calculating a metric value by applying a metric to the matching. The metric quantifies a coherence between the first statement and the second statement based on the matching. Here, coherence corresponds to the semantic connection, correlation, and / or relationship between the two statements. Two statements having a high degree of coherence would have clauses which are semantically related (e.g., relate to the same subject, the same topic, be identical or similar, etc.). The way in which this relationship is quantified is dependent upon the content of the two statements and the metric used to determine the metric value. In general, the metric is a precision metric or a recall metric. Precision and recall are known metrics typically used to measure errors in binary classification. More particularly, precision and recall measure errors in relation to a reference result and a test result. Within the context of the present disclosure, precision and recall measure errors in relation toclauses of a reference statement and a test statement. Throughout the following, the first clause in a matching (pairing) corresponds to a clause of the reference statement and the second clause in the matching corresponds to a clause of the test statement.

[0125] A precision metric calculates the number of true positive matchings (TP) between the two statements divided by the sum of the number of true positive matchings between the two statements and the number of false positive matchings (FP) between the two statements. A recall metric calculates the number of true positive matchings between the two statements divided by the sum of the number of true positive matchings between the two statements and the number of false negative matchings (FN) between the two statements. A true positive matching corresponds to a correct matching of clauses between two statements (e.g., the pair (1, 2) corresponds to a true positive matching because clause 1 of the reference statement matches to clause 2 of the test statement). A false positive matching corresponds to the situation whereby a clause within the test statement is not matched to a clause within the reference statement (e.g., the pair (-, 1) corresponds to a false positive matching because clause 1 in the test statement does not match to any clauses within reference first statement). That is, a false positive matching occurs when the test statement includes a clause which is not present, or matched to a clause, within the reference statement. A false negative matching corresponds to the situation where a clause within the reference statement is not matched to a clause within the test statement (e.g., the pair (3, -) corresponds to a false negative matching because the test statement does not include any clauses which match to clause 3 in the reference statement). That is, a false negative matching occurs when the reference statement includes a clause which is not present, or matched to a clause, within the test statement. As described below, the precision and recall metrics of the present disclosure quantify different aspects of the artificial agent’s performance depending on the content of the first statement and the second statement.

[0126] To illustrate the different metrics which can be calculated, Fig. 7 shows all pairwise combinations of statements between a query 702 provided by a user to an artificial agent, an output 704 provided by the artificial agent in response to the query 702, context specific data 706 related to the query 702 and used by the artificial agent to generate the output 704, and a ground truth response 708 to the query 702. Fig. 7 further illustrates a completeness metric 710, a relevance metric 712, a faithfulness metric 714, a context utilization metric 716, a context recall metric 718, a context precision metric 720, a question coverage metric 722, a question alignment metric 724, a context integrity metric 726, a context efficiency metric 728, a sensitivity metric 730, and a positive predictive value metric 732.

[0127] Throughout the discussion of Fig. 7, an example will be used whereby the query 702 (Q) is “What is my strain and my recovery?”, the output 704 (O) is “Your strain is 10. Great job, you’re doing great!”, the context specific data 706 (C) is the set of contextual statements “User’s Strain is 10. User’s Recovery is 34%. Stress is 2.5”, and the ground truth response 708 (G) is “Your strain is 10 and recovery is 34%”. The query 702 (Q) is decomposed into the set of clauses {QI : “What is my strain?”; Q2: “What is my recovery?”}, the output 704 (O) is decomposed into the set of clauses {01 : “Your strain is 10.”; 02: “Great job, you’re doing great!”}, the context specific data 706 (C) is decomposed into the set of clauses {Cl : “User’s strain is 10.”; C2: “User’s Recovery is 34%.”; C3: “User’s Stress is 2.5.”}, and the ground truth response 708 (G) is decomposed into the set of clauses {Gl : “Your strain is 10.”; G2: “Your recover is 34%.”}. The matchings are thus calculated as Q^O = {(QI, 01), (Q2, -), (-, 02)}, Q^C = {(QI, Cl), (Q2, C2), (-, C2)}, Q^G = {(QI, Gl), (Q2, G2)}, O^C = {(01, Cl), (02, -), (-, C2), (-, C3)}, G— >O = {(Gl, 01), (G2, -), (-, 02)}, and G^C = {(Gl, Cl), (G2, C2), (-, C3)}.

[0128] The completeness metric 710 is a recall metric defined when the first statement (i.e., the reference statement) is the query 702 and the second statement (i.e., the test statement) is the output 704 generated by the artificial agent in response to the query 702. The completeness metric 710 quantifies a degree of satisfaction of the query provided by the output (i.e., how much of the question is being satisfied by the response). In the above example, the completeness metric 710 computed on the matching Q^O = {(QI, 01), (Q2, -), (-, 02)} corresponds to the number of true positives (TP=1 because QI matches 01) divided by the sum of the number of true positives and the number of false negatives (FN=1 because Q2 in the reference statement — the query 702 — is not matched to a relevant clause within the test statement — the output 704). Therefore, the completeness metric 710 is calculated for this example as 1 / 2=50%.

[0129] The relevance metric 712 is a precision metric defined when the first statement (i.e., the reference statement) is the query 702 and the second statement (i.e., the test statement) is the output 704 generated by the artificial agent in response to the query 702. The relevance metric 712 quantifies a relevance of the output to the query (i.e., how much of the response is relevant to the query or question being asked). In the above example, the completeness metric 710 computed on the matching Q^O = {(QI, 01), (Q2, -), (-, 02)} corresponds to the number of true positives (TP=1 because QI matches 01) divided by the sum of the number of true positives and the number of false positive (FP=1 because 02 in the test statement — theoutput 704 — is not matched to a relevant clause within the reference statement — the query 702). Therefore, the relevance metric 712 is calculated for this example as 1 / 2=50%.

[0130] The faithfulness metric 714 is a recall metric defined when the first statement (i.e., the reference statement) is the output 704 generated by the artificial agent in response to the query 702 and the second statement (i.e., the test statement) is the context specific data 706 related the query 702. The faithfulness metric 714 quantifies a proportion of the output related to the context specific data (i.e., how much of the response generated by the artificial agent was sourced from the context specific data as opposed to the LLM). In the above example, the faithfulness metric 714 computed on the matching O^C = {(01, Cl), (02, -), (-, C2), (-, C3)} corresponds to the number of true positives (TP=1 because 01 matches Cl) divided by the sum of the number of true positives and the number of false negatives (FN=1 because 02 in the reference statement — the output 704 — is not matched to a relevant clause within the test statement — the context specific data 706). Therefore, the completeness metric 710 is calculated for this example as 1 / 2=50%. Advantageously, the faithfulness metric 714 can help approximate the hallucination rate of the LLM used by the artificial agent to generate the output 704. As is known, hallucinations occur when an LLM generates an output that is factually incorrect, nonsensical, and / or disconnected from the input query or prompt. By quantifying the proportion of the response that was sourced from the context specific data (as opposed to content introduced by the LLM), the faithfulness metric 714 provides an indication of the extent to which potential hallucinations are being included within the output 704. This in turn can provide a degree of confidence in which a user, or other systems, can place on the output 704.

[0131] The context utilization metric 716 is a precision metric defined when the first statement (i.e., the reference statement) is the output 704 generated by the artificial agent in response to the query 702 and the second statement (i.e., the test statement) is the context specific data 706 related the query 702. The context utilization metric 716 quantifies a proportion of the context specific data used to generate the output (i.e., how much of the context was used to generate the answer or response). In the above example, the context utilization metric 716 computed on the matching O^C = {(01, Cl), (02, -), (-, C2), (-, C3){ corresponds to the number of true positives (TP=1 because 01 matches Cl) divided by the sum of the number of true positives and the number of false positive (FP=2 because C2 and C3 in the test statement — the context specific data 706 — are not matched to relevant clauses within the reference statement — the output 704). Therefore, the context utilization metric 716 is calculated for this example as 1 / 3-33%.

[0132] The context recall metric 718 is a recall metric defined when the first statement (i.e., the reference statement) is the ground truth response 708 to the query 702 and the second statement (i.e., the test statement) is the context specific data 706 related the query 702. The context recall metric 718 quantifies the proportion of the ground truth response contained within the context specific data. In the above example, the context recall metric 718 computed on the matching G^C = {(Gl, Cl), (G2, C2), (-, C3){ corresponds to the number of true positives (TP=2 because Gl matches Cl and G2 matches C2) divided by the sum of the number of true positives and the number of false negatives (FN=0 because all clauses within the reference statement — the ground truth response 708 — are included within the test statement — the context specific data 706). Therefore, the context recall metric 718 is calculated for this example as 2 / 2=100%.

[0133] The context precision metric 720 is a precision metric defined when the first statement (i.e., the reference statement) is the ground truth response 708 to the query 702 and the second statement (i.e., the test statement) is the context specific data 706 related the query 702. The context precision metric 720 quantifies a proportion of the context specific data needed to generate the ground truth response (i.e., how much of the context is necessary to generate to the ground truth). In the above example, the context precision metric 720 computed on the matching G^C = {(Gl, Cl), (G2, C2), (-, C3)} corresponds to the number of true positives (TP=2 because Gl matches Cl and G2 matches C2) divided by the sum of the number of true positives and the number of false positive (FP=1 because C3 the test statement — the context specific data 706 — is not matched to relevant clauses within the reference statement — the ground truth response 708). Therefore, the context precision metric 720 is calculated for this example as 2 / 3-66%. Beneficially, the context precision metric 720 quantifies the amount of noise in the context specific data 706 and thus provides a measure of the performance of a context service such as the context service 514 as shown in Fig. 5. A higher value of context precision indicates that a lower amount of extra, or superfluous, context data is retrieved and included in the context specific data.

[0134] The question coverage metric 722 is a recall metric defined when the first statement (i.e., the reference statement) is the query 702 and the second statement (i.e., the test statement) is the ground truth response 708 to the query 702. The question coverage metric 722 quantifies a degree to which the ground truth covers the extent of the query. In the above example, the question coverage metric 722 computed on the matching Q^G = {(QI, Gl), (Q2, G2){ corresponds to the number of true positives (TP=2 because QI matches Gl and Q2 matches G2) divided by the sum of the number of true positives and the number of falsenegatives (FN=O because all clauses within the reference statement — the query 702 — are included within the test statement — the ground truth response 708). Therefore, the question coverage metric 722 is calculated for this example as 2 / 2=100%. As will be described in more detail below, the question coverage metric 722 is particularly useful when evaluating the quality of a test question-answer set used to perform an offline evaluation of the artificial agent’s performance.

[0135] The question alignment metric 724 is a precision metric defined when the first statement (i.e., the reference statement) is the query 702 and the second statement (i.e., the test statement) is the ground truth response 708 to the query 702. The question alignment metric 724 quantifies a degree to which the query 702 elicited the ground truth response 708. In the above example, the question alignment metric 724 computed on the matching Q^G = {(QI, Gl), (Q2, G2)} corresponds to the number of true positives (TP=2 because QI matches Gl and Q2 matches G2) divided by the sum of the number of true positives and the number of false positive (FP=0 because all clauses within the reference statement — the query 702 — are included within the test statement — the ground truth response 708). Therefore, the question alignment metric 724 is calculated for this example as 2 / 2=100%. Beneficially, the question alignment metric 724 quantifies the amount of noise in the context specific data 706 and thus provides a measure of the performance of a context service such as the context service 514 as shown in Fig. 5. The question alignment metric 724 is particularly useful when evaluating the quality of a test question-answer set used to perform an offline evaluation of the artificial agent’s performance.

[0136] The context integrity metric 726 is a recall metric defined when the first statement (i.e., the reference statement) is the query 702 and the second statement (i.e., the test statement) is the context specific data 706 related to the query 702. The context integrity metric 726 quantifies a degree to which the context specific data 706 provided an answer or response to the query. In the above example, the context integrity metric 726 computed on the matching Q^C = {(QI, Cl), (Q2, C2), (-, C2){ corresponds to the number of true positives (TP=2 because QI matches Cl and Q2 matches C2) divided by the sum of the number of true positives and the number of false negatives (FN=0 because all clauses within the reference statement — the query 702 — are included within the test statement — the context specific data 706). Therefore, the context integrity metric 726 is calculated for this example as 2 / 2=100%.

[0137] The context efficiency metric 728 is a precision metric defined when the first statement (i.e., the reference statement) is the query 702 and the second statement (i.e., the test statement) is the context specific data 706 related to the query 702. The context efficiencymetric 728 quantifies a proportion of the context specific data 706 needed to answer the query 702 (i.e., how much of the context was necessary to answer the question). In the above example, the context efficiency metric 728 computed on the matching Q^C = {(QI, Cl), (Q2, C2), (-, C2)} corresponds to the number of true positives (TP=2 because QI matches Cl and Q2 matches C2) divided by the sum of the number of true positives and the number of false positive (FP=1 because the test statement — the context specific data 706 — includes clause C2 which is not matched to a clause within the reference statement — the query 702). Therefore, the context efficiency metric 728 is calculated for this example as 2 / 3-66%.

[0138] The sensitivity metric 730 (alternatively referred to as recall or output recall) is a recall metric defined when the first statement (i.e., the reference statement) is the ground truth response 708 to the query 702 and the second statement (i.e., the test statement) is the output 704 provided by the artificial agent in response to the query 702. The sensitivity metric 730 quantifies the correctness of the output 704 (i.e., how much of the ground truth response 708 is contained within the output 704 which is a measure of similarity between the two statements). In the above example, the sensitivity metric 730 computed on the matching G— >O = {(Gl, 01), (G2, -), (-, 02)} corresponds to the number of true positives (TP=1 because G1 matches 01) divided by the sum of the number of true positives and the number of false negatives (FN=0 because clause G2 in the reference statement — the ground truth response 708 — does not match to a clause in the test statement — the output 704). Therefore, the sensitivity metric 730 is calculated for this example as 1 / 2=50%. The sensitivity metric 730 can be used as an indicator of the general accuracy, or performance, of the artificial agent in providing a suitable response to the query 702.

[0139] The positive predictive value metric 732 (alternatively referred to as precision or response precision) is a precision metric defined when the first statement (i.e., the reference statement) is the ground truth response 708 to the query 702 and the second statement (i.e., the test statement) is the output 704 provided by the artificial agent in response to the query 702. The positive predictive value metric 732 quantifies a redundancy of the output 704 in relation to the ground truth response 708 (i.e., how much of the response was necessary to meet the ground truth). In the above example, the positive predictive value metric 732 computed on the matching G— >O = {(Gl, 01), (G2, -), (-, 02)} corresponds to the number of true positives (TP=1 because Gl matches 01) divided by the sum of the number of true positives and the number of false positive (FP=1 because the test statement — the output 704 — includes clause 02 which is not matched to a clause within the reference statement — the ground truth response 708). Therefore, the positive predictive value metric 732 is calculated for this example as 1 / 2=50%.Advantageously, the positive predictive value metric 732 provides a quantification of the noise level of the output 704 (e.g., how much extra, superfluous, information has been included in the response).

[0140] Referring once again to Fig. 6, as shown in step 604, metric values can be calculated for one or more pairwise combinations of statements by performing steps 606 and 608 for each pairing. For example, if the plurality of statements include the query provided by the user to the artificial agent and the output generated by the artificial agent in response to the query, then the steps 606 and 608 can be performed to generate values for the completeness metric and / or the relevance metric (e.g., the completeness metric 710 and / or the relevance metric 712 shown in Fig. 7). As a further example, if the plurality of statements include the query provided by the user to the artificial agent, the output generated by the artificial agent in response to the query, and context data related to the query, then the steps 606 and 608 can be performed over the pairings {(query, output), (query, context), (output, context)} to generate values for the completeness metric, the relevance metric, the faithfulness metric, the context utilization metric, the context integrity metric, and / or the context efficiency metric (e.g., the completeness metric 710, the relevance metric 712, the faithfulness metric 714, the context utilization metric 716, the context integrity metric 726, and / or the context efficiency metric 728 shown in Fig. 7).

[0141] As shown in step 610, the method 600 may include calculating a performance score of the artificial agent in relation to the interaction based on the metric value determined for one or more pairwise combinations of the plurality statements. In one embodiment, the performance score is the combination, or concatenation, of all metric values calculated during step 604. For example, the performance score of the artificial agent in relation to the interaction may be values of the completeness metric and the relevance metric concatenated into a vector of values. Alternatively, the performance score is calculated from the metric values calculated during step 604. For example, an Fl score (or F-Measure) can be calculated from one or more pairs of precision and recall metrics calculated for the artificial agent. As is known, Fl score is calculated using the formula 2 x ((precision x recall) / (precision + recall)). The Fl score can be calculated for specific statement pairings. For example, the overall performance of the artificial agent in providing an accurate response to a query with a known ground truth can be calculated from the sensitivity metric (recall) and positive predictive power (precision) using the above Fl score formula. As an alternative example, the overall performance of the artificial agent in utilizing context data within a response can be calculated from the faithfulness metric (recall) and the context utilization metric (precision) using the above Fl score formula. In furtherembodiments, the performance score of the artificial agent is calculated using the Matthew correlation coefficient (MCC), micro-averaging, and / or macro-averaging. Additionally, or alternatively, one or more of the metric values are compared to ideal, or expected, metric values or ratios to determine the performance score. The performance score can thus be calculated as a similarity of one or more metric values to their corresponding ideal metric values. For example, an ideal faithfulness metric value may be in the range of 45-65% and so a positive performance score is assigned if the artificial agent has a faithfulness metric value in or near this range and a negative performance value if outside of this range. As a further example, the performance score can be calculated as the similarity between the ratio of context efficiency to context integrity and the ideal, or expected, ratio of context efficiency to context integrity.

[0142] In one embodiment, the performance score is output after being calculated at step 610. For example, the performance score is output to a user or another component or module of a physiological monitoring system. In the example shown in Fig. 5, the performance score may be output from the quality control module 544 to a logging service or a maintenance service of the one or more services 506. Outputting the performance score can include displaying the performance score on a device viewable by an administrator of the physiological monitoring system, storing the performance score at a device (e.g., within a persistent data store), and / or transmitting the performance score.

[0143] As shown in step 612, the method 600 may include the step of identifying one or more performance trends related to the artificial agent based on the performance score calculated at step 610 and one or more other performance scores calculated for the artificial agent. That is, the performance score calculated at step 610 can be used in conjunction with one or more other performance scores to identify trends and / or behaviors of the artificial agent. For example, the one or more other performance scores may be historical performance scores obtained for the artificial agent from interactions involving the artificial agent and one or more users over a time period (e.g., 1 day, 1 week, 1 month, etc.). The performance trends can then reveal any changes in the artificial agent’s performance over this time period, such as a performance degradation, which may require an adjustment to the operation of the artificial agent to remediate.

[0144] As shown in step 614, the method 600 may include the step of causing an adjustment to the artificial agent based on the one or more performance trends. For example, if the performance trends reveal a decrease in the artificial agent’s performance, then the artificial agent may be reconfigured to attempt to remediate this drop in performance. As a further example, if the one or more performance trends reveal that an increase in hallucinations generated by the artificial agent (e.g., a decrease in the faithfulness metric over time), then theartificial agent can be configured to use a different large language model to attempt to reduce the number of hallucinations. Alternatively, the adjustment to the artificial agent is based on the performance score.

[0145] Fig. 8 is a flow chart illustrating a method 800 for generating ground truth question-answer (QA) test sets related to a subject. The method 800 may be used in cooperation with any of the devices, systems, and methods described herein. In one embodiment, the method 800 is performed by an audit or quality control module such as the quality control module 544 or a service of the one or more services 506 shown in Fig. 5. In general, the method 800 utilizes the above described statement decomposition approach and precision / recall metrics to generate high quality synthetic datasets of test data. This improves the speed and efficiency of offline evaluation of an artificial agent’s performance because large amounts of high quality ground truth data (e.g., queries with corresponding ground truth statements such as query 702 and ground truth response 708 shown in Fig. 7) can be generated automatically and without human intervention.

[0146] As shown in step 802, the method 800 may include identifying a set of QA pairs related to the topic. Each QA pair comprises a question and a corresponding ground truth response to the question (i.e., a ground truth answer). In one embodiment, the set of QA pairs are identified from an article or source (e.g., a knowledge base article, an academic or scientific article, a blog post, etc.) by a large language model (LLM). A prompt is provided to the LLM, and the set of QA pairs related to the subject are obtained from the LLM in response to the prompt. The prompt comprises an article related to the topic and a command operable to cause the LLM to extract questions and corresponding answers from the article. In one embodiment, step 802 is repeated for multiple articles related to the topic such that the set of QA pairs comprises QA pairs extracted from multiple articles.

[0147] As shown in step 804, the method 800 may include decomposing the set of QA pairs into a plurality of sets of clauses such that each question and each answer is decomposed into a set of question clauses and a set of answer clauses respectively. Step 804 corresponds to the decomposition performed as part of the matching at step 606 of Fig. 6.

[0148] In one embodiment, a faithfulness metric (e.g., the faithfulness metric 714 shown in Fig. 7) is applied to the answer within each QA pair and the article from which the QA pair was generated. The skilled person will appreciate that both the answer and the article, or a portion thereof, are decomposed prior to the faithfulness being applied. Here, the answer corresponds to the reference statement (cf. output) and the article from which the QA pair was generated corresponds to the test statement (cf. context). Because the faithfulness metric isindicative of the extent to which the artificial agent has hallucinated, the faithfulness metric can be used to determine how faithful ground truth answers are to the article. Any QA pair having a faithfulness metric value less than a predetermined threshold can be removed from the set of QA pairs (e.g., below 1, 0.999, 0.95, 0.9, etc.).

[0149] As shown in step 806, the method 800 may include determining one or more matchings between the plurality of sets of clauses. A matching defines a potential connective relationship, or similarity, between a first clause forming part of a first question of a first QA pair and a second clause forming part of a first answer of the first QA pair. Step 806 corresponds to the matching performed at step 606 of Fig. 6, with the question of a QA pair corresponding to the query and the answer of the QA pair corresponding to the ground truth statement related to the query.

[0150] As shown in step 808, the method 800 may include applying at least one metric to the plurality of sets of clauses based on the one or more matchings. The at least one metric measures a predicted suitability of each answer to each corresponding question. In one embodiment, the metrics used are question coverage and question alignment (e.g., the question coverage metric 722 and the question alignment metric 724 shown in Fig. 7). These metrics are described in more detail above where the question of a QA pair corresponds to the query (e.g., the query 702) and the answer of the QA pair corresponds to the ground truth statement (e.g., the ground truth response 708). As described above, question coverage quantifies how well the ground truth answer covers the extent of the question while question alignment quantifies how well the question is designed to elicit the ground truth as a response. These metrics therefore provide an efficient, compact, and comparable measure of the suitability of a question-answer pair.

[0151] As shown in step 810, the method 800 may include filtering the QA pairs according to the at least one metric thereby generating a synthetic data set of QA pairs related to the topic such that values of the at least one metric for each QA pair within the synthetic data set of QA pairs satisfy a threshold criterion. In one embodiment, any QA pair having a metric value above a pre-determined threshold amount (e.g., 1.0, 0.99, 0.98, 0.95, etc.) is included in the synthetic data set of QA pairs. In a further embodiment, the metrics are combined into a combined measure such as Fl score and any QA pair having a combined measure above a predetermined threshold amount (e.g., 1.0, 0.99, 0.98, 0;95, etc.) is include in the synthetic data set of QA pairs.

[0152] In one embodiment, the synthetic data set of QA pairs are output. Outputting the synthetic data set of QA pairs can include saving the synthetic data set of QA pairs at a storagelocation, utilizing the synthetic data set of QA pairs to evaluate the offline performance of an artificial agent, transmitting the synthetic data set of QA pairs, and the like.

[0153] Fig. 9 is a flow chart illustrating a method 900 for quantifying expected behavior of an artificial agent (e.g., quantifying the expected behavior of an artificial agent such as the interaction module 502 shown in Fig. 5). The method 900 may be used in cooperation with any of the devices, systems, and methods described herein. In one embodiment, the method 900 is performed by an audit or quality control module such as the quality control module 544 shown in Fig. 5. In general, the method 900 classifies a query into one or more query classes and extracts response metric values from an output provided by an artificial agent in response to the query. A quality matrix is then generated which combines the response metric values with each query class thereby encoding the expected behavior of the artificial agent. The method 900 advantageously allows observed behaviors of the artificial agent to be correlated with different types of queries thereby allowing emergent behaviors of the artificial agent to be identified during both online and offline evaluation. As described in more detail below, this helps optimize performance of the artificial agent in an efficient manner with a high degree of granularity as to the scenarios (e.g., input states) in which specific behaviors occur.

[0154] As shown in step 902, the method 900 may include identifying a query provided to an artificial agent by a user. In one embodiment, the query corresponds to the message 538 shown in Fig. 5. Alternatively, the query corresponds to a query, or question, contained within a test set of question-answer (QA) pairs used to perform an offline evaluation of the expected behavior of an artificial agent. For example, a query / question generated by the method 800 described in relation to Fig. 8 above.

[0155] As shown in step 904, the method 900 may include identifying an output generated by the artificial agent in response to the query. The artificial agent is operable to interface with a first large language model (LLM) to determine the output. In one embodiment, the artificial agent is the interaction module 502 which is configured to utilize the first LLM 508 and / or the second LLM 510 to generate the response 540. Alternatively, the output is an answer of a QA pair such as a QA pair generated by the method 800 described in relation to Fig. 8 above.

[0156] As shown in step 906, the method 900 may include classifying the query into one or more query classes which each define a characteristic of the query. The characteristic of the query is a higher-level characteristic of the query which relates to the content, topic, and / or meaning of the query as opposed to lower-level properties of the query such as sentence structure and / or grammar. As such, the characteristic may be referred to as a subjectcharacteristic, a topic characteristic, a semantic characteristic, a high-level characteristic, or a higher-order characteristic. Examples of query classes include a query type class, a subject tag class, a support identifier class, and a language identifier class. A query type class is the type of query / question that is asked based on a pre-defined set of categories. For example, the query “How can I get my strain to 15.8 with an average HR of 170 bpm?” would be assigned the query type class WHOOP SPECIFIC RECOMMENDATION which defines a topic (characteristic) of the query (i.e., the query relates to a specific recommendation), while the query "Why is my recovery still yellow when my restorative sleep is so high and my HRV is average?" would be assigned the query type class METRIC S EXPL AN ATION which defines a topic of the query (i.e., the query relates to an explanation of a metric). A subject tag class is a tagging of subjects pertaining to the query and is used to assign subject matter tags to a query. For example, the query “How can I lower my cholesterol” would be assigned the subject matter tags HEALTH, NUTRITION, CARDIOVASCULAR. A support identifier class is used to indicate whether or not a query relates to a specific request for support (e.g., the query “How do I charge my WHOOP?” would be assigned a support identifier class of CHARGING). The language identifier class is used to identify the language of the query (e.g., English, French, Welsh, etc.). More generally, any group and / or hierarchy of classes useful for categorizing the subject matter of user requests may be used to classify a query as described herein.

[0157] In one aspect, a query can be classified into the one or more query classes using any suitable machine learning classifier or natural language processing method. Which may rely on the corresponding model, and / or use augmentation such as retrieval augmented generation to specify various known categories prior to requesting classification from a language model or the like.

[0158] In one embodiment, the query is classified into the one or more query classes using a large language model (LLM). A prompt is provided to the LLM and the one or more query classes are obtained from the LLM in response to the prompt. The prompt comprises the query and a command, or instruction, operable to cause the LLM to classify the query into the one or more query classes. For example, the command may include the pre-defined set of categories used to assign a query to one or more query type classes along with an instruction to classify the query into zero or more of these categories. Similarly, the command may include an instruction to the LLM to extract specific subject matter tags from the query.

[0159] As shown in step 908, the method 900 may include determining one or more response metric values for the output. A response metric value quantifies an expected behavior of the artificial agent according to a respective response metric. The one or more responsemetric values correspond to one or more of: a refusal value; a follow up value; a degree of personalization; a deny-list word rate; a word misuse ratio; a verbosity value; and a language value.

[0160] A refusal value may indicate whether the artificial agent has refused to answer the query directly. This may occur due to a lack of context data (e.g., “Apologies, but the specific data for your highest HRV week isn’t available at the moment”) or the artificial agent refusing something that is out of scope (e.g., “I am sorry, but I am unable to answer legal questions ”). In one embodiment the refusal value is a binary value indicating whether the artificial agent has refused to answer the query directly. The refusal value is set to True if the response (output) generated by the artificial agent contains keywords for a pre-defined list of keywords (e.g., [“unable”, “not available”, ...]). Alternatively, the refusal value is determined by providing a prompt to an LLM and obtaining the refusal value from the LLM in response to the prompt. The prompt contains the output and a command operable to cause the LLM to return a value of True if the output is a refusal and False if not. In other embodiments, the refusal value is indicative of a category or level of refusal. For example, a full refusal category is assigned if the artificial agent has refused to answer the query in full, a partial refusal category is assigned if the artificial agent has refused to answer a part, but not all, of the query, and a refusal with tangential answer is assigned if the artificial agent provided a tangential answer as part of the refusal.

[0161] A follow up value may indicate whether the output provided by the artificial agent in response to the query contains a follow up question or query. For example, the output “Your recovery was 1%. Would you like some tips on how to improve recovery?” contains the follow up question “Would you like some tips on how to improve recovery?”. In such an example, the follow up value would be set to True; whereas if no follow up questions are asked, then the follow up value would be set to False. The follow up value is set to True if the output provided by the artificial agent contains a “?” character. Alternatively, the follow up value is determined by providing a prompt to an LLM and obtaining the follow up value from the LLM in response to the prompt. The prompt contains the output and a command operable to cause the LLM to return a value of True if the output contains a question and False if not. In other embodiments, the command is operable to cause the LLM to classify the output into zero or more follow categories which define the type, scope, and / or suitability of the follow up question.

[0162] A degree of personalization value may indicate a degree to which the output provided by the artificial agent in response to the user’s query is personalized with the user’s data. Given the user query “What’s HRV?”, a personalized response would be “HRV measuresvariation in heartbeats. Your HRV of 42ms suggests a readiness for optimal performance” while a non-personalized response would be “HRV reflects your body’s readiness for physical and mental performance”. In one embodiment, the context utilization metric 716 described in relation to Fig. 7 above is used to determine the degree of personalization value (with a value of 0 indicating that the output contains no context data and is thus not personalized). Alternatively, the degree of personalization value is set to True if the output contains any numeric values. In one embodiment, the degree of personalization is calculated according to a hierarchy of personalization, with a degree of personalization being assigned to one or more levels in the hierarchy. For example, the lowest level of personalization in the hierarchy can be indicative of there being no user data provided, the next level can be indicative of user data being provided, the next level can be indicative of insights being provided, the next level can be indicative of actionable insights being provided, and so on.

[0163] A deny-list word rate may indicate the number of times that a word appearing on a deny-list of words appears in the output. For example, if the output contains 2 words which appear on the deny-list of words, then the deny-list word rate would be equal to 2. Similarly, a word misuse ratio indicates the number of times an incorrect word is used in relation to the correct word. The word misuse ratio is calculated using a list of word pairs, such as (“swole”, “swollen”) where “swole” is the incorrect word and “swollen” is the correct word. The word misuse ratio for a given word pair is then calculated by counting the number of occurrences in the output of each word in the word pair. A word misuse ratio of 2: 1 indicates that the output uses the incorrect word twice and the correct word once.

[0164] A verbosity value may be calculated using a compactness function which is a natural language processing function used to determine the degree of verbosity (e.g., length, variety of words used, redundancy, etc.) of the output. The language value may be used to identify the language of the output (e.g., English, French, etc.). The output can be classified into one or more query classes using any suitable machine learning classifier or natural language processing method.

[0165] As shown in step 910, the method 900 may include generating a quality matrix for the agent based on the one or more query classes and the one or more response metric values. The quality matrix encodes a set of expected behaviors of the artificial agent for each of the one or more query classes, e.g., by providing an array of pairwise relationships between each query class and each response metric value. That is, the response metric values identified at step 908 are associated, or linked, to each query class identified at step 906 to allow behaviors of the artificial agent to be correlated to specific categories, or classes, of query. As such, the matrixcan contain, for each query class, a list of response metric values (e.g., each row corresponds to a query class and each column corresponds to a response metric such that each value in the matrix corresponds to a response metric value for a query class according to a response metric). The matrix can contain multiple response metric values for each query class (e.g., due to the quality matrix containing the results obtained from multiple query / output pairs). A response metric value can have an ideal, or expected, value or ratio which can be used to quantify and / or compare the performance of the artificial agent to a baseline performance. The ideal, or expected, values or ratios can differ across different query classes.

[0166] In one embodiment, the method 900 may further comprise outputting the quality matrix. Outputting the quality matrix may include displaying, transmitting, or otherwise storing the quality matrix. The quality matrix may be output by being appended to a previously generated quality matrix to incorporate new results into the previously generated quality matrix.

[0167] In one embodiment, the method 900 may further comprise identifying one or more trends or patterns in the quality matrix. The one or more trends or patterns can be identified using any suitable statistical approach such as correlation analysis (e.g., determining the degree of correlation between a query class and a response metric), outlier analysis (e.g., identifying response metric values which are outliers), and / or machine learning approaches such as clustering algorithms, regression algorithms, and the like. Identifying trends within the quality matrix becomes particularly advantageous when observing the behavior of an artificial agent over a large number of outputs and query classes because emergent behaviors of the artificial agent can be identified and linked to specific query classes. For example, it may be determined from such analysis that the artificial agent is refusing to answer any queries which have a subject tag of LEGAL (which is a positive behavior) but does not provide any follow up questions for queries having a METRIC S EXPL AN ATION query type (which is a behavior which can be improved). As a further example, it may be determined that the artificial agent always responds in French to queries provided in Welsh because the artificial agent cannot respond in that language (although a more preferable behavior would be to respond in English). As described in more detail below, identifying trends in this manner allows the performance of the artificial agent to be optimized and improved by identifying the input states which lead to sub-optimal performance thereby guiding development to improve the outputs for those input states.

[0168] As shown in step 912, the method 900 may comprise causing an adjustment to the artificial agent based on the quality matrix. In one embodiment, the adjustment to the artificial agent is based on the one or more trends or patterns identified in the quality matrix (or a quality matrix containing the response values for the artificial agent over a large number ofqueries and outputs). Adjustment to the artificial agent can comprise turning off specific features of the artificial agent, causing the artificial agent to utilize a different LLM (e.g., switch from using the first LLM 508 shown in Fig. 5 to the second LLM 510), cause the artificial agent to access a different and / or wider selection of context data, etc.

[0169] Additionally, or alternatively, the adjustment to the artificial agent can comprise an adjustment to a response provided to the user. The adjustment to the artificial agent can be caused automatically or in response to a human input confirming that the adjustment is to take place. For example, the quality matrix and metrics may be used to define a score or scores that form a threshold quality measure, or fidelity score, for the LLM output (e.g. that the output of the LLM achieve at least fidelity score of 80% or similar). After the user-facing output is generated by the LLM, the system may, instead of displaying the output, send the output to a server to be scored against a matching quality matrix (the quality matrix associated with the query type identified in the earlier step). Where the score is above the pre-defined threshold, the output is then processed or displayed (e.g., the output is passed to the user’s device for display to the user). Where the score is below the threshold, the output and the quality score and a prompt to require an improvement in the below-threshold dimension are sent back to the LLM and the LLM generates a new message. The new message is then scored and, if above the threshold, is displayed (as described above), and if below, is sent back again along with the failed messages and another prompt to improve. This process may be repeated a predetermined number of times (e.g. 2 times, 3 times, 4 times, or more). Alternatively, the process may be repeated until there is no improvement in the quality score between iterations. If the system ultimately fails, the LLM may be prompted to provide a response that it cannot answer the question. Alternatively, the LLM may be prompted to ask the user to try rephrasing the question or other request to the user and the process repeated.

[0170] Thus in one aspect, the method 900 may include applying the quality matrix to a new query by classifying the new query, scoring a new response to the new query according to one or more response metrics in the matching quality matrix (a portion of the quality matrix containing response metric values for classes identified for the new query) to provide new response metric values, and comparing the new response metric values for the new query to the expected response metric values in the quality matrix. More generally, the quality matrix may be used to evaluate a new response by the artificial agent to a new query by comparing the new response to an expected behavior encoded in the response matrix.

[0171] In a further embodiment, the threshold scores may be adjusted depending on the topic identified for association with a pre-defined quality matrix. For example, the system maybe set to have a very high faithfulness and context utilization threshold (e.g., 0.98, 0.99, 0.999, etc.) where the user question is identified as medical in nature, while for a question about product support, a threshold for relevance and completeness may be set at a high level (e.g., 0.75, 0.8, 0.9, etc.). This may be useful to allow the system to function in different fields such as medical, education, and product support where different metrics are of different priority.

[0172] Alternatively, the system could be used to provide to the user a confidence score related to the response. For example, the actual metric score, or a combination of metric scores, may be converted, processed, or combined into one metric value representing the confidence in the response. The skilled person will appreciate that any suitable translation or combination of the scores may be made to determine the confidence score. For example, the actual metric score(s) may be directly used as the confidence score, or the average metric scores may be used as the confidence score. In this way the user can be informed regarding the quality of the response provided by the artificial agent - e.g. that it is faithful to the context. The confidence score can be provided as part of, or in conjunction with, the response provided to the user. For example, a response may be displayed along with an icon indicating the confidence (e.g., green for high-confidence, orange for medium confidence, red for low confidence, etc.). In an alternative example, the response may be displayed along with a textual representation of the response (e.g., “80% confidence”, “There is an 80% confidence in this response being faithful to the context”, etc.). Additionally, or alternatively, the multiple confidence scores can be presented to the user to allow the user to understand the different dimensions of confidence with respect to the different metrics. In one embodiment, the confidence score(s) are by default hidden to the user but the user is provided with a user interface (UI) element to display the confidence score(s) for a response. The user is thus provided with a mechanism to explore and understand the response provided by the LLM.

[0173] The above systems, devices, methods, processes, and the like may be realized in hardware, software, or any combination of these suitable for the control, data acquisition, and data processing described herein. This includes realization in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices or processing circuitry, along with internal and / or external memory. This may also, or instead, include one or more application specific integrated circuits, programmable gate arrays, programmable array logic components, or any other device or devices that may be configured to process electronic signals. It will further be appreciated that a realization of the processes or devices described above may include computer-executable code created using a structured programming language such as C, an object oriented programming language such asC++, or any other high-level or low-level programming language (including assembly languages, hardware description languages, and database programming languages and technologies) that may be stored, compiled or interpreted to run on one of the above devices, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software.

[0174] Thus, in one aspect, each method described above, and combinations thereof may be embodied in computer executable code that, when executing on one or more computing devices, performs the steps thereof. In another aspect, the methods may be embodied in systems that perform the steps thereof, and may be distributed across devices in a number of ways, or all of the functionality may be integrated into a dedicated, standalone device or other hardware. The code may be stored in a non-transitory fashion in a computer memory, which may be a memory from which the program executes (such as random access memory associated with a processor), or a storage device such as a disk drive, flash memory or any other optical, electromagnetic, magnetic, infrared, or other device or combination of devices. In another aspect, any of the systems and methods described above may be embodied in any suitable transmission or propagation medium carrying computer-executable code and / or any inputs or outputs from same. In another aspect, means for performing the steps associated with the processes described above may include any of the hardware and / or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.

[0175] The method steps of the implementations described herein are intended to include any suitable method of causing such method steps to be performed, consistent with the patentability of the following claims, unless a different meaning is expressly provided or otherwise clear from the context. So, for example, performing the step of X includes any suitable method for causing another party such as a remote user, a remote processing resource (e.g., a server or cloud computer) or a machine to perform the step of X. Similarly, performing steps X, Y, and Z may include any method of directing or controlling any combination of such other individuals or resources to perform steps X, Y, and Z to obtain the benefit of such steps. Thus, method steps of the implementations described herein are intended to include any suitable method of causing one or more other parties or entities to perform the steps, consistent with the patentability of the following claims, unless a different meaning is expressly provided or otherwise clear from the context. Such parties or entities need not be under the direction or control of any other party or entity and need not be located within a particular jurisdiction.

[0176] It will be appreciated that the methods and systems described above are set forth by way of example and not of limitation. Numerous variations, additions, omissions, and othermodifications will be apparent to one of ordinary skill in the art. In addition, the order or presentation of method steps in the description and drawings above is not intended to require this order of performing the recited steps unless a particular order is expressly required or otherwise clear from the context. Thus, while particular embodiments have been shown and described, it will be apparent to those skilled in the art that various changes and modifications in form and details may be made therein without departing from the spirit and scope of this disclosure and are intended to form a part of the invention as defined by the following claims.ENUMERATED ITEMS1. A computer program product comprising computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of: obtaining a message from a user of a physiological monitoring system, the message including a query associated with the physiological monitoring system; classifying the message into one or more topics, wherein each of the one or more topics is associated with the query included in the message; mapping, using one or more mapping functions, the one or more topics to a set of context specific data, wherein each of the one or more mapping functions obtains context specific data associated with a respective topic of the one or more topics, the context specific data comprising static data, and / or dynamic data associated with the physiological monitoring system; generating a command comprising a context portion based on the set of context specific data and an instruction portion, wherein the instruction portion comprises instructions for a first large language model (LLM) to perform a rephrasing task based on the context portion; providing the command to the first LLM and subsequently obtaining a text-based response from the first LLM, wherein the text-based response comprises a natural language representation of the context portion of the command; and causing a response based on the text-based response to be output to the user.2. The computer program product of item 1 wherein the message is classified into the one or more topics using a hierarchy of topic classifiers.3. The computer program product of item 2 wherein the hierarchy of topic classifiers comprise a top-level classifier for classifying the message into one or more top level topics.4. The computer program product of item 3 wherein the hierarchy of topic classifiers comprise one or more second-level classifiers, each second-level classifier classifies the message into one or more topics within an associated top level topic.5. The computer program product of item 1 wherein the message is classified into the one or more topics using a second LLM, the second LLM returning the one or more topics from an input comprising the message and a plurality of predetermined topics.6. The computer program product of item 1 wherein each of the one or more mapping functions obtains context specific data associated with the respective topic based on one or more parameters.7. The computer program product of item 6 wherein the one or more parameters are based at least in part on the respective topic.8. The computer program product of item 1 wherein the one or more mapping functions comprise at least one mapping function which obtains one or more contextual examples for the respective topic, wherein the one or more contextual examples act as a few-shot prompt for the first LLM.9. The computer program product of item 1 wherein the dynamic data comprises one or more metrics associated with the user of the physiological monitoring system.10. The computer program product of item 9 wherein the one or more metrics include one or more of heart rate, strain, sleep duration, and respiratory rate.11. The computer program product of item 1 wherein the dynamic data comprises one or more goals associated with the user and physiological data associated with the user.12. The computer program product of item 1 wherein the static data comprises a document obtained from a database of documents.13. The computer program product of item 12 wherein the document is obtained using a vector based search.14. The computer program product of item 13 wherein the vector based search compares a vector embedding associated with the respective topic and a plurality of vector embeddings associated with a corresponding plurality of documents within the database of documents.15. The computer program product of item 1 wherein the context data is aggregated prior to the command being generated.16. The computer program product of item 1 wherein the text-based response comprises a template token associated with an element to be included in the response.17. The computer program product of item 16 further comprising, prior to the step of causing the response to be output, replacing the template token with the element within the response.18. The computer program product of item 16 wherein the element includes one of: an image or graphic; a chart; a bullet point; a hyperlink; data related to the user; proprietary data; or a document.19. The computer program product of item 1 further comprising, after causing the response to be output, obtaining a feedback score from the user, wherein the feedback score is indicative of a quality of the response to the message.20. The computer program product of item 19 further comprising storing the message, the one or more topics, the context data, the response, and / or the feedback score as a response instance within a response database.21. The computer program product of item 20 wherein the command further comprises a few shot prompt comprising one or more response instances obtained from the response database, the one or more response instances each having a similar message to the message.22. The computer program product of item 1 wherein at least one topic of the one or more topics is represented as a structured query in a query language.23. The computer program product of item 22 wherein the structured query is generated using a third LLM based on the message from the user, and a schema for the query language.24. The computer program product of item 23 wherein the structured query is generated using the third LLM in accordance with a determination that the message from the user has a predetermined ambiguity level.25. The computer program product of item 23 wherein the computer executable code, when executing on one or more computing devices, further performs the steps of: generating a structured query by providing one or more prompts to the third LLM, wherein the one or more prompts comprise a predetermined instruction, the message from the user, and a schema for the query language.26. The computer program product of item 22 wherein context data related to the at least one topic is obtained using at least one mapping function of the one or more mapping functions determined from the structured query.27. The computer program product of item 26 wherein the computer executable code, when executing on one or more computing devices, further performs the steps of, as part of the step of mapping: parsing the structured query to generate an abstract syntax tree (AST) representation of the structured query; and evaluating the AST representation of the structured query to obtain context specific data related to the at least one topic, wherein the context specific data is obtained using at least one mapping function identified from evaluation of the AST representation of the structured query.28. The computer program product of item 1 wherein the command further comprises an optimization portion comprising instructions for the first LLM to limit a length of the text-based response according to a computational load on the first LLM.29. The computer program product of item 28 wherein the computer executable code, when executing on one or more computing devices, further performs the steps of: obtaining one or more load values indicative of the computational load on the first LLM; and generating the optimization portion comprising instructions for the first LLM to generate the text-based output according to an output length criterion, wherein the output length criterion is based on the one or more load values.30. A method comprising: obtaining a message from a user of a physiological monitoring system, the message including a query associated with the physiological monitoring system;classifying the message into one or more topics, wherein each of the one or more topics is associated with the query included in the message; mapping, using one or more mapping functions, the one or more topics to a set of context specific data, wherein each of the one or more mapping functions obtains context specific data associated with a respective topic of the one or more topics, the context specific data comprising static data, and / or dynamic data associated with the physiological monitoring system; generating a command comprising a context portion based on the set of context specific data and an instruction portion, wherein the instruction portion comprises instructions for first a large language model (LLM) to perform a rephrasing task based on the context portion; providing the command to the first LLM and subsequently obtaining a text-based response from the first LLM, wherein the text-based response comprises a natural language representation of the context portion; and causing a response based on the text-based response to be output to the user.31. A system comprising: a wearable physiological monitor including one or more sensors, a first processor configured to obtain a physiological metric for a user based on a signal from the one or more sensors, and a communications interface for coupling with a remote resource; a server coupled in a communicating relationship with the wearable physiological monitor, the server including a second processor configured by computer executable code to: obtain a message from the user, wherein the message includes a query; classify the message into one or more topics, wherein each of the one or more topics is associated with the query included in the message; map, using one or more mapping functions, the one or more topics to a set of context specific data, wherein each of the one or more mapping functions obtains context specific data associated with a respective topic of the one or more topics, the context specific data comprising static data, and / or the physiological metric for the user; generate a command comprising a context portion based on the set of context specific data and an instruction portion, wherein the instruction portion comprises instructions for a large language model (LLM) to perform a rephrasing task based on the context portion; and provide the command to the LLM and subsequently obtain a response from the LLM, wherein the response comprises a natural language representation of the context portion; anda user interface configured to present the response to the user.32. A method comprising: obtaining a message from a user, the message including an ambiguous query; generating a structured query in a query language by providing one or more prompts to a large language model (LLM), wherein the one or more prompts comprise a predetermined instruction, the message from the user, and a schema for the query language; parsing the structured query to generate an abstract syntax tree (AST) representation of the structured query; evaluating the AST representation of the structured query to obtain context specific data related to one or more topics associated with the ambiguous query, wherein the context specific data is obtained using one or more mapping functions identified from evaluation of the AST representation of the structured query; and outputting the context specific data as a resolution to the ambiguous query.33. A computer program product comprising computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of item 32.34. A device comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the device to perform the steps of item 32.35. A method for dynamically optimizing load on a large language model (LLM), the method comprising: identifying a prompt to be provided to an LLM for performing a task related to a physiological monitoring system; obtaining one or more load values indicative of a computational load on the LLM; generating a command comprising instructions for the LLM to generate an output according to an output length criterion, wherein the output length criterion is based on the one or more load values; and providing, to the LLM, the prompt and the command such that a subsequent output generated by the LLM satisfies the output length criterion.36. The method of item 35 wherein the one or more load values include a time to first token value.37. The method of item 35 wherein the one or more load values include a number of active requests value.38. The method of item 35 wherein the one or more load values include a time to last token value.39. The method of item 35 wherein at least one of the one or more load values is determined from a metric obtained from the LLM.40. The method of item 39 wherein the metric comprises a response time periodically obtained from the LLM.41. The method of item 40 wherein the response time comprises an average response time obtained from the LLM over a period of time.42. The method of item 35 wherein the one or more load values include a predicted load value.43. The method of item 42 wherein the predicted load value is associated with a predicted load of the LLM for a period of time and / or a geographic region.44. The method of item 35 wherein a number of tokens within the subsequent output is proportional to the load on the LLM as represented by the one or more load values.45. The method of item 44 wherein the output length criterion causes the LLM to limit the number of tokens to a first maximum number of tokens if at least one of the one or more load values satisfy a first load threshold.46. The method of item 45 wherein the output length criterion causes the LLM to limit the number of tokens to a second maximum number of tokens if at least one of the one or more load values satisfy a second load threshold.47. The method of item 46 wherein the second load threshold is greater than the first load threshold and the second number of tokens is less than the first number of tokens.48. The method of item 35 further comprising: trimming the prompt based on the one or more load values.49. The method of item 48 wherein the prompt is trimmed if the one or more load values satisfy a third load threshold.50. The method of item 35 further comprising: obtaining the subsequent output from the LLM.51. A computer program product comprising computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of item 35.52. A device comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the device to perform the steps of item 35.53. A method for generating cross-component responses to user requests, the method comprising: obtaining a portion of a natural language message received from a user of a physiological monitoring system, wherein the portion of the natural language message relates to a request from the user in relation to one or more operations performed by the physiological monitoring system; providing, to a large language model (LLM), a prompt operable to cause the LLM to output a code block which encodes a response to the request and is processable by one or more components of the physiological monitoring system to generate one or more component specific representations of the response, wherein the prompt comprises a predetermined instruction, the portion of the natural language message, a language schema for code within the code block, and an object schema related to the one or more operations performed by the physiological monitoring system; and obtaining the code block from the LLM.54. The method of item 53 further comprising: processing, by a first component of the physiological monitoring system, the code block thereby generating a first component specific representation of the response.55. The method of item 54 wherein the first component is an application of the physiological monitoring system.56. The method of item 55 wherein the first component specific representation comprises one or more elements of the response which are adjustable within the application.57. The method of item 55 wherein the application is a physiological training application.58. The method of item 54 wherein the first component is a user interface (UI) generation component.59. The method of item 58 wherein the first component specific representation is a user interface element associated with the response.60. The method of item 54 wherein the first component is a text generation component.61. The method of item 60 wherein the first component specific representation is a natural language representation of the response.62. The method of item 53 further comprising, prior to providing the prompt to the LLM: identifying the object schema from a set of object schemas based on the portion of the natural language message.63. The method of item 53 further comprising: storing the code block to a persistent storage location.64. The method of item 53 further comprising, prior to providing the prompt to the LLM: obtaining a previous code block previously generated in response to a prior user request; wherein the prompt further comprises the previous code block such that the code block output by the LLM is based at least in part on the previous code block.65. The method of item 64 wherein the request from the user relates to a requested change to the response previously generated in response to the prior user request.66. A computer program product comprising computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of item 53.67. A device comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the device to perform the steps of item 53.68. A method for quantifying artificial agent performance, the method comprising: identifying a plurality of statements related to an interaction between a user and an artificial agent, wherein the plurality of statements include at least two of: a query provided by the user to the artificial agent; an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; and / or one or more reference statements related to the query; for one or more pairwise combinations of the plurality of statements involving a first statement and a second statement: determining a matching between the first statement and the second statement, wherein the matching encodes similarities between one or more clauses of the first statement and one or more clauses of the second statement; and calculating a metric value by applying a metric to the matching, wherein the metric quantifies a coherence between the first statement and the second statement based on the matching; and calculating a performance score of the artificial agent in relation to the interaction based on the metric value determined for the one or more pairwise combinations of the plurality of statements.69. The method of item 68 further comprising: outputting the performance score.70. The method of item 68 further comprising:identifying one or more performance trends related to the artificial agent based on the performance score and one or more other performance scores calculated for the artificial agent.71. The method of item 70 further comprising: causing an adjustment to the artificial agent based on the one or more performance trends.72. The method of item 68 wherein the one or more reference statements related to the query include one or more of: a ground truth response to the query; and context data related to the query.73. The method of item 68 wherein the step of determining the matching between the first statement and the second statement comprises: decomposing the first statement into a first set of clauses; decomposing the second statement into a second set of clauses; and matching the first set of clauses to the second set of clauses thereby determining the matching, wherein a first clause of the first set of clauses matches a second clause of the second set of clauses if a similarity between the first clause and the second clause satisfy a similarity criterion.74. The method of item 73 wherein the step of decomposing the first statement into the first set of clauses comprises: providing, to a second LLM, a first prompt comprising the first statement and a first command operable to cause the second LLM to extract one or more clauses from the first statement; and obtaining, from the second LLM and in response to the first prompt, the first set of clauses for the first statement.75. The method of item 74 wherein the first command is further operable to cause the second LLM to extract overlapping clauses such that each of the one or more clauses form a grammatically complete claim.76. The method of item 73 wherein the step of matching the first set of clauses to the second set of clauses comprises:providing, to a third LLM, a second prompt comprising the first set of clauses, the second set of clauses, and a second command operable to cause the third LLM to determine potential matches between clauses within the first set of clauses and the second set of clauses; and obtaining, from the third LLM and in response to the second prompt, the matching between the first set of clauses and the second set of clauses.77. The method of item 68 wherein the matching comprises a first identifier of a first clause of the first statement and / or a second identifier of a second clause of the second statement.78. The method of item 68 wherein the metric is a recall metric corresponding to a number of true positive matching clauses between the first statement and the second statement divided by a sum of the number of true positive matching clauses and a number of false negative matching clauses between the first statement and the second statement.79. The method of item 78 wherein the first statement is the query and the second statement is the output generated by the artificial agent in response to the query such that the metric quantifies a degree of satisfaction of the query provided by the output.80. The method of item 78 wherein the first statement is the output generated by the artificial agent in response to the query and the second statement is context specific data related to the query such that the metric quantifies a proportion of the output related to the context specific data.81. The method of item 78 wherein the first statement is a ground truth response related to the query and the second statement is context specific data related to the query such that the metric quantifies a proportion of the ground truth response contained within the context specific data.82. The method of item 78 wherein the first statement is the query and the second statement is a ground truth response related to the query such that the metric quantifies a degree to which the ground truth response covers the extent of the query.83. The method of item 78 wherein the first statement is the query and the second statement is context specific data related to the query such that the metric quantifies a degree to which the context specific data provide an answer to the query.84. The method of item 78 wherein the first statement is a ground truth response related to the query and the second statement is an output generated by the artificial agent in response to the query such that the metric quantifies a correctness of the output.85. The method of item 68 wherein the metric is a precision metric corresponding to a number of true positive matching clauses between the first statement and the second statement divided by a sum of the number of true positive matching clauses and a number of false positive matching clauses between the first statement and the second statement.86. The method of item 85 wherein the first statement is the query and the second statement is the output generated by the artificial agent in response to the query such that the metric quantifies a relevance of the output to the query.87. The method of item 85 wherein the first statement is the output generated by the artificial agent in response to the query and the second statement is context specific data related to the query such that the metric quantifies a proportion of the context specific data used to generate the output.88. The method of item 85 wherein the first statement is the query and the second statement is a ground truth response related to the query such that the metric quantifies a degree to which the query elicited the ground truth response.89. The method of item 85 wherein the first statement is the query and the second statement is context specific data related to the query such that the metric quantifies a proportion of the context specific data needed to answer the query.90. The method of item 85 wherein the first statement is a ground truth response related to the query and the second statement is context specific data related to the query such that the metric quantifies a proportion of the context specific data needed to generate the ground truth response.91. The method of item 85 wherein the first statement is a ground truth response related to the query and the second statement is the output generated by the artificial agent in response to the query such that the metric quantifies a redundancy of the output in relation to the ground truth response.92. The method of item 68 wherein the user is a user of a physiological monitoring system and the artificial agent is a part of the physiological monitoring system.93. A computer program product comprising computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of: identifying a plurality of statements related to an interaction between a user and an artificial agent, wherein the plurality of statements include at least two of: a query provided by the user to the artificial agent; an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; and / or one or more reference statements related to the query; for one or more pairwise combinations of the plurality of statements involving a first statement and a second statement: determining a matching between the first statement and the second statement, wherein the matching encodes similarities between one or more clauses of the first statement and one or more clauses of the second statement; and calculating a metric value by applying a metric to the matching, wherein the metric quantifies a coherence between the first statement and the second statement based on the matching; and calculating a performance score of the artificial agent in relation to the interaction based on the metric value determined for one or more pairwise combinations of the plurality of statements.94. The computer program product of item 93 wherein the matching is determined from the first statement to the second statement such that the metric quantifies a coherence from the first statement to the second statement based on the matching.95. A device comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the device to perform the steps of: identifying a plurality of statements related to an interaction between a user and an artificial agent, wherein the plurality of statements include at least two of: a query provided by the user to the artificial agent; an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; and / or one or more reference statements related to the query; for one or more pairwise combinations of the plurality of statements involving a first statement and a second statement: determining a matching between the first statement and the second statement, wherein the matching encodes similarities between one or more clauses of the first statement and one or more clauses of the second statement; and calculating a metric value by applying a metric to the matching, wherein the metric quantifies a coherence between the first statement and the second statement based on the matching; and calculating a performance score of the artificial agent in relation to the interaction based on the metric value determined for one or more pairwise combinations of the plurality of statements.96. A method for quantifying expected behavior of artificial agents, the method comprising: identifying a query provided to an artificial agent by a user; identifying an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; classifying the query into one or more query classes, wherein each of the one or more query classes define a characteristic of the query; determining one or more response metric values for the output, wherein each of the one or more response metric values quantify an expected behavior of the artificial agent according to a respective response metric; andgenerating a quality matrix for the artificial agent based on the one or more query classes and the one or more response metric values, wherein the quality matrix encodes a set of expected behaviors of the artificial agent for each of the one or more query classes.97. The method of item 96 further comprising: outputting the quality matrix.98. The method of item 96 further comprising: causing an adjustment to the artificial agent based on the quality matrix.99. The method of item 96 wherein the step of classifying the query into one or more query classes comprises: providing, to a second LLM, a first prompt comprising the query and a first command operable to cause the second LLM to classify the query into the one or more query classes; and obtaining, from the second LLM and in response to the first prompt, the one or more query classes.100. The method of item 96 wherein the one or more query classes include one or more of: a query type class; a subject tag class; a support identifier class; and a language identifier class.101. The method of item 96 wherein respective response metric comprises a natural language processing function.102. The method of item 101 wherein the natural language processing function comprises a keyword matching function operable to identify one or more keywords within the output.103. The method of item 101 wherein the natural language processing function comprises a compactness function operable to identify a verbosity of the output.104. The method of item 101 wherein the natural language processing function comprises: providing, to a third LLM, a second prompt comprising the output and a second command operable to cause the third LLM to identify expected behaviors within the output; and obtaining, from the third LLM and in response to the second prompt, the one or more response metric values quantifying the expected behaviors identified within the output.105. The method of item 96 wherein the one or more response metric values comprise one or more of: a refusal value; a follow up value; a degree of personalization; a deny-list word rate; a word misuse ratio; a verbosity value; and a language value.106. The method of item 96 wherein the characteristic is a high-level characteristic related to a topic of the query.107. A computer program product comprising computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of: identifying a query provided to an artificial agent by a user; identifying an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; classifying the query into one or more query classes, wherein each of the one or more query classes define a characteristic of the query; determining one or more response metric values for the output, wherein each of the one or more response metric values quantify an expected behavior of the artificial agent according to a respective response metric; and generating a quality matrix for the artificial agent based on the one or more query classes and the one or more response metric values, wherein the quality matrix encodes a set of expected behaviors of the artificial agent for each of the one or more query classes.108. A device comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the device to perform the steps of: identifying a query provided to an artificial agent by a user; identifying an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; classifying the query into one or more query classes, wherein each of the one or more query classes define a characteristic of the query;determining one or more response metric values for the output, wherein each of the one or more response metric values quantify an expected behavior of the artificial agent according to a respective response metric; and generating a quality matrix for the artificial agent based on the one or more query classes and the one or more response metric values, wherein the quality matrix encodes a set of expected behaviors of the artificial agent for each of the one or more query classes.109. A method comprising: identifying a set of question-answer (QA) pairs related to a topic, each QA pair comprising a question and a corresponding ground truth response to the question; decomposing the set of QA pairs into a plurality of sets of clauses such that each question and each answer is decomposed into a set of question clauses and a set of answer clauses respectively; determining one or more matchings between the plurality of sets of clauses, wherein a matching defines a potential connective relationship between a first clause forming part of a first question of a first QA pair and a second clause forming part of a first answer of the first QA pair; applying at least one metric to the plurality of sets of clauses based on the one or more matchings, wherein the at least one metric measures a predicted suitability of each answer to each corresponding question; and filtering the QA pairs according to the at least one metric thereby generating a synthetic data set of QA pairs related to the topic such that values of the at least one metric for each QA pair within the synthetic data set of QA pairs satisfy a threshold criterion.110. The method of item 109 wherein identifying the set of QA pairs related to the topic comprises: providing, to a large language model (LLM), a prompt comprising an article related to the topic and a command operable to cause the LLM to extract questions and corresponding answers from the article; and obtaining, from the LLM and in response to the prompt, the set of QA pairs related to the topic.111. A computer program product comprising computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of: identifying a set of question-answer (QA) pairs related to a topic, each QA pair comprising a question and a corresponding ground truth response to the question; decomposing the set of QA pairs into a plurality of sets of clauses such that each question and each answer is decomposed into a set of question clauses and a set of answer clauses respectively; determining one or more matchings between the plurality of sets of clauses, wherein a matching defines a potential connective relationship between a first clause forming part of a first question of a first QA pair and a second clause forming part of a first answer of the first QA pair; applying at least one metric to the plurality of sets of clauses based on the one or more matchings, wherein the at least one metric measures a predicted suitability of each answer to each corresponding question; and filtering the QA pairs according to the at least one metric thereby generating a synthetic data set of QA pairs related to the topic such that values of the at least one metric for each QA pair within the synthetic data set of QA pairs satisfy a threshold criterion.112. A device comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the device to perform the steps of: identifying a set of question-answer (QA) pairs related to a topic, each QA pair comprising a question and a corresponding ground truth response to the question; decomposing the set of QA pairs into a plurality of sets of clauses such that each question and each answer is decomposed into a set of question clauses and a set of answer clauses respectively; determining one or more matchings between the plurality of sets of clauses, wherein a matching defines a potential connective relationship between a first clause forming part of a first question of a first QA pair and a second clause forming part of a first answer of the first QA pair; applying at least one metric to the plurality of sets of clauses based on the one or more matchings, wherein the at least one metric measures a predicted suitability of each answer to each corresponding question; andfiltering the QA pairs according to the at least one metric thereby generating a synthetic data set of QA pairs related to the topic such that values of the at least one metric for each QA pair within the synthetic data set of QA pairs satisfy a threshold criterion.

Claims

CLAIMSWhat is claimed is:

1. A method for quantifying artificial agent performance, the method comprising: identifying a plurality of statements related to an interaction between a user and an artificial agent, wherein the plurality of statements include: a query provided by the user to the artificial agent; an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; and one or more reference statements related to the query; for one or more pairwise combinations of the plurality of statements: determining a matching between a first statement and a second statement, wherein the matching encodes similarities between one or more clauses of the first statement and one or more clauses of the second statement; and calculating a metric value by applying a metric to the matching, wherein the metric quantifies a coherence between the first statement and the second statement based on the matching; and calculating a performance score of the artificial agent in relation to the interaction based on the metric value determined for the one or more pairwise combinations of the plurality of statements.

2. The method of claim 1, further comprising: outputting the performance score.

3. The method of any of the preceding claims, further comprising: identifying one or more performance trends related to the artificial agent based on the performance score and one or more other performance scores calculated for the artificial agent; and causing an adjustment to the artificial agent based on the one or more performance trends.

4. The method of any of the preceding claims, wherein the one or more reference statements related to the query include one or more of: a ground truth response to the query; and context data related to the query.

5. The method of any of the preceding claims, wherein the step of determining the matching between the first statement and the second statement comprises: decomposing the first statement into a first set of clauses; decomposing the second statement into a second set of clauses; and matching the first set of clauses to the second set of clauses thereby determining the matching, wherein a first clause of the first set of clauses matches a second clause of the second set of clauses if a similarity between the first clause and the second clause satisfies a similarity criterion.

6. The method of claim 5, wherein the step of decomposing the first statement into the first set of clauses comprises: providing, to a second LLM, a first prompt comprising the first statement and a first command operable to cause the second LLM to extract one or more clauses from the first statement; and obtaining, from the second LLM and in response to the first prompt, the first set of clauses for the first statement.

7. The method of claim 6, wherein the first command is further operable to cause the second LLM to extract overlapping clauses such that each of the one or more clauses form a grammatically complete claim.

8. The method of any of claims 5 to 7, wherein the step of matching the first set of clauses to the second set of clauses comprises: providing, to a third LLM, a second prompt comprising the first set of clauses, the second set of clauses, and a second command operable to cause the third LLM to determine potential matches between clauses within the first set of clauses and the second set of clauses; and obtaining, from the third LLM and in response to the second prompt, the matching between the first set of clauses and the second set of clauses.

9. The method of any of the preceding claims, wherein the metric is a recall metric corresponding to a number of true positive matching clauses between the first statement and the second statement divided by a sum of the number of true positive matching clauses and a number of false negative matching clauses between the first statement and the second statement.

10. The method of claim 9, wherein the first statement is the query and the second statement is one of: the output generated by the artificial agent in response to the query such that the metric quantifies a degree of satisfaction of the query provided by the output; a ground truth response related to the query such that the metric quantifies a degree to which the ground truth response covers an extent of the query; or context specific data related to the query such that the metric quantifies a degree to which the context specific data provide an answer to the query.

11. The method of claim 9, wherein the first statement is the output generated by the artificial agent in response to the query and the second statement is context specific data related to the query such that the metric quantifies a proportion of the output related to the context specific data.

12. The method of claim 9, wherein the first statement is a ground truth response related to the query and the second statement is one of: context specific data related to the query such that the metric quantifies a proportion of the ground truth response contained within the context specific data; or the output generated by the artificial agent in response to the query such that the metric quantifies a correctness of the output.

13. The method of any of the preceding claims, wherein the metric is a precision metric corresponding to a number of true positive matching clauses between the first statement and the second statement divided by a sum of the number of true positive matching clauses and a number of false positive matching clauses between the first statement and the second statement.

14. The method of claim 13, wherein the first statement is the query and the second statement is one of: the output generated by the artificial agent in response to the query such that the metric quantifies a relevance of the output to the query; a ground truth response related to the query such that the metric quantifies a degree to which the query elicited the ground truth response; or context specific data related to the query such that the metric quantifies a proportion of the context specific data needed to answer the query.

15. The method of claim 13, wherein the first statement is the output generated by the artificial agent in response to the query and the second statement is context specific data related to the query such that the metric quantifies a proportion of the context specific data used to generate the output.

16. The method of claim 13, wherein the first statement is a ground truth response related to the query and the second statement is one of: context specific data related to the query such that the metric quantifies a proportion of the context specific data needed to generate the ground truth response; or the output generated by the artificial agent in response to the query such that the metric quantifies a redundancy of the output in relation to the ground truth response.

17. The method of any of the preceding claims, wherein the user is a user of a physiological monitoring system and the artificial agent is a part of the physiological monitoring system.

18. The method of any of the preceding claims, wherein the matching is determined from the first statement to the second statement such that the metric quantifies the coherence from the first statement to the second statement based on the matching.

19. A computer program product comprising computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of: identifying a plurality of statements related to an interaction between a user and an artificial agent, wherein the plurality of statements includes at least two of: a query provided by the user to the artificial agent; an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; and one or more reference statements related to the query; for one or more pairwise combinations of the plurality of statements involving a first statement and a second statement: determining a matching between the first statement and the second statement, wherein the matching encodes similarities between one or more clauses of the first statement and one or more clauses of the second statement; andcalculating a metric value by applying a metric to the matching, wherein the metric quantifies a coherence between the first statement and the second statement based on the matching; and calculating a performance score of the artificial agent in relation to the interaction based on the metric value determined for the one or more pairwise combinations of the plurality of statements.

20. A system comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause a device to perform the steps of identifying a plurality of statements related to an interaction between a user and an artificial agent, wherein the plurality of statements includes at least two of a query provided by the user to the artificial agent; an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; and one or more reference statements related to the query; and for one or more pairwise combinations of the plurality of statements involving a first statement and a second statement: determining a matching between the first statement and the second statement, wherein the matching encodes similarities between one or more clauses of the first statement and one or more clauses of the second statement; calculating a metric value by applying a metric to the matching, wherein the metric quantifies a coherence between the first statement and the second statement based on the matching; and calculating a performance score of the artificial agent in relation to the interaction based on the metric value determined for the one or more pairwise combinations of the plurality of statements.

21. A method for quantifying expected behavior of artificial agents, the method comprising: identifying a query provided to an artificial agent by a user; identifying an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output;classifying the query into one or more query classes, wherein each of the one or more query classes defines a characteristic of the query; determining one or more response metric values for the output, wherein each of the one or more response metric values quantify an expected behavior of the artificial agent according to a respective response metric; and generating a quality matrix for the artificial agent based on the one or more query classes and the one or more response metric values, wherein the quality matrix encodes a set of expected behaviors of the artificial agent for each of the one or more query classes as an array of pairwise combinations between the one or more query classes and the one or more response metric values.

22. The method of claim 21, further comprising: outputting the quality matrix.

23. The method of any of claims 21 to 22, further comprising: causing an adjustment to the artificial agent based on the quality matrix.

24. The method of any of claims 21 to 23, wherein the step of classifying the query into the one or more query classes comprises: providing, to a second LLM, a first prompt comprising the query and a first command operable to cause the second LLM to classify the query into the one or more query classes; and obtaining, from the second LLM and in response to the first prompt, the one or more query classes.

25. The method of any of claims 21 to 24, wherein the one or more query classes include one or more of: a query type class; a subject tag class; a support identifier class; and a language identifier class.

26. The method of claim 25, wherein determining one or more response metric values for the output includes use of a keyword matching function operable to identify one or more keywords within the output.

27. The method of claim 25, wherein determining one or more response metric values for the output includes use of a compactness function operable to identify a verbosity of the output.

28. The method of claim 25, wherein determining one or more response metric values for the output includes: providing, to a third LLM, a second prompt comprising the output and a second command operable to cause the third LLM to identify expected behaviors within the output; and obtaining, from the third LLM and in response to the second prompt, the one or more response metric values quantifying the expected behaviors identified within the output.

29. The method of any of claims 21 to 28, wherein the one or more response metric values comprise one or more of: a refusal value; a follow up value; a degree of personalization; a deny- list word rate; a word misuse ratio; a verbosity value; and a language value.

30. The method of any of claims 21 to 29, wherein the characteristic is a high-level characteristic related to a topic of the query.

31. The method of any of claims 21 to 30, further comprising classifying a new request, identifying a matching quality matrix containing one or more corresponding response metrics, and evaluating a new response by the artificial agent based on the one or more corresponding response metrics.

32. The method of any of claims 21 to 31, further comprising evaluating a new response by the artificial agent to a new query by comparing the new response to a one of the set of expected behaviors encoded in the quality matrix.

33. A computer program product comprising computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of: identifying a query provided to an artificial agent by a user; identifying an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; classifying the query into one or more query classes, wherein each of the one or more query classes define a characteristic of the query;determining one or more response metric values for the output, wherein each of the one or more response metric values quantify an expected behavior of the artificial agent according to a respective response metric; and generating a quality matrix for the artificial agent based on the one or more query classes and the one or more response metric values, wherein the quality matrix encodes a set of expected behaviors of the artificial agent for each of the one or more query classes.

34. The computer program product of claim 33, further comprising code that performs the step of storing the quality matrix in a non-transitory medium.

35. The computer program product of any of claims 33 to 34, further comprising code that performs the step of causing an adjustment to the artificial agent based on the quality matrix.

36. The computer program product of any of claims 33 to 35, wherein the one or more query classes include one or more of: a query type class; a subject tag class; a support identifier class; and a language identifier class.

37. The computer program product of any of claims 33 to 36, wherein the one or more response metric values comprise one or more of: a refusal value; a follow up value; a degree of personalization; a deny-list word rate; a word misuse ratio; a verbosity value; and a language value.

38. The computer program product of any of claims 33 to 37, further comprising code that performs the steps of classifying a new request, identifying a matching quality matrix containing one or more corresponding response metrics, and evaluating a new response by the artificial agent based on the one or more corresponding response metrics.

39. The computer program product of any of claims 33 to 38, further comprising code that performs the step of evaluating a new response by the artificial agent to a new query by comparing the new response to a one of the set of expected behaviors encoded in the quality matrix.

40. A system comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, causes a device to perform the steps of:identifying a query provided to an artificial agent by a user; identifying an output generated by the artificial agent in response to the query, wherein the artificial agent is operable to interface with a first large language model (LLM) to determine the output; classifying the query into one or more query classes, wherein each of the one or more query classes defines a characteristic of the query; determining one or more response metric values for the output, wherein each of the one or more response metric values quantify an expected behavior of the artificial agent according to a respective response metric; and generating a quality matrix for the artificial agent based on the one or more query classes and the one or more response metric values, wherein the quality matrix encodes a set of expected behaviors of the artificial agent for each of the one or more query classes.

Images