Method, system, and product for parsing data into structured and formatted output data with iterative prompting

Abstract

A computer-implemented method ingests semi-structured dialog data obtained from a microphone or text channel, segments the dialog into utterances, and generates, via a large- language-model prompt module, structured records that include entity labels, time stamps, and confidence scores. When any confidence score is below a predefined threshold, the system automatically issues a clarification prompt to the user and re-runs inference on the response. The structured records are stored in a local or cloud data store and made available to downstream visualization software. Embodiments include a local GPU implementation that maintains protected-health-information on-device, as well as a cloud implementation secured with end-to- end encryption. Additional embodiments cover computer-readable media and training pipelines for fine-tuning the language model on domain-specific corpora.

Description

METHOD, SYSTEM, AND PRODUCT FOR PARSING DATA INTO STRUCTURED AND FORMATTED OUTPUT DATA WITH ITERATIVE PROMPTINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. provisional application No. 63 / 667,260 filed on July 3, 2024, incorporated herein by reference in its entirety.BACKGROUND OF THE INVENTION

[0002] The collection of structured data is important in many settings. Conventional processes usually involve reading a series of manual questions and then recording answers to boxes in a form. For example, when made straightforward, some questions may be offloaded to a user with self-report screeners. However, this approach is inefficient because the manualized entry is slow and then must be translated into a useful form for downstream analysis.Additionally, this type of process may be of annoyance to the patient or may yield erroneous and incomplete data.

[0003] When automating data collection, for example when using artificial intelligence, creating structured data is particularly difficult, especially for Large Language Models (LLMs). Additionally, data collection methods frequently extend beyond standardized forms, and may instead take the form of interactions or conversations thereby making it particularly difficult to standardize data collection given the nonstandard form of interactions. Therefore, it is desirable to standardize unstructured data from a variety of sources to structure data in a quick, predictable, and repeated manner in a number of settings, including legal depositions, court proceedings, educational settings, business meetings, corporate trainings, customer service, customer supportjournalism interviews, media interviews, market research, focus groups, human resources, recruitment, event planning, project management, and academic research.

[0004] One situation where the issue of standardized data collection is particularly relevant is clinical or research settings, which rely on structured data such as diagnostic criteria or pathwaysto care. For example, in the case of collecting pathways to care (PTC) data, it was found that 10% of pathways collected had to be excluded from analysis because they were incomplete or internally irreconcilable, (see Mathis, W. S., Ferrara, M., Burke, S., Hyun, E., Li, F., Zhou, B., Cahill, J., Kline, E. R., Keshavan, M. S., & Srihari, V. H.; Granular analysis of pathways to care and durations of untreated psychosis: A marginal delay model. PLOS ONE, 17(12), 2022)

[0005] Therefore, there is a need in the art for an automated process to be used in different settings during screening to create an overall more efficient process without missing information, and to create structured data that may be stored in standardized formats, such as electronic health records (EHR), Research Electronic Data Capture (REDCAP), etc.SUMMARY OF THE INVENTION

[0006] The present disclosure relates generally to methods, systems, and products for the parsing of interactions, and more particularly, the parsing of interactions using large language models that are prompted to output structured data that is capable of being easily processed downstream. Figure 3 and Example 1 below walk through an exemplary patient-intake utterance, the prompt applied, and the resulting JSON output.

[0007] Some embodiments of the invention disclosed herein are set forth below, and any combination of these embodiments (or portions thereof) may be made to define another embodiment.

[0008] In one aspect, a system for parsing interactions is broadly described. The system may include: at least one processor and an input device; a non-transitoiy memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: receiving semi -structured dialog data from one or more sources as text data; processing the text data in a prompt module, wherein the prompt module may generate a call to a large language model (LLM) to structure the text data; receiving structured output from the LLM corresponding to the forwarded text data; and storing the structured output from the LLM in a repository.

[0009] In one embodiment, the one or more sources may comprise a text source.

[0010] In one embodiment, the one or more sources may comprise an audio source, wherein the operations may further comprise: receiving audio data from the audio input device; and converting the audio data to the text data via a text transcription engine.

[0011] In one embodiment, the one or more sources may comprise an audio source and a text source.

[0012] In one embodiment, the input device may comprise an audio input device, wherein the operations may further comprise: receiving audio data from the audio input device; and converting the audio data to a first portion of the text data via a text transcription engine; converting text data received from the text source to a second portion of the text data; organizing the first portion of the text data and the second portion of the text data via an aggregation engine to create the text data; and forwarding the organized text data to the prompt module.

[0013] In one embodiment, the operations may further comprise: determining whether clarification is required based on the structured output from the LLM; when clarification is required, generating a prompt to request second semi-structured dialog data from the one or more sources as text data; processing the second text data in the prompt module, wherein the prompt module generates a call to the large language model (LLM) to structure the second text data; receiving a second structured output from the LLM corresponding to the second text data; updating at least a portion of the structured output with at least a portion of the second structured output; and storing the updated structured output from the LLM in the repository.

[0014] In another aspect, a system for parsing interactions is broadly described. The system may comprise: an output device configured to output a first set of one or more questions; an input device configured to receive a first set of one or more answers to the first set of one or more questions; an audio device configured to receive audio data, wherein the audio data comprises: a second set of one or more answers; a computing system communicatively coupled to the output device, the input device, and the audio device, the computing system comprising: one or more datastores storing a large language model (LLM), wherein the LLM is configured to standardize output based on one or more prompts; a text transcription engine configured to transcribe the audio data to text data; a prompt module configured to structure the text data into aseries of prompts and present the series of prompts to the LLM; and an aggregation engine configured to organize the data received from the LLM into a structured dataset.

[0015] In one embodiment, the audio data may further comprise a second set of one or more questions. In one embodiment, the input device may comprise a keyboard or a touchscreen, wherein the output device may comprise a display or a speaker. In one embodiment, the audio device may comprise a microphone. In one embodiment, the aggregation engine may be further configured to present at least a portion of the structured dataset as output on the output device.

[0016] In another aspect, a method for parsing interactions is broadly described. The method may comprise: receiving semi-structured dialog data from one or more sources as text data; organizing the text data into a set of prompts; generating at least one call to a large language model (LLM) comprising a prompt of the set of prompts; and aggregating the output from the LLM into a structured dataset.

[0017] In one embodiment, the one or more sources may comprise a text source. In one embodiment, the one or more sources may comprise an audio source, the method may further comprise the steps of: receiving audio data from the audio source; and converting the audio data to the text data. In one embodiment, the one or more sources may comprise an audio source and a text source.

[0018] In one embodiment, the method may further comprise the steps of: receiving audio data from the audio source; converting the audio data to a first portion of the text data; converting text data received from the text source to a second portion of the text data; aggregating the first portion of the text data and the second portion of the text data; and forwarding the organized text data to the prompt module.

[0019] In one embodiment, the method may further comprise the steps of: determining whether clarification is required based on the structured dataset; when clarification is required, requesting second semi-structured dialog data from the one or more sources as second text data; organizing the second text data into a second set of prompts; generating at least one second call to the LLM comprising a prompt of the second set of prompts and receiving a second structured output; and updating at least a portion of the structured dataset with at least a portion of the second structured output.

[0020] In one embodiment, the one or more sources may comprise a text source including one or more of: text message, messengers, web application, cell phone applications. In one embodiment, the structured dataset may comprise a data structure or markup language for easy integration with other software. In one embodiment, the output data may comprise JSON or XML. In one embodiment, the output data may be stored as a comma-separated-values (CSV) file.BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The foregoing purposes and features, as well as other purposes and features, will become apparent with reference to the description and accompanying figures below, which are included to provide an understanding of the invention and constitute a part of the specification, in which like numerals represent like elements, and in which:Fig. 1 depicts an illustrative computing device 100.Fig. 2A depicts an illustrative flow diagram of a method 200.Fig. 2B depicts an illustrative flow diagram of a method 200a.Fig. 3 depicts an illustrative computing system 300.DETAILED DESCRIPTION OF THE INVENTION

[0022] It is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clearer comprehension of the present invention, while eliminating, for the purpose of clarity, many other elements found in systems, methods, and products. Those of ordinary skill in the art may recognize that other elements and / or steps are desirable and / or required in implementing the present invention. However, because such elements and steps are well known in the art, and because they do not facilitate a better understanding of the present invention, a discussion of such elements and steps is not provided herein. The disclosure herein is directed to all such variations and modifications to such elements and methods known to those skilled in the art.

[0023] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, exemplary methods and materials are described.

[0024] As used herein, each of the following terms has the meaning associated with it in this section.

[0025] The articles “a” and “an” are used herein to refer to one or to more than one (i.e., to at least one) of the grammatical objects of the article. By way of example, “an element” means one element or more than one element.

[0026] The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. For example, as used herein, the singular forms “a”, “an” and “the” may be intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms “comprises,” “comprising,” “including,” and “having,” are inclusive and therefore specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order discussed or illustrated, unless specifically identified as an order of performance. It is also to be understood that additional or alternative steps may be employed.

[0027] When an element or layer is referred to as being “on,” “engaged to,” “connected to,” or “coupled to” another element or layer, it may be directly on, engaged, connected, or coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being “directly on,” “directly engaged to,” “directly connected to” or “directly coupled to” another element or layer, there may be no intervening elements or layers present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between” versus “directly between,” “adjacent” versus “directly adjacent,”etc ). As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0028] Although the terms first, second, third, etc., may be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms may be only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. That is, terms such as “first,” “second,” and other numerical terms, when used herein, do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer, or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the exemplary embodiments.

[0029] Terms concerning attachments, coupling and the like, such as “connected” and “interconnected,” refer to a relationship wherein structures are secured or attached to one another either directly or indirectly through intervening structures, as well as both movable or rigid attachments or relationships, unless expressly described otherwise. The term “operatively or operably connected” is such an attachment, coupling or connection that allows the pertinent structures to operate as intended by virtue of that relationship.

[0030] “About” as used herein when referring to a measurable value such as an amount, a temporal duration, and the like, is meant to encompass variations of ±20%, ±10%, ±5%, ±1%, and ±0.1% from the specified value, as such variations are appropriate.

[0031] Ranges: throughout this disclosure, various aspects of the invention can be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Where appropriate, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 2.7, 3, 4, 5, 5.3, and 6. This applies regardless of the breadth of the range.

[0032] The figures and descriptions provided herein may have been simplified to illustrate aspects that are relevant for a clear understanding of the herein described apparatuses, systems, and methods, while eliminating, for the purpose of clarity, other aspects that may be found in typical similar devices, systems, and methods. Those of ordinary skill may thus recognize that other elements and / or operations may be desirable and / or necessary to implement the devices, systems, and methods described herein. But because such elements and operations are known in the art, and because they do not facilitate a better understanding of the present disclosure, for the sake of brevity a discussion of such elements and operations may not be provided herein. However, the present disclosure is deemed to nevertheless include all such elements, variations, and modifications to the described aspects that would be known to those of ordinary skill in the art.

[0033] Embodiments are provided throughout so that this disclosure is sufficiently thorough and fully conveys the scope of the disclosed embodiments to those who are skilled in the art. Numerous specific details are set forth, such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. Nevertheless, it will be apparent to those skilled in the art that certain specific disclosed details need not be employed, and that embodiments may be embodied in different forms. As such, the embodiments should not be construed to limit the scope of the disclosure. As referenced above, in some embodiments, well-known processes, well-known device structures, and well-known technologies may not be described in detail.

[0034] Unless otherwise indicated, the following terms apply throughout this description. “Semi-structured dialog data” refers to digital records of a conversation in which each utterance is stored together with at least: (i) an absolute or relative time-stamp, (ii) a speaker identifier, and (iii) a sequential turn index. In some embodiments, the term “Semi-structured dialog data” can exclude fully structured databases (e.g., fixed-schema SQL tables) and completely unstructured raw audio waveforms.

[0035] Accordingly the aforementioned systems may include one or more computing devices communicatively and / or operatively connected to the systems for performing one or more steps of any of the disclosed methods. For example, in some embodiments, the computing devices enable deep learning, machine learning and / or artificial intelligence with various networks (e.g.,neural networks) and algorithms. In some aspects of the present invention, software executing the instructions provided herein may be stored on a non-transitory computer-readable medium, wherein the software performs some or all of the steps of the present invention when executed on a processor.

[0036] Aspects of the invention relate to algorithms executed in computer software. Though certain embodiments may be described as written in particular programming languages, or executed on particular operating systems or computing platforms, it is understood that the system and method of the present invention is not limited to any particular computing language, platform, or combination thereof. Software executing the algorithms described herein may be written in any programming language known in the art, compiled or interpreted, including but not limited to C, C++, C#, Objective-C, Java, JavaScript, MATLAB, Python, PHP, Perl, Ruby, or Visual Basic. It is further understood that elements of the present invention may be executed on any acceptable computing platform, including but not limited to a server, a cloud instance, a workstation, a thin client, a mobile device, an embedded microcontroller, a television, or any other suitable computing device known in the art.

[0037] Parts of this invention are described as software running on a computing device. Though software described herein may be disclosed as operating on one particular computing device (e.g. a dedicated server or a workstation), it is understood in the art that software is intrinsically portable and that most software running on a dedicated server may also be run, for the purposes of the present invention, on any of a wide range of devices including desktop or mobile devices, laptops, tablets, smartphones, watches, wearable electronics or other wireless digital / cellular phones, televisions, cloud instances, embedded microcontrollers, thin client devices, or any other suitable computing device known in the art.

[0038] Similarly, parts of this invention are described as communicating over a variety of wireless or wired computer networks. For the purposes of this invention, the words “network”, “networked”, and “networking” are understood to encompass wired Ethernet, fiber optic connections, wireless connections including any of the various 802.11 standards, cellular WAN infrastructures such as 3G, 4G / LTE, or 5G networks, Bluetooth®, Bluetooth® Low Energy (BLE) or Zigbee® communication links, or any other method by which one electronic device iscapable of communicating with another. In some embodiments, elements of the networked portion of the invention may be implemented over a Virtual Private Network (VPN).

[0039] Fig. 1 and the following discussion are intended to provide a brief, general description of a suitable computing environment in which the invention may be implemented. While the invention is described above in the general context of program modules that execute in conjunction with an application program that runs on an operating system on a computer, those skilled in the art will recognize that the invention may also be implemented in combination with other program modules.

[0040] Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0041] Fig. 1 depicts an illustrative computer architecture for a computer 100 for practicing the various embodiments of the invention. The computer architecture shown in Fig. 1 illustrates a conventional personal computer, including a central processing unit 150 (“CPU”), a system memory 105, including a random-access memory 110 (“RAM”) and a read-only memory (“ROM”) 115, and a system bus 135 that couples the system memory 105 to the CPU 150. A basic input / output system containing the basic routines that help to transfer information between elements within the computer, such as during startup, is stored in the ROM 115. The computer 100 further includes a storage device 120 for storing an operating system 125, application / program 130, and data.

[0042] The storage device 120 is connected to the CPU 150 through a storage controller (not shown) connected to the bus 135. The storage device 120 and its associated computer-readable media, provide non-volatile storage for the computer 100. Although the description ofcomputer-readable media contained herein refers to a storage device, such as a hard disk or CD- ROM drive, it should be appreciated by those skilled in the art that computer-readable media can be any available media that can be accessed by the computer 100.

[0043] By way of example, and not to be limiting, computer-readable media may comprise computer storage media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer.

[0044] According to various embodiments of the invention, the computer 100 may operate in a networked environment using logical connections to remote computers through a network 140, such as TCP / IP network such as the Internet or an intranet. The computer 100 may connect to the network 140 through a network interface unit 145 connected to the bus 135. It should be appreciated that the network interface unit 145 may also be utilized to connect to other types of networks and remote computer systems.

[0045] The computer 100 may also include an input / output controller 155 for receiving and processing input from a number of input / output devices 160, including a keyboard, a mouse, a touchscreen, a camera, a microphone, a controller, a joystick, or other type of input device. Similarly, the input / output controller 155 may provide output to a display screen, a printer, a speaker, or other type of output device. The computer 100 can connect to the input / output device 160 via a wired connection including, but not limited to, fiber optic, ethernet, or copper wire or wireless means including, but not limited to, Bluetooth, Near-Field Communication (NFC), infrared, or other suitable wired or wireless connections.

[0046] As mentioned briefly above, a number of program modules and data files may be stored in the storage device 120 and RAM 110 of the computer 100, including an operating system 125 suitable for controlling the operation of a networked computer. The storage device120 and RAM 110 may also store one or more applications / programs 130. In particular, the storage device 120 and RAM 110 may store an application / program 130 for providing a variety of functionalities to a user. For instance, the application / program 130 may comprise many types of programs such as a word processing application, a spreadsheet application, a desktop publishing application, a database application, a gaming application, internet browsing application, electronic mail application, messaging application, and the like. According to an embodiment of the present invention, the application / program 130 comprises a multiple functionality software application for providing word processing functionality, slide presentation functionality, spreadsheet functionality, database functionality and the like.

[0047] The computer 100 in some embodiments can include a variety of sensors 165 for monitoring the environment surrounding and the environment internal to the computer 100. These sensors 165 can include a Global Positioning System (GPS) sensor, a photosensitive sensor, a gyroscope, a magnetometer, thermometer, a proximity sensor, an accelerometer, a microphone, biometric sensor, barometer, humidity sensor, radiation sensor, or any other suitable sensor.

[0048] In some embodiments, the disclosed system may comprise a user interface (UI) such as a graphical user interface (GUI) for arranging and presenting any aspects of the disclosed systems and methods such as augmented reality, patient data, information, results and analysis. In some embodiments, the UI or GUI is produced to a wearable device, such as a set of smartglasses, or to a display. In some embodiments, the UI or GUI enables any of: displaying patient data, displaying patient imaging, stress modelling, overlays of live imaging, augmented reality, mixed reality, audiovisual artificial intelligence (AVAI) and video electroencephalography (vEEG).

[0049] Aspects of the invention relate to machine learning executed on a computing device, wherein the computing device may be computer 100. In some embodiments, the disclosed system and method utilize machine learning algorithms and models, including one or more neural networks, that may operate on at least one computing device (e.g., computer 100). The disclosed system may employ various types of neural networks known in the art, including but not limited to feedforward neural networks (FNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformer networks, autoencoders, generative adversarialnetworks (GANs), Radial Basis Function Networks (RBFNs), extreme learning machines (ELMs), quantum neural networks (QNNs), and deep neural networks (DNNs).

[0050] Machine learning is a branch of artificial intelligence (Al) that enables systems to learn and improve from experience without being explicitly programmed. Machine learning models analyze data sets to identify patterns and correlations, and then uses those patterns to make predictions or decisions. Machine learning models can generally be categorized into three primary types: supervised learning, unsupervised learning, and semi-supervised learning.

[0051] Supervised learning involves training a model using labeled datasets to classify data or predict outcomes accurately. As input data is fed into the model, the model adjusts its internal parameters (e.g., weights) to minimize prediction errors. Common methods used in supervised learning include neural networks, naive Bayes classifiers, linear regression, logistic regression, random forests, and support vector machines (SVMs).

[0052] Classification is a common task in supervised learning, where data inputs are categorized into distinct classes. Classification models may include binary classifiers (e.g., spam vs. non-spam) and multi-class classifiers (e.g., identifying different species of animals). A decision tree is a widely used classification method that applies a sequence of "if-then" conditions to narrow down possible outcomes.

[0053] Regression is another form of supervised learning where the output is a continuous variable rather than a discrete category. Linear regression predicts a continuous value based on a linear relationship between inputs and outputs, while logistic regression predicts categorical outcomes based on defined inputs.

[0054] Unsupervised learning involves analyzing unlabeled datasets to identify hidden patterns or groupings without human intervention. Principal component analysis (PCA) and singular value decomposition (SVD) are common techniques used to reduce data dimensionality and reveal underlying structures.

[0055] Clustering is a key unsupervised learning technique where data points are grouped based on shared features or proximity. K-means clustering is a widely used method where the number of clusters is defined by a variable "k," and the algorithm iteratively adjusts clustercentroids to minimize variance within each cluster. Other clustering methods include hierarchical clustering and probabilistic clustering.

[0056] Semi-supervised learning combines elements of both supervised and unsupervised learning. A model is initially trained using a smaller labeled dataset, which then guides the classification and feature extraction from a larger unlabeled dataset. Semi-supervised learning is particularly useful when acquiring large amounts of labeled data is costly or impractical.

[0057] Multi-modal sensing machine learning involves combining data from multiple sensors (like cameras, microphones, and radar) to create a more complete and accurate understanding of the environment. This approach leverages the strengths of different sensors, allowing machines to "see" and "hear" the world in a way that's more like human perception, and improves the performance of machine learning models in tasks like object recognition, scene understanding, robot navigation.

[0058] Deep learning is a subfield of machine learning that uses neural networks with multiple hidden layers to process and analyze complex data. Neural networks mimic the structure and function of the human brain, comprising layers of interconnected nodes (neurons). Each neuron receives input data, applies a transformation based on assigned weights, and passes the result to the next layer.

[0059] A typical neural network consists of: input layer - receives raw data inputs; hidden layer(s) - applies mathematical transformations using weighted connections; and output layer - generates the final prediction or classification.

[0060] In some embodiments, the disclosed system may include an Al model trained using reinforcement learning, where an agent learns to make decisions through trial and error by interacting with an environment and receiving feedback in the form of rewards or penalties.

[0061] Referring now to Figs. 2A-2B and Fig. 3, shown are exemplary flow diagram of methods (200, 200a) for parsing an interaction and an illustrative computing system 300 for generating structured output data 330 based on receiving input, according to an implementation. The interaction may be a semi-structured interview, such as question / answer format or othertypes of closed ended questions, and the interaction may pertain to a particular topic like patient- reported health history data (e.g., pathways to care (PTC)).

[0062] Other types of topics may include but are not limited to legal proceedings; court proceedings; depositions; lectures; tutoring sessions; business meetings; customer service; customer support interactions; journalism and media interviews; focus groups; human resource interviews and recruitment; project discussions; project management; event planning; qualitative interview for academic research; and social services and counseling.

[0063] However, the semi -structured interaction may be more flexible than simple question / answer format. The interaction may include topics or sections which allow the interaction to be broken up into multiple stages. Some examples include but are not limited tomedical history, surgical history, family history, medication usage, and allergies for the case of clinical data.

[0064] In addition to closed ended questions, the interaction may include a conversation that may be received via an input device, such as a microphone. For example, the interaction may be in-person between at least two people whereby a single computing device (e.g., a laptop) is capable of receiving the conversation from the at least two people. However, the interaction may also be remote, such as a virtual meeting or augmented reality, whereby the microphone is capable of receiving input from the local person speaking, and the application holding the virtual meeting is capable of forwarding the video-audio from the remote individual to be later transcribed. Therefore, both the audio from the microphone and audio-video from the application may be transcribed. In some embodiments, the conversation may be text based. For example, the users may exchange text-based messages with one another over a text-based instant messaging platform, web application, cell phone application, or cell phone text message. Whether the conversation is in-person, remote, text-based, or some combination thereof, the conversation happening in real time may be transcribed as shown in Step 203.

[0065] Referring now to Fig. 2B, in one embodiment, the method 200a may start by completely recording the interaction (e.g., a conversation) in Step 201a, and the completed recording, which may have occurred on another device, may be downloaded as shown in Step 201a. In this case, the video-audio or audio may be transcribed into text as shown in Step 203a.

[0066] In one embodiment, the interaction may include a series of questions (e.g., like an interview) which may comprise semi -structured document data that are presented on an output device, such as a computer screen or speaker. In one embodiment, the interaction may include answers to the questions that may be received via a keyboard or received via an audio device such as a microphone as discussed above. In the case where both document data and audio data have been received, the questions / answers or answers to closed-ended questions received may be aggregated in Step 204 ultimately in an acceptable format as text data for the LLM to process. The interaction may include a conversation; screening forms that may include closed-ended questions; or some combination of both.

[0067] Whether the interaction is a conversation, a screening form, or some combination of both, text data may be pushed to a prompt module 305 for the text data to be processed in Step 205. Where a conversation was included, the audio may be transcribed into text in Step 203, and a portion of the interaction that included audio may be reinserted, interposed, appended, or prepended to the interaction at the appropriate position during aggregation in Step 204. For example, the interaction may include a conversation component comprising audio and may additionally include a screening form with a closed-ended questions component comprising text or other user-provided data. Where a screening form is used, in some embodiments, no audio transcription is needed. Where multiple types of data are present, aggregation engine 311 may aggregate data from, for example, semi-structured document data 310 and audio data into a single data structure for processing. In various embodiments, the data received in Step 201 may have timestamps or markers to ensure that the aggregation engine 311 may be able to order or group the data properly. In one embodiment, such markers may act as types of “stops.” These stops may be based on specific events identified during the conversation. In a healthcare use case, the stops may be based on an interaction with an Emergency Department visit; an interaction with the police; an interaction with a Primary Care Provider; etc. Additionally, the stops may correspond to a timestamp.

[0068] By way of example, the aggregation engine 311 may take a stream of text data (which may include text input and / or transcribed audio text) and create groups / clusters such that the corresponding answers to similar questions or questions addressing a similar topic are in the same group. In some embodiments, screening forms may be labeled with section headers (e.g., medication listings, family history, surgical history) with corresponding questions and answers, such as yes / no, or fill in sections that require that the user list information (e.g., current medications being taken). Corresponding section headers in conversations may be spoken, for example, by a clinician to clearly delineate a series of questions / answers or to delineate a closed- ended question (e.g. asking what medications the patient is currently taking) that may require more than one answer in the form of a list.

[0069] In one embodiment, the transcribed text in Step 203 may be passed to, for example, the prompt module 305, which may utilize a separate internal or external (for example cloudbased) system for algorithmically improving language model prompts and weight. In someembodiments, the system comprises a software framework for programmatically interacting with LLMs and asserting qualities of LLM output (e.g., DSPy). In some embodiments, the prompt module 305 may interact with a locally or remotely-hosted Large Language Model 305a (LLM) as shown in Fig. 3. In turn, the LLM will parse the text and structure the data into a standardized format token by token. In one embodiment, some or the majority of the processing may run on GPU 151 to increase the output rate via parallel processing. Where the LLM 305a is executed locally, the LLM 305a may use GPU 151. Such a setup may be advantageous, for example to avoid breaches of privacy in a clinical setting and / or generally to avoid unintentional data sharing due to complicated Business Associate Agreements that are typical in cloud-based options.

[0070] Using, for example, the prompt module 305, the output from the LLM 305a may be controlled so that LLM 305a may be used and data may be processed reliably and quickly into a coherent, standardized format when output in Step 205. By way of example, the aggregation engine 311 may identify certain subsections of the conversation (e.g., family history, medication, medical history, allergies, surgical history) and group sections of the conversation into subsections as the group.

[0071] Other examples of groups may include but are not limited to: Drug dosage history (e g., what dose of meds patient was taking at a given time, dates of dose changes);Immunization Records (e.g., dates and types of vaccines administered, any adverse reactions, and upcoming scheduled immunizations); Lifestyle and Habits (e.g., details on diet, exercise routines, smoking status, alcohol consumption, and sleep patterns); Mental Health History (e.g., diagnoses, treatment history, therapy sessions, medications, and symptom timelines); Occupational Health (e.g., job history, exposure to hazards, occupational injuries, and related health screenings); Chronic Disease Management (e.g., disease progression, treatment plans, medication adherence, and monitoring results (e.g., blood glucose levels for diabetes); Reproductive Health (e.g., menstrual history, pregnancies, birth control methods, and gynecological exam results); Social Determinants of Health (e.g., housing status, socioeconomic factors, education level, and access to healthcare services); Allergy and Sensitivity Information (e g., types of allergies, severity of reactions, and any desensitization treatments undertaken); Emergency Contacts and Advanced Directives (e.g., list of emergency contacts, power ofattorney, and any living wills or advanced care plans); Diagnostic Test Results (e.g., dates and results of lab tests, imaging studies, and other diagnostic procedures, including follow-up actions); Patient-Reported Outcomes (e.g., patient feedback on symptoms, quality of life assessments, and treatment satisfaction scores); Pain Management History (e.g., pain levels over time, pain management strategies used, and effectiveness of treatments).

[0072] In one embodiment, the downstream processing in Step 207 may include a visual display such as a timeline.

[0073] Although the methods and systems disclosed herein may be described using examples in a particular use case (e.g. a clinical setting or clinical interview) it is understood that the methods and systems disclosed herein may further be used in other situations, including but not limited to the following. In legal depositions or court proceedings, real-time transcription and data extraction during legal proceedings can help in ensuring accuracy and completeness. Lawyers and clients can review and validate the data on the spot. In educational settings, during lectures or tutoring sessions, the system can transcribe the conversation, extract key points, and create summaries or study materials. Students and teachers can then review and correct the content in real time. In business meetings or corporate training, transcribing and extracting action items, decisions, and important points from meetings can streamline follow-up actions. Real-time validation ensures nothing is missed or misunderstood.

[0074] In customer service or customer support, the system can transcribe calls, extract relevant information, and prompt for any missing details. This can improve the accuracy of customer records and enhance service quality. In journalism or media interviews, journalists conducting interviews can use the system to transcribe and extract key quotes and information. Real-time validation ensures that the reported content is accurate and complete. In market research or focus groups, the system can capture and analyze responses, extracting trends and key insights. This can facilitate more accurate and immediate data analysis. In a human resources / recruitment setting, the system can transcribe interviews, extract qualifications and experiences, and ensure all necessary information is gathered and validated.

[0075] In event planning and project management, the system can capture and organize tasks, deadlines, and responsibilities. Real-time validation helps in maintaining clarity andaccountability. In the field of academic research, researchers conducting qualitative interviews can use the system to transcribe and code data in real time, ensuring data integrity and facilitating immediate analysis. In social services and counseling, social workers and counselors can use the system to document client interactions, ensuring accurate records and facilitating better service delivery through real-time data validation.

[0076] In one embodiment, the method 200 may include an iterative clarification mechanism, where ambiguities or gaps in the data from the LLM 305a are identified in Step 206. For example, ambiguities or gaps may be identified if there is a corresponding “stop” without a date or a timestamp, or if there was an event that was identified to have two conflicting dates of occurrence. In one embodiment, a feedback to Step 206 may not necessarily need to be executed. For example, the system may be able to record a correct date between two disparate provided dates by prompting the user with discrete choices (e.g., by displaying a selection on items on the screen as A, B, C, etc.). In some embodiments, the clarification may be in natural language asking the user to type in the answer to be filled in (e.g., asking for the date and entering it). For example, the screen may display a prompt that reads: “Excuse me. I am confused. What date did the Emergency Department visit occur?” In some embodiments, the prompt may comprise some other form of visual prompt, for example a blinking red dot next to a stop where it is unclear when on the timeline the event occurred.

[0077] In some embodiments, each confidence score Cs is compared to a configurable confidence threshold (Ct); if Cs > Ct (e.g., Ct = 0.90) the record proceeds to storage, while any record with Cs < Ct triggers a clarification branch. The threshold may be tuned between 0.50 and 0.99; values > 0.90 balance precision with user burden in clinical-intake trials conducted by the inventors.

[0078] In situations where the recording (e.g., video-audio or audio) was recorded, the user may choose to confirm or accept the output from the LLM. After confirmation, the output may proceed to downstream processing in Step 207a. The confirmation may include confirmation of all or some of the parts of the output. That is, the user may select all or one or more portions of the output they find acceptable or not.

[0079] For example, the user may indicate the output is unacceptable because the parsed text may not have grouped to their liking. In such a case, Step 205a is re-executed with different prompts, based on different parameter inputs, to generate a different standardized output. Additionally, the user may continue to regenerate portions of the output until those portions are acceptable. In one embodiment, an algorithmically-generated request for clarification may be output to one or more parties in the interaction for example via a display.

[0080] In one embodiment, the request for clarification may be output via audio, for example as a text-to-speech prompt. In one embodiment, the request for clarification may include audio or video and may include a number of output devices such as displays, monitors, projectors, speakers, and headphones as discussed above in relation to Fig. 1. In the case where clarification is requested, structured output data 330 may be partially fed back into computing system 300 and / or Step 201 may be re-executed referencing a corresponding marker, timestamp, stop, or cluster such that new semi -structured data, as document data 310 or audio data 330, may replace or supplement the previous text with the same marker, timestamp, stop, or cluster during Step 204.

[0081] The interaction may include one person, two people, three people, or any number of people. In a medical setting, the interaction may include one person (e.g., a patient). A semistructured interaction may include a series of questions / answers which may be presented via display on a screen, via audio or video prompts, or the like, or any combination thereof and may be received using a keyboard, microphone, or any combination thereof. In some embodiments, the interaction may include only one person talking, with or without any prompting question(s), for example where the person is merely asked to free-associate, meditate, or record a personal journal entry. In one example, a single person may recite a free-form list of tasks or goals, which the system may convert into a clearly formatted to-do list.

[0082] In one embodiment, the downstream processing may include an output to a screen in Step 207 whereby the single user may be able to provide clarification if needed. For example, the screen may include a series of discrete action items to be confirmed, where each of the action items may correspond to a grouping section as discussed above. Additionally, the action items may correspond generally to any of the structures created by the LLM based on the prompts.

[0083] In various settings, the interaction may include at least two people. The semistructured interaction may include a series of questions / answers which may be presented via an output device, e.g. a display or a speaker, or may be directly posed by the parties (e.g., interviewer / interviewee, lawyer / client, customer support / customer, HR / applicant, counselor / counselee). In one embodiment, the downstream processing may include an output to a screen in Step 207 whereby the first party, the second party, or both (e.g., patient / clinician) may be able to validate the processed structured data and provide clarification or correction if needed.

[0084] Upon validation, the data may be committed into one or more downstream repositories (e.g., EHR in clinical settings or REDCAP in research settings) or in any data store such as a relational database that takes structured data.

[0085] Referring in detail to Fig. 3, a data flow diagram of a software platform is disclosed. The software platform may in some embodiments comprise a text transcription engine 309, being connected to an input device, which may include text or audio. In the case of audio, the input device may include a microphone 301b, and in the case of text, may include SMS text messages from an exemplary cell phone 301a, or may include any other devices capable of collecting textbased input, including but not limited to text applications, web applications, cell phone applications, and the like.

[0086] The text transcription engine 309 may take an audio file or audio stream as an input, for example an audio file of a pre-recorded conversation or an audio stream comprising recorded audio data transmitted over a communication network to the transcription engine 309. That is, the transcription engine 309 may receive a wholly pre-recorded conversation, which may have been transferred, for example via a network connection, to the computing device for processing, having been recorded on another device. The transcription engine 309 may parse the incoming audio, presented in whatever format, into text data 320, where text data 320 is a text-based representation of the recorded or transmitted audio. Text data 320 may be stored in any suitable format, for example as a simple ASCII text file, or may alternatively be stored in a semistructured dialog data file such as XML, or a token-based or tuple-based text format such as JSON. In some embodiments, the text data may be diarized, either manually or in an automated fashion, in order to track which speaker said which portions of the transcript.

[0087] The software may further comprise a prompt module 305 taking the text data 320 as an input, wherein the prompt module 305 is configured to generate prompts from the text data 320. In some embodiments, the prompt module 305 may be further configured to accept document data 310, for example the results of a user-submitted survey, as an input, either in addition to or alternatively to the text data 320. The prompt module may in some embodiments comprise a locally- or remotely-executed software for generating prompts from semi-structured data, for example DSPy. The prompt module 305 may be further configured to communicate the generated prompts to an LLM 305a, where the LLM 305a may be locally or remotely executed. The prompt module 305 may be configured to receive the results of the queries to the LLM and pass the results to aggregation engine 311.

[0088] Aggregation engine 311 may be configured to receive LLM results from prompt module 305 or directly from LLM 305a, and may further be configured to accept the prompts themselves as an additional input from the prompt module 305. In some embodiments, some or all data passed to the aggregation engine 311 may be timestamped in order to preserve ordering of data. In some embodiments, some or all data passed to aggregation engine 311 may comprise a confidence score, for example a confidence score returned from the text transcription engine 309 representing a confidence in the accuracy of the text transcription, and / or a confidence score returned from the LLM representing a confidence in the accuracy of the response from the LLM based on the prompt presented by prompt module 305. Aggregation engine 311 may in some embodiments be further configured to accept the text data 320 and / or document data 310 as an additional input. The aggregation engine 311 is configured to collect, sort, and structure the data presented from the various data sources into a structured output data 330, and may be further configured to present some or all of the structured or input data as a human-readable output via an output device, for example a display 302a and / or a speaker 302b. In some embodiments, output devices may include a virtual or augmented-reality display or headset.

[0089] In some embodiments, aggregation engine 311 may be further configured to determine whether clarification may be required in any of the data received, for example if a response from the LLM 305a has a low confidence score and / or where some text data 320 has a low confidence score. In such situations, the aggregation engine 311 may be configured to prompt a user, for example via display 302a or speaker 302b, to request clarification of one ormore questions, or to request additional information based on the responses collected so far. In some embodiments, the clarification request may be made after the interaction has been completed, for example an aggregation engine may determine that a confidence score in certain text data 320 may be below a certain threshold, and may request that a user verify the content of a portion of the text data 320 by playing back some portion of collected audio data for a human to verify the transcribed text data 320 is correct. In some embodiments, a clarification may be required due to an algorithmic exception or failure, for example if a computer-implemented method attempts to assert a particular piece of interpreted data into a particular format and the assertion fails, the fail routine may comprise the step of asking for clarification. In some embodiments, as described above, feedback may be implemented as a real-time back and forth during the interaction. In other embodiments, where a recorded conversation is analyzed and processed after the fact, clarification may be requested and provided asynchronously. In such situations, clarification questions may be formulated and sent to one or more parties in the interaction afterwards. In some embodiments, data subject to clarification may be flagged and / or discarded, with clarification scheduled for the next interaction or follow-up visit.REFERENCES

[0090] The following publications are each hereby incorporated herein by reference in their entirety:

[0091] Mathis, W. S., Ferrara, M., Burke, S., Hyun, E., Li, F., Zhou, B., Cahill, J., Kline, E. R., Keshavan, M. S., & Srihari, V. H.; Granular analysis of pathways to care and durations of untreated psychosis: A marginal delay model. PLOS ONE, 17(12), 2022.

[0092] The disclosures of each and every patent, patent application, and publication cited herein are hereby incorporated herein by reference in their entirety. While this invention has been disclosed with reference to specific embodiments, it is apparent that other embodiments and variations of this invention may be devised by others skilled in the art without departing from the true spirit and scope of the invention. The appended claims are intended to be construed to include all such embodiments and equivalent variations.

Claims

1. CLAIMSWhat is claimed is:

1. A system comprising: at least one processor and an input device; a non-transitory memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising ingest- segment-generate steps comprising: receiving semi -structured dialog data from one or more sources as text data; processing the text data in a prompt module, wherein the prompt module generates a call to a large language model (LLM) to structure the text data; receiving structured output from the LLM corresponding to the forwarded text data; and storing the structured output from the LLM in a repository.

2. The system of Claim 1, wherein the one or more sources comprise a text source.

3. The system of Claim 1, wherein the one or more sources comprise an audio source, wherein the operations further comprise: receiving audio data from the audio input device; and converting the audio data to the text data via a text transcription engine.

4. The system of Claim 1, wherein the one or more sources comprise an audio source and a text source.

5. The system of Claim 4, wherein the input device comprises an audio input device, wherein the operations further comprise: receiving audio data from the audio input device; and converting the audio data to a first portion of the text data via a text transcription engine; converting text data received from the text source to a second portion of the text data;organizing the first portion of the text data and the second portion of the text data via an aggregation engine to create the text data; and forwarding the organized text data to the prompt module.

6. The system of Claim 1, wherein the operations further comprise: determining whether clarification is required based on the structured output from the LLM; when clarification is required, generating a prompt to request second semi-structured dialog data from the one or more sources as text data; processing the second text data in the prompt module, wherein the prompt module generates a call to the large language model (LLM) to structure the second text data; receiving a second structured output from the LLM corresponding to the second text data; updating at least a portion of the structured output with at least a portion of the second structured output; and storing the updated structured output from the LLM in the repository.

7. The system of claim 1, wherein the operations execute on a locally attached graphics processing unit (GPU) and no raw dialog data is transmitted outside the device.

8. The system of claim 1, the operations further comprising: receiving a domain-specific training corpus; updating parameters of the LLM using gradient-descent fine-tuning; and repeating the ingest-segment-generate steps with the fine-tuned model.

9. The system of claim 1, wherein the semi-structured dialog data is obtained via a prompt template selected from a library of domain-specific templates that include at least a healthcare template and a legal-deposition template.

10. The system of claim 1, further comprising a visualization layer configured to generate, in real time, a timeline graphic or checklist based on the structured output. re11. A system comprising: an output device configured to output a first set of one or more questions; an input device configured to receive a first set of one or more answers to the first set of one or more questions; an audio device configured to receive audio data, wherein the audio data comprises: a second set of one or more answers; a computing system communicatively coupled to the output device, the input device, and the audio device, the computing system comprising: one or more datastores storing a large language model (LLM), wherein the LLM is configured to standardize output based on one or more prompts; a text transcription engine configured to transcribe the audio data to text data; a prompt module configured to structure the text data into a series of prompts and present the series of prompts to the LLM; and an aggregation engine configured to organize the data received from the LLM into a structured dataset.

12. The system of Claim 11, wherein the audio data further comprises a second set of one or more questions.

13. The system of Claim 11, wherein the input device comprises a keyboard or a touchscreen, wherein the output device comprises a display or a speaker.

14. The system of Claim 11, wherein the audio device comprises a microphone.

15. The system of Claim 11, wherein the aggregation engine is further configured to present at least a portion of the structured dataset as output on the output device.

16. The system of claim 11, wherein the LLM executes on a locally attached graphics processing unit (GPU) and no raw dialog data is transmitted outside the device.

17. A method comprising: receiving semi -structured dialog data from one or more sources as text data; organizing the text data into a set of prompts; generating at least one call to a large language model (LLM) comprising a prompt of the set of prompts; and aggregating the output from the LLM into a structured dataset.

18. The method of Claim 17, wherein the one or more sources comprise a text source.

19. The method of Claim 17, wherein the one or more sources comprise an audio source, further comprising the steps of: receiving audio data from the audio source; and converting the audio data to the text data.

20. The method of Claim 17, wherein the one or more sources comprise an audio source and a text source.

21. The method of Claim 20, further comprising the steps of: receiving audio data from the audio source; converting the audio data to a first portion of the text data; converting text data received from the text source to a second portion of the text data; aggregating the first portion of the text data and the second portion of the text data; and forwarding the organized text data to the prompt module.

22. The method of Claim 17, further comprising the steps of: determining whether clarification is required based on the structured dataset; when clarification is required, requesting second semi-structured dialog data from the one or more sources as second text data; organizing the second text data into a second set of prompts; generating at least one second call to the LLM comprising a prompt of the second set of prompts and receiving a second structured output; andupdating at least a portion of the structured dataset with at least a portion of the second structured output.

23. The method of Claim 17, wherein the one or more sources comprise one or more of: a text message, a text-based messaging app, or a web-based application.

24. The method of Claim 17, wherein the structured dataset comprises a markup language selected from JSON or XML.

25. The method of Claim 17, wherein the structured dataset comprises a fde format selected from comma-separated values (CSV) or tab-separated values (TSV).

26. The method of claim 17, wherein the LLM executes on a locally attached graphics processing unit (GPU) and no raw dialog data is transmitted outside the device.

27. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the processors to perform operations comprising: receiving semi -structured dialog data from one or more sources as text data; processing the text data in a prompt module, wherein the prompt module generates a call to a large language model (LLM) to structure the text data; receiving structured output from the LLM corresponding to the forwarded text data; and storing the structured output from the LLM in a repository.

Images