Intelligent Transcription and Biomarker Analysis
An AI-driven system for transcribing and analyzing therapy sessions addresses the inefficiencies of traditional methods by providing accurate, real-time emotional insights, improving therapy fidelity and scalability.
Patent Information
- Application Number
- JP2024552098
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-10-10
- Filing Date
- 2022-11-21
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Traditional transcription techniques in mental health therapy sessions are distracting for therapists, prone to human error, and fail to capture emotional nuances, making them inefficient and non-scalable.
Implementing an AI-driven system for transcribing, tagging, and analyzing therapy sessions using natural language processing (NLP) and machine learning to automatically generate transcriptions, identify biomarkers, and provide real-time insights.
Enhances therapy session fidelity by reducing human distraction, minimizing errors, and providing real-time emotional analysis, enabling more effective therapy delivery.
Smart Images

Figure 2024543238000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This PCT application claims priority to U.S. Provisional Patent Applications No. 63 / 282,638 (APPARATUSES, SYSTEMS, AND METHODS FOR A LANGUAGE AGNOSTIC AI APPROACH TO ENSURE THERAPEUTIC FIDELITY, filed November 23, 2021) and No. 63 / 414,772 (INTELLIGENT TRANSCRIPTION AND BIOMARKER ANALYSIS, filed October 10, 2022), which are incorporated herein in their entireties for all purposes. [Background technology]
[0002]
[0002] Every year, millions of people undergo some form of mental health treatment or counseling. There have been many advances made in mental health techniques and therapies over time. However, there are still some shortcomings in therapy sessions. For example, during a therapy session, the therapist may attempt to take notes during the session or create their own written record after the session. However, when the therapist needs to take their own notes during the session, they are both distracted and distanced from the patient. Furthermore, when the therapist creates a written record after the session, the record may inadvertently leave out parts of the session due to human memory. Furthermore, the written record may not take into account the emotions of the therapist or the patient during the session.
[0003]
[0003] Conventional transcription techniques may also not take into account emotions during a session or provide insight or analysis regarding the session. Furthermore, given the number of patients a therapist may see in a given time period and the amount of data the therapist will parse through, human-based transcription may introduce errors and manual tagging of therapist sessions may be very time-consuming and not easily scalable. Summary of the Invention
[0004]
[0004] Various embodiments according to the present disclosure are described with reference to the drawings. [Brief description of the drawings]
[0005] [Figure 1]
[0005] FIG. 1 illustrates an example user interface that may be used to implement aspects of various embodiments. [Diagram 2]
[0006] 1A-1C illustrate example audio file waveforms that may be utilized to implement aspects of various embodiments. [Diagram 3]
[0007] 1A-1C illustrate example analytics that may be extracted, generated, or calculated in accordance with various embodiments. [Figure 4]
[0008] FIG. 1 illustrates an example system that can be used to implement aspects of various embodiments. [Diagram 5]
[0009] FIG. 1 illustrates an example methodology that may be used to implement one or more aspects of various embodiments. [Figure 6]
[0010] FIG. 1 illustrates an example environment for implementing aspects according to various embodiments. [Figure 7]
[0011] 1 is an exemplary block diagram of an electronic device that may be utilized to implement one or more aspects of the various embodiments. [Figure 8]
[0012] FIG. 1 illustrates components of another exemplary environment in which aspects of various embodiments may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0006]
[0013] Techniques are provided for transcribing, translating, reviewing, tagging, and providing analysis of therapist sessions. One or more interactions between one or more providers and recipients of services may be recorded and automatically transcribed according to natural language processing (NLP) or artificial intelligence techniques. The interactions may include patient interactions with the patient's medical team, including, but not limited to, interactions between the patient and a therapist or between the patient and a general practitioner, among other options. According to an exemplary embodiment, individual utterances may have timestamps associated with them that indicate where the utterance may be in the audio file or transcript. The timestamps may be associated with a speaker identifier. In some embodiments, a confidence score may be provided that indicates how confident the system is in determining the timestamp. Additionally, session type, length of recording, location, and date of recording may be provided, among other options. Use of the system may generate a rich database of transcriptions and automated analysis, which allows NLP algorithms and / or other artificial intelligence techniques to be utilized for automatic tagging and / or biomarker discovery. The analysis may be provided in near real-time or in real-time.
[0007]
[0014] In the following description, various embodiments are described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may be practiced without these specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the embodiments being described.
[0008]
[0015] "Artificial intelligence" as used herein may include, but is not limited to, machine learning, natural language processing, neural networks, random forest models, and other such algorithms or models.
[0009]
[0016] FIG. 1 illustrates an exemplary user interface 100 that may be used to implement aspects of various embodiments. As shown, the exemplary interface 100 may include a feature 110 for controlling audio. The audio may be audio from a media file taken during a therapist session. In some embodiments, the therapist session may occur before, during, and / or after an initial treatment, such as a medication session with psilocybin, for the treatment of a medical condition. A table 120 may also be provided for display. The table 120 may include utterances provided under a "#" column, timestamps provided under a "Time" column, speaker IDs provided under a "Speaker ID" column, and transcribed text corresponding to the media file, shown under a "Text" column. One or more indicators 130 may also be provided. In an exemplary embodiment, the indicators 130 may correspond to instructions associated with the therapy session or tags that may identify a biomarker of the patient and one or more corresponding features of the patient that allow a system or user to find meta-biomarkers for that person. For example, the patient's emotional state, which may be a biomarker in at least some embodiments, may be used to train a tagging model. The machine learning model may utilize the tagged data to generate one or more new biomarkers that the user may not have initially considered or disclosed.
[0010]
[0017] In at least one embodiment, the tagging system may be provided for display as an interactive table that allows a user to apply one or more tags to one or more strings of text. The interactive table may display the time the corresponding text was spoken, the individual who spoke (based on an identifier), and the content of the speech. In at least one embodiment, the user may be presented with a pop-up window or another screen that allows the user to select one or more of a plurality of tags. In one exemplary embodiment, the tag may include an instruction such as "therapist discusses grounding techniques" or "therapist discusses external support," among other such options. Individual tags may be displayed with an associated color or other indicator, such as a symbol. If the user elects to assign a tag to a string of text, the string of text may be displayed with a color and / or indicator that correlates to the selected tag, according to one exemplary embodiment. In embodiments in which a video recording of the session is recorded, the video data may be tagged or labeled using computer vision techniques.
[0011]
[0018] In some embodiments, tagging may be automated, such that a set of training data comprising tagged items may be utilized with a machine learning algorithm to associate some text from the transcript with tags. The trained machine learning model may predict one or more tags for the transcript. In at least some exemplary embodiments, the predicted tags may be provided for display.
[0012]
[0019] According to an exemplary embodiment, NLP and AI techniques may be utilized to identify one or more biomarkers indicative of a patient's progress in their therapy journey and / or flag those biomarkers for the therapist's review. In an exemplary embodiment, the biomarkers may correspond to any digitally identifiable signal of voice, speech pattern, tonality, frequency / pitch, emotion, syntax, pronunciation, or any sensory input, including but not limited to audio, visual, text, or tactile input. In some exemplary embodiments, the biomarkers may be provided for display. The biomarkers may be generated based on one or more data inputs, according to an exemplary embodiment, or the biomarkers may be manually entered by a therapist or administrator. Machine learning may be utilized to analyze the sensor data and determine if patterns can be detected to predict the outcome of interest, according to an exemplary embodiment. Supervised or unsupervised learning techniques may be utilized.
[0013]
[0020] The data input may take into account sensor data, including data received through one or more smartphones, tablets, cellular phones, touch screen devices, smart watches, personal computers, accelerometers, gyroscopes, respiration sensors, body movement sensors, proximity sensors, motion sensors, ambient light sensors, moisture sensors, temperature sensors, compasses, barometers, fingerprint sensors, face identification sensors, cameras, pulse sensors, heart rate variability (HRV) sensors, beats per minute (BPM) heart rate sensors, microphones or other sound sensors, speakers, and GPS or other location sensors. In at least some embodiments, multiple sensors may be utilized to collect a combination of data. In some embodiments, multiple sensors may be communicatively coupled.
[0014]
[0021] In one exemplary embodiment, a patient may have a therapy session with a therapist. Such a session may occur shortly (e.g., a day or a week) after an initial therapy session, such as after a medication session using psilocybin for the treatment of a health condition. This example is not limiting, and one or more aspects of various embodiments may be utilized and / or performed prior to or during a therapy treatment session. In this exemplary embodiment, the therapist session may be recorded and transcribed, translated, and / or annotated. The therapy session may be analyzed in an automated manner to determine whether the patient or therapist followed a predetermined treatment model.
[0015]
[0022] In an exemplary embodiment, a therapy session may be translated from one language to another. For example, a text-based representation of the session may be maintained in any language. In some embodiments, the session may be transcribed prior to translation. In other embodiments, the session may be transcribed after translation.
[0016]
[0023] One or more aspects of the present embodiment may be provided through a web application. According to one or more embodiments, a user may manually rate and tag therapy sessions. Additionally, a user may view sessions that have been automatically rated and tagged, such as sessions that the computing system has analyzed and tagged. For example, the computing system may be configured to apply natural language processing (NLP) and / or artificial intelligence (AI) techniques to transcribe, translate, review, rate, and / or flag items of interest from one or more sessions and / or therapists. NLP may be utilized to automatically generate session transcriptions, according to an exemplary embodiment. Using AI, one or more biomarkers related to a patient's progress may be identified and flagged for review by the therapist. AI may be configured to improve understanding of mechanisms of action and / or mechanisms of change for a particular therapy. For example, AI may be utilized to predict how a patient is responding to psilocybin therapy or to determine how a patient may be feeling during the preclinical phase of the patient's therapy. Biomarkers can be any digitally identifiable signal of voice, speech pattern, tonality, frequency / pitch, syntax, pronunciation, or any sensory input, including but not limited to audio, visual, text, and tactile input. Biomarkers can be generated based on data input, such as sensor data. In some embodiments, biomarkers can be manually entered by a therapist and / or administrator. In an exemplary embodiment, biomarkers can be determined and evaluated at a given time. For example, biomarkers can be analyzed prior to, during, and / or after treatment. Additionally, the system can predict patient suitability for certain types of treatment.For example, the system may determine a suitability score for a patient based on various factors, including whether the patient has asked more questions and may therefore be more open to experience prior to treatment, or whether / how much the patient has reviewed literature related to the treatment. The suitability score may be provided for display to the therapist via a mobile or web-based application, among other options.
[0017]
[0024] In NLP, an "utterance" may be defined as a spoken group of words preceded and followed by a pause. In contrast, a "sentence" may refer to a group of words that expresses a complete thought. According to one or more embodiments described herein, an "utterance" may be an amalgamation of two utterances. For example, if a sentence occurs across several utterances, the utterances may be combined to form a single utterance. Alternatively, if an utterance contains several sentences, each sentence may be extracted and treated as a single utterance.
[0018]
[0025] After using NLP to transcribe an audio file, individual utterances within the transcription may be determined. For example, a language model may be utilized to parse the transcript into individual utterances. The language model may consider factors such as punctuation or terminology that may relate to the same topic within a given time period. Additional processing steps may be taken after transcribing. For example, Unicode characters may be converted to ASCII characters. Redacted information such as names may be replaced with non-personally identifiable alternatives (e.g., Jane / John Doe). According to another exemplary embodiment, transcriber comments may be replaced with text familiar to the model. For example, the transcription note "[LAUGHING]" may be replaced with "haha" so the model may better recognize the term and use it in generating the score.
[0019]
[0026] Sentiment analysis of a text may include scoring the text as positive or negative. Text may be scored in two dimensions: valence (e.g., positivity) and arousal (e.g., energy or activation). In this way, sentiment may capture intensity, not just positivity or negativity. Sentiment scores of a text may be distinguishable from attempting to infer the emotional state of a speaker. For example, a text stating "I love broccoli" may be scored as positive by a traditional sentiment model. However, if voiced in a sarcastic manner, the text would signal a negative attitude toward broccoli. Furthermore, a text stating "I love broccoli" may be ranked as 97% positive in one model, while a text stating "I like broccoli" may be ranked as 98% positive in the same model. The similarity in the positivity scores may be a result of the sentiment analysis problem being treated as a classification problem and not considering intensity. In such a case, the model may only care whether the "positive" or "negative" label is correct. According to one or more embodiments described herein, a classifier may be used to score, for each utterance, the likelihood that the utterance should belong to one of the following four classes: "happy", "angry", "pessimistic" and "calm". This exemplary embodiment describes these four classes, but other classes that indicate emotions may be utilized. For a given utterance, the model may provide a score for each class. To determine the session sentiment score, a set of utterances of a given speaker (e.g., participant or patient, therapist, etc.) during a single visit (e.g., initial preparation, initial integration, etc.) may be given a probability measure, where the measure for a given utterance is proportional to the number of words in the utterance. From this, an average valence score and an average arousal score may be obtained.The scores, in at least one exemplary embodiment, may be provided for display along with the transcription or as a separate page.
[0020]
[0027] In at least one exemplary embodiment, the therapy session may be recorded via one or more sensors, such as a microphone. In one exemplary embodiment, the microphone may be positioned such that it produces at least two different audio tracks, which allows for spatial source separation for various users within the session. The recorded audio may be stored, such as on a local device with memory that may be coupled to the computing device, or directly on the computing device. In other embodiments, the recorded audio may be uploaded to cloud-based storage. Prior to transcription, the audio may be preprocessed to reduce background noise. Preprocessing may also include ensuring that each track is dedicated to a single speaker. Examples of such preprocessing may include, but are not limited to, blind source separation or winner-takes-all sound audio filtering, among other such options.
[0021]
[0028] According to an exemplary embodiment, visualization of audio may be provided, such as in the form of sound waves as shown. FIG. 2 shows an exemplary audio file waveform 200 that may be utilized to implement aspects of various embodiments. In this example, audio from the therapist and audio from the patient 240 may be presented in an easily identifiable format, such as by presenting individual waveforms 250, 260 in different colors or line formats. The waveforms in this exemplary embodiment may be provided as amplitude 220 over time 230. The waveforms may be utilized in an exemplary embodiment to determine one or more biomarkers regarding how the patient or therapist may be feeling during the session. The user may also be presented with the option to isolate and / or independently adjust the volume of each party. For example, the user may reduce the volume of the therapist to better hear the patient. Additionally, the volume of the therapy for the therapist and the patient may be automatically adjusted and / or normalized.
[0022]
[0029] In some exemplary embodiments, transcription may be performed on the fly in real time while the therapy session is taking place. Transcription may convert the audio file into a text file (e.g., JSON or similar format). The transcription may indicate or link to one or more time points related to the time points in the audio file. Transcription may include neural net transcription and translation, according to one exemplary embodiment, whereby a neural net is utilized to generate a transcript and / or generate a translation of the session recording. In some embodiments, the therapy session may be video recorded and biomarkers may be collected based on detected visual characteristics, such as facial expressions and body language interpreted through a computing system.
[0023]
[0030] In some embodiments, the therapy session may be conducted in person or via a computing device, such as a device having a camera and associated microphone, hi other embodiments, the therapy session may be conducted through the use of a virtual reality device.
[0024]
[0031] The transcript may be uploaded to a web or mobile application for the therapist or reviewer to reference. Additionally, the audio file may be uploaded so that the therapist or reviewer can listen to the file and make any corrections to the file. Corrections may also be automatically flagged by the system based on one or more predictions, according to an exemplary embodiment. For example, the system may detect that portions of the transcript may not have been transcribed with sufficient confidence (e.g., a threshold level of confidence), indicating that those portions of the transcript may not have been accurately transcribed. Based on those flags, the system may predict alternative transcripts from which the therapist or reviewer may choose, or the therapist / reviewer may listen to the audio file at various, flagged time points and manually correct the transcript. For example, a user may state, "I was feeling ... um ... well this morning," but the system may have determined that the user may have said, "um ... well" or "unwell." Such a discrepancy may result in a low confidence value, which causes the system to raise this as an issue. By flagging this potential discrepancy to the therapist or reviewer, the transcript may have a higher overall fidelity because the corrected transcript may more accurately represent how the patient was feeling during the therapist session. In some exemplary embodiments, the system may automatically implement corrections to the transcript based on the determined alternatives. For example, a machine learning model may utilize training data to analyze the transcript and predict one or more corrections to the transcript. According to one or more embodiments, the corrections may be applied to utterances that may have been incorrectly split from the transcript.
[0025]
[0032] Tagging, according to an exemplary embodiment, may be utilized to provide analyses of fidelity and predicted patient outcomes, as well as for NLP and AI analyses of potential digital biomarkers. For example, tagging may involve prompting the user to ask or respond to follow-up questions related to a corresponding piece of text. For example, if a sleep disorder is identified or noted in the user's data, a follow-up question may be asked regarding the user's sleep. Furthermore, tagging may be utilized to request the therapist to follow a particular portion of a treatment model. Tagging may further provide the user with information that may be relevant to the identified portion of text. Such questions and information may be configured to assist the user in categorizing, rating, reviewing, and / or tagging portions of text. Furthermore, tagging may be utilized to determine and ensure that the therapist is following a particular treatment model and / or implementing all necessary key points for a therapy session. In this manner, fidelity within a session may be ensured, which is an important aspect of a particularly sensitive therapy session, such as the first session after initial medication treatment. The analysis may be provided in near real-time or real-time. This example is not intended to be limiting, and one or more aspects of various embodiments may be utilized and / or performed prior to or during a therapy treatment session.
[0026]
[0033] FIG. 3 illustrates an exemplary analysis 300 that may be extracted, generated, or possibly calculated according to various embodiments. This exemplary embodiment is illustrated with respect to a particular analysis of words per minute over time, and is not limiting. The calculated analysis may be processed using one or more techniques, including, but not limited to, mathematical models, statistical models, and computer science analysis, including dimensionality reduction, clustering, NLP, and / or generalized linear modeling. Machine learning techniques may be utilized to help predict one or more treatment outcomes associated with a user, according to an exemplary embodiment. The analysis may be provided in a dashboard format to one or more users, such as authenticated users, for review. According to an exemplary embodiment, AI may be utilized to identify potential language biomarkers. The system may consider various analyses, including, but not limited to, pitch / frequency, words per minute 310, pause frequency, and / or pause duration for individual speakers 330 over time 320. The analysis may show the analysis associated with individual speakers as graphs 340, 350, as shown in FIG. 3. Additionally, the analysis may be categorized based on when the data was collected. For example, the system may compute analyses before, during, and / or after administration of therapy, and both sets of analyses may be provided for display. The system may extract differences between sets of data and provide visual indicators suggesting potential explanations or other insights for those differences. For example, using machine learning, the system may generate insights based on differences between pre- and post-medication. In another exemplary embodiment, dynamic support for the therapist may be provided for display in real-time or pseudo-real-time to assist the therapist in making clinical decisions and improving patient care. Such support may be generated using artificial intelligence. As a non-limiting example, when the therapist is meeting with the patient, a machine learning model may prompt areas for exploration or areas of the treatment model that were not robustly covered in the previous session.Alternatively, if the system detects that needed content was not covered in a previous time frame of the session, real-time feedback can be provided to the therapist to prompt the therapist to cover that content.
[0027]
[0034] The analysis may include multiple metrics in an exemplary embodiment. For example, the audio recording may be analyzed for speaking rate, including the patient's rate and optionally the therapist's rate. The rate may be measured in words per minute or any other suitable units. The system may present a graph representing the speaking rate, such as one in the form of "words per minute versus time points." The patient's and therapist's speaking rates may be presented as different color line graphs or by other visual indicators.
[0028]
[0035] In an exemplary embodiment, the tags may include both a patient's biomarkers and one or more corresponding features of the patient that allow a system or user to find meta-biomarkers for that person. For example, the patient's emotional state, which may be a biomarker in at least some embodiments, may be used to train a tagging model. A machine learning model may utilize the tagged data to generate one or more new biomarkers that the user may not have initially considered or disclosed.
[0029]
[0036] In at least one embodiment, the tagging system may be provided for display as an interactive table that allows a user to apply one or more tags to one or more strings of text. The interactive table may display the time when the corresponding text was spoken, the individual who spoke (based on an identifier), and the content of the speech. In at least one embodiment, the user may be presented with a pop-up window or another screen that allows the user to select one or more of a plurality of tags. In one exemplary embodiment, the tag may include an instruction such as "The therapist described grounding techniques" or "The therapist described external support," among other such options. Individual tags may be displayed with an associated color or other indicator, such as a symbol. If the user elects to assign a tag to a string of text, the string of text may be displayed with a color and / or indicator that correlates to the selected tag, according to one exemplary embodiment. In embodiments in which a video recording of the session is recorded, the video data may be tagged or labeled using computer vision techniques. For example, using computer vision techniques, biomarkers may be labeled at a given time point and provided for display. In this example, the patient's video data may be analyzed using machine learning and digital biomarkers may be extracted. In this embodiment, the digital biomarkers may include, but are not limited to, physical agitation, facial expression, flushing response, and eye contact, among other such biomarkers. Tabs or filters may be provided within the interactive table so that specific categories or subsets of information may be displayed. The tabs or filters may include, but are not limited to, "Transcribed," "Anonymized," "Redacted," and "Labeled," according to one exemplary embodiment.
[0030]
[0037] In addition to tagging, other features may be provided for display, including, but not limited to, commenting, tagging other users to specific modules within the service, highlighting sections for review, and providing space for collaborative notes.
[0031]
[0038] According to an exemplary embodiment, a user may select a particular string of text. One or more parameters, including but not limited to the user's selection and various identifiers associated with the session, may be sent to a back-end database. Thus, the user selection may be verified and communicated to the database. In such an embodiment, the tags may be sorted and / or filtered algorithmically, either deterministically or by using artificial intelligence. The tags may be sorted and / or filtered to provide the most appropriate tags for user selection based on NLP or AI and one or more of the parameters. The sorted tags may be returned to a front-end web / mobile application where they may be provided for display to the user.
[0032]
[0039] The tagging system and machine learning models can be General Data Protection Regulation (GDPR) compliant and Health Insurance Portability and Accountability Act (HIPAA) compliant. For example, audio files may be streamed but not copied by users. Additionally, users may be assigned specific roles with associated access permissions. Users may also be assigned identification numbers so that personal information such as names are not compromised. The system may utilize data from any number of patients without disclosing the identity and / or source of the data. In this manner, the machine learning models utilized in the system may train on collaborative patient data without compromising the patient's personal identity. Additionally, steps performed on the data may be captured and logged along with version history. Additionally, the process for users to sign in may be user specific with policies in place to help ensure that data is protected. In some exemplary embodiments, verified users may be provided with the option to utilize a Quick Response (QR) code or other embedded identifier to sign in and access permissible features.
[0033]
[0040] According to an exemplary embodiment, principal component analysis (PCA) may be employed when analyzing session recordings. PCA may be utilized to visualize the data sets. For example, patterns or variations among the data sets may be visualized using PCA. Patterns may include the number of words used, the types of words used, or other related factors. A two-dimensional PCA graph may be provided for presentation to a user, where data from individual speakers may be color coded or otherwise provided to indicate the speaker.
[0034]
[0041] In at least one embodiment, the system may calculate the average speed duration of each speaker during a session. The system may further compare the average duration of speech by each individual against the average duration of speech of the other party (e.g., in the case of different patient-therapist pairings). The system may present the average duration of speech of the parties by therapist identifier to the user. The system may also provide for display a visual representation of the actual speech duration for both parties from a particular session.
[0035]
[0042] In some exemplary embodiments, a user may be able to review a given session and rate its fidelity. For example, a user may be provided with multiple prompts and sorted by category, which allows the user to rate the completeness of each prompt. The categories in this example may include, but are not limited to, preparation, medication, and integration. One exemplary prompt may be "Did the therapist cover relevant information about the treatment?" If the user determines that the therapist did not cover relevant information about the treatment, the user may enter an answer such as "Didn't." In some embodiments, the user may respond to the prompts using a slider bar or other such interface. Additionally, the user may be provided with the option to include their own notes after each prompt. Responses for the prompts may be aggregated to determine an overall fidelity rating. Additionally, responses for each prompt in each category may be aggregated to determine a category-specific fidelity rating.
[0036]
[0043] FIG. 4 illustrates an exemplary system 400 that may be used to implement aspects of various embodiments. In this exemplary embodiment, a user may be provided with a tagging feature that may be presented to the user as an interactive table through one or more user interfaces 402, 404, allowing the user to apply one or more tags to a string of text. Prior to referencing the interactive table, one or more access control files 416 may be sent and verified at one or more backend servers 406, 410 to ensure that the user is able to view the content. In response to a request for access 420, the user may have access to view the content through the cloud infrastructure 412, 414. Stored objects may be returned 422 and an updated or consolidated access control file 408 may be stored. The consolidated results may be provided 418 to the user interfaces 402, 404. The interactive table may display the time the text was spoken, the individual who spoke, and the content of the talk, among other options. In an exemplary embodiment, the user may be presented with a pop-up window or another screen that allows the user to choose from a list of multiple tags. The interactive table may include several tabs and / or filters that may be configured to display categories or sets of text strings.
[0037]
[0044] FIG. 5 illustrates an example method 500 that may be used to implement one or more aspects of various embodiments. According to one or more embodiments, a media file capturing one or more interactions between one or more providers and a recipient of a service may be obtained 510. According to an example embodiment, the one or more providers may be members of a medical team providing a medical service, and the recipient may be a patient of the service. A transcript of at least a portion of the one or more interactions captured in the audio file may be generated 520. Using a neural network, a plurality of analyses may be inferred based at least in part on the content contained in the transcript 530. One or more biomarkers for the patient may be determined 540 based at least in part on the plurality of analyses. A predicted response to the service to be provided for display may be generated 550 based at least in part on the one or more biomarkers and the analyses.
[0038]
[0045] For any process herein, unless otherwise stated, it should be understood that there may be additional, fewer, or alternative steps that are performed in similar or alternative orders or in parallel within the scope of various embodiments.
[0039]
[0046] As described, different approaches may be implemented in various environments according to the described embodiments. For example, FIG. 6 illustrates an example of an environment 600 for implementing aspects according to various embodiments. As will be appreciated, although a web-based environment is used for purposes of explanation, different environments may be used to implement various embodiments, as appropriate. The system includes electronic client devices 602, 608, which may include any suitable device operable to send and receive requests, messages or information over a suitable network 604 and to communicate information to a user of the device. Examples of such client devices include personal computers, cell phones, handheld messaging devices, laptop computers, set-top boxes, personal digital assistants, e-book readers, and the like. The network may include any suitable network, including an intranet, the Internet, a cellular network, a local area network, or any other such network, or combinations thereof. The components used for such a system may depend at least in part on the type of network and / or environment selected. Protocols and components for communicating over such networks are well known and will not be described in detail herein. Communication over the network may be enabled via wired or wireless connections and combinations thereof. In this example, the network includes the Internet as the environment includes one or more servers 606 for receiving requests and serving content in response thereto, although, as will be apparent to one skilled in the art, in other networks, alternative devices serving similar purposes may be used.
[0040]
[0047] The exemplary environment includes at least one application server 610 and a data store 612. It should be understood that there may be several application servers, layers or other elements, processes or components, which may be coupled or otherwise configured, that may interact to perform a task, such as retrieving data from an appropriate data store. The term "data store" as used herein refers to any device or combination of devices capable of storing, accessing and retrieving data, which may include any combination of data servers, databases, data storage devices and data storage media, and any number of such devices, in any standard, distributed or clustered environment. The application server 610 may include any suitable hardware and software for integrating with the data store 612 as needed to execute aspects of one or more applications for client devices, and for handling much of the data access and business logic for the applications. The application server may cooperate with the data store to service access control and generate content, such as text, graphics, audio and / or video to be transferred to the user, which may be served to the user by one or more servers 606, including a web server, in this example, in the form of HTML, XML or another suitable structured language. Handling of all requests and responses, as well as delivery of content between the client devices 602, 608 and the application server 610 may be handled by the web server of the server 606. It should be understood that the web server and application server are not required and are merely exemplary components, as the structured code described herein may be executed on any suitable device or host machine, as described elsewhere herein.
[0041]
[0048] The data store 612 may include several separate data tables, databases, or other data storage mechanisms and media for storing data related to a particular aspect. For example, the illustrated data store includes mechanisms for storing production data 614 and user information 618, which may be used to serve content for the manufacturing side. The data store is also shown to include mechanisms for storing log or application session data 616. It should be understood that there may be many other aspects that may need to be stored in the data store, such as page image information and access rights information, which may be stored in any of the mechanisms listed above or in additional mechanisms in the data store 612, as appropriate. The data store 612, through logic associated therewith, is operable to receive instructions from the application server 610 and retrieve, update, or otherwise process data in response thereto. In one example, a user may submit a request to transcribe, tag, and / or label a media file. In this case, the data store may access the user information to verify the user's identity and provide a transcript including the tags and / or labels along with analytics associated with the media file. The information may then be returned to the user, such as in a results listing on a web page that the user can view via a browser on the user device 602, 608. Information about a particular feature of interest may be viewed in a dedicated page or window of the browser.
[0042]
[0049] Each server will typically include an operating system that provides executable program instructions for the general administration and operation of the server, and will typically include a computer-readable medium that stores instructions that, when executed by the server's processor, enable the server to perform its intended functions. Suitable implementations for server operating systems and general functions are known or commercially available, and will be readily implemented by those of skill in the art, especially in light of the disclosure herein.
[0043]
[0050] The environment in one embodiment is a distributed computing environment utilizing several computer systems and components interconnected via communication links using one or more computer networks or direct connections. However, those skilled in the art will appreciate that such a system may operate equally well in a system having fewer or a greater number of components than are shown in Figure 6. Thus, the illustration of system 600 in Figure 6 should be taken as illustrative in nature and not limiting on the scope of the present disclosure.
[0044]
[0051] 7 illustrates an example block diagram of an electronic device that may be utilized to implement one or more aspects of various embodiments. An instance of an electronic device 700 may include one or more servers and one or more client devices. In general, the electronic device may include a processor / CPU 702, a memory 704, a power supply 706, and input / output (I / O) components / devices 710 that may be operable to provide, for example, a graphical or textual user interface, such as a microphone, speaker, display, touch screen, keyboard, mouse, keypad, microscope, GPS components, camera, heart rate sensor, light sensor, accelerometer, target biometric sensor, etc.
[0045]
[0052] A user may provide input via a touchscreen of the electronic device 700. The touchscreen may determine whether a user is providing input, for example, by determining whether the user is touching the touchscreen with a part of the user's body, such as the user's finger. The electronic device 700 may also include a communication bus 712 that connects to the above-mentioned elements of the electronic device 700. The network interface 708 may include a receiver and transmitter (or transceiver) and one or more antennas for wireless communication.
[0046]
[0053] The processor 702 may include any type of processing device, such as one or more of a central processing unit (CPU) and a graphics processing unit (GPU). Also, for example, the processor may utilize central processing logic, or other logic, for performing one or more functions or actions or for initiating one or more functions or actions from one or more other components, and may include hardware, firmware, software, or a combination thereof. Also, based on a desired application or need, the central processing logic, or other logic may include, for example, a software-controlled microprocessor, discrete logic, such as an application specific integrated circuit (ASIC), a programmable / programmed logic device, a memory device containing instructions, or the like, or combinational logic embedded in hardware. Additionally, logic may also be embodied entirely as software.
[0047]
[0054] Memory 704, which may include random access memory (RAM) 714 and read only memory (ROM) 716, may be enabled by any type of memory device, for example, one or more of a primary (directly accessible by the CPU) or secondary (indirectly accessible by the CPU) storage device (e.g., flash memory, magnetic disk, optical disk, etc.). The RAM may include an operating system 718, data storage 720, which may include one or more databases, and programs and / or applications 722, which may include, for example, software aspects of programs 724. The ROM 716 may also include a basic input / output system (BIOS) 726 of the electronic device 700.
[0048]
[0055] The software aspects of the programs 722 are intended to broadly include or represent all programming, applications, algorithms, models, software, and other tools necessary to implement or facilitate the methods and systems according to embodiments of the present invention, the elements of which may reside on a single computer or be distributed among multiple computers, servers, devices, or entities.
[0049]
[0056] The power supply 706 may include one or more power components and may help facilitate the supply and management of power to the electronic device 700.
[0050]
[0057] Input / output components including input / output (I / O) interface 710 may include any interface for facilitating communication between, for example, components of electronic device 700, components of external devices, and end users. For example, such components may include a network card, which may be an integration of a receiver, a transmitter, a transceiver, and one or more input / output interfaces. The network card may facilitate wired or wireless communication, for example, with other devices of a network. In the case of wireless communication, an antenna may facilitate such communication. Also, some of the input / output interfaces 710 and the bus 712 may facilitate communication between components of electronic device 700 and, in one example, may facilitate processing performed by processor 702.
[0051]
[0058] If the electronic device 700 is a server, the server may include a computing device that may be capable of sending or receiving signals, e.g., a wired or wireless network, or may be capable of processing or storing signals, e.g., in memory, as physical memory states. The server may be an application server that includes configurations for providing one or more applications to another device over a network. The application server may also host a website that may provide, for example, a user interface for administration of the exemplary embodiment.
[0052]
[0059] FIG. 8 illustrates an example environment 800 in which aspects of various embodiments may be implemented. In this example, a user may utilize one or more client devices 802 to submit requests to a multitenant resource provider environment 806 over at least one network 804. The client devices may include any suitable electronic device operable to send and receive requests, messages, or other such information over an appropriate network and to communicate information to a user of the device. Examples of such client devices include personal computers, tablet computers, smartphones, notebook computers, and the like. The at least one network 804 may include any suitable network, including an intranet, the Internet, a cellular network, a local area network (LAN), or any other such network, or combination, and communication over the network may be enabled via wired and / or wireless connections. The resource provider environment 806 may include any suitable components for receiving requests and returning information or performing actions in response to those requests. As an example, the provider environment may include web servers and / or application servers for receiving and processing requests and then returning data, web pages, video, audio, or other such content or information in response to the request.
[0053]
[0060] In various embodiments, the provider environment may include various types of resources that may be utilized by multiple users for various different purposes. As used herein, computing resources and other electronic resources utilized in a network environment may be referred to as "network resources." These may include, for example, servers, databases, load balancers, routers, and the like that may perform tasks, such as receiving, transmitting, and / or processing data and / or executable instructions. In at least some embodiments, all or a portion of a given resource or set of resources may be allocated to a particular user or for a particular task, at least for a determined period of time. The sharing of these multi-tenant resources from the provider environment is often referred to as resource sharing, web services, or "cloud computing," among other terms, as well as depending on the particular environment and / or implementation. In this example, the provider environment includes multiple resources 814 of one or more types. These types may include, for example, application servers operable to process instructions provided by users, or database servers operable to process data stored in one or more data stores 816 in response to user requests. As is known for such purposes, a user may also reserve at least a portion of the data storage in a given data store. Methods for enabling users to reserve various resources and resource instances are well known in the art, and thus a detailed description of the entire process, and an explanation of all possible components, will not be described in detail herein.
[0054]
[0061] In at least some embodiments, a user desiring to utilize a portion of the resource 814 can submit a received request to an interface layer 808 of the provider environment 806. The interface layer can include an application programming interface (API) or other exposed interface that allows a user to submit a request to the provider environment. The interface layer 808 in this example can also include other components, such as at least one web server, a routing component, a load balancer, etc. When a request to provision a resource is received in the interface layer 808, information about the request can be directed to a service manager 810, or other such system, service, or component, configured to manage user accounts and information, resource provisioning and usage, and other such aspects. The service manager 810 that receives the request can perform tasks such as authenticating the identity of the user who submitted the request and determining whether the user has an existing account with the resource provider, if account data can be stored in at least one data store 812 in the provider environment. The user can provide any of a variety of types of credentials to the provider to authenticate the user's identity. These credentials may include, for example, a username and password pair, biometric data, a digital signature, a QR-based certificate, or other such information.
[0055]
[0062] The provider can verify this information against information stored about the user. If the user has an account with the proper permissions, status, etc., the resource manager can determine whether there are sufficient resources available to suit the user's request, and if there are sufficient resources, can provision those resources for use by the user for the amount specified by the request, or possibly grant access to a corresponding portion of those resources. This amount can include, for example, the ability to process a single request or perform a single task, a specified time period, or a cycling / renewable period, among other values. If the user does not have a valid account with the provider, or the user account does not allow access to the type of resource specified in the request, or another such reason prevents the user from gaining access to such resources, a communication can be sent to the user to enable the user to create or modify an account or change the resource specified in the request, among other options. In at least some exemplary embodiments, a user can be authenticated to access an entire fleet of services offered within the service provider environment. In other exemplary embodiments, the user's access can be limited to specific services within the service provider environment using one or more access policies tied to the user's credential(s).
[0056]
[0063] Once a user has been authenticated, an account verified, and resources allocated, the user may utilize the allocated resource(s) for a specified capacity, amount of data transfer, time period, or other such value. In at least some embodiments, the user may provide a session token or other such proof with subsequent requests to allow those requests to be processed on that user session. The user may receive a resource identifier, specific address, or other such information that may enable the client device 802 to communicate with the allocated resource without having to communicate with the service manager 810, at least until such time that a relevant aspect of the user account changes, the user is no longer granted access to the resource, or another such aspect changes.
[0057]
[0064] Also, the service manager 810 in this example (or another such system or service) can function as a virtual layer of hardware and software components that handles control functions in addition to management actions, such as may include provisioning, scaling, replication, etc. The resource manager can utilize dedicated APIs in the interface layer 808, where each API can be provided to receive a request for at least one specific action to be performed with respect to the data environment, such as provisioning, scaling, cloning, or quiescing an instance. Upon receiving a request to one of the APIs, the web services portion of the interface layer can parse or otherwise analyze the request to act on the call or determine the steps or actions required to process the call. For example, a web services call can be received that includes a request to create a data repository.
[0058]
[0065] The interface layer 808 in at least one embodiment includes a scalable set of user-facing servers that can offer various APIs and return appropriate responses based on the API specifications. The interface layer, in one embodiment, can also include at least one API services layer, consisting of stateless, replicated servers that handle external-facing user APIs. The interface layer can be responsible for web services front-end features such as authenticating users based on credentials, authorizing users, throttling user requests to the API server, validating user input, and marshalling or unmarshalling requests and responses. The API layer can also be responsible for reading and writing database configuration data to / from an administration data store in response to API calls. In many embodiments, the web services layer and / or API services layer will be the only externally visible components or the only components that are visible to and accessible by users of the control services. The servers of the web services layer are stateless and can be horizontally scaled as known in the art. The API servers, as well as the persistent data stores, may be spread across multiple data centers across a region, for example, so that they are resilient to single data center failures.
[0059]
[0066] Various embodiments may also be implemented in a wide variety of operating environments that may, in some cases, include one or more user computers or computing devices that may be used to operate any of a number of applications. User or client devices may include any of a number of general-purpose personal computers, such as desktop or laptop computers running standard operating systems, as well as any of a number of cellular, wireless and handheld devices capable of running mobile software and supporting a number of networking and messaging protocols. Such systems may also include a number of workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as dummy terminals, thin clients, gaming systems and other devices capable of communicating over a network.
[0060]
[0067] Most embodiments utilize at least one network, as will be familiar to those skilled in the art, for supporting communications using any of a variety of commercially available protocols, such as TCP / IP, FTP, UPnP, NFS, and CIFS. The network may be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof. In embodiments utilizing a web server, the web server may run any of a variety of server or mid-tier applications, including an HTTP server, an FTP server, a CGI server, a data server, a Java server, and a business application server. The server(s) may also be capable of executing programs or scripts in response requests from user devices, such as by executing one or more web applications, which may be implemented as one or more scripts or programs written in any programming language, such as Java, C, C#, or C++, or any scripting language, such as Perl, Python, or TCL, and combinations thereof. The server(s) may also include database servers, including, but not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, and IBM®.
[0061]
[0068] The environment may include various data stores and other memory and storage media, as described above. These may be present in various locations, such as across a network, on storage media local to (and / or present in) one or more of the computers, or remote from any or all of the computers. In a particular set of embodiments, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, files necessary to perform functions ascribed to a computer, server, or other network device may be stored locally and / or remotely, as appropriate. Where the system includes computerized devices, each such device may include hardware elements that may be electrically coupled via a bus, including, for example, at least one central processing unit (CPU), at least one input device (e.g., a mouse, keyboard, controller, touch-sensitive display element or keypad), and at least one output device (e.g., a display device, printer, or speaker). Such a system may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices, such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash cards, etc. Such devices may also include a computer-readable storage medium reader and a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device), and a working memory as described above. A computer-readable storage medium reader may be connected to or configured to accept computer-readable storage media representing remote, local, fixed and / or removable storage devices, as well as storage media for temporarily and / or more permanently containing, storing, transmitting and retrieving computer-readable information.
[0062]
[0069] The systems and various devices will also include several software applications, modules, services or other elements located in at least one working memory device, including an operating system and application programs such as client applications or web browsers. It should be appreciated that alternative embodiments may have numerous variations from the embodiments described above. For example, customized hardware may also be used and / or particular elements may be implemented in hardware, software (including portable software such as applets), or both. Additionally, connections to other computing devices, such as network input / output devices, may be employed. Storage media and other non-transitory computer readable media for containing code or portions of code may include any suitable media known or used in the art, such as volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data, including, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media that may be used to store the desired information and that may be accessed by the system devices. Based on the disclosure and teachings provided herein, those skilled in the art will appreciate other ways and / or methods for implementing the various embodiments.
[0063]
[0070] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense, however it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.
Claims
1. obtaining audio files capturing one or more interactions between one or more providers and recipients of a service; generating a transcript of at least a portion of the one or more interactions captured in the audio file; using machine learning to infer a plurality of analytics based at least in part on content contained in the transcript; determining one or more biomarkers for the recipient based at least in part on the plurality of analyses; and generating a predicted response to the service to be provided for display based at least in part on the one or more biomarkers and the plurality of analyses; A computer-implemented method comprising:
2. 10. The computer-implemented method of claim 1, wherein the recording is transcribed using natural language processing (NLP).
3. detecting that the transcript contains errors; and providing an indication of said error; and suggesting one or more corrections to said errors; and The computer-implemented method of claim 1 , further comprising:
4. analyzing one or more utterances present in the transcript; and generating one or more tags associated with the one or more utterances; inferring the plurality of analyses based at least in part on the generated tags; and The computer-implemented method of claim 1 , further comprising:
5. 2. The computer-implemented method of claim 1, wherein the one or more biomarkers are determined based at least in part on at least one of a detected emotion, a detected pitch, a detected frequency, a determined words per minute, a detected pause, and a pause duration in the audio file.
6. assigning one or more labels to the audio file based at least in part on audio cues detected in the audio file; The computer-implemented method of claim 1 , further comprising:
7. At least one processor; at least one memory for storing instructions; wherein the instructions, when executed by the at least one processor, cause the at least one processor to: Obtaining a media file capturing one or more interactions between one or more providers and recipients of a service; generating a transcript of at least a portion of the one or more interactions captured in the media file; using machine learning to infer a plurality of analytics based at least in part on content contained in the transcript; determining one or more biomarkers for the recipient based at least in part on the plurality of analyses; and generating a predicted response to the service to be provided for display based at least in part on the one or more biomarkers and the plurality of analyses; A system that allows the user to:
8. The instructions, when executed by the at least one processor, cause the at least one processor to: detecting that the transcript contains errors; and providing an indication of said error; and suggesting one or more corrections to said errors; and The system of claim 7 , further comprising:
9. The system of claim 7 , wherein the recording is transcribed using natural language processing (NLP).
10. The instructions, when executed by the at least one processor, cause the at least one processor to: analyzing one or more utterances present in the transcript; and generating one or more tags associated with the one or more utterances; inferring the plurality of analyses based at least in part on the generated tags; and The system of claim 7 , further comprising:
11. 8. The system of claim 7, wherein the one or more biomarkers are determined based at least in part on at least one of a detected emotion, a detected pitch, a detected frequency, a determined words per minute, a detected pause, and a duration of a pause in the media file.
12. The instructions, when executed by the at least one processor, cause the at least one processor to: assigning one or more labels to the media file based at least in part on audio or visual cues detected in the media file; The system of claim 7 , further comprising:
13. The system of claim 7 , wherein the media file is pre-processed prior to transcription to filter out unwanted noise from the media file.
14. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to: Obtaining a media file capturing one or more interactions between one or more providers and recipients of a service; generating a transcript of at least a portion of the one or more interactions captured in the media file; using machine learning to infer a plurality of analytics based at least in part on content contained in the transcript; determining one or more biomarkers for the recipient based at least in part on the plurality of analyses; and generating a predicted response to the service to be provided for display based at least in part on the one or more biomarkers and the plurality of analyses; A non-transitory computer-readable medium for causing
15. The instructions, when executed by the at least one processor, cause the at least one processor to: detecting that the transcript contains errors; and providing an indication of said error; and suggesting one or more corrections to said errors; and 15. The non-transitory computer-readable medium of claim 14, further comprising:
16. 15. The non-transitory computer-readable medium of claim 14, wherein the recording is transcribed using natural language processing (NLP).
17. The instructions, when executed by the at least one processor, cause the at least one processor to: analyzing one or more utterances present in the transcript; and generating one or more tags associated with the one or more utterances; inferring the plurality of analyses based at least in part on the generated tags; and 15. The non-transitory computer-readable medium of claim 14, further comprising:
18. 15. The non-transitory computer readable medium of claim 14, wherein the one or more biomarkers are determined based at least in part on at least one of a detected emotion, a detected pitch, a detected frequency, a determined words per minute, a detected pause, and a duration of a pause in the media file.
19. The instructions, when executed by the at least one processor, cause the at least one processor to: assigning one or more labels to the media file based at least in part on audio or visual cues detected in the media file; 15. The non-transitory computer-readable medium of claim 14, further comprising:
20. 15. The non-transitory computer-readable medium of claim 14, wherein the media file is pre-processed prior to transcription to filter out unwanted noise from the media file.
Citation Information
Patent Citations
Sound acquisition device, sound acquisition method, and sound acquisition program
JP2018191234A
Machine learning classifications of aphasia
US11004462B1
Error Correction In Automatic Speech Recognition Transcripts
US20070208567A1
Optical biomodule for detection of diseases at an early onset
US20170316487A1
Methods of treating neurocognitive disorders, chronic pain and reducing inflammation
WO2020212948A1
