Paired Neural Networks for Diagnosing Health Conditions via Voice

Abstract

To improve health diagnosis by processing human voice.SOLUTION: A method includes: a step 610 of receiving a first audio signal corresponding to a first period; a step 620 of calculating a first feature vector from the first audio signal; a step 630 of processing the first feature vector to calculate a first voice embedded vector; a step 640 of receiving a second audio signal corresponding to a second period; a step 650 of calculating a second feature vector from the second audio signal; a step 660 of processing the second feature vector to calculate a second voice embedded vector; a step 670 of calculating element-wise difference between the first voice embedded vector and the second voice embedded vector; and a step 680 of using a mathematical model to process the element-wise difference, and thus calculating a variation value indicative of change in health state between the first period and the second period.SELECTED DRAWING: Figure 6

Description

[Background technology]

[0001] Improved diagnosis of health conditions has many benefits to society, including improved quality of life, increased life expectancy, and potentially reduced health care costs if early diagnosis and treatment are more effective than late diagnosis and treatment.

[0002]

[0002] Health conditions may be diagnosed in a variety of ways. Some health conditions are diagnosed using a patient's voice. For example, a person's voice may be used in diagnosing mental health conditions (stress, depression, anxiety), concussion, Alzheimer's disease, and congestive heart failure.

[0003]

[0003] In some cases, a person may listen to a person's voice and use the characteristics of that voice in determining a diagnosis. In some cases, a mathematical model (such as a neural network) may process the voice to determine a diagnosis, sometimes providing a more accurate diagnosis than a trained medical professional. Improved techniques for diagnosing health conditions using mathematical models could provide many additional benefits to society. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 10,152,988 Summary of the Invention [Means for solving the problem]

[0005]

[0004] The present invention and the following detailed description of specific embodiments thereof can be understood by reference to the following drawings. [Brief explanation of the drawings]

[0006] [Figure 1]

[0005] Figure 1A is a diagram of an example system for processing audio using a mathematical model to determine a health condition label.

[0006] Figure 1B is a diagram of an example system for processing first audio from a first time period and second audio from a second time period using a mathematical model to determine a change in health condition between the first time period and the second time period.

[0007]

[0007] FIG. 1C is a diagram of an exemplary system for processing first audio from a first time period, a first health status label from the first time period, and second audio from a second time period using a mathematical model to determine a second health status label for the second time period.

[0008] FIG. 1D is a diagram of an example system for processing multiple previous pairs of audio and health condition labels from previous time periods and a current audio sample from the current time using a mathematical model to determine a health condition label for the current time. [Figure 2]

[0009] FIG. 2 is a diagram of an exemplary system for processing a first audio from a first time period and a second audio from a second time period using a mathematical model to determine a change in health status between the first time period and the second time period. [Figure 3]

[0010] FIG. 3 is a diagram of an exemplary system for processing audio from two periods using a mathematical model to determine changes in health status using element-by-element differences. [Figure 4]

[0011] FIG. 4 is a diagram of an example system for processing first audio and a first health condition label from a first time period and second audio from a second time period using a mathematical model to determine a second health condition label for the second time period. [Figure 5]

[0012] FIG. 5 is a diagram of an example system for processing multiple pairs of previous audio and health status labels from previous time periods and current audio from a current time period using a mathematical model to determine a current health status label for a current time period. [Figure 6]

[0013] FIG. 6 is a flow diagram of an exemplary method for processing audio from two periods using a mathematical model to determine changes in health status using element-by-element differences. [Figure 7]

[0014] FIG. 7 is a flow diagram of an example method for processing first audio and a first health condition label from a first time period and second audio from a second time period using a mathematical model to determine a second health condition label for the second time period. [Figure 8]

[0015] FIG. 8 illustrates components of one embodiment of a computing device 800 for implementing any of the techniques described herein. DETAILED DESCRIPTION OF THE INVENTION

[0008]

[0016] Voices sound different from person to person and have a variety of different qualities and aspects. Because voices sound different from person to person, diagnosing health conditions can be difficult. In a simple example, with a first person, their voice may normally sound smooth, but after speaking for a long time, their voice may become raspy and "lost." However, with a second person, their voice may always be raspy, and that is their normal way of speaking.

[0009]

[0017] Samples of a person's voice from multiple time periods may be used to improve the diagnosis of a health condition through processing of the person's voice. Continuing with the example above, a sample of a person's voice from a first time period in which the person's voice is not hoarse would help determine whether the person has lost their voice in a second time period. Techniques for improving the diagnosis of a health condition by processing a person's voice from two or more time periods are described herein.

[0010]

[0018] Any suitable health condition can be diagnosed using the techniques described herein. For example, the health condition can include mental health conditions (e.g., stress, depression, anxiety, and post-traumatic stress disorder), concussion, Parkinson's disease, Alzheimer's disease, and congestive heart failure. In some embodiments, the health condition can include the likelihood of a health-related event occurring, such as the likelihood of a patient being readmitted to the hospital after being discharged from the hospital or the likelihood of being readmitted to the hospital after being treated for heart failure.

[0011]

[0019] As used herein, a period may be bounded by any suitable interval used by a medical professional in treating a patient. In some circumstances, the periods may be separated by months or years, while in some circumstances, multiple periods may fall on the same day.

[0012]

[0020] As used herein, speech includes any sound produced by a person's vocal tract, and these sounds need not include sounds intended as intelligible speech or speech. For example, speech can include sighs, breath sounds, or grunts.

[0013]

[0021] 1A-1D show an example architecture for processing audio using a mathematical model to diagnose a health condition. The mathematical model in FIG. 1A-1D can include any suitable mathematical model, such as a neural network.

[0014]

[0022] 1A is an example system 100 for processing audio using a mathematical model component 110 to determine a health condition label. The health condition label may include any suitable label associated with a medical diagnosis, such as a Boolean value (indicating whether or not a person has the condition), a selection from a set of labels (e.g., "mild," "moderate," or "severe"), an integer value (e.g., on a scale of 1 to 10), or a floating-point value (e.g., a temperature of 98.6 degrees).

[0015]

[0023] 1B is an example system 102 for processing a first audio from a first time period and a second audio from a second time period to determine a change in health state between the first and second time periods using a mathematical model component 112. The change in health state may be any suitable value that may be used to indicate a change in health state, such as a Boolean value (indicating the presence or absence of a change), an integer value, or a floating point value.

[0016]

[0024] 1C illustrates an example system 104 for processing a first audio signal from a first time period, a first health condition label from the first time period, and a second audio signal from a second time period to determine a second health condition label for the second time period using a mathematical model component 114. The first health condition label may be determined using any suitable technique, such as by a human or a mathematical model. The first and second health condition labels may include any of the labels described herein.

[0017]

[0025] FIG. 1D illustrates an example system 106 for processing multiple previous pairs of audio and health condition labels from previous time periods and a current audio sample from the current time using a mathematical model component 116 to determine a health condition label for the current time. The example of FIG. 1D illustrates N previous pairs of audio and health condition labels, where N may be any number greater than 1. The previous health condition labels may have been determined using any suitable technique, such as by a human or a mathematical model. The previous and current health condition labels may include any of the labels described herein. The current time period may include any suitable period for which it is desired to calculate a health condition label, and the processing of the system 106 need not be performed at the time the current audio is received.

[0018]

[0026] Further details of the embodiment of Figures 1A-1D will now be described.

[0027] FIG. 2 is an example system 200 for processing a first audio from a first time period and a second audio from a second time period using a mathematical model to determine a change in health status between the first and second time periods.

[0019]

[0028] 2, a first audio is processed by feature extraction component 210 to compute a first feature vector (or possibly a first sequence of feature vectors), and a second audio is processed by feature extraction component 212 to compute a second feature vector (or possibly a second sequence of feature vectors). Feature extraction component 210 and feature extraction component 212 may compute the same type of features or may compute different types of features. The feature vectors may include any suitable type of features, including, but not limited to, any of the features described in U.S. Pat. No. 10,152,988, which is incorporated herein by reference.

[0020]

[0029] The features may include acoustic features, which are any features computed from speech data that do not involve or rely on performing speech recognition on the speech data (e.g., the acoustic features do not use information about spoken words in the speech data). For example, the acoustic features may include Mel-frequency cepstral coefficients, perceptual linear prediction features, Wav2Vec features, prosodic features (such as pitch, energy, or probability of utterance), timbre features (such as jitter, jitter of jitter, shimmer, or harmonic to noise ratio), or entropy.

[0021]

[0030] The features can include linguistic features, which are computed using recognized text obtained by automatic speech recognition. For example, the linguistic features can include the words spoken in the audio, the speaking rate (e.g., the number of vowels or syllables per second), the number of pause fillers (e.g., "ums" and "ahs"), the difficulty of the word (e.g., an uncommon word), or the part of speech of the word following the pause filler. In some implementations, the linguistic features can include a determination of whether a person correctly answered a question. For example, a person may be asked what year it is or who the President of the United States is. The person's audio can also be processed to determine what the person said in response to the question to determine whether the person correctly answered the question.

[0022]

[0031] In some implementations, feature extraction component 210 and feature extraction component 212 can perform speech recognition to obtain text corresponding to the speech and then output the tokenized text as features such as word-piece encoding, byte-pair encoding, or sentence-piece encoding. The tokenized text may be combined with any of the other features described herein.

[0023]

[0032] A pair of mathematical models can then process the feature vectors. An audio embedding component 220 can process the first feature vector calculated by the feature extraction component 210 and calculate a first audio embedding vector. Similarly, an audio embedding component 222 can process the second feature vector calculated by the feature extraction component 212 and calculate a second audio embedding vector.

[0024]

[0033] As used herein, a phonetic embedding vector is a representation in a vector space of a corresponding speech, and the location of the phonetic embedding vector in the vector space corresponds to information, properties, or other aspects of the speech. For example, in some implementations, the location of the phonetic embedding vector may correspond to the meaning of words in the speech (e.g., "hello" and "good morning"), such that phonetic embedding vectors for speech with similar meanings are close to each other in the vector space.

[0025]

[0034] The audio embedding component 220 and the audio embedding component 222 may have the same architecture and parameters, the same architecture and different parameters, or different architectures and different parameters.

[0026]

[0035] The audio embedding component 220 and the audio embedding component 222 may be implemented using any suitable technique, such as a transformer neural network (such as a Bidirectional Encoder Representation from Transformer, or BERT, neural network), a fully connected neural network (e.g., a multi-layer perceptron), a recurrent neural network, a convolutional neural network, or any combination of the aforementioned neural networks. In some implementations, the audio embedding component 220 and the audio embedding component 222 may include one or more feedforward neural network layers and one or more self-attention neural network layers.

[0027]

[0036] The mathematical model component 240 processes the first audio embedding vector calculated by the audio embedding component 220 and the second audio embedding vector calculated by the audio embedding component 222 to calculate a change value indicative of a change in health state, such as any of the health state change values ​​described herein. In some implementations, the mathematical model component 240 can concatenate the first audio embedding vector with the second audio embedding vector and process the concatenated vector with a mathematical model. The mathematical model component 240 may be implemented using any suitable mathematical model, such as a linear model (e.g., matrix-vector multiplication or dot product) or a neural network (e.g., a fully connected neural network, a feedforward fully connected neural network, a multilayer perceptron, a transformer neural network, a recurrent neural network, a convolutional neural network, or any of the other neural networks described herein).

[0028]

[0037] In some implementations, the mathematical model component 240 may be implemented using a Transformer Neural Network, such as a BERT neural network. For example, the first and second speech coding vectors and the first label may be concatenated to form an input vector for the Transformer Neural Network. The change value may be an element of an output vector of the Transformer Neural Network, or the output of the Transformer Neural Network may be followed by one or more layers (e.g., linear layers) for computing the change value from the output of the Transformer Neural Network.

[0029]

[0038] In some implementations, the mathematical model component 240 may be implemented using a recurrent neural network. For example, the first and second audio encoding vectors may be processed sequentially (in any suitable order, optionally with a separator token) by the recurrent neural network. The change values ​​may be elements of the output vector of the recurrent neural network, or the output of the recurrent neural network may be followed by one or more layers (e.g., linear layers) for calculating the change values ​​from the output of the recurrent neural network.

[0030]

[0039] 3 is an example system 300 for processing audio from two time periods using a mathematical model to determine a health state change value using element-by-element differences. In FIG. 3, feature extraction component 210, feature extraction component 212, audio embedding component 220, and audio embedding component 222 may be implemented as described above.

[0031]

[0040] The difference component 330 receives the first audio embedding from the audio embedding component 220 and the second audio embedding from the audio embedding component 222 and calculates a difference vector that is the element-wise difference of the two audio embedding vectors. For example, if the first element of the first audio embedding vector is "a" and the first element of the second audio embedding vector is "b", then the first element of the difference vector is "ab".

[0032]

[0041] The mathematical model component 340 processes the difference vector calculated by the difference component 330 and calculates a change value indicative of a change in health state, such as any of the health state change values ​​described herein. The mathematical model component 340 may be implemented using any suitable technique, such as any of the techniques described above for the mathematical model component 240.

[0033]

[0042] In some implementations, the mathematical model component 340 can calculate a health state change value that is antisymmetric given the input, meaning that if the audio input is swapped, the output health state change is of the same magnitude but opposite sign (e.g., the health state change switches from +3 to -3). For example, if the mathematical model component 340 calculates the health state change by calculating the dot product of a difference vector and a vector of parameters, the health state change calculation is antisymmetric.

[0034]

[0043] 4 is an example system 400 for processing a first audio and a first health condition label from a first time period and a second audio from a second time period using a mathematical model to determine a second health condition label for the second time period. The first and second health condition labels may include any suitable labels, such as any of the labels described herein.

[0035]

[0044] In FIG. 4, feature extraction component 210, feature extraction component 212, audio embedding component 220, and audio embedding component 222 may be implemented as described above.

[0036]

[0045] The mathematical model component 440 processes the first audio embedding vector calculated by the audio embedding component 220, the second audio embedding vector calculated by the audio embedding component 222, and a first health condition label corresponding to the first time period, and calculates a second health condition label corresponding to the second time period. The first health condition label may be combined with the first and second audio embedding vectors using any suitable technique, such as concatenation (e.g., with a transformer neural network) or sequential processing (e.g., with a recurrent neural network). The mathematical model component 440 may be implemented using any suitable technique, such as any of the techniques described above for the mathematical model component 240.

[0037]

[0046] In some embodiments, the mathematical model component 440 may implement regression techniques such as linear regression, nonlinear regression, multiple regression, multivariate regression, semiparametric regression, or any combination of nonparametric regression (e.g., using nearest neighbor, regression trees, kernel regression, local regression, multivariate adaptive regression splines, neural networks, support vector regression, or smoothing splines).

[0038]

[0047] In some implementations, the mathematical model component 440 can calculate a difference vector as the element-wise difference between the first audio embedding and the second audio embedding, and then use the difference vector to calculate a value indicative of a change in health state between the first time period and the second time period. The mathematical model component 440 can then calculate a second health state label for the second time period using the first label and the value indicative of the change in health state. For example, the second health state label may be calculated by adding the first health state label and the change.

[0039]

[0048] 5 is an example system 500 for processing multiple pairs of previous audio and health condition labels from previous time periods and current audio from a current time period using a mathematical model to determine a current health condition label for the current time period. The health condition label may include any suitable label, such as any of the labels described herein.

[0040]

[0049] 5, N speech inputs and N labels are shown, where N corresponds to any number greater than 1. The N speech inputs and labels may be obtained from the patient's medical records and may correspond to previous visits to the patient at different time periods.

[0041]

[0050] The N+1 speech input may be the speech input for which it is desired to calculate a health condition label, and the N+1 speech input may correspond to the patient's most recent visit or the current time. System 500 processes the N speech input / label pairs and the N+1 speech input to calculate the N+1 health condition label.

[0042]

[0051] 5, feature extraction component 210, feature extraction component 212, feature extraction component 214, and feature extraction component 216 may be implemented using any of the feature extraction techniques described herein. Each instance of the feature extraction component may compute the same type of feature or different types of features.

[0043]

[0052] Audio embedding component 220, audio embedding component 222, audio embedding component 224, and audio embedding component 226 can compute audio embedding vectors from the feature vectors using any of the techniques described herein. Each instance of the audio embedding component can use the same or different neural network architectures and parameters.

[0044]

[0053] Mathematical model component 540 processes the N labels and the N+1 audio embedding vectors to calculate an N+1 health condition label for the N+1 audio input. Mathematical model component 540 may be implemented using any suitable technique. For example, mathematical model component 540 may be implemented using any of the techniques described above for mathematical model component 240 or mathematical model component 440, adapted to additional input values ​​using techniques known to those skilled in the art.

[0045]

[0054] FIG. 6 is a flow diagram of an exemplary method for processing audio from two periods using a mathematical model to determine changes in health status using element-by-element differences.

[0055] A first audio signal corresponding to a first time period is received, the first audio signal including the voice of a first person, at step 610. The first audio signal may be received using any suitable technique, such as via an API call or retrieved from storage.

[0046]

[0056] In step 620, a first feature vector is calculated from the first audio signal. The first feature vector may include any suitable features, such as any of the features described herein. In some implementations, the features may include word-piece encodings corresponding to text of speech in the audio signal determined using automatic speech recognition. In some implementations, the features may include acoustic features.

[0047]

[0057] In step 630, a first audio embedding vector is computed by processing the first feature vector with a neural network. The neural network may be any suitable neural network, such as any of the neural networks described herein. In some implementations, the neural network may include a transformer neural network. In some implementations, the neural network may include one or more feedforward neural network layers and one or more self-attention neural network layers.

[0048]

[0058] A second audio signal corresponding to a second time period is received, the second audio signal including the voice of the first person, at step 640. The first audio signal may be received as described above for step 610.

[0049]

[0059] A second feature vector is calculated from the second audio signal in step 650. The second feature vector may be calculated as described above for step 620.

[0050]

[0060] In step 660, a second audio embedding vector is computed by processing the second feature vector with a neural network. The second audio embedding may be computed as described above for step 630.

[0051]

[0061] An element-wise difference vector is calculated between the first and second audio embedding vectors in step 670. The element-wise difference vector may be calculated using any of the techniques described herein.

[0052]

[0062] In step 680, a change value indicative of a change in health state is calculated by processing the element-wise difference vector with a mathematical model. The change value may be indicative of a change in health state between the first time period and the second time period. The mathematical model may be any suitable mathematical model, such as any of the mathematical models described herein. In some implementations, the change value may be an antisymmetric change value. In some implementations, the mathematical model may calculate a dot product or a matrix-vector multiplication using the element-wise difference vector.

[0053]

[0063] In some implementations, some of steps 610-630 may be performed in advance, and the first feature vector or first audio embedding may be stored for later use. For example, some of steps 610-630 may be performed shortly after the first consultation, when the first audio sample is obtained. The first feature vector or first audio embedding may be stored so that it can be used the next time the person visits the clinic. Steps 640-680 may be performed after a subsequent consultation, which may be days, weeks, months, or years after the initial consultation.

[0054]

[0064] In some implementations, the steps of Figure 6 may be performed for multiple previous visits to calculate multiple change values. For example, the third audio signal may be obtained from another previous visit. The second change value may be calculated using the third audio signal, the third feature vector, the third speech embedding vector, and a second element-wise difference vector calculated using the second speech embedding vector and the third speech embedding vector.

[0055]

[0065] In some implementations, a first health state label may be obtained corresponding to a first time period. The first health state label may be determined using any suitable technique, such as any of the techniques described herein. A second health state label for a second time period may then be calculated using the first health state label and the change value. The second health state label may be calculated using any suitable technique, such as by adding the first health state label and the change value.

[0056]

[0066] FIG. 7 is a flow diagram of an example method for processing first audio and a first health condition label from a first time period and second audio from a second time period using a mathematical model to determine a second health condition label for the second time period.

[0057]

[0067] In FIG. 7, steps 710-760 may be performed as described above for steps 610-660 in FIG.

[0068] In step 770, a first health condition label corresponding to the first time period is obtained. The first health condition label may be any of the health condition labels described herein and may be obtained using any suitable technique. For example, the first health condition label may be stored together with the first audio signal (or the first feature vector or the first speech embedding vector).

[0058]

[0069] In step 780, a second health condition label corresponding to the second time period is calculated by processing the first health condition label and the first and second audio embedding vectors using a mathematical model. The mathematical model may be any suitable mathematical model, such as a neural network. In some implementations, the mathematical model may use linear or nonlinear regression techniques to calculate the second health condition label.

[0059]

[0070] For any of the techniques described herein, parameters of the mathematical model (including neural networks) may be learned or trained using a training process. Any suitable training process may be used, such as supervised or unsupervised training. The training process may include a training corpus of audio files, which may also include training labels indicating health state labels corresponding to audio files or change values ​​corresponding to pairs of audio files. The parameters of the mathematical model may be learned through an iterative training process. For example, the training process may include a forward pass that processes training data to calculate predicted values ​​(e.g., health state labels or change values), error values ​​may be calculated using the predicted values ​​and training labels, and a backward pass may be performed that uses the error values ​​to update the mathematical model parameters (e.g., using stochastic gradient descent). The training process may continue until a desired convergence criterion is achieved.

[0060]

[0071] Figure 8 illustrates components of one embodiment of a computing device 800 for implementing any of the techniques described herein. While the components are illustrated in Figure 8 as being on a single computing device, the components may be distributed among multiple computing devices, such as, for example, a system of computing devices including end-user computing devices (e.g., smartphones or tablets) and / or server computers (e.g., cloud computing).

[0061]

[0072] The computing device 800 may include any components typical of a computing device, such as volatile or non-volatile memory 810, one or more processors 811, and one or more network interfaces 812. The computing device 800 may also include any input and output components, such as a display, a keyboard, and a touchscreen. The computing device 800 may also include various components or modules that provide specific functionality, and these components or modules may be implemented in software, hardware, or a combination thereof. The computing device 800 may include one or more non-transitory computer-readable media that include computer-executable instructions that, when executed, cause the processor to perform operations corresponding to any of the techniques described herein. Some example components are described below for one exemplary embodiment; other embodiments may include additional components or exclude some of the components described below.

[0062]

[0073] The computing device 800 may have a feature extraction component 820 that can compute a feature vector from the audio signal using any of the techniques described herein. The computing device 800 may have an audio embedding component 821 that can compute an audio embedding vector from the feature vector using any of the techniques described herein. The computing device 800 may have a mathematical model component 822 that can compute a health state label or a health state change value using any of the techniques described herein. The computing device 800 may have an element-wise differencing component 823 that can compute an element-wise difference between two vectors using any of the techniques described herein.

[0063]

[0074] Computing device 800 may include or have access to a variety of data stores. The data stores may use any known storage technology, such as files, relational databases, non-relational databases, or any non-transitory computer-readable medium. Computing device 800 may have a training data store 830 that stores training data that may be used to train any of the mathematical models described herein.

[0064]

[0075] The methods and systems described herein may be deployed, in part or in whole, through a machine executing computer software, program code, and / or instructions on a processor. As used herein, "processor" is meant to include at least one processor, and the plural and singular forms should be understood as interchangeable unless the context clearly indicates otherwise. Any aspect of the present disclosure may be implemented as a computer-implemented method on a machine, as a system or apparatus as part of or associated with a machine, or as a computer program product embodied in a computer-readable medium running on one or more of the machines. The processor may be part of a server, client, network infrastructure, mobile computing platform, stationary computing platform, or other computing platform. The processor may be any type of computational or processing device capable of executing program instructions, code, binary instructions, etc. The processor may be or include any variation, such as a signal processor, digital processor, embedded processor, microprocessor, or coprocessor (such as a mathematical coprocessor, graphics coprocessor, communication coprocessor, etc.) that can directly or indirectly facilitate the execution of stored program code or program instructions. Additionally, the processor may enable the execution of multiple programs, threads, and code. Threads may be executed simultaneously to improve processor performance and facilitate simultaneous operation of applications. In some embodiments, the methods, program codes, program instructions, etc. described herein may be implemented with one or more threads. Threads may spawn other threads, which may have assigned priorities associated with them, and the processor may execute these threads based on priorities based on instructions provided in the program code, or any other order.The processor may include memory that stores the methods, codes, instructions, and programs described herein and elsewhere. The processor may access, via an interface, a storage medium that may store the methods, codes, and instructions described herein and elsewhere. Storage media associated with the processor for storing methods, programs, codes, program instructions, or other types of instructions executable by a computing or processing device may include, but are not limited to, one or more of a CD-ROM, a DVD, memory, a hard disk, a flash drive, RAM, ROM, cache, etc.

[0065]

[0076] A processor may include one or more cores, which can increase the speed and performance of a multiprocessor. In embodiments, the processor may be a dual-core processor, a quad-core processor, or other chip-level multiprocessor that combines two or more independent cores (called dies).

[0066]

[0077] The methods and systems described herein may be deployed, in part or in whole, through machines executing computer software on servers, clients, firewalls, gateways, hubs, routers, or other such computer and / or networking hardware. Software programs may be associated with servers, which may include file servers, print servers, domain servers, Internet servers, intranet servers, and other variations such as secondary servers, host servers, and distributed servers. A server may include one or more of the following: memory, processors, computer-readable media, storage media, ports (physical and virtual), communication devices, and interfaces capable of accessing other servers, clients, machines, and devices via wired or wireless media. The methods, programs, or codes described herein and elsewhere may be executed by the server. Additionally, other devices necessary for the execution of the methods described herein may be considered part of the infrastructure associated with the server.

[0067]

[0078] A server may provide an interface to other devices, including, but not limited to, clients, other servers, printers, database servers, print servers, file servers, communication servers, distributed servers, etc. Furthermore, this coupling and / or connection may facilitate remote execution of programs over a network. Networking some or all of these devices may facilitate parallel processing of a program or method at one or more locations without departing from the scope of this disclosure. Additionally, any of the devices attached to the server via an interface may include at least one storage medium capable of storing methods, programs, code, and / or instructions. A central repository may provide program instructions to be executed on different devices. In this embodiment, the remote repository may serve as a storage medium for program code, instructions, and programs.

[0068]

[0079] A software program may be associated with a client, which may include a file client, a print client, a domain client, an Internet client, an intranet client, and other variations such as a secondary client, a host client, a distributed client, etc. A client may include one or more of a memory, a processor, a computer-readable medium, a storage medium, a port (physical and virtual), a communication device, and an interface capable of accessing other clients, servers, machines, and devices via a wired or wireless medium, etc. The methods, programs, or codes described herein and elsewhere may be executed by a client. Additionally, other devices necessary for execution of the methods described in this application may be considered part of the infrastructure associated with the client.

[0069]

[0080] A client may provide an interface to other devices, including, but not limited to, a server, other clients, printers, database servers, print servers, file servers, communication servers, distributed servers, etc. Furthermore, this coupling and / or connection may facilitate remote execution of a program over a network. Networking some or all of these devices may facilitate parallel processing of a program or method at one or more locations without departing from the scope of this disclosure. Additionally, any of the devices attached to a client via an interface may include at least one storage medium capable of storing methods, programs, applications, code, and / or instructions. A central repository may provide program instructions to be executed on different devices. In this embodiment, a remote repository may serve as a storage medium for program code, instructions, and programs.

[0070]

[0081] The methods and systems described herein may be deployed partially or entirely through a network infrastructure. The network infrastructure may include elements such as computing devices, servers, routers, hubs, firewalls, clients, personal computers, communication devices, routing devices, and other active and passive devices, modules, and / or components known in the art. The computing and / or non-computing devices associated with the network infrastructure may include storage media such as flash memory, buffers, stacks, RAM, ROM, etc., apart from other components. The processes, methods, program codes, instructions described herein and elsewhere may be executed by one or more of the network infrastructure elements.

[0071]

[0082] The methods, program codes, and instructions described herein and elsewhere may be implemented on a cellular network having multiple cells. The cellular network may be either a frequency division multiple access (FDMA) network or a code division multiple access (CDMA) network. The cellular network may include mobile devices, cell sites, base stations, repeaters, antennas, towers, etc. The cell network may be a GSM, GPRS, 3G, EVDO, mesh, or other network type.

[0072]

[0083] The methods, program codes, and instructions described herein and elsewhere may be implemented on or via a mobile device. Mobile devices may include navigation devices, cell phones, mobile phones, mobile personal digital assistants, laptops, palmtops, netbooks, pagers, e-book readers, music players, and the like. These devices may include, among other components, storage media such as flash memory, buffers, RAM, ROM, and one or more computing devices. The computing devices associated with the mobile devices may be adapted to execute program codes, methods, and instructions stored thereon. Alternatively, the mobile devices may be configured to execute instructions in cooperation with other devices. The mobile devices may communicate with a base station interfaced with a server and configured to execute program code. The mobile devices may communicate over a peer-to-peer network, a mesh network, or other communication network. The program code may be stored on a storage medium associated with the server and executed by a computing device embedded within the server. The base station may include a computing device and a storage medium. The storage device may store program codes and instructions executed by the computing devices associated with the base station.

[0073]

[0084] The computer software, program code, and / or instructions may be stored in and / or accessed from machine-readable media, which may include computer components, devices, and other computer memory such as recording media that hold digital data used for computing for some interval of time, semiconductor storage known as random access memory (RAM), mass storage typically for more permanent storage such as optical disks, hard disks, tapes, drums, cards, and other types of magnetic storage, processor registers, cache memory, volatile memory, non-volatile memory, optical storage such as CDs, DVDs, flash memory (e.g., USB sticks or keys), floppy disks, magnetic tape, paper tape, punch cards, standalone RAM disks, Zip drives, removable mass storage, offline, etc., dynamic memory, static memory, read / write storage, mutable storage, read-only, random access, sequential access, location addressable, file addressable, content addressable, network attached storage, storage area networks, barcodes, magnetic ink, etc.

[0074]

[0085] The methods and systems described herein can transform physical and / or intangible items from one state to another. The methods and systems described herein can also transform data representing physical and / or intangible items from one state to another.

[0075]

[0086] The elements described and depicted herein, including the flowcharts and block diagrams throughout the figures, imply logical boundaries between the elements. However, in accordance with software or hardware engineering practices, the depicted elements and their functionality may be implemented on a machine having a processor capable of executing stored program instructions through a computer-executable medium, as a monolithic software structure, as a standalone software module, as a module employing external routines, code, services, etc., or as any combination thereof; all such implementations may be within the scope of the present disclosure. Examples of such machines may include, but are not limited to, personal digital assistants, laptops, personal computers, mobile phones, other handheld computing devices, medical equipment, wired or wireless communication devices, transducers, chips, calculators, satellites, tablet PCs, e-books, gadgets, electronic devices, devices with artificial intelligence, computing devices, networking equipment, servers, routers, etc. Furthermore, the elements depicted in the flowcharts and block diagrams or any other logical components may be implemented on a machine capable of executing program instructions. Thus, while the foregoing figures and description describe functional aspects of the disclosed system, the specific configuration of software for implementing those functional aspects should not be inferred from these descriptions unless expressly stated or otherwise apparent from the context. Similarly, it will be understood that the various steps identified and described above may be varied, and the order of steps may be adapted to particular applications of the technology disclosed herein. All such variations and modifications are intended to fall within the scope of the present disclosure. As such, the depiction and / or description of the order of various steps should not be understood to require a particular order of performance of those steps unless required by a particular application or unless expressly stated or otherwise apparent from the context.

[0076]

[0087] The above-described methods and / or processes, and steps thereof, may be implemented in hardware, software, or any combination of hardware and software suitable for a particular application. Hardware may include general-purpose computers and / or special-purpose or specific computing devices, or specific aspects or components of specific computing devices. The processes may be implemented in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, or other programmable devices, along with internal and / or external memory. The processes may also, or instead, be embodied in application-specific integrated circuits, programmable gate arrays, programmable array logic, or any other device or combination of devices that can be configured to process electronic signals. Furthermore, it will be understood that one or more of the processes may be implemented as computer-executable code executable on a machine-readable medium.

[0077]

[0088] The computer-executable code may be written using a structured programming language such as C, an object-oriented programming language such as C++, or any other high-level or low-level programming language (including assembly language, hardware description languages, and database programming languages ​​and techniques), which may be stored, compiled, or interpreted to run on one of the above devices, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software, or any other machine capable of executing program instructions.

[0078]

[0089] Thus, in one aspect, each of the methods and combinations thereof described above may be embodied in computer-executable code that performs its steps when executed on one or more computing devices. In another aspect, the method may be embodied in a system that performs its steps, may be distributed in some manner across devices, or all of the functionality may be integrated into a dedicated stand-alone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any of the hardware and / or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.

[0079]

[0090] While the present invention has been disclosed in connection with the preferred embodiments shown and described in detail, various modifications and improvements thereon will be readily apparent to those skilled in the art. Accordingly, the spirit and scope of the present invention is not intended to be limited by the foregoing examples, but is to be understood in the broadest sense permitted by law.

[0080]

[0091] All documents referenced herein are incorporated by reference in their entirety. [Explanation of symbols]

[0081] 100 systems 102 System 104 System 106 System 110 Mathematical Model Components 112 Mathematical Model Components 114 Mathematical Model Components 116 Mathematical Model Components 200 systems 210 Feature Extraction Component 212 Feature Extraction Component 214 Feature Extraction Component 216 Feature Extraction Component 220 Audio Embedding Component 222 Audio Embedding Component 224 Audio Embedding Component 226 Audio Embedding Component 240 Mathematical Model Components 300 System 330 Difference Components 340 Mathematical Model Components 400 System 440 Mathematical Model Components 500 Systems 540 Mathematical Model Components 800 Computing Devices 810 Volatile or Non-Volatile Memory 811 processor 812 network interface 820 Feature Extraction Component 821 Audio Embedding Component 822 Mathematical Model Components 823 Element-wise Differential Components 830 Training Data Store

Claims

1. receiving a first audio signal corresponding to a first time period, the first audio signal comprising a human voice; calculating a first feature vector from the first audio signal; computing a first audio embedding vector by processing the first feature vector using a neural network; receiving a second audio signal corresponding to a second time period, the second audio signal including the person's voice; calculating a second feature vector from the second audio signal; computing a second audio embedding vector by processing the second feature vector with the neural network; calculating an element-wise difference between the first audio embedding vector and the second audio embedding vector; calculating a change value indicative of a change in health status between the first time period and the second time period by processing the component-by-component differences using a mathematical model; 11. A computer-implemented method comprising:

2. obtaining a first health condition label indicative of a health condition during the first time period; calculating a second health state label indicative of a health state during the second time period by processing the first health state label and the change value; The computer-implemented method of claim 1 , comprising:

3. The computer-implemented method of claim 2 , wherein calculating the second health label comprises adding the first health label and the variance value.

4. wherein computing the first feature vector comprises: (i) performing speech recognition on the first audio signal to obtain recognized text; and (ii) obtaining a word-piece encoding corresponding to the recognized text; The neural network comprises a plurality of feedforward neural network layers, a plurality of self-attention neural network layers, The computer-implemented method of claim 1 , comprising:

5. The computer-implemented method of claim 1 , wherein the mathematical model comprises a second neural network.

6. The computer-implemented method of claim 1 , wherein the mathematical model comprises a fully connected neural network.

7. The computer-implemented method of claim 1 , wherein the health condition corresponds to stress, depression, anxiety, post-traumatic stress disorder, concussion, Parkinson's disease, Alzheimer's disease, or congestive heart failure.

8. The computer-implemented method of claim 1 , wherein calculating the change value comprises calculating an anti-symmetric change value.

9. receiving a first audio signal corresponding to a first time period, the first audio signal including a human voice; calculating a first feature vector from the first audio signal; processing the first feature vector using a neural network to calculate a first audio embedding vector; receiving a second audio signal corresponding to a second time period, the second audio signal including the person's voice; calculating a second feature vector from the second audio signal; processing the second feature vector using the neural network to calculate a second audio embedding vector; and calculating an element-wise difference between the first audio embedding vector and the second audio embedding vector; processing the component-by-component differences using a mathematical model to calculate a change value indicative of a change in health status between the first time period and the second time period; 1. A system comprising: at least one computer configured to:

10. The system of claim 9 , wherein the first feature vector comprises acoustic features.

11. The system of claim 9 , wherein the neural network comprises a transformer neural network.

12. The system of claim 9 , wherein the mathematical model comprises a fully connected neural network.

13. the at least one computer receiving a third audio signal corresponding to a third time period, the third audio signal including the person's voice; calculating a third feature vector from the third audio signal; processing the third feature vector using the neural network to calculate a third audio embedding vector; and Calculating a second element-wise difference between the third audio embedding vector and the second audio embedding vector; and processing the second component-by-component difference using the mathematical model to calculate a second change value indicative of a change in health status between the third time period and the second time period; The system of claim 9 configured to:

14. The system of claim 9 , wherein the at least one computer is configured to calculate the change value by calculating an anti-symmetric change value.

15. 10. The system of claim 9, wherein the at least one computer is configured to calculate the first feature vector by: (i) performing speech recognition on the first audio signal to obtain recognized text; and (ii) obtaining a word-piece encoding corresponding to the recognized text.

16. When executed, the method causes at least one processor to: receiving a first audio signal corresponding to a first time period, the first audio signal including a human voice; calculating a first feature vector from the first audio signal; processing the first feature vector using a neural network to calculate a first audio embedding vector; receiving a second audio signal corresponding to a second time period, the second audio signal including the person's voice; calculating a second feature vector from the second audio signal; processing the second feature vector using the neural network to calculate a second audio embedding vector; and calculating an element-wise difference between the first audio embedding vector and the second audio embedding vector; processing the component-by-component differences using a mathematical model to calculate a change value indicative of a change in health status between the first time period and the second time period; One or more non-transitory computer-readable media containing computer-executable instructions for performing operations including:

17. 17. The one or more non-transitory computer-readable media of claim 16, wherein the mathematical model comprises a fully connected neural network.

18. 17. The one or more non-transitory computer-readable media of claim 16, wherein calculating the change value comprises calculating an anti-symmetric change value.

19. The one or more non-transitory computer-readable media of claim 16 , wherein the first feature vector comprises acoustic features.

20. 17. The one or more non-transitory computer-readable media of claim 16, wherein calculating the first feature vector comprises: (i) performing speech recognition on the first audio signal to obtain recognized text; and (ii) obtaining a wordpiece encoding corresponding to the recognized text.

Images