Systems and methods of audio compression for physiological parameter sensing and analysis

Abstract

Encoder-decoder systems and methods of compressing and processing audio data are provided herein. The encoder-decoder systems and methods may find particular application in the context of physiological parameter sensing and analysis. The devices, systems, and methods described herein improve transmission and processing of audio data between an audio recording device and a remote / backend device. As described, an audio signal is captured by a first device (i.e., frontend device) that is then encoded using a compression algorithm trained to minimize a bitrate of the audio signal while preserving one or more predictive features that are used by a second device (i.e., a backend device) for the purpose of physiological monitoring.

Description

SYSTEMS AND METHODS OF AUDIO COMPRESSION FOR PHYSIOLOGICAL PARAMETER SENSING AND ANALYSISField of the Disclosure

[0001] The present disclosure relates generally to audio compression, and more specifically to systems and methods for audio compression used for physiological parameter sensing and analysis, such as for respiratory sensing applications.Background

[0002] The CO VID- 19 pandemic has had significant effects in reshaping the landscape of healthcare delivery. In one aspect, the demand for telehealth services has drastically increased because lockdowns and social distancing measurements made traditional in-person healthcare visits more challenging or unsafe. Beyond the pandemic, offering patients remote access to healthcare professionals through telehealth services has remained a growing field. In important respects, telehealth services offer increased flexibility, convenience, and accessibility. However, robust patient monitoring in remote settings is a significant challenge. For example, telehealth services are currently limited due to significant technical hurdles concerning the collection, handling, transmission, and processing of patient health data.

[0003] In recent years, wearable personal monitoring devices have shown promise in supporting telehealth services by providing individuals with real-time access to their health and wellness data. These wearable devices, often referred to as “wearables”, are portable electronic devices that can be worn on the body and are typically equipped with sensors, processors, and communication capabilities to perform various functions such as tracking biometric data, monitoring health metrics, providing notifications, and / or interacting with other devices. Wearables can take various forms, including smartwatches, fitness trackers, smart glasses, and dedicated purpose-built medical monitoring devices. They are designed to be worn comfortably for extended periods and are often used to augment the user's daily activities or provide personalized services based on collected data. However, the utility of these devices in clinical applications remains limited for a number of reasons including concerns with accuracy and reliability of the data they collect and process.

[0004] For example, and more specifically, it has been demonstrated that it is possible to use deep learning techniques to develop computational models that can evaluate an individual’s respiratory function from normal conversational speech. However, the estimation of respiratory function typically requires computational resources that are not necessarily available in common wearable or nearby devices equipped to capture the audio signal. Moreover, conventional audio compression techniques may alter the audio signal in such a way that destroys its value for evaluating respiratory function (or another type of physiological function).Summary of the Disclosure

[0005] Accordingly, it is an object of the present disclosure to provide systems and methods for audio compression for physiological sensing and analysis that enables efficient transmission from local devices (i.e., “edge” devices) to remote devices (i.e., “backend” devices) for processing. It is also an object of the present disclosure to encode the audio data from the local devices such that the features necessary for the physiological sensing and analysis are preserved while still minimizing the size and / or bitrate of the compressed data stream. As a result, the embodiments described herein may find particular application in telehealth services where physiological data is collected in a remote setting and transmitted to a backend or centralized processing system for analysis.

[0006] According to an embodiment of the present disclosure, a frontend audio processing device is provided. The frontend audio processing device may include: an audio receiving device configured to detect sound and generate an audio signal having a first bitrate based thereon; a memory configured to store the audio signal generated; and an audio encoder comprising one or more processors configured to encode the audio signal using a trained compression algorithm to obtain a compressed bit stream, the compressed bit stream having a second bitrate that is lower than the first bitrate. The trained compression algorithm may be configured to minimize the second bitrate while preserving one or more predictive features of the audio signal.

[0007] In an aspect, the one or more processors of the audio encoder may be configured to: designate a plurality of windowed segments of the audio signal; compute a short-time Fourier transform vector for each of the plurality of windowed segments of the audio signal; and generate a plurality of short-time log Mel spectrum vectors by converting each of the short-time Fourier transform vectors to short-time log Mel spectrum vectors.

[0008] In an aspect, the trained compression algorithm may receive as input the plurality of short-time log Mel spectrum vectors generated by the audio encoder.

[0009] In an aspect, the trained compression algorithm may include a neural network layer having a hard sigmoid activation function that generates as an output an encoded binary vector for each input to the trained compression algorithm.

[0010] In an aspect, the compressed bit stream may include a plurality of encoded binary vectors generated using the trained compression algorithm.

[0011] In an aspect, the frontend audio processing device may further comprise: a networking unit configured to transmit electronic signals from the frontend audio processing device to a backend audio processing device over a transmission channel, wherein the electronic signals include at least the compressed bit stream generated by the audio encoder.

[0012] In an aspect, the audio signal may include a speech signal of an individual.

[0013] In an aspect, the one or more predictive features of the audio signal may include features that are predictive of one or more respiratory parameters of the individual.

[0014] In an aspect, the one or more predictive features of the audio signal may include features that are predictive of a virtual respiratory belt of the individual.

[0015] In an aspect, the trained compression algorithm and a trained predictive algorithm may form a neural network trained using a Siamese network configuration including a differentiable network and a non-differentiable network, wherein the differentiable and non-differentiable networks receive a common audio input. The neural network may be trained to minimize a distance measure between an output of the differentiable network, an output of the non-differentiable network, and a ground truth output.

[0016] According to another embodiment of the present disclosure, a backend audio processing device is provided. The backend audio processing device may include: a networking unit configured to receive electronic signals from a frontend audio processing device over a transmission channel, wherein the electronic signals include at least a compressed bit stream generated by an audio encoder of the frontend audio processing device; a memory configured to store at least the compressed bit stream; and an audio decoder comprising one or more processors configured to decode the compressed bit stream using a trained predictive algorithm to obtain a decoded output signal, wherein the compressed bit stream comprises a plurality of encoded binary vectors.

[0017] In an aspect, the audio encoder of the frontend audio processing device may receive an audio signal comprising a speech signal of an individual and may output the compressed bit stream. The decoded output signal may be a predicted respiratory parameter or signal.

[0018] According to still another embodiment of the present disclosure, a method of compressing and processing audio data is provided. The method may include: generating, at a frontend audio processing device, an audio signal having a first bitrate; and encoding, at the frontend audio processing device, the generated audio signal using a trained compression algorithm to obtain a compressed bit stream having a second bitrate that is lower than the first bitrate. The trained compression algorithm may be a lossy audio encoding algorithm configured to minimize the second bitrate of the compressed bit stream while preserving one or more predictive features of the audio signal. The compressed bit stream may include a plurality of encoded binary activation vectors generated using the trained compression algorithm.

[0019] In an aspect, the method may further include: transmitting, from the frontend audio processing device, the compressed bit stream over a transmission channel to a backend audio processing device; receiving, at a backend audio processing device, the compressed bit stream; decoding, at the backend audio processing device, the compressed bit stream to generate a decoded audio signal representation; and processing, the decoded audio signal representation using a trained predictive algorithm to obtain a physiological signal or parameter associated with an individual.

[0020] In an aspect, the audio signal may include a speech signal of the individual and the physiological signal or parameter output by the trained predictive algorithm may be a predicted respiratory signal or parameter of the individual.

[0021] These and other aspects of the various embodiments will be apparent from and elucidated with reference to the embodiments described hereinafter.Brief Description of the Drawings

[0022] In the drawings, like reference characters generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the various embodiments.

[0023] FIG. 1 is a block diagram illustrating an encoder-decoder system in accordance with aspects of the present disclosure.

[0024] FIG. 2 is a flowchart illustrating a method of compressing and processing audio data in accordance with aspects of the present disclosure.

[0025] FIG. 3 is a block diagram illustrating the training architecture of an encoder-decoder algorithm in accordance with aspects of the present disclosure.

[0026] FIG. 4 is a block diagram illustrating the implementation of an encoder-decoder algorithm in accordance with aspects of the present disclosure.

[0027] FIG. 5 is a block diagram illustrating an audio processing device in accordance with aspects of the present disclosure.Detailed Description of Embodiments

[0028] Systems and methods for audio compression for physiological parameter sensing and analysis are provided herein. More specifically, it is an object of the present disclosure to provide systems and methods that improve transmission of audio signals from an audio recording device to a remote / backend device for processing which improves the transmission efficiency (e.g., speed) without sacrificing the digital information contained within the audio signal needed for physiological evaluation. In accordance with various embodiments of the present disclosure, the audio signal captured by a first device (i.e., frontend device) can include speech from an individual that is encoded using a compression algorithm trained to minimize a bitrate of the audio signal while preserving one or more predictive features that are used by a second device (i.e., a backend device) for the purpose of evaluating the respiratory function of the individual. Also provided herein are methods of training a machine learning algorithm for encoding and decoding audio signals.

[0029] Turning now to FIG. 1, an audio encoder-decoder system 100 is illustrated in accordance with aspects of the present disclosure. As shown, the audio encoder-decoder system 100 includes an edge device 110 and a backend device 160. In embodiments, the audio encoderdecoder system 100 includes at least one edge device 110, but may also include more than one edge device 110. For example, in some embodiments, the audio encoder-decoder system 100 may include a plurality of edge devices 110. As described herein, each of the edge devices 110 can include an audio receiving device 112 configured to detect sound 102 and generate an audio signal based thereon. Each of the edge devices 110 can include also include a memory 114 configured tostore the audio signals generates, an audio encoder 116 configured to encode the audio signals to obtain a compressed bit stream, and / or a networking unit 118 configured to transmit information including the compressed bit stream. As described in more detail below, the audio encoder 116 of an edge device 110 can include one or more processors configured to encode the audio signal using a trained compression algorithm. In embodiments, the compression algorithm is trained to minimize the bitrate of the compressed bit stream while preserving one or more predictive features of the audio signal.

[0030] The audio encoder-decoder system 100 can also include a backend system or device, such as backend device 160 shown in the example of FIG. 1. The backend device 160 of the audio encoder-decoder system 100 can include one or more electronic devices located remotely from the one or more edge devices 110 and configured to compute a physiological analysis based on the audio signals encoded at the edge devices 110. In particular embodiments, the backend device 160 can include a networking unit 162 configured to receive electronic signals from one or more edge devices 110 including audio signals encoded by the edge devices 110, a memory 164 configured to store at least the encoded audio signals received from the edge devices 110, an audio decoder 166 configured to decode a representation of the audio signal from the encoded audio signal, and a predictive algorithm 168 used to compute a physiological analysis based on the decoded information.

[0031] According to various aspects of the present disclosure, the encoder-decoder system 100 is configured to execute a method compressing and processing signals, including for the purposes of physiological sensing as described herein. For example, with reference to FIG. 2, an exemplary method 200 for physiological sensing and analysis is illustrated in accordance with certain aspects of the present disclosure. In embodiments, the method 200 can include: in a step 210, generating an audio signal at a first device; in a step 220, encoding the audio signal at the first device; in a step 230, transmitting the encoded audio signal to a second device; in a step 240, receiving the encoded audio signal at the second device; in a step 250, decoding the encoded audio signal at the second device; and in a step 260, processing the decoded audio signal at the second device to determine a physiological metric.

[0032] More specifically, in the step 210, the method 200 may include generating an audio signal using at least a first device, such as one or more edge devices 110. As described above, the edge device(s) 110 can include an audio receiving device 112 configured to detect sound 102within proximity of the edge device(s) 110 and generate an audio signal based thereon. In embodiments, the audio signal generated by the audio receiving device 112 may have a bitrate of at least about 50 kilobits per second (Kbps), including at least about 96 Kbps, at least about 160 Kbps, or at least about 320 Kbps. In particular embodiments, the audio signal generated by the audio receiving device 112 may have a bitrate of between about 96 Kbps and about 1,411 Kbps. This audio signal may be stored in the memory 114 of the edge device(s) 110.

[0033] In the step 220, the method 200 may then include encoding the audio signal generated by the audio receiving device 112. In embodiments, the audio signal may be encoded using an audio encoder 116 of the edge device(s) 110. The audio encoder 116 can include one or more computer processors configured to receive the audio signal, apply a compression algorithm to the audio signal, and generate a compressed bit stream. In embodiments, the compressed bit stream comprises a representation of the audio signal that may be used by a machine learning algorithm in order to predict or re-construct a physiological signal or parameter associated with an individual (e.g., a patient undergoing remote patient monitoring through a telehealth service, etc.). As described in more detail below, the compressed bit stream can include one or more binary activation vectors (e.g., such as binary activation vector 120 shown in FIG. 1) for use in a trained predictive algorithm (e.g., predictive algorithm 168 shown in FIG. 1).

[0034] In embodiments, the one or more processors of the audio encoder 116 may also be configured to covert the audio signal into a plurality of short-time log Mel spectrum vectors prior to applying the trained compression algorithm. That is, the audio encoder 116 of the edge device(s) 110 may generate a plurality of short-time log Mel spectrum vectors based on the input audio signal 102, which are then used as the input to the trained compression algorithm 117. In such embodiments, the trained compression algorithm 117 can receive as an input the plurality of short- time log Mel spectrum vectors and may output a compressed bit stream comprising an encoded binary activation vector for each of the plurality of short-time log Mel spectrum vectors.

[0035] In specific embodiments, generating the short-time log Mel spectrum vectors may include: (i) designating a plurality of windowed segments of the audio signal 102; (ii) computing a short-time Fourier transform vector for each of the plurality of windowed segments of the audio signal; and (iii) generating a plurality of short-time log Mel spectrum vectors by converting each of the short-time Fourier transform vectors to short-time log Mel spectrum vectors. Spectrograms and log Mel spectrograms are described further in Venkata Srikanth Nallanthighal et al., “Deeplearning architectures for estimating breathing signal and respiratory parameters from speech recordings,” NEURAL NETWORKS, Volume 141, 2021, Pages 211-224, ISSN 0893-6080, https: / / doi.Org / 10.1016 / j.neunet.2021.03.029, the contents of which are incorporated herein in its entirety.

[0036] In still further embodiments, the one or more processors of the audio encoder 116 are configured to apply one or more compression / encoding techniques in addition to the trained compression algorithm. For example, in some embodiments, the one or more processors of the audio encoder 116 may be configured to apply a block- wise lossless entropy encoder to the output of the trained compression algorithm. In embodiments, one or more of these compression / encoding techniques may be applied before and / or after generating the short-time log Mel spectrum vectors and / or the compressed bit stream.

[0037] In embodiments, the compressed bit stream output by the audio encoder 116 can have a bitrate that is less than the bitrate of the input audio signal. In particular embodiments, the compressed bit stream may have a bitrate of less than about 1,411 Kbps, including less than about 320 Kbps, less than about 160 Kbps, less than about 96 Kbps, or less than about 50 Kbps. In other embodiments, the compressed bit stream may have a bitrate that is less than 90% of the bitrate of the input audio signal, including less than about 80%, less than about 70%, less than about 60%, less than about 50%, less than about 40%, and / or less than about 30% of the bitrate of the input audio signal.

[0038] In embodiments, the compression algorithm executed by the audio encoder 116 may be a machine learning algorithm (or a portion thereof) that is trained to minimize the bitrate of the compressed bit stream while preserving one or more predictive features of the audio signal. As described herein, the term “predictive features” refers to one or more characteristics and / or statistical properties of the audio signal which may be used by a trained machine learning algorithm (described in more detail below) in order to predict a physiological signal or parameter associated with an individual. For example, in particular embodiments, the audio signal can include a speech signal of an individual and the one or more predictive features are characteristics and / or statistical properties of the audio signal that are used by a trained machine learning algorithm in order to predict a respiratory signal or parameter of the individual, a cardiac signal or parameter of the individual (e.g., heart rate), and / or a pulmonary signal or parameter of the individual (e.g., bloodoxygen saturation). In specific embodiments, the predictive features may be predictive of a virtual respiratory belt of the individual.

[0039] In the steps 230 and 240, the method 200 can then include transmitting at least the encoded audio signal (i.e., the compressed bit stream) from the edge device(s) 110 and receiving at least the encoded audio signal (i.e., the compressed bit stream) at a backend device 160. As described above, the edge device(s) 110 can include a first networking unit 118 and the backend device(s) 160 can include a second networking unit 162 that are configured to send and / or receive electronic signals through wired and / or wireless connections. In particular embodiments, the networking units 118, 162 may operatively connect these devices 110, 160 to one or more types of communications networks, which can include a direction interconnection, the Internet, a local area network (“LAN”), a metropolitan area network (“MAN”), a wide area network (“WAN”), a wired or Ethernet connection, a wireless connection, a cellular network, a Bluetooth connection, and similar types of communications networks, including combinations thereof.

[0040] In the step 250, the method 200 then includes decoding the encoded audio signal (i.e., the compressed bit stream). In embodiments, the encoded audio signal may be decoded using an audio decoder 166 of the backend device(s) 160. The audio decoder 166 can include one or more computer processors configured to receive the encoded audio signal, apply one or more decompression / decoding techniques, and generate a decoded audio signal representation. For example, in embodiments, the one or more processors of the audio decoder 166 may be configured to apply a block-wise lossless entropy decoder to the encoded audio signal received in order to generate the decoded audio signal representation. In embodiments, the decoded audio signal can include a representation of the audio signal for use by a predictive algorithm 168 of the backend device(s) 160 in order to predict or re-construct a physiological signal or parameter associated with an individual (e.g., a patient undergoing remote patient monitoring through a telehealth service, etc.). In particular embodiments, the decoded audio signal can include one or more binary activation vectors (e.g., such as binary activation vector 120 shown in FIG. 1) for use in a trained predictive algorithm (e.g., predictive algorithm 168 shown in FIG. 1).

[0041] In the step 260, the method 200 then includes processing the decoded audio signal representation at the backend device(s) 160 in order to predict / determine a physiological signal or parameter associated with an individual, including in order to predict a respiratory signal or respiratory parameter of the individual. In particular embodiments, the step 260 can includeapplying a trained predictive algorithm 168 using the decoded audio signal representation in order to predict / determine the physiological signal or parameter.

[0042] The predictive algorithm 168 may form part of a machine learning algorithm that is trained to predict / determine a physiological signal or parameter based on an audio signal, such as an audio signal containing human speech which may be recorded by the edge device(s) 110. In some embodiments, the predictive algorithm 168 may be trained to estimate a respiratory signal or respiratory parameter of an individual based on speech of the individual contained in the audio signal. In embodiments, the respiratory parameter can include, but is not limited to, breath events, breathing rates, speech tidal volumes, and / or the like. In further embodiments, the respiratory signal can include, but is not limited to, a virtual respiratory belt signal.

[0043] As described herein, the trained compression algorithm of the audio encoder 116 of the edge device(s) 110 and the trained predictive algorithm 168 of the backend device(s) 160 may be components of a neural network (e.g., neural network 302 shown in FIG. 3) that is trained using deep learning techniques to develop a computational model that can synthesize physiological signals or parameters based on an audio input. More specifically, the neural network (e.g., neural network 302) may be trained to develop a computation model that can synthesize a virtual respiratory belt (VRB) signal from normal conversational speech.

[0044] In embodiments, the neural network (e.g., neural network 302) can include a first component comprising one or more feedforward neural network layers, including at least one feedforward neural network layer having a hard sigmoid activation function that generates as an output an encoded binary activation vector of for each input. In embodiments, the first component may be the trained compression algorithm executed by the edge device(s) 110. As such, in embodiments, the first component of the neural network may receive as an input an audio signal in the form of a plurality of short-time log Mel spectrum vectors generated by an audio encoder 116 of the edge device(s) 110, which then outputs a binary vector of activations for each short- time log Mel spectrum vector.

[0045] In further embodiments, the neural network (e.g., neural network 302) can include a second component comprising one or more recurrent neural network (RNN) / transformer layers and a linear layer configured to use the binary activation vectors and generate an estimated physiological signal or parameter (e.g., an estimated virtual respiratory belt signal). Inembodiments, the second component may be the trained predictive algorithm 168 executed by the backend device(s) 160.

[0046] With reference to FIG. 3, the training architecture 300 for training the neural network 302 is illustrated in accordance with various aspects of the present disclosure. As shown, in some embodiments, the training architecture 300 may be a Siamese network configuration, i.e., a type of neural network architecture that consists of two identical or similar subnetworks (or twin networks) that share the same parameters and weights. Two subnetworks process two different inputs independently, and then their outputs are compared or combined in some way. The advantage of deploying the Siamese network configuration in conjunction with the present disclosure is to learn a representation (or embedding) of the input data in a shared feature space, where similar inputs are mapped close together and dissimilar inputs are mapped farther apart. By training the network to minimize the distance between similar pairs of inputs and maximize the distance between dissimilar pairs, the network learns to distinguish between different classes or categories of inputs.

[0047] Accordingly, as shown in FIG. 3, the training architecture 300 of a Siamese network configuration includes a differential network 304 and a non-differentiable network 302. Each of the networks 302, 304 may be configured to receive some audio input 306 and output an estimated / predicted physiological signal or parameter. In particular embodiments, the audio input 306 includes short-term speech spectrum features (e.g., short-time log Mel spectrum vectors, etc.) and the output is an estimated virtual respiratory belt signal 308, 310. Because the network 304 uses a feedforward neural network layer with a softsign activation function (which produces pseudobinary activation vectors), the network 304 is differentiable, and can therefore be trained using back-propagation techniques to model a ground-truth physiological signal (such as a measured virtual respiratory belt signal). As shown in FIG. 3, the network 302 is not differentiable because of the feedforward neural network layer having a hard sigmoid activation function, which produces a binary activation vector for each of the inputs. However, it should be appreciated that the hard sigmoid activation function in the network 302 may be replaced by another non-differentiable quantizer, such as a fixed or an adaptive quantizer, and a corresponding differentiable quantizer may be used in the network 304.

[0048] As the differentiable network 304 is trained, the layer coefficients are copied to the non- differentiable network 302, and both networks 302, 304 output a corresponding estimatedphysiological signal 308, 310. In particular embodiments, the Siamese network configuration 300 is trained by simultaneously minimizing the difference measurements between the estimated physiological signals 308, 310 and a ground-truth physiological signal. For example, the Siamese network configuration 300 may be trained to minimize the parameter dsum, which is given by the following equation:wherein veis the physiological signal 308 from the differentiable network 304, vbis the physiological signal 310 from the non-differentiable network 302, y is the ground-truth signal, dsis the difference between veand vbto be minimized, and dmis the difference between veand y to be minimized, and wherein operations between vectors are point-wise Hadamard products.

[0049] It should be appreciated that the cost function can also be weighted, such that the two parts are weighted differently. In particular embodiments, the cost function may also be expressed in a logarithmic domain.

[0050] As described herein, the goal of the training architecture 300 is to find the minimum length n of the binary activation vectors 120 required to generate estimated physiological signals 308, 310 that differ from the ground-truth signal by less than a predetermined threshold. For example, the minimum length n of the binary activation vectors 120 may be found such that, for a given test data set of audio signals, gives less than a 5% change in the accuracy of detecting inspiration events (“IES”) between the ground-truth VRB signal as compared with the estimated VRB signals 308, 310. In other embodiments, the minimum length n of the binary activation vectors 120 may be found such that, for a given test data set of audio signals, gives less than a 5% change in the accuracy of detecting respiration rate, inspiration time and / or gradient, expiration time and / or gradient, and / or the like.

[0051] Although the training of a particular neural network has been described, it should be appreciated that variations to this architecture may be implemented. For example, in particular embodiments, the neural network may contain convolutional neural network (CNN), recurrent neural network (RNN), and / or dense network components in different layers of the system.

[0052] In further embodiments, two or more distinct networks with a varying binary vector lengths. For example, in one embodiment, a testing dataset comprising speech activity of different individuals may be used to train a neural network and develop an encoder that has been optimized for speech audio content. In another embodiment, a testing dataset comprising non-speechbreathing sounds of different individuals may be used to train a neural network and develop an encoder that has been optimized for non-speech audio content. As such, the first encoder may generate binary activation vectors of a first length B1whereas the second encoder generates binary activation vectors of a second length B2that is different from B . In embodiments, the audio encoder 116 of the edge device(s) 110 and the audio decoder 166 of the backend device(s) 160 can be configured to detect the type of content within the audio signal and execute one or more different neural networks based thereon.

[0053] Turning to FIG. 4, a flowchart illustrating the use an encoder-decoder system 100 implementing of a neural network 302 that is trained using deep learning techniques to develop a computational model that can synthesize physiological signals or parameters based on an audio input. As shown, an edge device 110 captures and develops an audio signal comprising short-term speech spectrum components 306 (e.g., log Mel spectrum vectors), which are used an inputs to an encoder 116. The encoder 116 can include one or more computer processors configured to execute one or more neural network layers, including a feedforward neural network layer with a hard sigmoid activation function that produces one or more binary activation vectors 120. As mentioned above, the one or more processors of the encoder 116 may be configured to apply a block-wise lossless entropy encoder 402 to the binary activation vectors 120 to generate a compressed bit stream comprising the one or more binary activation vectors 120. The edge device 110 may then transmit the compressed bit stream via a transmission channel 404 to an audio decoder 406 located at a backend device 160.

[0054] The backend device 160 may receive the compressed bit stream from the edge device 110. An audio decoder 166 comprising one or more computer processors may then apply a blockwise lossless entropy decoder to the compressed bit stream to obtain a decoded audio signal. This decoded audio signal may then be processed by the decoder 166 using one or more neural network layers, transformer layers, and / or linear layers in order to predict or re-construct a physiological signal or parameter associated with an individual (e.g., a patient undergoing remote patient monitoring through a telehealth service, etc.). As shown in the example of FIG. 4, the physiological signal generated by the encoder-decoder system 100 is an estimated virtual respiratory belt signal 410 of an individual.

[0055] Turning to FIG. 5, an exemplary processing device 500 is illustrated in accordance with various aspects of the present disclosure. As described herein, each of the edge device(s) 110and / or the backend device(s) 160 may be a processing device, such as processing device 500. In the example of FIG. 5, the processing device 500 can include one or more processors 502 and a computer-readable memory 504 interconnected and / or in communication via a system bus 506 containing conductive circuit pathways through which instructions (e.g., machine-readable signals) may travel to effectuate communication, tasks, storage, and the like. The processing device 500 can be connected to a power source (not shown), which can include an internal power supply and / or an external power supply. In embodiments, the processing device 500 can also include one or more additional components, such as a user interface 508, a display 510, an input / output (I / O) interface 512, a networking unit 514, and the like, including combinations thereof. As shown, each of these components may be interconnected and / or in communication via the system bus 506, for example.

[0056] In embodiments, the one or more processors 502 can include one or more high-speed data processors adequate to execute the program components described herein and / or perform one or more operations of the methods described herein. The one or more processors 502 may include a microprocessor, a multi-core processor, a multithreaded processor, an ultra-low voltage processor, an embedded processor, and / or the like, including combinations thereof. The one or more processors 502 can include multiple processor cores on a single die and / or may be a part of a system on a chip (SoC) in which the processor 502 and other components are formed into a single integrated circuit, or a single package. That is, the one or more processors 502 may be a single processor, multiple independent processors, or multiple processor cores on a single die.

[0057] In embodiments, the user interface 508 may be configured to receive various forms of input from a user associated with the processing device 500. The user interface 508 can include, but is not limited to, one or more of a keyboard, keypad, trackpad, trackball(s), capacitive keyboard, controller (e.g., a gaming controller), computer mouse, computer stylus / pen, and / or the like, including combinations thereof. In particular embodiments, the processing device 500 may be an edge device 110 and the user interface 508 may include an audio capturing device 112.

[0058] In embodiments, the display device 510 may be configured to display information, including text, graphs, and / or the like. The display device 510 can include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, a touch screen or other touch- enabled display, a foldable display, a projection display, and so on, or combinations thereof.

[0059] In embodiments, the input / output (I / O) interface 512 may be configured to connect and / or enable communication with one or more peripheral devices (not shown), including but not limited to additional machine-readable memory devices, diagnostic equipment, and other attachable devices. The I / O interface 512 may include one or more I / O ports that provide a physical connection to the one or more peripheral devices. In some embodiments, the I / O interface 512 may include one or more serial ports.

[0060] In embodiments, the networking unit 514 may include one or more types of networking interfaces that facilitate wired and / or wireless communication between the processing device 500 and one or more external devices. That is, the networking unit 514 may operatively connect the processing device 500 to one or more types of communications networks 516, which can include a direction interconnection, the Internet, a local area network (“LAN”), a metropolitan area network (“MAN”), a wide area network (“WAN”), a wired or Ethernet connection, a wireless connection, a cellular network, and similar types of communications networks, including combinations thereof. In some embodiments, the networking unit 514 may be a networking unit 118 of an edge device 110 that is configured to connect the edge device 110 to a backend device 160. In other embodiments, the networking unit 514 may be a networking unit 162 of a backend device 160 that is configured to connect the backend device 160 to one or more edge devices 110.

[0061] In embodiments, the memory 504 (e.g., memory 114, 164) can be variously embodied in one or more forms of machine accessible and machine-readable memory. In some embodiments, the memory 504 includes a storage device (not shown), which can include, but is not limited to, a non-transitory storage medium, a magnetic disk storage, an optical disk storage, an array of storage devices, a solid-state memory device, and / or the like, as well as combinations thereof. The memory 504 may also include one or more other types of memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), Flash memory, and / or the like, as well as combinations thereof. In embodiments, the memory 504 may include one or more types of transitory and / or non-transitory memory.

[0062] The processing devices 500 described herein (e.g., the edge devices 110 and / or the backend devices 160) can be configured by software components stored in their respective memory 504 to perform one or more processes of the methods described herein.

[0063] For example, in some embodiments, the processing device 500 may be an edge device 110 and the memory 504 can be configured to store data / information 520 and computer-readable instructions 522 that, when executed by the one or more processors 502, causes the edge device 110 to detect sound, generate an audio signal, encode the audio signal, and transmit the encoded audio signal to another device. Such data 520 and the computer-readable instructions 522 stored in the memory 504 may form an audio encoding package 524A that may be incorporated into, loaded from, loaded onto, or otherwise operatively available to and from the edge device 110. Thus, in some embodiments, the audio encoding package 524A and / or one or more individual software packages may be stored in a local storage device of the memory 504. However, in other embodiments, the audio encoding package 524A and / or one or more individual software packages may be loaded onto and / or updated from a remote server or service via the communications network 516.

[0064] In other embodiments, the processing device 500 may be a backend device 160 and the memory 504 can be configured to store data / information 520 and computer-readable instructions 522 that, when executed by the one or more processors 502, causes the backend device 160 to receive an encoded audio signal, decode the encoded audio signal, and process the decoded audio signal to generate an estimated physiological signal. Such data 520 and the computer-readable instructions 522 stored in the memory 504 may form an audio decoding package 524B that may be incorporated into, loaded from, loaded onto, or otherwise operatively available to and from the backend device 160. Thus, in some embodiments, the audio decoding package 524B and / or one or more individual software packages may be stored in a local storage device of the memory 504. However, in other embodiments, the audio decoding package 524B and / or one or more individual software packages may be loaded onto and / or updated from a remote server or service via the communications network 516.

[0065] The processing devices 500 may also include an operating system component 526, which may be stored in the memory 504. The operating system component 524 may be an executable program facilitating the operation of the processing device 500. Typically, the operating system component 526 can facilitate access of the I / O interface 512, network interface 514, the user interface 508, and the display 510, and can communicate or control other components of the processing device 500.

[0066] As described herein, the edge device 110 may be a type of processing device, such as processing device 500. In specific embodiments, such edge devices 110 may be embodied as one or more a portable computers, laptop computers, mobile devices (e.g., smartphones, media players, game players, mobile phones, and the like), desktop computers, wearable devices (e.g., smart watches, smart rings, mobile telemetry devices, wristbands, and the like), and other such processing devices. In further embodiments, the backend device 160 may be a type of processing device, such as a separate processing device 500. In specific embodiments, such backend devices 160 may be embodied as one or more a portable computers, laptop computers, desktop computers, servers, and other such processing devices.

[0067] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein. It should also be appreciated that terminology explicitly employed herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.

[0068] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0069] The terms “edge device” and “frontend device” are used synonymously and interchangeably throughout the disclosure.

[0070] The term “bitrate” as used herein should be understood to mean the measure of bits of data being processed into audio over a given period of time.

[0071] The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”

[0072] The phrase “and / or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specificallyidentified by the “and / or” clause, whether related or unrelated to those elements specifically identified.

[0073] As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of’ or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e., “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.”

[0074] As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified.

[0075] As used herein, although the terms first, second, third, etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another element or component. Thus, a first element or component discussed below could be termed a second element or component without departing from the teachings of the inventive concept.

[0076] Unless otherwise noted, when an element or component is said to be “connected to,” “coupled to,” or “adjacent to” another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or intervening elements or components may be present. That is, these and similar terms encompass cases where one or more intermediate elements or components may be employed to connect two elements or components. However, when an element or component is said to be “directly connected” to another element or component, this encompasses only cases where the two elementsor components are connected to each other without any intermediate or intervening elements or components.

[0077] In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of’ and “consisting essentially of’ shall be closed or semi-closed transitional phrases, respectively.

[0078] It should also be understood that, unless clearly indicated to the contrary, in any methods claimed herein that include more than one step or act, the order of the steps or acts of the method is not necessarily limited to the order in which the steps or acts of the method are recited.

[0079] The above-described examples of the described subject matter can be implemented in any of numerous ways. For example, some aspects can be implemented using hardware, software or a combination thereof. When any aspect is implemented at least in part in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single device or computer or distributed among multiple devices / computers.

[0080] The present disclosure can be implemented as a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0081] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium comprises the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitorysignals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0082] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0083] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, comprising an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions can execute entirely on the user’s computer, partly on the user’s computer, as a standalone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, comprising a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some examples, electronic circuitry comprising, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0084] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer programproducts according to examples of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0085] The computer readable program instructions can be provided to a processor of a, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture comprising instructions which implement aspects of the function / act specified in the flowchart and / or block diagram or blocks.

[0086] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0087] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various examples of the present disclosure. In this regard, each block in the flowchart or block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the Figures. For example, two blocks shown in succession can, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can beimplemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0088] Other implementations are within the scope of the following claims and other claims to which the applicant can be entitled.

[0089] While several inventive embodiments have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to be within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the inventive teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure.

Claims

ClaimsWhat is claimed is:

1. A frontend audio processing device comprising: an audio receiving device configured to detect sound and generate an audio signal having a first bitrate based thereon; a memory configured to store the audio signal generated; and an audio encoder comprising one or more processors configured to encode the audio signal using a trained compression algorithm to obtain a compressed bit stream, the compressed bit stream having a second bitrate that is lower than the first bitrate; wherein the trained compression algorithm is configured to minimize the second bitrate while preserving one or more predictive features of the audio signal.

2. The frontend audio processing device of claim 1, wherein the one or more processors of the audio encoder are configured to: designate a plurality of windowed segments of the audio signal; compute a short-time Fourier transform vector for each of the plurality of windowed segments of the audio signal; and generate a plurality of short-time log Mel spectrum vectors by converting each of the short-time Fourier transform vectors to short-time log Mel spectrum vectors.

3. The frontend audio processing device of claim 2, wherein the trained compression algorithm receives as input the plurality of short-time log Mel spectrum vectors generated by the audio encoder.

4. The frontend audio processing device of claim 1 , wherein the trained compression algorithm includes a neural network layer having a hard sigmoid activation function that generates as an output an encoded binary vector for each input to the trained compression algorithm.

5. The frontend audio processing device of claim 4, wherein the compressed bit stream comprises a plurality of encoded binary vectors generated using the trained compression algorithm.

6. The frontend audio processing device of claim 1, further comprising: a networking unit configured to transmit electronic signals from the frontend audio processing device to a backend audio processing device over a transmission channel, wherein the electronic signals include at least the compressed bit stream generated by the audio encoder.

7. The frontend audio processing device of claim 1, wherein the audio signal comprises a speech signal of an individual.

8. The frontend audio processing device of claim 7, wherein the one or more predictive features of the audio signal includes features that are predictive of one or more respiratory parameters of the individual.

9. The frontend audio processing device of claim 7, wherein the one or more predictive features of the audio signal includes features that are predictive of a virtual respiratory belt of the individual.

10. The frontend audio processing device of claim 1 , wherein the trained compression algorithm and a trained predictive algorithm comprise a neural network trained using a Siamese network configuration including a differentiable network and a non-differentiable network, wherein the differentiable and non-differentiable networks receive a common audio input and the neural network is trained to minimize a distance measure between an output of the differentiable network, an output of the non-differentiable network, and a ground truth output.

11. A backend audio processing device comprising: a networking unit configured to receive electronic signals from a frontend audio processing device over a transmission channel, wherein the electronic signals include at least a compressed bit stream generated by an audio encoder of the frontend audio processing device; a memory configured to store at least the compressed bit stream; andan audio decoder comprising one or more processors configured to decode the compressed bit stream using a trained predictive algorithm to obtain a decoded output signal, wherein the compressed bit stream comprises a plurality of encoded binary vectors.

12. The backend audio processing device of claim 11, wherein the audio encoder of the frontend audio processing device receives an audio signal comprising a speech signal of an individual and outputs the compressed bit stream, and wherein the decoded output signal is a predicted respiratory parameter or signal.

13. A method of compressing and processing audio data, the method comprising: generating, at a frontend audio processing device, an audio signal having a first bitrate; and encoding, at the frontend audio processing device, the generated audio signal using a trained compression algorithm to obtain a compressed bit stream having a second bitrate that is lower than the first bitrate; wherein the trained compression algorithm is a lossy audio encoding algorithm configured to minimize the second bitrate of the compressed bit stream while preserving one or more predictive features of the audio signal; and wherein the compressed bit stream comprises a plurality of encoded binary activation vectors generated using the trained compression algorithm.

14. The method of claim 13, further comprising: transmitting, from the frontend audio processing device, the compressed bit stream over a transmission channel to a backend audio processing device; receiving, at a backend audio processing device, the compressed bit stream; decoding, at the backend audio processing device, the compressed bit stream to generate a decoded audio signal representation; and processing, the decoded audio signal representation using a trained predictive algorithm to obtain a physiological signal or parameter associated with an individual.

15. The method of claim 14, wherein the audio signal comprises a speech signal of the individual and the physiological signal or parameter output by the trained predictive algorithm is a predicted respiratory signal or parameter of the individual.

Images