Machine learning based encoder model for health acoustic representations

Abstract

An example method includes a computer-implemented method. The method involves receiving, by a computing device, training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data. The method also involves training, based on the training data and in a self-supervised manner, an audio encoder to predict a feature representation for an input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks. The method further involves providing, by the computing device, the trained audio encoder to generate feature representations for audio clips.

Description

MACHINE LEARNING BASED ENCODER MODEL FORHEALTH ACOUSTIC REPRESENTATIONSCROSS-REFERENCE TO RELATED APPLICATIONS[1] This present application claims priority to U.S. provisional application serial no. 63 / 537,069 filed September 7, 2023, the full disclosure of which is incorporated herein by reference.BACKGROUND[2] Neural networks can be trained to predict representations of input data. In some aspects, input data can be represented as feature vectors. Health acoustic representations can be useful for detecting and / or monitoring an individual’s health status.SUMMARY[3] Health-related acoustic signals, such as coughs and breathing sounds, are informative for medical diagnosis and continuous health monitoring. Existing machine learning approaches for health acoustics are generally trained and evaluated on specific tasks, limiting their generalizability across various healthcare applications. As described herein, a self-supervised learning framework may be configured to learn health acoustics. In some embodiments, the framework could be a generative learning based framework (e.g., a masked autoencoder), or a contrastive learning based framework (e.g., Slowfast NFNets that are ResNet50-ish networks with SimCLR loss). One aspect of optimizing the contrastive learning based framework may involve identifying effective audio augmentations. An in-depth analysis of various audio augmentation strategies may be performed to demonstrate that an appropriate augmentation strategy enhances the performance of the audio encoder across a diverse set of health acoustic tasks. The model is shown to outperform baseline models on evaluation tasks across six health acoustic datasets.[4] In a first aspect, a computer-implemented method is provided. The method involves receiving, by a computing device, training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data. The method also involves training, based on the training data and in a self-supervised manner, an audio encoder to predict a feature representation for an input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcarerelated tasks. The method further involves providing, by the computing device, the trained audio encoder to generate feature representations for audio clips.[5] In a second aspect, a computing device is provided. The computing device includes one or more processors and data storage. The data storage has stored thereon computerexecutable instructions that, when executed by one or more processors, cause the computing device to carry out functions. The functions include: receiving, by the computing device, training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data; training, based on the training data and in a self-supervised manner, an audio encoder to predict a feature representation for an input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks; and providing, by the computing device, the trained audio encoder to generate feature representations for audio clips.[6] In a third aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to carry out functions. The functions include: receiving, by the computing device, training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data; training, based on the training data and in a self-supervised manner, an audio encoder to predict a feature representation for an input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks; and providing, by the computing device, the trained audio encoder to generate feature representations for audio clips.[7] In a fourth aspect, an article of manufacture is provided. The article of manufacture includes one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions. The functions include: receiving, by the computing device, training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data; training, based on the training data and in a self-supervised manner, an audio encoder to predict a feature representation for an input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks; and providing, by the computing device, the trained audio encoder to generate feature representations for audio clips.[8] In a fifth aspect, a system is provided. The system includes means for receiving, by a computing device, training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data; means fortraining, based on the training data and in a self-supervised manner, an audio encoder to predict a feature representation for an input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks; and means for providing, by the computing device, the trained audio encoder to generate feature representations for audio clips.[9] In a sixth aspect, a computer-implemented method is provided. The method involves receiving, by a computing device, an input audio clip. The method also involves predicting, by a trained audio encoder, a feature representation for the input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks, the trained audio encoder having been trained in a self-supervised manner on training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data. The method further involves providing, by the computing device, the feature representation to a machine learning model performing detection inference for the plurality of healthcare related tasks.

[0010] In a seventh aspect, a computing device is provided. The computing device includes one or more processors and data storage. The data storage has stored thereon computerexecutable instructions that, when executed by one or more processors, cause the computing device to carry out functions. The functions include: receiving, by a computing device, an input audio clip; predicting, by a trained audio encoder, a feature representation for the input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks, the trained audio encoder having been trained in a selfsupervised manner on training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data; and providing, by the computing device, the feature representation to a machine learning model performing detection inference for the plurality of healthcare related tasks.

[0011] In an eighth aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to carry out functions. The functions include: receiving, by a computing device, an input audio clip;predicting, by a trained audio encoder, a feature representation for the input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks, the trained audio encoder having been trained in a self-supervised manner on training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data; and providing, by the computing device, the feature representation to a machine learning model performing detection inference for the plurality of healthcare related tasks.

[0012] In a ninth aspect, an article of manufacture is provided. The article of manufacture includes one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions. The functions include: receiving, by a computing device, an input audio clip; predicting, by a trained audio encoder, a feature representation for the input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks, the trained audio encoder having been trained in a self-supervised manner on training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data; and providing, by the computing device, the feature representation to a machine learning model performing detection inference for the plurality of healthcare related tasks.

[0013] In a tenth aspect, a system is provided. The system includes means for receiving, by a computing device, an input audio clip; means for predicting, by a trained audio encoder, a feature representation for the input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks, the trained audio encoder having been trained in a self-supervised manner on training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data; and means for providing, by the computing device, the feature representation to a machine learning model performing detection inference for the plurality of healthcare related tasks.

[0014] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description and the accompanying drawings.BRIEF DESCRIPTION OF THE FIGURES

[0015] FIG. 1 is an example overview of a Health Acoustic Representations (HeAR) system, in accordance with example embodiments.

[0016] FIG. 2 is a table illustrating detection yield for health acoustic events, in accordance with example embodiments.

[0017] FIG. 3 A is a table illustrating example performance comparison on health acoustic event classifications on a Freesound Dataset 50k (FSD50K), in accordance with example embodiments.

[0018] FIG. 3B is a table illustrating example performance comparison on cough relevant tasks, in accordance with example embodiments.

[0019] FIG. 4 is a table illustrating details of five evaluation datasets, in accordance with example embodiments.

[0020] FIG. 5 is a table illustrating CIDRZ cohort descriptive statistics per split, in accordance with example embodiments.

[0021] FIG. 6A illustrates a graphical representation of a scaling effect of increasing the YT-NS data size for training HeAR, in accordance with example embodiments.

[0022] FIG. 6B illustrates another graphical representation of a scaling effect of increasing the YT-NS data size for training HeAR, in accordance with example embodiments.

[0023] FIG. 6C illustrates another graphical representation of a scaling effect of increasing the YT-NS data size for training HeAR, in accordance with example embodiments.

[0024] FIG. 7 illustrates example images of applying different audio augmentation strategies on the same health acoustic sample, in accordance with example embodiments.

[0025] FIG. 8 is a table illustrating example descriptions of augmentation strategies, in accordance with example embodiments.

[0026] FIG. 9 is a table illustrating example evaluation performance for comparing combinations of augmentation strategies, in accordance with example embodiments.

[0027] FIG. 10 is a table illustrating estimated numbers of audio clips for health acoustic events, in accordance with example embodiments.

[0028] FIG. 11 is a table illustrating example evaluation dataset statistics, in accordance with example embodiments.

[0029] FIG. 12 is a table illustrating example performance comparisons on downstream tasks, in accordance with example embodiments.

[0030] FIG. 13 is another table illustrating example performance comparisons on downstream tasks, in accordance with example embodiments.

[0031] FIG. 14 is a diagram illustrating training and inference phases of a machine learning model, in accordance with example embodiments.

[0032] FIG. 15 depicts a distributed computing architecture, in accordance with example embodiments.

[0033] FIG. 16 is a block diagram of a computing device, in accordance with example embodiments.

[0034] FIG. 17 depicts a network of computing clusters arranged as a cloud-based server system, in accordance with example embodiments.

[0035] FIG. 18 is a flowchart of a method, in accordance with example embodiments.

[0036] FIG. 19 is another flowchart of a method, in accordance with example embodiments.DETAILED DESCRIPTION

[0037] This application relates, in one aspect, to a system for learning health acoustic representations. The system may include two components: a health acoustic event detector and an audio encoder. The system can allow audio clips with various lengths as inputs, and outputs the corresponding audio embeddings for downstream task use.

[0038] Non-semantic speech sounds from humans may include different kinds of respiratory sounds such as crying, coughing, sneezing, snoring, and so forth. As described herein, such sound patterns from the respiratory system may be used to train a large machine learning based model that can encode the sound patterns a vector representation, and then use that representation to predict a healthcare related task, such as a diagnostic and / or a health monitoring task, as a downstream task.Overview

[0039] Health acoustic sounds such as coughs and breaths are known to contain useful health signals with significant potential for monitoring health and disease, yet are underexplored in the medical machine learning community. The existing deep learning systems for health acoustics are often narrowly trained and evaluated on a single task, which is limited by data and may hinder generalization to other tasks. To mitigate these gaps, HeAR, as described herein, is a scalable self-supervised learning-based deep learning system using masked autoencoders trained on a large dataset of two-second long audio clips. Through linearprobes, a Health Acoustic Representations (HeAR) system is presented as a state-of-the-art health audio embedding model on a benchmark of 20 health acoustic tasks across five (5) datasets.

[0040] Acoustic non-semantic attributes of speech may enable machine learning models to perform paralinguistic tasks, including emotion recognition, speaker identification, and dementia detection. Cerebrovascular and neurodegenerative diseases like stroke, Parkinson’s, Alzheimer’s, cerebral palsy and amyotrophic lateral sclerosis (ALS) may also be detected and monitored using non-semantic patterns of speech, such as articulation, resonation, and phonation. Non-semantic acoustic signals related to health are not confined solely to conversational speech data. Health-related acoustic cues, originating from human respiratory system’s airflow, including sounds like coughs and breathing patterns may be harnessed for health monitoring purposes. For example, clinicians use sounds such as “whoop” -like coughing to diagnose pertussis, and agonal breathing for detecting acute cardiovascular events. Such health sounds can also be collected via ambient sensing technologies on ubiquitous devices such as mobile phones, which may augment healthcare workers in low- medium income countries (LMICs) with improved screening capabilities.

[0041] With advancements in deep learning, neural networks may be designed to learn high-quality general representations directly from raw speech data, and use them for various semantic and non-semantic speech-related tasks. Health acoustics, specifically non-semantic respiratory sounds, also have potential as biomarkers to detect various respiratory diseases. However, current machine learning (ML) systems for health acoustics are task-specific and may not generalize well to out-of-distribution (OOD) settings and are often limited by data quantity.

[0042] Self-supervised learning (SSL) has demonstrated potential for building robust and capable systems by learning general representations from large, unlabeled sources. There is some progress on learning general, universal representations in vision, language and speech. SSL can have numerous training objectives including contrastive, such as SimCLR, BYOL, and generative, like masked autoencoder. Such approaches are also used for learning representations of biomedical language, medical images and even physiological waveforms. However, these approaches are still underexplored in the field of health acoustics. Data-driven audio SSL approached have made progress, specifically for semantic speech. There are also approached focusing on non-semantic speech. For example, TRILL uses a triplet loss as a training objective, TRILLsson and FRILL further distill the TRILL encoder to make it smallerand faster. Researchers also adopted different neural network architectures, such as Conformer, and Slowfast NFNet to develop performant audio encoders.

[0043] Some approaches use respiratory sounds for health monitoring and disease detection. For example, cough sound patterns can be used as a biomarker to identify coughers, detect various respiratory diseases, such as COVID-19, bronchitis, bronchiolitis, pertussis, obstructive versus restrictive lung diseases, and tuberculosis (TB). However, these approaches are based on developing a single task-specific system trained in a supervised learning framework that may not generalize as well to new settings.

[0044] As described herein, HeAR is a self-supervised generative learning-based system trained on a large dataset of two-second long audio clips for learning low-dimensional representations that can transfer well across health acoustic tasks and generalize to OOD data. HeAR may be benchmarked on a diverse set ofhealth acoustic tasks spanning thirteen (13) health acoustic event detection tasks and seven cough relevant tasks across five datasets and demonstrate that simple linear classifiers trained on top of our representations outperform the state-of-the-art on many tasks.

[0045] A full network is described, including a health acoustic event detector and an audio encoder described herein. A multilabel sound event classification is described. The labels can relate to audio for coughing, baby coughing, breathing, throat cleaning, laughing, and non- semantic speech. In some embodiments, the health acoustic event detector can be a convolutional neural network (CNN). In some embodiments, multiple interconnected separable convolution layers may be utilized. Also, for example, the CNN may be optimized by balanced binary cross entropy.

[0046] In some embodiments, the trained health acoustic event detector model may be converted to a low latency, portable model (e.g., a TFLite model). Different detection thresholds may be applied for different event classes. In some embodiments, the audio encoders may be of two types: a contrastive learning-based encoder or a generative learning-based encoder.

[0047] In some embodiments, a combination of audio augmentation strategies may be identified in the SimCLR encoder. For example, the audio augmentation strategies may include applying a simple data augmentation method for speech recognition (e.g., SpecAugment). SpecAugment may be applied directly to the feature inputs of a neural network. The augmentation may involve warping the features, masking blocks of frequency channels, and / or masking blocks of time steps. Some embodiments involve applying a Brownian tape speedalong with SpecAugment. Some embodiments involve applying a circular time shift, followed by a time stretch.

[0048] In some embodiments, contrastive loss may be used to learn the representations.

[0049] In the masked autoencoder (MAE), approximately 75% of inputs may be masked, and a pixelwise reconstruction loss may be used to predict the missing part of the input and compare against the ground truth. Also, for example, in the MAE, learnable mask tokens may be added to the sequence of encoded tokens, and a multi-layered (e.g., 8 layers) transformer decoder can be configured to reconstruct the missing patches, by minimizing the / distance between normalized masked patches and its predictions.

[0050] In some embodiments, the MAE model may be optimized using AdamW, using an exponentially decaying learning rate ( / IO every epoch). Generally, an AdamW optimization refers to a stochastic gradient descent method that involves adaptive estimation of first-order and higher order (e.g., second-order) moments with decaying weights.

[0051] In some embodiments, the quality of representations may be evaluated throughout training using linear probing, implemented using the stop gradient operator. This enables evaluation of the performance of the representation on the validation splits of the downstream evaluation tasks.

[0052] In some embodiments, the HeAR system may be trained on a collected large web scale video dataset of non-semantic (NS) audio tracks (e.g., two-second long audio clips of random one billion YouTube (YT) videos including non-semantic (YT-NS) audio clips).

[0053] Some embodiments involve generating the training dataset. In some embodiments, the system may be trained in a self-supervised manner. In some embodiments, the system may allow audio clips with various lengths as inputs, and outputs the corresponding audio embeddings for downstream task use.Example HeAR systems

[0054] FIG. 1 is an example overview of a HeAR system 100, in accordance with example embodiments. The HeAR system 100 may be trained on an audio database 105, such as, for example, a curated dataset, YouTube Non-Semantic (YT-NS), consisting of two-second long audio clips extracted from one billion non-copyrighted YOUTUBE™ videos using the health acoustic event detector 115, totaling 341 million two-second clips or 189k hours of audio. The HeAR audio encoder may be trained on this dataset. The audio encoder may be trained on YT- NS, without the requirement of human and / or expert data curation.

[0055] Some embodiments involve generating the training dataset. For example, audio clips that are considered to include non-semantic audio, such as sounds emanating from a human respiratory system, may be identified and extracted. Accordingly, the training dataset may include shorter audio clips that include the relevant non-semantic audio. Such a training dataset may be generated in a self-supervised, and / or unsupervised, manner.

[0056] The health acoustic event detector 115 may resample the audio to mono channel (e.g., a 16 kHz or 32 kHz sample rate), then crop the audio into two-second long health acoustic clips 120 that include log-mel spectrogram features (e.g., with 48 frequency bins ranging from 125Hz to 7.5 kHz) with per-channel energy normalization (PCEN). In situations where the health acoustic event detector 115 is adapted for use on mobile devices, an optional resampling may be performed to generate lower quality images.

[0057] In some embodiments, the health acoustic event detector 115 is a multilabel classification convolutional neural network (CNN) that identifies the presence of any of six types of non-speech health acoustic events in two-second long health acoustic clips 120: coughing, baby coughing, breathing, throat clearing, laughing, and speaking (non-semantic).

[0058] The health acoustic event detector 115 may be trained on two publicly available FSD50K and Flusense datasets, and another health acoustic dataset. FSD50K has over 50K audio clips (over 100 hours) annotated using AudioSet ontology, and Flusense is the subset of AudioSet dataset including sounds related to flu illnesses, which has six labels: cough, sneeze, sniffle, throat clearing, speech, and “etc.” (everything else). The private dataset is collected from a variety of sources. The health acoustic event detector 115 may use the audio clips with labels such as “throat clearing”, “coughing”, “sneezing”, and “breathing” for training.

[0059] The health acoustic event detector 115 first converts and resamples the audio to mono channel 16 kHz sampling rate, then crops the audio into two-second log-mel spectrogram features with 48 frequency bins ranging from 125Hz to 7.5kHz with per-channel energy normalization (PCEN). These features may be passed into a small convolutional neural network (CNN). The loss may be balanced binary cross entropy, and the output of CNN may be the logits for each prediction class.

[0060] FIG. 2 is a table 200 illustrating detection yield for health acoustic events, in accordance with example embodiments. For example, a yield is provided for each sound type. The term “yield” as used herein, corresponds to a number of two-second audio clips.

[0061] These two-second long health acoustic clips 120 may be passed into a CNN (e.g., with eight interconnected separable convolution layers). The CNN can be optimized by balanced binary cross entropy, and the output of CNN may be the logits for each predictionclass. The trained model may be converted to a low latency, portable TFLite model. A different detection threshold may be applied for each event class. Generally, the health acoustic event detector 115 identifies the audio that may be relevant to the downstream healthcare related tasks. Such healthcare related tasks can include detection of an illness (e.g., a respiratory illness), monitoring a patient’s health, and so forth. The healthcare related tasks can also include detecting a potential exposure to smoke, a cough disambiguation task, and so forth.

[0062] In some embodiments, an audio encoder 125 (e.g., an MAE-based audio encoder) may be used to learn audio representations by training an autoencoder to reconstruct masked 16 x 16 spectrogram patches. For example, 75% of inputs (e.g., two-second log-mel spectrograms) may be masked out, and pixelwise reconstruction loss may be used to predict the missing part of the input, and subsequently compared against the ground truth. In some embodiments, learnable mask tokens may be added to the sequence of encoded tokens, and an 8-layer transformer decoder may be tasked with reconstructing the missing patches, by minimizing the L2distance between normalized masked patches and its predictions. The model may be trained using an AdamW optimizer, and appropriate hyperparameters. Generally, a learning rate may follow an exponential decay schedule ( / IO every epoch). Checkpoint selection may be performed using an early stopping criterion on the validation loss. The encoded audio representations may be provided to a decoder 130 (e.g., an MAE-based decoder).

[0063] FIG. 2 is a table 200 illustrating example evaluation dataset statistics and datasetspecific preprocessing metrics, in accordance with example embodiments. The quality of representations may be evaluated during the training phase using linear probing (e.g., implemented using the stop gradient operator). This enables evaluation of the performance of the representation on the validation splits of the downstream evaluation tasks.

[0064] In some embodiments, the quality of representations may be evaluated throughout training using linear probing, implemented using the stop gradient operator. This enables evaluation of the performance of the representation on the validation splits of the downstream evaluation tasks. In some embodiments, task-specific evaluation may be performed. For example, two-second audio clips 140 may be extracted from a dataset of evaluation audio 135. Health acoustic event detector 145 may share one or more aspects in common with health acoustic event detector 115 and may generate two-second long acoustic clips 150. The two- second long acoustic clips 150 may be provided to a trained audio encoder 155 (e.g., a trained CNN) to generate embeddings 160. Embeddings 160 are then provided for downstreamhealthcare related tasks 165. As previously indicated, such healthcare related tasks can include detection of an illness (e.g., a respiratory illness), monitoring a patient’s health, and so forth. The healthcare related tasks can also include detecting a potential exposure to smoke, a cough disambiguation task, and so forth.

[0065] For example, to evaluate tuberculosis (TB) prediction as an evaluation task, weights of the trained audio encoder may be frozen for the task of TB prediction and then used to generate output vector embeddings. These embedding may then be used for the comparison / evaluation task. The weights inside the audio encoder can be optimized (finetuned) using the particular task and run on the downstream TB detection task. Such an approach generally provides better results on the data specific task. Although TB detection was used for illustrative purposes, the trained audio encoder may be fine-tuned for other healthcare related tasks as well. As described herein, the trained audio recorder is not biased toward any particular kind of task.

[0066] The HeAR system 100 may be compared with several state-of-the-art models for learning health acoustic representations, including a TRipLet Loss network (TRILL), a fast TRILL (FRILL), and a Big Semi-Supervised Learning (BigSSL) model. The TRILL, FRILL and BigSSL are generally trained on speech data. HeAR, trained on non-semantic acoustic data, generally outperforms and / or is comparable to other baseline audio encoders. For example, comparisons may be performed on several human acoustic event classification tasks, and cough-specific classification tasks across multiple datasets. As illustrated herein, a model trained on non-semantic sounds works better for non-semantic tasks.

[0067] HeAR may be benchmarked both on general health acoustic event classification and on cough relevant tasks. For general health acoustic event classification, a Freesound Dataset 50k (FSD50K) and Flusense may be used. For cough specific evaluation, Cough VID, and Coswara, and a prospective, tuberculosis-specific dataset from the Centre for Infectious Disease Research in Zambia (CIDRZ) may be used. For CIDRZ, to obtain audio from microphones of varying quality, three devices may be used to collect the cough audio, representing three mobile phone tiers with different costs and potentially different recording quality: a low-tier (PIXEL™ 3a), a mid-tier (GALAXY™ A12), and a high-tier (GALAXY™ A22). The datasets may be referred to herein as CIDRZ low-tier, CIDRZ mid-tier, and CIDRZ high-tier, respectively.

[0068] FIG. 3A is a table 300A illustrating example performance comparison on health acoustic event classifications on a Freesound Dataset 50k (FSD50K), in accordance with example embodiments. For example, performance comparison may include AUROC with 95%confidence intervals. “All Respiratory sounds” in FSD50K is the superset class of other five FSD50K classes in table 300A.

[0069] FIG. 3B is a table 300B illustrating example performance comparison on cough relevant tasks, in accordance with example embodiments. For example, when evaluated on cough tasks, HeAR performed on par with or better than other baseline audio embedding models. Specifically, HeAR performed better than TRILL, FRILL and BigSSL-CAP12 in this study at COVID-related health status prediction on Coswara dataset and TB prediction on CIDRZ low to mid-tier phones; while BigSSL-CAP12 appears to have better performance at COVID prediction and TB prediction on CIDRZ high-tier phones. HeAR is in general good at smoking status and sex classifications, and comparable with BigSSL-CAP12 for age and BMI, from cough sounds across different datasets and devices. HeAR and BigSSL models appear to perform better than TRILL and FRILL across tasks except age on CIDRZ high-tier phones.

[0070] FIG. 4 is a table 400 illustrating details of five evaluation datasets, in accordance with example embodiments. The CIDRZ TB dataset is collected by the Centre for Infectious Disease Research in Zambia. The study was approved by the University of Zambia Biomedical Ethics Committee, and all participants provided written informed consent prior to enrollment in the study. Adults who had symptoms suggestive of tuberculosis, were identified as close contacts of tuberculosis patients, or were newly diagnosed with HIV were recruited at three clinical sites (Chawama, Chainda-South, and Kanyama) in Zambia (trial NCT05139940). Audio recordings of cough sounds were obtained from 599 consenting patients. To ensure robustness across different microphones, the sounds were recorded by four devices: ZOOM H2N microphone (high quality audio recorder), SAMSUNG GALAXY A22 (high-tier phone), SAMSUNG GALAXY A12 (mid-tier phone), and PIXEL 3a (low-tier phone). The audio clips were recorded by an audio recorder and encoded in the wav file format with 24-bit PCM, sampling rate of 192 kHz, and in stereo channels under a quiet environment.

[0071] To collect cough sounds, the participant was asked to remove his / her mask and generate four cough events (three single coughs and one sequence of multiple coughs). There may be a 10-15 seconds gap between cough events to enable a return to “baseline” before the next cough.

[0072] FIG. 5 is a table 500 illustrating CIDRZ cohort descriptive statistics per split, in accordance with example embodiments.

[0073] FIGs. 6A-C illustrate graphical representations of a scaling effect of increasing the YT-NS data size for training HeAR, in accordance with example embodiments. For example, scaling up the training data (YT-NS) used for training the HeAR audio encoder may helpimprove the linear probing performance across different downstream tasks. Graph 600A of FIG. 6 A illustrates the scaling effect for Cough VID sex classification. Graph 600B of FIG. 6B illustrates the scaling effect for Coswara sex classification. Graph 600C of FIG. 6C illustrates the scaling effect for CIDRZ tuberculosis prediction tasks.Optimization of Audio Augmentations

[0074] Health-related acoustic signals, such as cough and breathing sounds, are relevant for medical diagnosis and continuous health monitoring. Many existing machine learning approaches for health acoustics are trained and evaluated on specific tasks, limiting their generalizability across various healthcare applications. As described herein, a self-supervised learning framework, SimCLR with a Slowfast NFNet backbone, may be used for contrastive learning of health acoustics. One aspect of optimizing Slow-fast NFNet for this application lies in identifying effective audio augmentations. Various audio augmentation strategies may be described and an appropriate augmentation strategy may be identified that enhances the performance of the Slowfast NFNet audio encoder across a diverse set of health acoustic tasks. Generally speaking, when augmentations are combined, synergistic effects may be produced that exceed the benefits seen when each is applied individually.

[0075] Non-speech, non-semantic sounds, like coughing and breathing, may provide information for doctors to detect various respiratory diseases, cardiovascular diseases and neurological diseases. Advances in deep learning-based machine learning (ML) enables development of medical assistants and continuous health monitoring applications by learning effective acoustic data representations.

[0076] Current approaches for learning health acoustic representations are mostly trained and evaluated on specific tasks. For example, models may be trained to detect tuberculosis using cough sounds via supervised learning. However, it can be challenging to adopt such models directly for other acoustic health tasks. Retraining task specific health acoustic models requires manual data collection and labeling by clinical experts, which can be time consuming and costly. Some approaches involving contrastive learning have proven effective for generating robust representations across multiple data modalities, including images, videos, speech, audio, and periodic data. Significant research has been conducted on the utility of various augmentations for images, videos, and speech / audio. However, the unique characteristics of health-related acoustic signals, such as coughs and breathing sounds, which differ in pitch and tone from speech and music, raise questions about the applicability of existing contrastive learning and augmentation strategies in this specialized domain.

[0077] The approach described herein explores eight distinct audio augmentation techniques and their combinations in the context of health acoustic representation learning. In some embodiments, a self-supervised contrastive learning framework, SimCLR, may be used, with a Slowfast NFNet backbone. After identifying a suitable combination of augmentations, the performance of the resulting Slowfast NFNet may be compared against other state-of-the- art off-the-shelf audio encoders on 21 unique binary classification tasks across five datasets. Augmentation parameters that work optimally when applied to health acoustics may be identified, and synergistic effects of combining audio augmentations for enhancing health acoustic representations using SimCLR may be determined.

[0078] Data augmentation serves as a regularization technique to mitigate the risk of model over-fitting. Within the framework of contrastive learning, the objective may be to learn data representations that minimizes the distance between representations of semantically similar inputs and maximizes the distance between representations of semantically dissimilar inputs. Data augmentations are critical for contrastive learning-based self-supervised learning (SSL), and eliminates the need for labeled data for representation learning. By applying a variety of augmentations to a single input, semantically consistent but distinct variations, commonly referred to as views, may be generated.

[0079] The task then amounts to bringing these related views closer together in the representational space, while concurrently pushing views derived from different, unrelated inputs farther apart, via a contrastive loss, such as InfoNCE in SimCLR. Such an approach may establish a form of invariance in the model, rendering it robust to the augmentations applied during the training process. Augmentations have been widely explored as part of contrastive learning -based SSL methods such as SimCLR, BYOL, MoCo, and SwAV. Data augmentations also enhance the performance of SSL methods broadly across different data modalities, including images, videos, audio, speech, and 1 -dimensional signals (e.g., human physiological signals). By contrast, an application of data augmentation strategies for contrastive learning of health acoustic signals is described herein.

[0080] Existing approaches involve generating two augmented speech signals with speeds relative to the original of 0.9 and 1.1. This may yield performance improvements across four speech recognition tasks. This strategy may be expanded upon this by introducing a triplet loss for audio representation learning, incorporating random noise, time / frequency translation, example mixing, and temporal proximity augmentations. Another existing approach may involve an adaptation of SimCLR for speech data, termed Speech SimCLR, where a diverse set of augmentations such as random pitch shift, speed perturbation, room reverberation andadditive noise to the original waveform, as well as time and frequency masking to the spectrogram, may be used. Also, for example, an audio augmentation module including prenormalization, foreground acoustic event mixup, random resize cropping and postnormalization may be designed. A multi-modal approach may involve adopting augmentations from both vision and audio domains, including random resized cropping, random time / frequency shifts, compression, SpecAugment, Gaussian noise addition, and Gaussian blurring. Sound separation techniques may be utilized for sound event detection to enable targeted data augmentations. Another approach involves noise injection as an augmentation strategy to bolster the robustness of speech models. CLAR identified six augmentation operations: pitch shift, noise injection in frequency domain, and fade in / out, time masking, time shift, time stretching in temporal domain, and explored their utility for audio contrastive learning. Described herein is an improved approach that identifies an optimal combination and sequence of augmentation strategies, with a specific focus on developing robust representations for health acoustics.

[0081] In some embodiments, one or more phases may be used to determine augmentation parameters. One phase (Phase 1) may include finding the best parameters for each augmentation for use with SimCLR. In another phase (Phase 2), various combinations of augmentations may be analyzed, where one or two successive augmentations may be combined to create a view of the input. The augmentation parameters may be the same as Phase 1. In another phase (Phase 3), the results of an optimally performing model may be compared to other state-of-the-art audio encoder models on the validation set used for comparing augmentations. In some embodiments, an evaluation may involve 21 unique downstream tasks across five datasets and the quality of embeddings generated from each audio encoder may be analyzed using linear probing. For example, a SimCLR model with a 63 million parameter SlowFast NFNet as the neural network backbone may be used.

[0082] Several types of augmentation strategies for health acoustics may be deployed. For example, a circular time shift, Brownian tape speed, random interval masking (e.g., temporal transformations), random noising, Gaussian smoothing, and random scaling (e.g., frequency transformations) may be used.

[0083] FIG. 7 illustrates example images 700 of applying different audio augmentation strategies on the same health acoustic sample, in accordance with example embodiments. Mel spectrograms generated from various augmentations applied to the same health acoustic sample are shown. One two-second example from the Cough VID dataset may be acquired and modified by each augmentation method. In some embodiments, eight (8) augmentations maybe used. These may include time-domain augmentations such as crop and pad, noising, Brownian tape speed, scaling, pitch shift, time stretch, and circular time shift. Additionally, SpecAugment may be applied after the transformation of audio inputs into spectrograms.

[0084] FIG. 8 is a table 800 illustrating example descriptions of augmentation strategies, in accordance with example embodiments. For example, the first row of table 800 describes a crop and pad augmentation strategy which may be applied temporally. The strategy may involve cropping an audio signal and zero-padding to the input length. The parameters may include a probability of 1.0, a minimum fraction of 0.1, and a maximum fraction of 0.5. Also, for example, various associated grid search parameters are provided. Generally, up to a certain threshold, more intense augmentation parameters may yield better performance results.

[0085] Each of the augmentations may be associated with a tunable parameter space to allow for varying degrees of transformational intensity. To identify the optimal hyperparameters for each specific augmentation, a grid search may be performed. After determining the best augmentation parameters, various potential synergistic effects from the sequential application of either one or two successive augmentations may be analyzed. With eight (8) augmentations, experimenting with every permutation of one or two augmentations would result in 64 experiments. In some embodiments, SpecAugment may be applied after the time domain augmentations which reduced the number of 2-step augmentations to 57.

[0086] FIG. 9 is a table 900 illustrating example evaluation performance for comparing combinations of augmentation strategies, in accordance with example embodiments. For example, single augmentations 905 are illustrated and two augmentations 910 applied where rows represent the first augmentation and columns represent the second augmentation. As illustrated, many augmentations may perform better in combination than individually. The analysis indicates that the optimally effective single augmentation strategy may be SpecAugment. The optimally effective 2- step augmentation strategy may involve applying a circular time shift, followed by time stretch. Circular time shift does not perform well on its own and each of these augmentations individually may underperform SpecAugment. However, circular time shift and time stretch appear to be synergistic when applied in combination. On average, time stretch may be a useful first augmentation, excluding SpecAugment (which may be applied second or alone). Also, for example, SpecAugment appears to be a useful second augmentation on average.

[0087] The dataset may include a curated dataset such as a YouTube Non-Semantic (YT- NS) dataset consisting of two-second long audio clips extracted from one billion noncopyrighted YOUTUBE™ videos totaling 255 million two-second clips or 142k hours ofaudio. In some embodiments, a convolutional neural network-based health acoustic detector model may be applied on two public health acoustic AudioSet derivatives, FSD50K and Flusense, as well as another health acoustic dataset. The model may be used to filter two-second audio clips from the one billion videos for health acoustic signals such as coughs, speech, laughing, throat-clearing, baby coughs, and breathing. In some embodiments, the Slowfast NFNet encoder may be trained using this dataset.

[0088] FIG. 10 is a table 1000 illustrating estimated numbers of audio clips for health acoustic events, in accordance with example embodiments. For example, the health acoustic event “cough” corresponds to 77,000,000 audio clips.

[0089] In some embodiments, five (5) publicly available datasets, FSD50K, Flusense, PSG, Cough VID, and Coswara may be used for evaluation.

[0090] FIG. 11 is a table 1100 illustrating example evaluation dataset statistics, in accordance with example embodiments. For example, the first row corresponds to the FSD50K dataset and includes health acoustics events with six (6) tasks. The number of examples per training linear probes are 32,652, and the number of examples for evaluation are 8,313.

[0091] Evaluation may be performed based on twenty-one (21) unique downstream binary classification tasks across five ( 5 ) datasets to evaluate the quality of health acoustic representations generated from the learned audio encoders, including thirteen (13) human acoustic event classifications from two datasets, FSD50K and Flusense, five (5) sleep apneaspecific tasks, and three (3) cough relevant tasks. The cough tasks may include COVID detection, sex classification, and smoking status classification.

[0092] For Phases 1 and 2 optimal parameters may be identified for each augmentation, as well as an optimal combination of augmentations. In some embodiments, a composite score may be determined that aggregates performance across various downstream tasks. The PSG, Cough Vid, and Coswara datasets may be segmented into two-second clips. For Flusense, the data may be pre- processed by segmenting variable length clips using labeled timestamps. For FSD50K and Flusense, a lightweight evaluation strategy may be used by randomly sampling a single two second long clip from each clip. For example, an average area under the receiver operating characteristic curve (AUROC) may be determined across the tasks and such a composite measure may be used to rank augmentation strategies.

[0093] For Phase 3, the PSG data may be segmented into 10 second clips and for FSD50K and Flusense, clips may be cropped and / or zero padded to 10 seconds. In some embodiments, a sliding window approach may be applied for FSD50K, Flusense, and PSG, where embeddings may be generated for two-second windows with a step size of one second. Also,for example, mean pooling may be applied to the resulting embeddings to generate an output embedding.

[0094] In some embodiments, linear probing may be applied to evaluate the quality of the generated representations. Logistic regression with cross-validated ridge penalty may be used, which is trained to predict binary labels from the frozen precomputed embeddings. AUROC may be determined for the tasks and a DeLong method may be used to compute 95% confidence intervals (Cis).

[0095] For comparative evaluation, several audio encoders (e.g., off-the-shelf audio encoders) may be used, each trained on semantic or non-semantic speech data. Specifically, the baseline models may include TRILL, a publicly available ResNet50 architecture trained on an AudioSet subset that is enriched with speech labels. Another baseline model may be FRILL, a light-weight MobileNet-based encoder distilled from TRILL. Another baseline model may be BigSSL-CAP12 that leverages a Conformer-based architecture, trained on YouTube and LibriLight.

[0096] FIG. 12 is a table 1200 illustrating example performance comparisons on downstream tasks, in accordance with example embodiments. For example, performance comparison (AUROC with 95% confidence intervals) on downstream tasks in FSD50K, Flusense and PSG datasets are illustrated. In table 1200, OSA is an acronym for obstructive sleep apnea.

[0097] FIG. 13 is another table 1600 illustrating example performance comparisons on downstream tasks, in accordance with example embodiments. For example, performance comparison (AUROC with 95% confidence intervals) on downstream tasks in Cough VID and Coswara datasets are illustrated.

[0098] FIGs. 12 and 13 demonstrate performance of an optimal SimCLR model versus the baseline models on the validation set used for the comparison of augmentations. Overall, the performance of the SimCLR model appears to be similar to BigSSL-CAP12, despite training on about lOx less hours of data and using a model that is nearly lOx smaller, and outperforms off-the-shelf audio encoders.Example Applications

[0099] A wide range of health acoustic event related, and / or cough-specific encodings may be generated (e.g., 32 binary classification tasks). For example, the encodings may enable thirteen (13) health acoustic event detection tasks, five (5), apnea and arousal tasks, and seven (7) unique cough relevant tasks.

[0100] Model distillation may be performed for real-world deployment, especially for edge devices.

[0101] For example, a small, lower power model may be configured to to run on a mobile device. The foundation model described herein (e.g., the detector, and / or the encoder) may be a large model that may not run efficiently on a mobile device. Accordingly, a student-teacher relationship may be used whereby the larger model transfers learning to a smaller model. An output of the smaller model matches an output of the larger model. The smaller model distills knowledge from the larger model without actually having to be trained on the training dataset.

[0102] The distilled model can be deployed on a mobile device such as, for example, a wearable device (e.g., smart watch), a mobile phone, and so forth.

[0103] Some downstream healthcare related tasks can include detection of an illness (e.g., a respiratory illness), monitoring a patient’s health, and so forth. The healthcare related tasks can also include detecting a potential exposure to smoke (e.g., from a fire), a cough disambiguation task, and so forth.

[0104] The system capabilities may be expanded beyond general health acoustics and cough detection. For example, additional acoustic patterns for more health conditions such as breathing patterns (e.g., agonal breathing for cardiovascular emergencies, abnormal sound for asthma exacerbation), and non-semantic speech patterns (e.g., for neurodegenerative or cerebrovascular diseases) may be used, and the system may be applied to other health acoustic applications such as cough disambiguation.

[0105] Non-semantic audio data, such as from products (e.g., machines, automobiles, computing devices, storage devices, fans, and so forth), may be analyzed and embeddings can be generated for downstream detection of malfunctioning devices.

[0106] These and other example applications are contemplated within a scope of this disclosure.Training Machine Learning Models for Generating Inferences / Predictions

[0107] FIG. 14 shows diagram 1400 illustrating a training phase 1402 and an inference phase 1404 of trained machine learning model(s) 1432, in accordance with example embodiments. Some machine learning techniques involve training one or more machine learning algorithms, on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in the) training data. The resulting trained machine learning algorithm can be termed as a trained machine learning model. For example, FIG. 14 shows training phase 1402 where one or more machine learningalgorithms 1420 are being trained on training data 1410 to become trained machine learning model(s) 1432. Then, during inference phase 1404, trained machine learning model(s) 1432 can receive input data 1430 and one or more inference / prediction requests 1440 (perhaps as part of input data 1430) and responsively provide as an output one or more inferences and / or prediction(s) 1450.

[0108] As such, trained machine learning model(s) 1432 can include one or more models of one or more machine learning algorithms 1420. Machine learning algorithm(s) 1420 may include, but are not limited to: an artificial neural network (e.g., a herein-described convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system). Machine learning algorithm(s) 1420 may be supervised or unsupervised, and may implement any suitable combination of online and offline learning.

[0109] In some examples, machine learning algorithm(s) 1420 and / or trained machine learning model(s) 1432 can be accelerated using on-device coprocessors, such as graphic processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application specific integrated circuits (ASICs). Such on-device coprocessors can be used to speed up machine learning algorithm(s) 1420 and / or trained machine learning model(s) 1432. In some examples, trained machine learning model(s) 1432 can be trained, reside and execute to provide inferences on a particular computing device, and / or otherwise can make inferences for the particular computing device.

[0110] During training phase 1402, machine learning algorithm(s) 1420 can be trained by providing at least training data 1410 as training input using unsupervised, supervised, semisupervised, and / or reinforcement learning techniques. Unsupervised learning involves providing a portion (or all) of training data 1410 to machine learning algorithm(s) 1420 and machine learning algorithm(s) 1420 determining one or more output inferences based on the provided portion (or all) of training data 1410. Supervised learning involves providing a portion of training data 1410 to machine learning algorithm(s) 1420, with machine learning algorithm(s) 1420 determining one or more output inferences based on the provided portion of training data 1410, and the output inference(s) are either accepted or corrected based on correct results associated with training data 1410. In some examples, supervised learning of machine learning algorithm(s) 1420 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or set of labels may be used to correct inferences of machine learning algorithm(s) 1420.[Hl] Semi-supervised learning involves having correct results for part, but not all, of training data 1410. During semi-supervised learning, supervised learning is used for a portion of training data 1410 having correct results, and unsupervised learning is used for a portion of training data 1410 not having correct results. Reinforcement learning involves machine learning algorithm(s) 1420 receiving a reward signal regarding a prior inference, where the reward signal can be a numerical value. During reinforcement learning, machine learning algorithm(s) 1420 can output an inference and receive a reward signal in response, where machine learning algorithm(s) 1420 are configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, machine learning algorithm(s) 1420 and / or trained machine learning model(s) 1432 can be trained using other machine learning techniques, including but not limited to, incremental learning and curriculum learning.

[0112] In some examples, machine learning algorithm(s) 1420 and / or trained machine learning model(s) 1432 can use transfer learning techniques. For example, transfer learning techniques can involve trained machine learning model(s) 1432 being pre-trained on one set of data and additionally trained using training data 1410. More particularly, machine learning algorithm(s) 1420 can be pre-trained on data from one or more computing devices and a resulting trained machine learning model provided to computing device CD1, where CD1 is intended to execute the trained machine learning model during inference phase 1404. Then, during training phase 1402, the pre-trained machine learning model can be additionally trained using training data 1410, where training data 1410 can be derived from kernel and non-kernel data of computing device CD1. This further training of the machine learning algorithm(s) 1420 and / or the pre-trained machine learning model using training data 1410 of CDl’s data can be performed using either supervised or unsupervised learning. Once machine learning algorithm(s) 1420 and / or the pre-trained machine learning model has been trained on at least training data 1410, training phase 1402 can be completed. The trained resulting machine learning model can be utilized as at least one of trained machine learning model(s) 1432.

[0113] In particular, once training phase 1402 has been completed, trained machine learning model(s) 1432 can be provided to a computing device, if not already on the computing device. Inference phase 1404 can begin after trained machine learning model(s) 1432 are provided to computing device CD1.

[0114] During inference phase 1404, trained machine learning model(s) 1432 can receive input data 1430 and generate and output one or more corresponding inferences and / orprediction(s) 1450 about input data 1430. As such, input data 1430 can be used as an input to trained machine learning model(s) 1432 for providing corresponding inference(s) and / or prediction(s) 1450 to kernel components and non-kemel components. For example, trained machine learning model(s) 1432 can generate inference(s) and / or prediction(s) 1450 in response to one or more inference / prediction requests 1440. In some examples, trained machine learning model(s) 1432 can be executed by a portion of other software. For example, trained machine learning model(s) 1432 can be executed by an inference or prediction daemon to be readily available to provide inferences and / or predictions upon request. Input data 1430 can include data from computing device CD1 executing trained machine learning model(s) 1432 and / or input data from one or more computing devices other than CD1.

[0115] Input data 1430 can include a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data. The collection of images can include video frames, images resident on computing device CD1, and / or other images. Other types of input data are possible as well.

[0116] Inference(s) and / or prediction(s) 1450 can include output images, output intermediate images, numerical values, and / or other output data produced by trained machine learning model(s) 1432 operating on input data 1430 (and training data 1410). In some examples, trained machine learning model(s) 1432 can use output inference(s) and / or prediction(s) 1450 as input feedback 1460. Trained machine learning model(s) 1432 can also rely on past inferences as inputs for generating new inferences.

[0117] A neural network comprising a detector and / or an audio encoder can be an example of machine learning algorithm(s) 1420. After training, the trained version of the neural network can be an example of trained machine learning model(s) 1432. In this approach, an example of the one or more inference / prediction request(s) 1440 can be a request to predict a feature representation for input data and a corresponding example of inferences and / or prediction(s) 1450 can be a predicted feature representation for the input data.

[0118] In some examples, one computing device CD SOLO can include the trained version of the detector and / or an audio encoder, perhaps after training. Then, computing device CD SOLO can receive a request to predict a feature representation for input data, and use the trained version of the neural network to predict the feature representation for the input data.

[0119] In some examples, two or more computing devices CD CLI and CD SRV can be used to provide feature representations; e.g., a first computing device CD CLI can generate and send requests to predict a feature representation for input data to a second computing device CD SRV. Then, CD SRV can use the trained version of the neural network, to generate thefeature representation, and respond to the requests from CD CLI. Then, upon reception of responses to the requests, CD CLI can provide the requested feature representation (e.g., using a user interface and / or a display, a printed copy, an electronic communication, etc.).Example Data Network

[0120] FIG. 15 depicts a distributed computing architecture 1500, in accordance with example embodiments. Distributed computing architecture 1500 includes server devices 1508, 1510 that are configured to communicate, via network 1506, with programmable devices 1504a, 1504b, 1504c, 1504d, 1504e. Network 1506 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public Internet, or any other type of network configured to provide a communications path between networked computing devices. Network 1506 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public Internet.

[0121] Although FIG. 15 only shows five programmable devices, distributed application architectures may serve tens, hundreds, or thousands of programmable devices. Moreover, programmable devices 1504a, 1504b, 1504c, 1504d, 1504e (or any additional programmable devices) may be any sort of computing device, such as a mobile computing device, desktop computer, wearable computing device, head-mountable device (HMD), network terminal, a mobile computing device, and so on. In some examples, such as illustrated by programmable devices 1504a, 1504b, 1504c, 1504e, programmable devices can be directly connected to network 1506. In other examples, such as illustrated by programmable device 1504d, programmable devices can be indirectly connected to network 1506 via an associated computing device, such as programmable device 1504c. In this example, programmable device 1504c can act as an associated computing device to pass electronic communications between programmable device 1504d and network 1506. In other examples, such as illustrated by programmable device 1504e, a computing device can be part of and / or inside a vehicle, such as a car, a truck, a bus, a boat or ship, an airplane, etc. In other examples not shown in FIG. 15, a programmable device can be both directly and indirectly connected to network 1506.

[0122] Server devices 1508, 1510 can be configured to perform one or more services, as requested by programmable devices 1504a-1504e. For example, server device 1508 and / or 1510 can provide content to programmable devices 1504a-1504e. The content can include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content can include compressed and / or uncompressed content. The content can be encrypted and / or unencrypted. Other types of content are possible as well.

[0123] As another example, server device 1508 and / or 1510 can provide programmable devices 1504a-1504e with access to software for database, search, computation, graphical, audio, video, World Wide Web / Intemet utilization, and / or other functions. Many other examples of server devices are possible as well.Computing Device Architecture

[0124] FIG. 16 is a block diagram of an example computing device 1600, in accordance with example embodiments. In particular, computing device 1600 shown in FIG. 16 can be configured to perform at least one function of and / or related to a detector and / or an audio encoder, method 1800, and / or method 1900.

[0125] Computing device 1600 may include a user interface module 1601, a network communications module 1602, one or more processors 1603, data storage 1604, one or more camera(s) 1618, one or more sensors 1620, and power system 1622, all of which may be linked together via a system bus, network, or other connection mechanism 1605.

[0126] User interface module 1601 can be operable to send data to and / or receive data from external user input / output devices. For example, user interface module 1601 can be configured to send and / or receive data to and / or from user input devices such as a touch screen, a computer mouse, a keyboard, a keypad, a touch pad, a track ball, a joystick, a voice recognition module, and / or other similar devices. User interface module 1601 can also be configured to provide output to user display devices, such as one or more cathode ray tubes (CRT), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, either now known or later developed. User interface module 1601 can also be configured to generate audible outputs, with devices such as a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices. User interface module 1601 can further be configured with one or more haptic devices that can generate haptic outputs, such as vibrations and / or other outputs detectable by touch and / or physical contact with computing device 1600. In some examples, user interface module 1601 can be used to provide a graphical user interface (GUI) for utilizing computing device 1600, such as, for example, a graphical user interface of a mobile phone device.

[0127] Network communications module 1602 can include one or more devices that provide one or more wireless interface(s) 1607 and / or one or more wireline interface(s) 1608 that are configurable to communicate via a network. Wireless interface(s) 1607 can include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, an LTE™ transceiver, and / or other type of wireless transceiver configurable to communicate via a wireless network. Wireline interface(s) 1608 can include one or more wireline transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver configurable to communicate via a twisted pair wire, a coaxial cable, a fiber-optic link, or a similar physical connection to a wireline network.

[0128] In some examples, network communications module 1602 can be configured to provide reliable, secured, and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., guaranteed message delivery) can be provided, perhaps as part of a message header and / or footer (e.g., packet / message sequencing information, encapsulation headers and / or footers, size / time information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check values). Communications can be made secure (e.g., be encoded or encrypted) and / or decry pted / decoded using one or more cryptographic protocols and / or algorithms, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), a Rivest-Shamir-Adelman (RSA) algorithm, a Diffie-Hellman algorithm, a secure sockets protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms can be used as well or in addition to those listed herein to secure (and then decry pt / decode) communications.

[0129] One or more processors 1603 can include one or more general purpose processors, and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). One or more processors 1603 can be configured to execute computer-readable instructions 1606 that are contained in data storage 1604 and / or other instructions as described herein.

[0130] Data storage 1604 can include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of one or more processors 1603. The one or more computer-readable storage media can include volatile and / or non-volatile storage components, such as optical, magnetic, organic or other memory or disc storage, which can be integrated in whole or in part with at least one of one or more processors 1603. In some examples, data storage 1604 can be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc storage unit), while in other examples, data storage 1604 can be implemented using two or more physical devices.

[0131] Data storage 1604 can include computer-readable instructions 1606 and perhaps additional data. In some examples, data storage 1604 can include storage required to perform at least part of the herein-described methods, scenarios, and techniques and / or at least part of the functionality of the herein-described devices and networks. In some examples, data storage 1604 can include storage for a trained neural network model 1612 (e.g., a model of a trained detector and / or an audio encoder). In particular of these examples, computer-readable instructions 1606 can include instructions that, when executed by one or more processors 1603, enable computing device 1600 to provide for some or all of the functionality of trained neural network model 1612.

[0132] In some examples, computing device 1600 can include one or more camera(s) 1618. Camera(s) 1618 can include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light in one or more images; that is, camera(s) 1618 can generate image(s) of captured light. The one or more images can be one or more still images and / or one or more images utilized in video imagery. Camera(s) 1618 can capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.

[0133] In some examples, computing device 1600 can include one or more sensors 1620. Sensors 1620 can be configured to measure conditions within computing device 1600 and / or conditions in an environment of computing device 1600 and provide data about these conditions. For example, sensors 1620 can include one or more of: (i) sensors for obtaining data about computing device 1600, such as, but not limited to, a thermometer for measuring a temperature of computing device 1600, a battery sensor for measuring power of one or more batteries of power system 1622, and / or other sensors measuring conditions of computing device 1600; (ii) an identification sensor to identify other objects and / or devices, such as, but not limited to, a Radio Frequency Identification (RFID) reader, proximity sensor, one-dimensional barcode reader, two-dimensional barcode (e.g., Quick Response (QR) code) reader, and a laser tracker, where the identification sensors can be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or object configured to be read and provide at least identifying information; (iii) sensors to measure locations and / or movements of computing device 1600, such as, but not limited to, a tilt sensor, a gyroscope, an accelerometer, a Doppler sensor, a GPS device, a sonar sensor, a radar device, a laser-displacement sensor, and a compass; (iv) an environmental sensor to obtain data indicative of an environment of computing device 1600, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor,a radio sensor, a movement sensor, a microphone, a sound sensor, an ultrasound sensor and / or a smoke sensor; and / or (v) a force sensor to measure one or more forces (e.g., inertial forces and / or G-forces) acting about computing device 1600, such as, but not limited to one or more sensors that measure: forces in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor that identifies ZMPs and / or locations of the ZMPs. Many other examples of sensors 1620 are possible as well.

[0134] Power system 1622 can include one or more batteries 1624 and / or one or more external power interfaces 1626 for providing electrical power to computing device 1600. Each battery of the one or more batteries 1624 can, when electrically coupled to the computing device 1600, act as a source of stored electrical power for computing device 1600. One or more batteries 1624 of power system 1622 can be configured to be portable. Some or all of one or more batteries 1624 can be readily removable from computing device 1600. In other examples, some or all of one or more batteries 1624 can be internal to computing device 1600, and so may not be readily removable from computing device 1600. Some or all of one or more batteries 1624 can be rechargeable. For example, a rechargeable battery can be recharged via a wired connection between the battery and another power supply, such as by one or more power supplies that are external to computing device 1600 and connected to computing device 1600 via the one or more external power interfaces. In other examples, some or all of one or more batteries 1624 can be non-rechargeable batteries.

[0135] One or more external power interfaces 1626 of power system 1622 can include one or more wired-power interfaces, such as a USB cable and / or a power cord, that enable wired electrical power connections to one or more power supplies that are external to computing device 1600. One or more external power interfaces 1626 can include one or more wireless power interfaces, such as a Qi wireless charger, that enable wireless electrical power connections, such as via a Qi wireless charger, to one or more external power supplies. Once an electrical power connection is established to an external power source using one or more external power interfaces 1626, computing device 1600 can draw electrical power from the external power source the established electrical power connection. In some examples, power system 1622 can include related sensors, such as battery sensors associated with the one or more batteries or other types of electrical power sensors.Cloud-Based Servers

[0136] FIG. 17 depicts a cloud-based server system in accordance with an example embodiment. In FIG. 17, functionality of a detector and / or an audio encoder, and / or acomputing device can be distributed among computing clusters 1709a, 1709b, 1709c. Computing cluster 1709a can include one or more computing devices 1700a, cluster storage arrays 1710a, and cluster routers 1711a connected by a local cluster network 1712a. Similarly, computing cluster 1709b can include one or more computing devices 1700b, cluster storage arrays 1710b, and cluster routers 1711b connected by a local cluster network 1712b. Likewise, computing cluster 1709c can include one or more computing devices 1700c, cluster storage arrays 1710c, and cluster routers 1711c connected by a local cluster network 1712c.

[0137] In some embodiments, computing clusters 1709a, 1709b, 1709c can be a single computing device residing in a single computing center. In other embodiments, computing clusters 1709a, 1709b, 1709c can include multiple computing devices in a single computing center, or even multiple computing devices located in multiple computing centers located in diverse geographic locations. For example, FIG. 17 depicts each of computing clusters 1709a, 1709b, 1709c residing in different physical locations.

[0138] In some embodiments, data and services at computing clusters 1709a, 1709b, 1709c can be encoded as computer readable information stored in non-transitory, tangible computer readable media (or computer readable storage media) and accessible by other computing devices. In some embodiments, computing clusters 1709a, 1709b, 1709c can be stored on a single disk drive or other tangible storage media, or can be implemented on multiple disk drives or other tangible storage media located at one or more diverse geographic locations.

[0139] In some embodiments, each of computing clusters 1709a, 1709b, and 1709c can have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. In other embodiments, however, each computing cluster can have different numbers of computing devices, different numbers of cluster storage arrays, and different numbers of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster can depend on the computing task or tasks assigned to each computing cluster.

[0140] In computing cluster 1709a, for example, computing devices 1700a can be configured to perform various computing tasks of a conditioned, axial self-attention based neural network, and / or a computing device. In one embodiment, the various functionalities of a neural network, and / or a computing device can be distributed among one or more of computing devices 1700a, 1700b, 1700c. Computing devices 1700b and 1700c in respective computing clusters 1709b and 1709c can be configured similarly to computing devices 1700a in computing cluster 1709a. On the other hand, in some embodiments, computing devices 1700a, 1700b, and 1700c can be configured to perform different functions.

[0141] In some embodiments, computing tasks and stored data associated with a neural network, and / or a computing device can be distributed across computing devices 1700a, 1700b, and 1700c based at least in part on the processing requirements of a neural network, and / or a computing device, the processing capabilities of computing devices 1700a, 1700b, 1700c, the latency of the network links between the computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that can contribute to the cost, speed, fault-tolerance, resiliency, efficiency, and / or other design goals of the overall system architecture.

[0142] Cluster storage arrays 1710a, 1710b, 1710c of computing clusters 1709a, 1709b, 1709c can be data storage arrays that include disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, can also be configured to manage backup or redundant copies of the data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0143] Similar to the manner in which the functions of a conditioned, axial self-attention based neural network, and / or a computing device can be distributed across computing devices 1700a, 1700b, 1700c of computing clusters 1709a, 1709b, 1709c, various active portions and / or backup portions of these components can be distributed across cluster storage arrays 1710a, 1710b, 1710c. For example, some cluster storage arrays can be configured to store one portion of the data of a first layer of a neural network, and / or a computing device, while other cluster storage arrays can store other portion(s) of data of second layer of a neural network, and / or a computing device. Also, for example, some cluster storage arrays can be configured to store the data of an encoder of a neural network, while other cluster storage arrays can store the data of a decoder of a neural network. Additionally, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0144] Cluster routers 1711a, 1711b, 1711c in computing clusters 1709a, 1709b, 1709c can include networking equipment configured to provide internal and external communications for the computing clusters. For example, cluster routers 1711a in computing cluster 1709a can include one or more internet switching and routing devices configured to provide (i) local area network communications between computing devices 1700a and cluster storage arrays 1710a via local cluster network 1712a, and (ii) wide area network communications between computing cluster 1709a and computing clusters 1709b and 1709c via wide area network link 1713a to network 1406. Cluster routers 1711b and 1711c can include network equipmentsimilar to cluster routers 1711a, and cluster routers 1711b and 1711c can perform similar networking functions for computing clusters 1709b and 1709b that cluster routers 1711a perform for computing cluster 1709a.

[0145] In some embodiments, the configuration of cluster routers 1711a, 1711b, 1711c can be based at least in part on the data communication requirements of the computing devices and cluster storage arrays, the data communications capabilities of the network equipment in cluster routers 1711a, 1711b, 1711c, the latency and throughput of local cluster networks 1712a, 1712b, 1712c, the latency, throughput, and cost of wide area network links 1713a, 1713b, 1713c, and / or other factors that can contribute to the cost, speed, fault-tolerance, resiliency, efficiency and / or other design criteria of the moderation system architecture.Example Methods of Operation

[0146] FIG. 18 is a flowchart of a method 1800, in accordance with example embodiments. Method 1800 can be executed by a computing device, such as computing device 1600. Method 1800 can begin at block 1810. Block 1810 involves receiving, by a computing device, training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data.

[0147] Block 1820 involves training, based on the training data and in a self-supervised manner, an audio encoder to predict a feature representation for an input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks.

[0148] Block 1830 involves providing, by the computing device, the trained audio encoder to generate feature representations for audio clips.

[0149] In some embodiments, the audio encoder may be a contrastive learning-based encoder. In some embodiments, the contrastive learning-based encoder may be a Slowfast Normalizer-Free Networks (NFNets) encoder including a contrastive learning of visual representations (SimCLR) objective.

[0150] Some embodiments involve applying at least one augmentation strategy to the input audio clip.

[0151] In some embodiments, the at least one augmentation strategy may include a combination of a Brownian tape speed and a simple data augmentation method for speech recognition.

[0152] In some embodiments, the at least one augmentation strategy may be based on one or more of a circular time shift, a Brownian tape speed, a random interval masking, a random noising, a Gaussian smoothing, or a random scaling.

[0153] In some embodiments, the audio encoder may be a generative learning-based encoder. In some embodiments, the generative learning-based encoder may be a masked autoencoder (MAE). Such embodiments involve masking a portion of the input audio clip. Such embodiments further involve applying pixelwise reconstruction loss to predict the masked portion. Such embodiments also involve comparing with a ground truth.

[0154] Some embodiments involve applying learnable mask tokens to a sequence of encoded tokens. The predicting of the masked portion may be performed by a layered transformer decoder. The training of the encoder may be based on an L2distance between normalized masked portions and predicted masked portions.

[0155] Some embodiments involve applying knowledge distillation for real-world deployment of the trained encoder.

[0156] Some embodiments involve fine-tuning one or more weights of the audio encoder to perform a particular healthcare related task of the plurality of healthcare related tasks.

[0157] Some embodiments involve providing the trained audio encoder as a pre-processing network for another machine learning model.

[0158] Some embodiments involve providing the feature representation to a machine learning model performing detection inference for the plurality of healthcare related tasks.

[0159] In some embodiments, the video dataset may include YOUTUBE™ (YT) videos including non-semantic (YT-NS) audio clips.

[0160] Some embodiments involve generating the training dataset in a self-supervised manner, wherein the generating of the training dataset comprises extracting the audio clips comprising the non-semantic acoustic data from the video dataset.

[0161] In some embodiments, the non-semantic acoustic data may include one or more of coughing, snoring, baby coughing, breathing, sneezing, throat clearing, laughing, burping, or hiccupping.

[0162] In some embodiments, the training data may include log-mel spectrograms of the plurality of audio clips.

[0163] Some embodiments involve training a health acoustic event detector to generate the log-mel spectrograms of the plurality of audio clips.

[0164] In some embodiments, the health acoustic event detector may be a mobile system. Such embodiments involve resampling each audio clip of the plurality of audio clips to a monochannel sample rate. Such embodiments further involve cropping each audio clip into the log- mel spectrograms based on per-channel energy normalization (PCEN). Such embodiments also involve outputting logits associated with a prediction class.

[0165] Some embodiments involve applying a detection threshold based on the prediction class.

[0166] In some embodiments, the health acoustic event detector is a convolutional neural network (CNN). In some embodiments, the CNN may include a plurality of convolution layers.

[0167] Some embodiments involve optimizing the CNN based on a balanced binary cross entropy.

[0168] Some embodiments involve converting the health acoustic event detector to a low latency, portable model.

[0169] In some embodiments, the low latency, portable model may be a Tensor Flow lite (TFLite) model.

[0170] FIG. 19 is a flowchart of a method 1900, in accordance with example embodiments. Method 1900 can be executed by a computing device, such as computing device 1600. Method 1900 can begin at block 1910. Block 1910 involves receiving, by a computing device, an input audio clip.

[0171] Block 1920 involves predicting, by a trained audio encoder, a feature representation for the input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks, the trained audio encoder having been trained in a self-supervised manner on training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data.

[0172] Block 1930 involves providing, by the computing device, the feature representation to a machine learning model performing detection inference for the plurality of healthcare related tasks.

[0173] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its spirit and scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those enumerated herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

[0174] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

[0175] With respect to any or all of the ladder diagrams, scenarios, and flow charts in the figures and as discussed herein, each block and / or communication may represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and / or messages may be executed out of order from that shown or discussed, including substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or functions may be used with any of the ladder diagrams, scenarios, and flow charts discussed herein, and these ladder diagrams, scenarios, and flow charts may be combined with one another, in part or in whole.

[0176] A block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and / or related data may be stored on any type of computer readable medium such as a storage device including a disk or hard drive or other storage medium.

[0177] The computer readable medium may also include non-transitory computer readable media such as non-transitory computer-readable media that stores data for short periods of time like register memory, processor cache, and random access memory (RAM). The computer readable media may also include non-transitory computer readable media that stores program code and / or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory(CD-ROM), for example. The computer readable media may also be any other volatile or nonvolatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

[0178] Moreover, a block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0179] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for explanatory purposes and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method, comprising: receiving, by a computing device, training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data; training, based on the training data and in a self-supervised manner, an audio encoder to predict a feature representation for an input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks; and providing, by the computing device, the trained audio encoder to generate feature representations for audio clips.

2. The computer-implemented method of claim 1, wherein the audio encoder is a contrastive learning-based encoder.

3. The computer-implemented method of claim 2, wherein the contrastive learning-based encoder is a Slowfast Normalizer-Free Networks (NFNets) encoder comprising a contrastive learning of visual representations (SimCLR) objective.

4. The computer-implemented method of claim 3, further comprising: applying at least one augmentation strategy to the input audio clip.

5. The computer-implemented method of claim 4, wherein the at least one augmentation strategy comprises a combination of a Brownian tape speed and a simple data augmentation method for speech recognition.

6. The computer-implemented method of claim 4, wherein the at least one augmentation strategy is based on one or more of a circular time shift, a Brownian tape speed, a random interval masking, a random noising, a Gaussian smoothing, or a random scaling.

7. The computer-implemented method of claim 1, wherein the audio encoder is a generative learning-based encoder.

8. The computer-implemented method of claim 7, wherein the generative learning-based encoder is a masked autoencoder (MAE), and further comprising: masking a portion of the input audio clip; applying pixelwise reconstruction loss to predict the masked portion; and comparing with a ground truth.

9. The computer-implemented method of claim 87, further comprising: applying learnable mask tokens to a sequence of encoded tokens, wherein the predicting of the masked portion is performed by a layered transformer decoder, and wherein the training of the encoder is based on an L2distance between normalized masked portions and predicted masked portions.

10. The computer-implemented method of claim 1, further comprising: applying knowledge distillation for real-world deployment of the trained encoder.

11. The computer-implemented method of claim 1, further comprising: fine-tuning one or more weights of the audio encoder to perform a particular healthcare related task of the plurality of healthcare related tasks.

12. The computer-implemented method of claim 1, further comprising: providing the trained audio encoder as a pre-processing network for another machine learning model.

13. The computer-implemented method of claim 1, further comprising: providing the feature representation to a machine learning model performing detection inference for the plurality of healthcare related tasks.

14. The computer-implemented method of claim 1, wherein the video dataset comprises YOUTUBE™ (YT) videos including non-semantic (YT-NS) audio clips.

15. The computer-implemented method of claim 1, further comprising:generating the training dataset in a self-supervised manner, wherein the generating of the training dataset comprises extracting the audio clips comprising the non-semantic acoustic data from the video dataset.

16. The computer-implemented method of claim 1, wherein the non-semantic acoustic data comprise one or more of coughing, snoring, baby coughing, breathing, sneezing, throat clearing, laughing, burping, or hiccupping.

17. The computer-implemented method of claim 1, wherein the training data comprises log-mel spectrograms of the plurality of audio clips.

18. The computer-implemented method of claim 17, further comprising: training a health acoustic event detector to generate the log-mel spectrograms of the plurality of audio clips.

19. The computer-implemented method of claim 18, wherein the health acoustic event detector is a mobile system, and further comprising: resampling each audio clip of the plurality of audio clips to a mono channel sample rate; cropping each audio clip into the log-mel spectrograms based on per-channel energy normalization (PCEN); and outputting logits associated with a prediction class.

20. The computer-implemented method of claim 19, further comprising: applying a detection threshold based on the prediction class.

21. The computer-implemented method of claim 20, wherein the health acoustic event detector is a convolutional neural network (CNN).

22. The computer-implemented method of claim 21, wherein the CNN comprises a plurality of convolution layers.

23. The computer-implemented method of claim 21, further comprising: optimizing the CNN based on a balanced binary cross entropy.

24. The computer-implemented method of claim 18, further comprising: converting the health acoustic event detector to a low latency, portable model.

25. The computer-implemented method of claim 24, wherein the low latency, portable model is a Tensor Flow lite (TFLite) model.

26. A computer-implemented method, comprising: receiving, by a computing device, an input audio clip; predicting, by a trained audio encoder, a feature representation for the input audio clip, wherein the feature representation encodes one or more features based on non-semantic acoustic data in the input audio clip, and wherein the feature representation is associated with a plurality of healthcare related tasks, the trained audio encoder having been trained in a self- supervised manner on training data comprising a plurality of audio clips extracted from a video dataset, wherein the audio clips comprise non-semantic acoustic data; and providing, by the computing device, the feature representation to a machine learning model performing detection inference for the plurality of healthcare related tasks.

27. A computing device, comprising: one or more processors; and data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out functions comprising the computer-implemented method of any one of claims 1-26.

28. The computing device of claim 27, wherein the computing device is a mobile device.

29. A computer program comprising instructions that, when executed by a computer, cause the computer to perform steps in accordance with the method of any one of claims 1-26.

30. An article of manufacture comprising one or more non-transitory computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions that comprise the computer-implemented method of any one of claims 1-26.

31. A system, comprising: means for carrying out the computer-implemented method of any one of claims 1-26.

Images