Image-based identification of pharyngitis and uses thereof

Abstract

A method for automated discrimination between Group A Streptococcus (GAS) and non-GAS pharyngitis, including segmenting and tracking a region of interest including a throat in a displayed image or video; automatically acquiring an input image based on the segmentation of the displayed image or video; receiving at least one patient symptom; removing shadows in the input image; segmenting the input image to isolate the region of interest including the throat; determining an image quality of the isolated region of interest; and classifying, via a machine learning classifier, pharyngitis in the isolated region of interest as being caused by GAS or a non-GAS cause based on the isolated region of interest and the at least one patient symptom.

Description

IMAGE-BASED IDENTIFICATION OF PHARYNGITIS AND USES THEREOFBACKGROUNDCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority to U.S. Provisional Application No. 63 / 510,712, filed June 28, 2023, which is incorporated herein by reference in its entirety for all purposes.FIELD OF THE DISCLOSURE

[0002] The present disclosure is related to diagnosis of pharyngitis causes.DESCRIPTION OF THE RELATED ART

[0003] Pharyngitis is a common pediatric symptom and can have various causes. Treatment of pharyngitis can depend on an underlying cause. Diagnosis of a bacterial cause of pharyngitis is typically dependent on a throat swab to obtain material for microbial testing.

[0004] The foregoing “Background” description is for the purpose of generally presenting the context of the disclosure. Work of the inventors, to the extent it is described in this background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present disclosure.SUMMARY

[0005] The foregoing paragraphs have been provided by way of general introduction, and are not intended to limit the scope of the following claims. The described embodiments, together with further advantages, will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0006] In one embodiment, the present disclosure is related to a method for automated discrimination between Group A Streptococcus (GAS) and non-GAS pharyngitis, comprising segmenting and tracking, via processing circuitry, a region of interest including a throat in a displayed image or video; automatically acquiring, via the processing circuitry, an input image based on the segmentation of the displayed image or video; receiving, via the processing circuitry, at least one patient symptom; removing, via the processing circuitry', shadows in the input image; segmenting, via the processing circuitry, the input image to isolate the region of interest including the throat; determining, via the processing circuitry', an image quality' of the isolated region of interest; and classifying, via a machine learning classifier executed by the processing circuitry, pharyngitis in the isolated region of interest as being caused by GAS or a non-GAS cause based on the isolated region of interest and the at least one patient symptom.

[0007] In one embodiment, the present disclosure is related to a non-transitory computer- readable storage medium for storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method for discriminating between Group A Streptococcus (GAS) and non-GAS pharyngitis, the method comprising: segmenting and tracking a region of interest including a throat in a displayed image or video; automatically acquiring an input image based on the segmentation of the displayed image or video; receiving at least one patient symptom; removing shadows in the input image; deblurring the input image; segmenting the input image to isolate the region of interest including the throat; determining an image quality of the isolated region of interest; and classifying, via a machine learning classifier, pharyngitis in the isolated region of interest as being caused by GAS or a non-GAS cause based on the isolated region of interest and the at least one patient symptom.

[0008] In one embodiment, the present disclosure is related to an apparatus for discriminating between Group A Streptococcus (GAS) and non-GAS pharyngitis,comprising: processing circuitry configured to segment and track a region of interest including a throat in a displayed image or video; automatically acquire an input image based on the segmentation of the displayed image or video; receive at least one patient symptom; remove shadows in the input image; segment the input image to isolate the region of interest including the throat; determine an image quality of the isolated region of interest; and classify, via a machine learning classifier, pharyngitis in the isolated region of interest as being caused by GAS or a non-GAS cause based on the isolated region of interest and the at least one patient symptom.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] A more complete appreciation of the invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

[0010] FIG. 1 A is an illustration of a labeled image, according to one embodiment of the present disclosure;

[0011] FIG. IB is an illustration of a labeled image, according to one embodiment of the present disclosure;

[0012] FIG. 1C is an illustration of a labeled image, according to one embodiment of the present disclosure;

[0013] FIG. 2 is a table of model performance results, according to one embodiment of the present disclosure;

[0014] FIG. 3 is an illustration of a segmented image, according to one embodiment of the present disclosure;

[0015] FIG. 4 is a table of model performance results, according to one embodiment of the present disclosure;

[0016] FIG. 5 is a method of GAS classification, according to one embodiment of the present disclosure;

[0017] FIG. 6 is a schematic of a user device for performing a method, according to one embodiment of the present disclosure;

[0018] FIG. 7 is a schematic of a hardware system for performing a method, according to one embodiment of the present disclosure; and

[0019] FIG. 8 is a schematic of a hardware configuration of a device for performing a method, according to one embodiment of the present disclosure.DETAILED DESCRIPTION

[0020] The terms “a” or “an”, as used herein, are defined as one or more than one. The term “plurality”, as used herein, is defined as two or more than two. The term “another”, as used herein, is defined as at least a second or more. The terms “including” and / or “having”, as used herein, are defined as comprising (i.e., open language). Reference throughout this document to "one embodiment", “certain embodiments”, "an embodiment", “an implementation”, “an example” or similar terms means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of such phrases or in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments without limitation.

[0021] In one embodiment, the present disclosure is directed to systems and methods for automated discrimination of pharyngitis caused by Group A Streptococcus (GAS) from other(non-GAS) causes of pharyngitis. Pharyngitis, or inflammation of the pharynx, can result in a sore throat and related symptoms, such as trouble swallowing and cough. These symptoms are common with pediatric patients and can result in ambulatory visits. Pharyngitis can have a number of causes, including a GAS bacterial infection (GAS pharyngitis). GAS pharyngitis can necessitate an antibiotic treatment regimen, while other causes of pharyngitis (non-GAS pharyngitis) can be addressed using less severe methods (e.g., without antibiotics). GAS pharyngitis is typically diagnosed using microbiological testing, such as a bacterial culture, antigen test, or molecular test. However, microbiological testing requires in-person sample collection and test equipment, which are not available in telemedicine applications. Visual diagnosis of GAS pharyngitis by human medical professionals can be inaccurate, leading to overprescription of antibiotics. Therefore, there is a need to improve the diagnostic accuracy of GAS pharyngitis using image analysis and classification.

[0022] In one embodiment, the present disclosure is directed to an automated method for acquiring images of the throat and assessing the images of the throat to discriminate between GAS and non-GAS pharyngitis. The method can include the use of one or more machine learning models. In one embodiment, the one or more machine learning models can include deep learning models. The machine learning models can be trained to segment an image of a throat and classify the image as being indicative of GAS or a non-GAS pharyngitis diagnosis. The method can be performed by a mobile device, such as the user device 20 described with reference to FIG. 6. In one embodiment, the mobile device can include a mobile application configured to execute the method for discriminating between GAS and non-GAS pharyngitis diagnosis as described herein. In one embodiment, the method can be performed by a combination of devices, such as a mobile device and a computer.

[0023] In one embodiment, the method can include acquiring one or more images of the throat. In one embodiment, the image acquisition can include processing a live view from animage acquisition device (e.g., a camera). For example, an image can be acquired using a mobile device (e.g., user device 20) having a camera. When the camera is open, the mobile device can display the live view of the camera before the image is acquired by the mobile device. The method can include processing the live view to ensure that the acquired image will result in accurate automated GAS pharyngitis diagnosis. The image processing can include, but is not limited to, image correction or modification, evaluation of image quality, and image segmentation and annotation. Additionally or alternatively, each of these image processing techniques can be applied to an acquired image.

[0024] In one embodiment, the image acquisition can be automated. For example, image processing can be applied to assess the quality of a live view of an image acquisition device and segment the live view. When the live view includes the throat with sufficient image quality, the image can automatically be acquired. In this manner, the acquired image is likely to result in accurate discrimination between GAS and non-GAS pharyngitis diagnosis because the image will include the anatomical features and level of detail needed for accurate classification. In one embodiment, the user device can display an assigned image quality label and / or image annotations.

[0025] In one embodiment, the image correction or modification can include shadow detection and removal. A shadow in an image can be caused by an obstruction of light and can be removed by adjusting the luminosity of the shadowed region and''or restoring image features or properties that are present when the light is not obstructed. In one embodiment, a deep learning model can be used to detect and remove shadows in an image. In one embodiment, the model can be an encoder-decoder network such as a convolutional neural network (CNN), a generative artificial intelligence model (e.g., stacked conditional generative adversarial network (GAN)), or an attention-based model such as a transformer. Further examples of model architectures for shadow detection and removal can includeDeshadowNet, SegNet, ShadowFormer, and CSDFormer. Shadow detection and removal can be performed by in two steps (detection by a first model followed by removal by a second model) or in a single step by an end-to-end model. In one embodiment, a shadow detection and removal deep learning model can be trained using (fully or in part) a synthetic dataset including shadow, shadow-free, and matte image triplets. In one embodiment, image correction can further include image alignment and quality enhancement (e.g., deblurring) in order to enhance the visual quality of the image. In one embodiment, shadow detection and removal can be applied to a feature in the image, such as the throat.

[0026] In one embodiment, the image quality can depend on characteristics such as an amount of blur in the image and a resolution of features. An unobstructed image of the throat can result in more accurate automated discrimination between GAS and non-GAS pharyngitis diagnosis. Therefore, the image quality can depend on a visibility of the throat in the image and the size of the throat region in the image. In one embodiment, a deep learning model (“quality assessment model”) can be used to assess image quality. The quality assessment model can be an image classification network. As an example, the quality' assessment model can be a residual network (ResNet), such as ResNet-18, ResNet-34, ResNet-101, ResNet-152, EfficientNetV2, a Harmonic Network, a Wide Residue Network, or via a combination of a signal processing model and the output of a segmentation model. The model can be trained using a training dataset comprising images of the throat and surrounding regions. The images in the training dataset can be of varying image quality and can be labeled according to their quality. In one embodiment, the quality assessment model can be trained via transfer learning.

[0027] In one embodiment, the quality assessment model can be trained to assign a quality label to an image. The quality label can be, for example, a value between 0 and 1. In one embodiment, the quality of an image can be evaluated in order to determine whether animage is usable for discrimination between GAS and non-GAS pharyngitis diagnosis. For example, the image can be usable when the quality label assigned by the quality assessment model meets a certain threshold. When the quality assessment model determines that the image is usable, the image can then be acquired and analyzed. In one embodiment, the image quality and automatic acquisition can depend on the presence and quality of a segmented feature in the image.

[0028] In one embodiment, the image segmentation and annotation can include identifying features in an input image of the throat and surrounding anatomy. The features can include anatomical structures (e.g., the throat, tongue, teeth, lips, nose, eyes, etc.) and objects (e.g., tongue depressor or spatula, medical instruments) in the input image. In one embodiment, a deep learning model can be used for image segmentation and annotation. The model can be a single-pass model or a multi-stage detection model. A single-pass model can evaluate an entire image in a single model pass-through to identify features, while a multi-stage detection model can first identify regions using a first model (or portion of a model) and then classify the regions using a second model (or portion of a model). Tire model can output bounding boxes around detected anatomical structures and objects in the image, predicted labels of the structures and objects contained in each bounding box, and a probability of the predictions. In one embodiment, the image segmentation and annotation model can identify a region of interest (ROI) including the throat in an image.

[0029] Examples of image segmentation and annotation models can include, but are not limited to, a “You Only Look Once” (YOLO) architecture (e.g., YOLOv8, YOLOv9, YOLOvlO and variants thereof); a transformer (e.g., Swin transformer, segmentation transformer (SegFormer), a real-time end-to-end detection transformer (RT-DETR)); or a convolutional neural network (CNN) (e.g., the U-Net image segmentation model, a mask region-based CNN (Mask R-CNN)). The models can be used for real-time segmentation of aninput image. In one embodiment, the model can be a mobile-friendly model version, e.g., a faster or less computationally intensive model version. In one embodiment, the image segmentation and annotation model can be trained on a dataset of images of the throat (“segmentation dataset”). Tire images can include the throat (pharynx) and surrounding anatomy. In one embodiment, the images can be acquired from pediatric patients in order to train the model for discrimination of pediatric GAS from non-GAS pharyngitis diagnosis. In one embodiment, the images can include combinations of facial and oral anatomical features (e.g., nose, lips, teeth, throat, tongue, eye) and instruments (e.g., tongue depressor) and can be annotated with the anatomical features and instruments. In one embodiment, the segmentation dataset can include throats with pharyngitis and throats without pharyngitis. The segmentation dataset can include images of varying image quality. In one embodiment, the image segmentation and annotation model can track a segment, such as a region of interest. For example, the live view of the camera can change over time due to motion of the patient or camera. The image segmentation and annotation model can identify the region of interest including the throat in the live view. The image segmentation and annotation model can continue identifying the region of interest when the live view changes. In one embodiment, the image segmentation and annotation model can generate a bounding box around the region of interest. The location of the bounding box can be updated as the live view changes to continue surrounding and tracking the region of interest.

[0030] In one embodiment, the image segmentation and annotation model can be a YOLO model including a backbone network, neck network, and head network in sequence. The networks can extract features from an input image (backbone), sample and transform the features into feature maps (neck), and predict bounding boxes and classes of objects in the image based on the feature maps (head). The YOLO architecture can be used to process an image in a single pass through the networks, resulting in faster detection. In one embodiment,the model can be an anchor-free model. In one embodiment, the model can include a spatial pyramid pooling feature (SPPF) and non-maximum suppression (NMS). The model can also be NMS-free such as YOLOvlO.

[0031] In one embodiment, the image segmentation and annotation model can be an RT- DETR. The RT-DETR can use a hybrid encoder to extract features from an input image and transform multi-scale features into a sequence of features based on feature interactions and feature fusion, hr one embodiment, the RT-DETR can model adaptive inter-task feature interactions to transform the features. In one embodiment, the RT-DETR can include a collaborative cross-fusion module (CCFM).

[0032] In one embodiment, the image segmentation and annotation model can be a Swin transformer. The Swin transformer can be a hierarchical transformer. The Swin transformer can use a shifting window to process the input image in localized regions. The transformer can adaptively merge the regions to extract multi-scale features while determining spatial relationships in the input image. The Swin transformer can include a self-attention mechanism, wherein features are weighted based on their relationships with each other.

[0033] In one embodiment, the image segmentation and annotation model can be a U-net model. The U-net model can have a CNN architecture wherein pooling operations (layers) are replaced by upsampling operators (layers). The upsampling increases the resolution of tire output for more precise segmentation. In one embodiment, the image segmentation and annotation model can be a mask R-CNN. The mask R-CNN can generate segmentation masks for features in the input image, resulting in more accurate feature detection and definition.

[0034] In one embodiment, the image segmentation and annotation model can be a SegFormer model including an encoder that can extract multi-scale features from an input image. The model can further include a multilayer perceptron (MLP) decoder that can use local attention and global attention to decode the features. In one example, tire SegFormermodel can have an MIT-BO backbone. The model can be pretrained on a separate image dataset and fine-tuned using the segmentation dataset. In one example, the number of epochs during training of the SegFormer model can be 15.

[0035] According to one example, the model architectures described herein were trained on a segmentation dataset having labeled images of the throat and surrounding anatomy. The labeling can include an identification of features in the image and the boundaries of each feature. FIGs. 1 A-1C are illustrations of labeled images used for segmentation training. In one example, a YOLO model can be trained with an initial learning rate of 0.01, a stochastic gradient descent (SGD) / Adam betal with momentum of 0.397, an optimizer weight decay of 0.0005, warmup epochs of 3.0, warmup initial momentum of 0.8, and warmup initial bias learning rate of 0.1. These parameters can change based on the size of the dataset, the model, and its performance. These parameters can be fine-tuned to achieve higher performance.

[0036] FIG. 2 is a table showing examples of precision (P), recall (R), and mean average precision (mAP) at 0.5 and 0.95 thresholds for a YOLOv8 segmentation model, an RT-DETR segmentation model, a YOLOv9 segmentation model, and a YOLOvlO segmentation model in segmenting features in an unseen dataset of images according to one example. YOLOv8 correctly detected “Nose” in 96.47% of the cases, “Lips” in 96.8%, “Teeth” in 96%, “Throat” has an accuracy of 95%. For the “Tongue Depressor”, the accuracy was 100%, and for “Eye”, it was 87%. The “Throat” class, which is a target class, had a high true positive rate, which suggests that the model is accurately detecting the ROI. RT-DETR had superior performance compared with YOLOv8 for all classes except for the “Tongue” class where YOLOv8 displayed 100% accuracy while RT-DETR displayed 94.44%. YOLOv9 displayed 100% accuracy on “Nose”, “Tongue”, and “Tongue Depressor”, 97.2% accuracy on “Lips”, 97.76% accuracy on “Eyes”, and 93.21% on “Teeth”. In all classes, YOLOv8 had a 0.94 Fl-score at 0.48, precision-confidence of 1 at 0.93, and precision-recall of 0.97 mAP at 0.50. RT-DETRhad 0.93 Fl-score at 0.66 and 0.98 mAP at 0.50. YOLOv9 had a 0.95 Fl-score, a precisionconfidence of 1 at 0.98, and precision-recall of 0.97 mAP at 0.5. In one example a YOLOvlO segmentation model was used to segment features in an unseen dataset of images. In all classes, YOLOvlO had a 0.91 Fl-score at 0.544, precision-confidence of 1 at I, and precision-recall of 0.957 mAP at 0.50.

[0037] FIG. 3 is an illustration of an image of the throat that is segmented and annotated using a segmentation model as described herein. Features such as the lips, teeth, throat, and tongue can be identified with a certain accuracy. In one embodiment, the segmentation model can be applied to a live view of an image capture device to trigger automatic image acquisition when the quality of the image is sufficient, e.g., when the throat is in view. In one embodiment, more than one image can be automatically acquired based on the image quality and segmentation to improve the robustness of classification.

[0038] In one embodiment, the image segmentation and annotation model can be used to identify an ROI containing the throat in the input image. The ROI can then be processed by a GAS pharyngitis classifier to determine whether the patient has GAS pharyngitis or non-GAS pharyngitis. Examples of the classifier can include, but are not limited to, a Swin transformer; a data-efficient image transformer (DeiT); a support vector machine (SVM); a residual network (e.g., ResNetl8); and a masked autoencoder; self-distillation with no labels (DINO) architecture. In one embodiment, the GAS pharyngitis classifier can use one or more input images including an ROI and the patient data (symptoms, clinical data, medical history, etc.) to discriminate between GAS pharyngitis and non-GAS pharyngitis. The patient data can be an input to the GAS pharyngitis classifier along with the image of the throat.

[0039] In one embodiment, the GAS pharyngitis classifier can be trained using transfer learning. For example, a DeiT model can be trained using unsupervised learning on a known dataset of images. The trained DeiT model can then be fine-tuned using a smaller dataset ofGAS and non-GAS pharyngitis images (“GAS dataset”). The GAS pharyngitis images can be throat images of patients with positive microbiologic GAS test and an absence of upper respiratory symptoms, while non-GAS pharyngitis images can be throat images of patients with negative microbiologic tests. The images can include the throat and surrounding features. In one embodiment, the GAS dataset images can be acquired from pediatric patients in order to train the model for discrimination between pediatric GAS and non-GAS pharyngitis diagnosis . In one embodiment, the learning rate during the fine-tuning training can be low, e.g., 0.0001. In one embodiment, the GAS dataset can include images that are generated via data augmentation methods, such as transforming existing images in the dataset or using a generative adversarial network (GAN) to generate an image for training. The GAS dataset can include synthetic images that are generated using scalable diffusion-based models.

[0040] The DeiT can be a transformer-based neural network architecture that is optimized for efficiency in image classification. A DeiT can use self-attention mechanisms to process global dependencies within an input image, resulting in pattern recognition that is usefill for medical diagnosis. In one embodiment, the DeiT can handle varying image resolutions and differences in lighting and background. The flexibility of the DeiT can be useful for evaluating images that are acquired hr a telemedicine context.

[0041] In one embodiment, the GAS pharyngitis classifier can be an SVM. The SVM can be trained to classify an input image based on the input image itself as well as patient symptoms. For example, symptoms can be encoded as a vector, wherein a vector value of 1 indicates the presence of a symptom and a vector value of 0 indicates the absence of the symptom. In one embodiment, scale-invariant feature transform (SIFT) can be applied to the input image to extract features that are invariant to scaling, rotation, and viewing conditions. The extracted features can include keypoint locations and keypoint descriptors. In one embodiment, thefeatures can be clustered via k-means clustering to generate a fixed number of clusters. A histogram of the clusters can be generated and used as a fixed-length feature vector. In one embodiment, k=75. In one embodiment, the histogram of clustered image features (the feature vector) can be scaled or normalized to a mean of 0 and a unit variance. The clustered image features can then be concatenated with the symptom vectors, which are between 0 and 1. The concatenated vector can be input to the SVM for classification.

[0042] In one embodiment, the GAS pharyngitis classifier can be a ResNetl8 model. The ResNetl8 model can include a number of convolutional layers followed by a pooling layer and one or more fully connected layers. The input image can be input to the first layer of the ResNetl8 model and processed. In one embodiment, the patient symptoms can be input to the ResNet 18 model at the first fully connected layer. The model can then use the patient symptoms to classify the input image as positive (GAS pharyngitis) or negative (non-GAS pharyngitis).

[0043] FIG. 4 is a table of performance metrics for an SVM, a ResNet 18 model, a Swin transformer, and a DeiT model in classifying images as GAS pharyngitis or non-GAS pharyngitis, according to one example. The performance results include accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). In one embodiment, the DeiT model can identify GAS pharyngitis with high specificity and high negative predictive value.

[0044] FIG. 5 is a method for GAS pharyngitis diagnosis according to one embodiment. At step 5100, a live view of an image acquisition device (e.g., user device 20) can be segmented in real time to identify whether the throat is visible. The live view can be a displayed image or video. The segmentation at step 5100 can be performed using an image segmentation and annotation deep learning model, e.g., a YOLO model. The segmentation can be a real-time process and can be repeated or updated for each frame of the live view. At step 5200, animage of the throat can be acquired. In one embodiment, the image of the throat can be acquired automatically when the image segmentation and annotation model identifies the throat in the displayed image of the live view. In one embodiment, the image of the throat can be acquired automatically when the quality of the image meets a certain threshold based on the image segmentation. In one embodiment, more than one image of the throat can be automatically acquired at step 5200. At step 5300, the image acquisition device can receive patient data, including symptoms, medical history, clinical data, and known or potential contact with an individual with GAS infection. The patient data can include, but is not limited to, symptoms such as sore throat, fever, headaches, abdominal pain, difficulty swallowing, cough, congestion, rhinorrhea, hoarseness, conjunctivitis, scarlatinifomi rash, enlarged cervical lymph nodes, tender cervical lymph nodes, history of tonsillectomy, and antibiotics usage history (e.g., in the last 48 hours). The patient data can be used in conjunction with the acquired image to determine a probability of GAS.

[0045] In one embodiment, patient data such as a symptom can be determined by a deep learning model. In one embodiment, the deep learning model can be an image analysis model. For example, a deep learning model (“symptom classification model”) can be used to identify a symptom such as conjunctivitis (pink eye) in an image of the patient’s eye. In one embodiment, the symptom classification model can be a deep learning classifier. The classifier can classify an image as including conjunctivitis (“yes”) or not including conjunctivitis (“no”).

[0046] In one example, the symptom classification model can be used to identify a symptom based on audio data. For example, the symptom classification model can be used to detect hoarseness in a patient’s voice based on an audio sample and classify the audio sample according to the presence of hoarseness as a symptom. One or more symptom classification models can be used to identify any number of symptoms having visible or audiblepresentations. In one embodiment, the symptom classification model can be a convolutional neural network (CNN) or an attention-based model. The symptom classification model can be trained using a symptom training dataset including positive examples (images, audio, etc.) of a symptom and negative examples of a symptom.

[0047] At step 5400, shadows can be detected and removed from the acquired image (input image). In one embodiment, the shadows can be detected and removed by one or more deep learning models, as described herein. At step 5500, the input image can be segmented to isolate an ROI containing the throat after shadow detection and removal. In one embodiment, the ROI can be segmented by the same model used in step 5100. In one embodiment, the segmentation can further include identifying and censoring features that are used for patient identification, such as the eyes and nose. Censoring features can include modifying or removing image data corresponding to the features. The images can then be safely stored and / or transmitted while maintaining patient anonymity.

[0048] At step 5600, the quality of the ROI can be assessed by the quality assessment model. Tire quality assessment model can assign a quality score to the ROI. In one embodiment, the ROI can be used for discrimination between GAS and non-GAS pharyngitis when the quality score meets or exceeds a certain threshold. When the quality score does not meet the threshold, the image acquisition device can repeat the method from step 5100 to acquire a new image. At step 5700, an ROI of sufficient quality can be classified by a GAS pharyngitis classifier to determine whether the patient has GAS pharyngitis or non-GAS pharyngitis. The GAS pharyngitis classifier can discriminate between pharyngitis that is caused by GAS and pharyngitis that is not caused by GAS based on the input image. In one embodiment, the GAS pharyngitis classifier can use the patient symptoms and input images (or videos) of the throat to classify the ROI and predict the class of positive or negative GAS pharyngitis.

[0049] In one embodiment, the method can include displaying the classification of the ROI. In one embodiment, the method can include providing a treatment recommendation or plan based on the classification of the ROI. hi one embodiment, the input image and the classification can be used to further train and refine the models used to classify the ROI. For example, the input image can be included in a training dataset. The use of image segmentation prior to image acquisition in the present method can result in acquisition of images that are more likely to result in accurate GAS classification. In addition, real-time image enhancement and processing can result in a more efficient method that can be applied in various contexts, such as live diagnosis or telemedicine. The models described herein are also compatible with mobile devices, such as the user device 20 of FIG. 6. The method can then be executed in any environment without specialized equipment.

[0050] Further advantages include the use of a combination of convolutional neural networks and attention-based models, emphasizing real-time processing and mobile deployment. In particular, the real-time image enhancement and quality adjustment directly in the image capture device results in improved accuracy of discrimination between GAS and non-GAS pharyngitis . The present disclosure can use high-resolution throat images or videos coupled with clinical symptoms, enhancing the relevance and specificity of the data for pediatric pharyngitis. In one embodiment, the models described herein can be trained by an electronic device such as the computer 500 illustrated in FIG. 7 or the device 601 of FIG. 8. A trained model can then be used by a mobile device (e.g., mobile phone) such as the user device 20 illustrated in FIG. 5 to execute one or more steps of the method of FIG. 4. In one embodiment, the present disclosure can be directed to a mobile app configured to execute the method for discrimination between GAS and non-GAS pharyngitis. The mobile app can be stored or accessed by a mobile device (e.g., user device 20). In one embodiment, the methods described herein can be performed by more than one device. For example, image processingcan be performed by a combination of a user device 20, a computer 500, a networked device (e.g., a server, a cloud-based computer) 601, etc. The devices can be remote from each other. In one embodiment, the devices can be connected via a network connection.

[0051] Embodiments of the subject matter and the functional operations described in this specification can be implemented by digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0052] The term “data processing apparatus” refers to data processing hardware and may encompass all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0053] A computer program, which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages,or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, Subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0054] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA an ASIC.

[0055] Computers suitable for the execution of a computer program include, by way of example, general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a CPU will receive instructions and data from a read-only memory or a random access memory or both. Elements of a computer are a CPU for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus(USB) flash drive, to name just a few. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0056] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser.

[0057] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0058] The computing system can include clients (user devices) and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In an embodiment, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the user device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received from the user device at the server.

[0059] Electronic user device 20 shown hi FIG. 6 can be an example of one or more of the devices described herein. In an embodiment, the electronic user device 20 may be a smartphone. However, the skilled artisan will appreciate that the features described herein may be adapted to be implemented on other devices (e.g., a laptop, a tablet, a server, an e- reader, a camera, a navigation device, etc.). The user device 20 of FIG. 6 includes processing circuitry, as discussed above. The processing circuitry' includes one or more of the elements discussed next with reference to FIG. 6. The electronic user device 20 may include other components not explicitly illustrated in FIG. 6 such as a CPU, GPU, frame buffer, etc. The electronic user device 20 includes a controller 410 and a wireless communication processor 402 connected to an antenna 401. A speaker 404 and a microphone 405 are connected to a voice processor 403.

[0060] The controller 410 may include one or more processors / processing circuitry (CPU, GPU, or other circuitry) and may control each element in the user device 20 to perform functions related to communication control, audio signal processing, graphics processing, control for the audio signal processing, still and moving image processing and control, andother kinds of signal processing. The controller 410 may perform these functions by executing instructions stored in a memory 450. Alternatively or in addition to the local storage of the memory 450, the functions may be executed using instructions stored on an external device accessed on a network or on a non-transitory computer readable medium.

[0061] The memory 450 includes but is not limited to Read Only Memory (ROM), Random Access Memory (RAM), or a memory array including a combination of volatile and nonvolatile memory units. The memory 450 may be utilized as working memory by the controller 410 while executing the processes and algorithms of the present disclosure. Additionally, the memory 450 may be used for long-term storage, e.g., of image data and information related thereto.

[0062] The user device 20 includes a control line CL and data line DL as internal communication bus lines. Control data to / from the controller 410 may be transmitted through the control line CL. The data line DL may be used for transmission of voice data, displayed data, etc.

[0063] The antenna 401 transmits / receives electromagnetic wave signals between base stations for performing radio-based communication, such as the various forms of cellular telephone communication. The wireless communication processor 402 controls the communication performed between the user device 20 and other external devices via the antenna 401. For example, the wireless communication processor 402 may control communication between base stations for cellular phone communication.

[0064] The speaker 404 emits an audio signal corresponding to audio data supplied from the voice processor 403. The microphone 405 detects surrounding audio and converts the detected audio into an audio signal. The audio signal may then be output to the voice processor 403 for further processing. The voice processor 403 demodulates and / or decodes the audio data read from the memory 450 or audio data received by the wirelesscommunication processor 402 and / or a short-distance wireless communication processor 407. Additionally, the voice processor 403 may decode audio signals obtained by the microphone 405.

[0065] The user device 20 may also include a display 420, a touch panel 430, an operation key 440, and a short-distance communication processor 407 connected to an antenna 406. The display 420 may be a Liquid Crystal Display (LCD), an organic electroluminescence display panel, or another display screen technology. In addition to displaying still and moving image data, the display 420 may display operational inputs, such as numbers or icons which may be used for control of the user device 20. The display 420 may additionally display a GUI for a user to control aspects of the user device 20 and / or other devices. Further, the display 420 may display characters and images received by the user device 20 and / or stored in the memory 450 or accessed from an external device on a network. For example, the user device 20 may access a network such as the Internet and display text and / or images transmitted from a Web server.

[0066] The touch panel 430 may include a physical touch panel display screen and a touch panel driver. The touch panel 430 may include one or more touch sensors for detecting an input operation on an operation surface of the touch panel display screen. The touch panel 430 also detects a touch shape and a touch area. Used herein, the phrase “touch operation” refers to an input operation performed by touching an operation surface of the touch panel display with an instruction object, such as a finger, thumb, or stylus-type instrument. In the case where a stylus or the like is used in a touch operation, the stylus may include a conductive material at least at the tip of the stylus such that the sensors included in the touch panel 430 may detect when the stylus approaches / contacts the operation surface of the touch panel display (similar to the case in which a finger is used for the touch operation).

[0067] In certain aspects of the present disclosure, the touch panel 430 may be disposed adjacent to the display 420 (e.g., laminated) or may be formed integrally with the display 420. For simplicity', the present disclosure assumes the touch panel 430 is formed integrally with the display 420 and therefore, examples discussed herein may describe touch operations being performed on the surface of the display 420 rather than the touch panel 430. However, the skilled artisan will appreciate that this is not limiting.

[0068] For simplicity, the present disclosure assumes the touch panel 430 is a capacitancetype touch panel technology. However, it should be appreciated that aspects of the present disclosure may easily be applied to other touch panel types (e.g., resistance-type touch panels) with alternate structures. In certain aspects of the present disclosure, the touch panel 430 may include transparent electrode touch sensors arranged in the X-Y direction on the surface of transparent sensor glass.

[0069] The touch panel driver may be included in the touch panel 430 for control processing related to the touch panel 430, such as scanning control. For example, the touch panel driver may scan each sensor in an electrostatic capacitance transparent electrode pattern in the X- direction and Y-direction and detect the electrostatic capacitance value of each sensor to determine when a touch operation is performed. The touch panel driver may output a coordinate and corresponding electrostatic capacitance value for each sensor. The touch panel driver may also output a sensor identifier that may be mapped to a coordinate on the touch panel display screen. Additionally, the touch panel driver and touch panel sensors may detect when an instruction object, such as a finger is within a predetermined distance from an operation surface of the touch panel display screen. That is, the instruction object does not necessarily need to directly contact the operation surface of the touch panel display screen for touch sensors to detect the instruction object and perform processing described herein. For example, in an embodiment, the touch panel 430 may detect a position of a user’s fingeraround an edge of the display panel 420 (e.g., gripping a protective case that surrounds the display / touch panel). Signals may be transmitted by the touch panel driver, e.g. in response to a detection of a touch operation, in response to a query from another element based on timed data exchange, etc.

[0070] The touch panel 430 and the display 420 may be surrounded by a protective casing, which may also enclose the other elements included in the user device 20. In an embodiment, a position of the user’s fingers on the protective casing (but not directly on the surface of the display 420) may be detected by the touch panel 430 sensors. Accordingly, the controller 410 may perform display control processing described herein based on the detected position of the user’s fingers gripping the casing. For example, an element in an interface may be moved to a new location within the interface (e.g., closer to one or more of the fingers) based on the detected finger position.

[0071] Further, in an embodiment, the controller 410 may be configured to detect which hand is holding the user device 20, based on the detected finger position. For example, the touch panel 430 sensors may detect fingers on the left side of the user device 20 (e.g., on an edge of the display 420 or on the protective casing), and detect a single finger on the right side of the user device 20. In this scenario, the controller 410 may determine that the user is holding the user device 20 with his / her right hand because the detected grip pattern corresponds to an expected pattern when the user device 20 is held only with the right hand.

[0072] The operation key 440 may include one or more buttons or similar external control elements, which may generate an operation signal based on a detected input by the user. In addition to outputs from the touch panel 430, these operation signals may be supplied to the controller 410 for performing related processing and control. In certain aspects of the present disclosure, the processing and / or functions associated with external buttons and the like may be performed by the controller 410 in response to an input operation on the touch panel 430display screen rather than the external button, key, etc. In this way, external buttons on the user device 20 may be eliminated in lieu of performing inputs via touch operations, thereby improving watertightness.

[0073] The antenna 406 may transmit / receive electromagnetic wave signals to / from other external apparatuses, and the short-distance wireless communication processor 407 may control the wireless communication performed between the other external apparatuses. Bluetooth, IEEE 802.11, and near- field communication (NFC) are non-limiting examples of wireless communication protocols that may be used for inter-device communication via the short-distance wireless communication processor 407.

[0074] The user device 20 may include a motion sensor 408. The motion sensor 408 may detect features of motion (i.e., one or more movements) of the user device 20. For example, the motion sensor 408 may include an accelerometer to detect acceleration, a gyroscope to detect angular velocity, a geomagnetic sensor to detect direction, a geo-location sensor to detect location, etc., or a combination thereof to detect motion of the user device 20. In an embodiment, the motion sensor 408 may generate a detection signal that includes data representing the detected motion. For example, the motion sensor 408 may determine a number of distinct movements in a motion (e.g., from start of the series of movements to the stop, within a predetermined time interval, etc.), a number of physical shocks on the user device 20 (e.g., a jarring, hitting, etc., of the electronic device), a speed and / or acceleration of the motion (instantaneous and / or temporal), or other motion features. The detected motion features may be included in the generated detection signal. The detection signal may be transmitted, e.g., to the controller 410, whereby further processing may be performed based on data included in the detection signal. The motion sensor 408 can work in conjunction with a Global Positioning System (GPS) section 460. The information of the present position detected by the GPS section 460 is transmitted to the controller 410. An antenna 461 isconnected to the GPS section 460 for receiving and transmitting signals to and from a GPS satellite.

[0075] The user device 20 may include a camera section 409, which includes a lens and shutter for capturing photographs of the surroundings around the user device 20. In an embodiment, the camera section 409 captures surroundings of an opposite side of the user device 20 from the user. The images of the captured photographs can be displayed on the display panel 420. A memory section saves the captured photographs. The memory section may reside within the camera section 109 or it may be part of the memory 450. The camera section 409 can be a separate feature attached to the user device 20 or it can be a built-in camera feature.

[0076] An example of a type of computer is shown in FIG. 7. The computer 500 can be used for the operations described in association with any of the computer-implement methods described previously, according to one implementation. The processing circuitry includes one or more of the elements discussed next with reference to FIG. 7. In FIG. 7, the computer 500 includes a processor 510, a memory 520, a storage device 530, and an input / output device 540. Each of the components 510, 520, 530, and 540 are interconnected using a system bus 550. The processor 510 is capable of processing instructions for execution within the system 500. In one implementation, the processor 510 is a single -threaded processor. In another implementation, the processor 510 is a multi-threaded processor. The processor 510 is capable of processing instructions stored in the memory 520 or on the storage device 530 to display graphical information for a user interface on the input / output device 540.

[0077] The memory 520 stores information within the computer 500. In one implementation, the memory 520 is a computer-readable medium. In one implementation, the memory 520 is a volatile memory unit. In another implementation, the memory 520 is a non-volatile memory unit.

[0078] The storage device 530 is capable of providing mass storage for the computer 500. In one implementation, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device.

[0079] The input / output device 540 provides input / output operations for the computer 500. In one implementation, the input / output device 540 includes a keyboard and / or pointing device. In another implementation, the input / output device 540 includes a display unit for displaying graphical user interfaces.

[0080] Next, a hardware description of a device 601 according to embodiments is described with reference to FIG. 8. In FIG. 8, the device 601, which can be any of the above described devices, including the server 1500, any of the sensors 1001, 1002, lOOn, the user device 1100, includes processing circuitry. The processing circuitry includes one or more of the elements discussed next with reference to FIG. 8. The process data and instructions may be stored in memory 602. These processes and instructions may also be stored on a storage medium disk 604 such as a hard drive (HDD) or portable storage medium or may be stored remotely. Further, the claimed advancements are not limited by the form of the computer- readable media on which the instructions of the inventive process are stored. For example, the instructions may be stored on CDs, DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other information processing device with which the device 601 communicates, such as a server or computer.

[0081] Further, the claimed advancements may be provided as a utility application, background daemon, or component of an operating system, or combination thereof, executing in conjunction with CPU 600 and an operating system such as Microsoft Windows, UNIX, Solaris, LINUX, Apple MAC-OS and other systems known to those skilled in the art.

[0082] The hardware elements in order to achieve the device 601 may be realized by various circuitry elements, known to those skilled in the art. For example, CPU 600 may be a Xenon or Core processor from Intel of America or an Opteron processor from AMD of America, or may be other processor types that would be recognized by one of ordinary skill in the art. Alternatively, the CPU 600 may be implemented on an FPGA, ASIC, PLD or using discrete logic circuits, as one of ordinary skill in the art would recognize. Further, CPU 600 may be implemented as multiple processors cooperatively working in parallel to perform the instructions of the processes described above.

[0083] The device 601 in FIG. 8 also includes a network controller 606, such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing with network 650. and to communicate with the other devices. As can be appreciated, the network 650 can be a public network, such as the Internet, or a private network such as an LAN or WAN network, or any combination thereof and can also include PSTN or ISDN subnetworks. The network 650 can also be wired, such as an Ethernet network, or can be wireless such as a cellular network including EDGE, 3G, 4G and 5G wireless cellular systems. The wireless network can also be WiFi, Bluetooth, or any other wireless form of communication that is known.

[0084] The device 601 further includes a display controller 608, such as a NVIDIA GeForce GTX or Quadro graphics adaptor from NVIDIA Corporation of America for interfacing with display 610, such as an LCD monitor. A general purpose I / O interface 612 interfaces with a keyboard and / or mouse 614 as well as a touch screen panel 616 on or separate from display 610. General purpose I / O interface also connects to a variety of peripherals 618 including printers and scanners.

[0085] A sound controller 620 is also provided in the device 601 to interface with speakers / microphone 622 thereby providing sounds and / or music.

[0086] The general purpose storage controller 624 connects the storage medium disk 604 with communication bus 626, which may be an ISA, EISA, VESA, PCI, or similar, for interconnecting all of the components of the device 601. A description of the general features and functionality of the display 610, keyboard and / or mouse 614, as well as the display controller 608, storage controller 624, network controller 606, sound controller 620, and general purpose CO interface 612 is omitted herein for brevity as these features are known.

[0087] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments.

[0088] Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

[0089] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components andsystems can generally be integrated together in a single software product or packaged into multiple software products.

[0090] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0091] Embodiments of the present disclosure may also be as set forth in the following parentheticals:

[0092] (1) A method for automated discrimination between Group A Streptococcus (GAS) and non-GAS pharyngitis, comprising segmenting and tracking, via processing circuitry, a region of interest including a throat in a displayed image or video; automatically acquiring, via the processing circuitry, an input image based on the segmentation of the displayed image or video; receiving, via the processing circuitry, at least one patient symptom; removing, via the processing circuitry, shadows in the input image; segmenting, via the processing circuitry, the input image to isolate the region of interest including the throat; determining, via the processing circuitry, an image quality of the isolated region of interest; and classifying, via a machine learning classifier executed by the processing circuitry, pharyngitis in the isolated region of interest as being caused by GAS or a non-GAS cause based on the isolated region of interest and the at least one patient symptom.

[0093] (2) The method of (1), wherein the input image is automatically acquired when the throat is segmented in the displayed image or video.

[0094] (3) The method of (1) to (2), wherein the segmentation of the input image includes identifying at least one of a nose, lips, the throat, a tongue, an eye, and teeth as an identified feature and tracking the identified feature over time.

[0095] (4) The method of (1) to (3), wherein the displayed image or video and the input image are segmented using a machine learning-based segmentation model.

[0096] (5) The method of (1) to (4), wherein the machine learning segmentation model includes a convolutional neural network or attention-based architecture.

[0097] (6) The method of (1) to (5), wherein the machine learning classifier is a transformer and / or a convolutional neural network.

[0098] (7) The method of (1) to (6), wherein the at least one patient symptom includes sore throat, fever, headaches, abdominal pain, difficulty swallowing, cough, congestion, rhinorrhea, hoarseness, conjunctivitis, a number of days with symptoms, or a scarlatiniform rash.

[0099] (8) A non-transitory computer-readable storage medium for storing computer- readable instructions that, when executed by a computer, cause the computer to perform a method for discriminating between Group A Streptococcus (GAS) and non-GAS pharyngitis, the method comprising: segmenting and tracking a region of interest including a throat in a displayed image or video; automatically acquiring an input image based on the segmentation of the displayed image or video; receiving at least one patient symptom; removing shadows in the input image; deblurring the input image; segmenting the input image to isolate the region of interest including the throat; determining an image quality of the isolated region of interest; and classifying, via a machine learning classifier, pharyngitis in the isolated region of interest as being caused by GAS or a non-GAS cause based on the isolated region of interest and the at least one patient symptom.

[0100] (9) The non-transitory computer-readable storage medium of (8), wherein the input image is automatically acquired when the throat is segmented in the displayed image or video.

[0101] (10) The non-transitory computer-readable storage medium of (8) to (9), wherein the segmentation of the input image includes identifying at least one of a nose, lips, the throat, a tongue, an eye, and teeth as an identified feature and tracking the identified feature over time.

[0102] (11) The non-transitory computer-readable storage medium of (8) to (10), wherein the displayed image or video and the input image are segmented using a machine learningbased segmentation model.

[0103] (12) The non-transitory computer-readable storage medium of (8) to (11), wherein the machine learning segmentation model includes a convolutional neural network or attention-based architecture.

[0104] (13) The non-transitory computer-readable storage medium of (8) to (12), wherein the machine learning classifier is a transformer and / or a convolutional neural network.

[0105] (14) The non-transitory computer-readable storage medium of (8) to (13), wherein the at least one patient symptom includes sore throat, fever, headaches, abdominal pain, difficulty swallowing, cough, congestion, rhinorrhea, hoarseness, conjunctivitis, a number of days with symptoms, or a scarlatiniform rash.

[0106] (15) An apparatus for discriminating between Group A Streptococcus (GAS) and non-GAS pharyngitis, comprising: processing circuitry configured to segment and track a region of interest including a throat in a displayed image or video; automatically acquire an input image based on the segmentation of the displayed image or video; receive at least one patient symptom; remove shadows in the input image; segment the input image to isolate the region of interest including the throat; determine an image quality of the isolated region of interest; and classify, via a machine learning classifier, pharyngitis in the isolated region ofinterest as being caused by GAS or a non-GAS cause based on the isolated region of interest and the at least one patient symptom.

[0107] (16) The apparatus of (15), wherein the input image is automatically acquired when the throat is segmented in the displayed image or video.

[0108] (17) The apparatus of (15) to (16), wherein the segmentation of the input image includes identifying at least one of a nose, lips, the throat, a tongue, an eye, and teeth as an identified feature and tracking the identified feature over time.

[0109] (18) The apparatus of (15) to (17), wherein the displayed image and the input image are segmented using a machine learning-based segmentation model.

[0110] (19) The apparatus of (15) to (18), wherein the machine learning classifier is a transformer and / or a convolutional neural network.

[0111] (20) The apparatus of (15) to (19), wherein the at least one patient symptom includes sore throat, fever, headaches, abdominal pain, difficulty swallowing, cough, congestion, rhinorrhea, hoarseness, conjunctivitis, a number of days without symptoms, or a scarlatiniform rash.

[0112] Thus, the foregoing discussion discloses and describes merely exemplary embodiments of the present disclosure. As will be understood by those skilled in the art, the present disclosure may be embodied in other specific forms without departing from the spirit thereof. Accordingly, the disclosure of the present disclosure is intended to be illustrative, but not limiting of the scope of the disclosure, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.

Claims

CLAIMS:

1. A method for automated discrimination between Group A Streptococcus (GAS) and non-GAS pharyngitis, comprising: segmenting and tracking, via processing circuitry, a region of interest including a throat in a displayed image or video; automatically acquiring, via the processing circuitry, an input image based on the segmentation of the displayed image or video; receiving, via the processing circuitry, at least one patient symptom; removing, via the processing circuitry, shadows in the input image; segmenting, via the processing circuitry, the input image to isolate the region of interest including the throat; determining, via the processing circuitry, an image quality of the isolated region of interest; and classifying, via a machine learning classifier executed by the processing circuitry, pharyngitis in the isolated region of interest as being caused by GAS or a non-GAS cause based on the isolated region of interest and the at least one patient symptom.

2. The method of claim 1, wherein the input image is automatically acquired when the throat is segmented in the displayed image or video.

3. The method of claim 1, wherein the segmentation of the input image includes identifying at least one of a nose, lips, the throat, a tongue, an eye, and teeth as an identified feature and tracking the identified feature over time.

4. The method of claim 1, wherein the displayed image or video and the input image are segmented using a machine learning-based segmentation model.

5. The method of claim 4, wherein the machine learning segmentation model includes a convolutional neural network or attention-based architecture.

6. The method of claim 1, wherein the machine learning classifier is a transformer and / or a convolutional neural network.

7. The method of claim 1, wherein the at least one patient symptom includes sore throat, fever, headaches, abdominal pain, difficulty swallowing, cough, congestion, rhinorrhea, hoarseness, conjunctivitis, a number of days with symptoms, or a scarlatiniform rash.

8. A non-transitory computer-readable storage medium for storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method for discriminating between Group A Streptococcus (GAS) and non-GAS pharyngitis, the method comprising: segmenting and tracking a region of interest including a throat in a displayed image or video; automatically acquiring an input image based on the segmentation of the displayed image or video; receiving at least one patient symptom; removing shadows in the input image; deblurring the input image;segmenting the input image to isolate the region of interest including the throat; determining an image quality of the isolated region of interest; and classifying, via a machine learning classifier, pharyngitis in the isolated region of interest as being caused by GAS or a non-GAS cause based on the isolated region of interest and the at least one patient symptom.

9. The non-transitory computer-readable storage medium of claim 8, wherein the input image is automatically acquired when the throat is segmented in the displayed image or video.

10. The non-transitory computer-readable storage medium of claim 8, wherein the segmentation of the input image includes identifying at least one of a nose, lips, the throat, a tongue, an eye, and teeth as an identified feature and tracking the identified feature over time.

11. The non-transitory computer-readable storage medium of claim 8, wherein the displayed image or video and the input image are segmented using a machine learning-based segmentation model.

12. The non-transitory computer-readable storage medium of claim 11, wherein the machine learning segmentation model includes a convolutional neural network or attentionbased architecture.

13. The non-transitory computer-readable storage medium of claim 8, wherein the machine learning classifier is a transformer and / or a convolutional neural network.

14. The non-transitory computer-readable storage medium of claim 8, wherein the at least one patient symptom includes sore throat, fever, headaches, abdominal pain, difficulty swallowing, cough, congestion, rhinorrhea, hoarseness, conjunctivitis, a number of days with symptoms, or a scarlatiniform rash.

15. An apparatus for discriminating between Group A Streptococcus (GAS) and non- GAS pharyngitis, comprising: processing circuitry configured to segment and track a region of interest including a throat in a displayed image or video; automatically acquire an input image based on the segmentation of the displayed image or video; receive at least one patient symptom; remove shadows in the input image; segment the input image to isolate the region of interest including the throat; determine an image quality of the isolated region of interest; and classify, via a machine learning classifier, pharyngitis in the isolated region of interest as being caused by GAS or a non-GAS cause based on the isolated region of interest and the at least one patient symptom.

16. The apparatus of claim 15, wherein the input image is automatically acquired when the throat is segmented in the displayed image or video.

17. The apparatus of claim 15, wherein the segmentation of the input image includes identifying at least one of a nose, lips, the throat, a tongue, an eye, and teeth as an identified feature and tracking the identified feature over time.

18. The apparatus of claim 15, wherein the displayed image and the input image are segmented using a machine learning-based segmentation model.

19. The apparatus of claim 15, wherein the machine learning classifier is a transformer and / or a convolutional neural network.

20. The apparatus of claim 15, wherein the at least one patient symptom includes sore throat, fever, headaches, abdominal pain, difficulty swallowing, cough, congestion, rhinorrhea, hoarseness, conjunctivitis, a number of days without symptoms, or a scarlatiniform rash.

Images