Multi-supervised framework for detection and recognition of cellular anomalies

Abstract

Disclosed herein is a multi-supervised deep framework that can automatically detect, segment, and classify abnormal cervical cells. The multi-supervised framework has semi-supervised, weakly-supervised and unsupervised modes which are effective in detecting cervical cells and classifying the cells as being normal or abnormal. The various modes address the situation in which labelled training data is sparse compared to unlabeled training data.

Description

Patent Application Filed Under the Patent Cooperation Treaty at the United States Receiving Office for MULTI-SUPERVISED FRAMEWORK FOR DETECTION AND RECOGNITION OF CELLULAR ANOMALIES Applicant Carnegie Mellon University ID2023-246 Inventors Ibrahim Alkanhal Marios Savvides PREPARED BY: KDWFIRM.COM Dennis M. Carleton, Principal KDW Firm PLLC 2601 WESTON PKWY. SUITE 103 CARY, NC 27513Attorney Docket: 8350.2023-246WO MULTI-SUPERVISED FRAMEWORK FOR DETECTION AND RECOGNITION OF CELLULAR ANOMALIES Related Applications

[0001] This application claims the benefit of U.S. Provisional Patent Applications Nos. 63 / 543,415, filed October 10, 2023, 63 / 544,296, filed October 16, 2023 and 63 / 543,426, filed October 10, 2023, the contents of which are incorporated herein in their entireties. Background

[0002] Pathology images are widely used in diagnosis, clinical studies and treatment planning. However, interpreting pathology images requires the expertise of an experienced medical professional and is time-consuming. Thus, there is an emerging need for automated approaches to improve the efficiency, accuracy and cost effectiveness of the image analysis and imaging evaluation of the medical image data, and more particularly for pathology images.

[0003] Some medical images like pathology images are created from tissue samples with different staining and visualization methods for various diagnosing purposes. Solid tissue or biopsy of cytology suspension from the solid tissue can be used to generate pathology images for histology section or smear slides. In evaluating whether any abnormal tissues or cells exist in the image,Attorney Docket: 8350.2023-246WO any odd morphology from tissue debris or pigment particles will become background noises and will interfere with the interpretation. Some images are generated from radiology methods and are in color or gray scale. In some cases, the abnormal tissue structure or cell morphology are sparse compare with the normal population. It is time consuming to screen through large amounts of the samples before a few positive diagnoses are be made.

[0004] For example, cervical cancer is one of the most common cancer types among women. Many signs may appear, but at late stages, which may not be helpful for cancer prevention or early treatment. The standard diagnosis technique for cervical cancer is a test called a Pap smear, which is a visual examination of the cervix smears, examples of which are shown in FIG.1. However, Pap smears must be analyzed by highly trained medical professionals who may need several hours to analyze a single patient. It is time consuming. The diagnostics conclusion is subjected to the experience of the individual pathologist. The quality of the slides and the extremely low percentage of positive cells in early stage patient’s biopsy add additional challenges for the final precise diagnosis. Therefore, there is increasing interest in using machine learning and artificial intelligence algorithms to assist with pathology image’s abnormality detection.

[0005] Most prior art methods are focused on abnormal cell classification over cropped single-cell images. These methods show decent performance over binary classification, which can only classify normal and abnormal cells.Attorney Docket: 8350.2023-246WO However, the performance of these classifiers depends mainly on the segmentation and feature extraction accuracy. Due to the limitation of the segmentation accuracy and hand-crafted features representation, multi-class classification for abnormal cell types is still not as good as binary classification.

[0006] Recently, deep neural networks (DNNs) have become state-of-the-art approaches in many medical vision applications. Many recent papers have proposed utilizing convolutional neural networks (CNNs) for cervical cancer recognition. Although they improve performance, DNNs often require large numbers of labeled training examples to achieve a superior performance over traditional machine learning algorithms or to match human performance. Some fundamental challenges in data imperfection such as missing data, imbalanced data, and sparse data have prevented DNNs from achieving their full potential in many medical vision applications as well as cervical cancer recognition.

[0007] From a clinical point of view, the primary objective of screening is to detect abnormal cells and classify them according to their abnormality level. In general, pixel-level annotation for segmentation methods is more challenging and expensive compared to instance-level bounding box annotation used in object detection methods. Therefore, direct detection of abnormal cells from smears which does not require exact cell segmentation may overcome the difficulties and improve the results. Hence, several deep object detection-Attorney Docket: 8350.2023-246WO based methods have been developed to directly detect abnormal cervical cells.

[0008] While these methods have shown significant improvements in performance, they still pose several challenges. The first challenge is that the limited availability of high-quality datasets makes it difficult to design effective deep learning models for abnormal cell detection. In general, medical imaging datasets are often smaller and more challenging to collect compared to other domains. Most publicly available Pap smear datasets are composed from cropped single cells from smears. The second challenge is high complexity and imbalance due to medical imaging data’s large variations scale, shape, and appearance. In general, it is more expensive to annotate all objects in medical imaging datasets. Thus, having a limited amount of labeled data (i.e., a larger amount of unlabeled data) for training made it hard to learn meaningful representations of abnormal cells. The third challenge is the difficulty of obtaining high-quality cell bounding box annotations. As opposed to natural images annotation, cervical cell bounding boxes contain more diverse information about cell size, shape, and between-cell / between-class relations. Therefore, such noisy bounding boxes prevent deep object detectors from achieving their potential. The fourth challenge is the high cost of abnormal cell bounding box annotations and labeling makes it difficult to obtain good deep detectors for abnormal cells recognition systems. In general, medical imaging datasets are challenging to collect and even moreAttorney Docket: 8350.2023-246WO expensive to annotate, compared to other domains. Therefore, having large amount of unlabeled data hinders deep object detectors from reaching their maximum capability. The fifth challenge is the severe imbalance that exists in cervical cell data over different abnormal classes. Therefore, it is difficult to learn robust feature representations from smear images that would tolerate the differences between abnormal classes and the noise associated with Pap smear data. Summary of the Invention

[0009] To address the issues identified above, disclosed herein is a novel multi- supervised framework for recognizing abnormal cervical cells, shown in block form in FIG.1. The framework improves the performance of Pap smear analysis by designing a multi-supervised deep framework that can automatically detect, segment, and classify abnormal cervical cells. The multi- supervised framework is motivated by the recent development of deep detection and classification neural networks. As opposed to previous works that focused on classification and segmentation of individual cell images, the framework is modelled to start with abnormal cell detection directly from smears. Further, several feature learning approaches are presented to improve the training performance of the deep detection and classification networks.Attorney Docket: 8350.2023-246WO

[0010] A supervised framework is first disclosed. Utilizing a newly created dataset, the performance of Pap smear analysis is improved by the design of the supervised deep framework that automatically detects abnormal cervical cells. The framework starts with abnormal cell detection directly from smears.

[0011] A novel semi-supervised framework is then disclosed. By utilizing deep detectors to identify missing labels and performing abnormal cell recognition using deep classification networks, more accurate and reliable cell feature learning results.

[0012] Next, a novel weakly-supervised framework is disclosed. By utilizing deep weakly-supervised detectors to leverage bounding box weak annotations and generate segmentation mask labels, a precise delineation of the cell boundary and more accurate and reliable cell feature learning is achieved. By designing a weakly-supervised deep framework that can automatically segment abnormal cervical cells, deep classification networks can be utilized to improve abnormal cell recognition.

[0013] Lastly, a novel approach for abnormal cervical cell recognition using an unsupervised framework is disclosed. The method involves leveraging deep unsupervised methods and raw data to generate cell bounding box labels. This is a more cost-efficient approach to cervical cell recognition. By designing an unsupervised deep framework that can automatically detectAttorney Docket: 8350.2023-246WO cervical cells, deep classification networks can be utilized to perform abnormal cell recognition without needing bounding box annotations.

[0014] Different feature learning techniques are presented to solve the imperfections in the dataset to achieve an optimized training process for deep neural networks. First, the “Softer-max” loss function is disclosed as an effective loss function that enhances the discriminative power of deeply learned features by minimizing intraclass variations while maintaining separation of different class features. Softer-max is formulated to maximize interclass and minimize intraclass variance utilizing a Softmax loss in addition to a newly proposed term loss mean term. Second, the “CovCos” loss function is disclosed wherein a Euclidean angular distance-based loss is modelled to optimize deep features along both Euclidean and angular spaces that enhance discriminative power. Third, a factor k that measures the ratio of the probabilities of an image’s normality or abnormality may be used to further enhance classification accuracy. Brief Description of the Drawings

[0015] FIG.1 is an illustration of the multi-supervised deep learning framework disclosed herein.

[0016] FIG.2 shows an example of a Pap smear slide with different multi-cell images.Attorney Docket: 8350.2023-246WO

[0017] FIG.3. shows examples of multi-cell images with annotations.

[0018] FIG.4 is a block diagram illustration of the supervised recognition embodiment of the invention.

[0019] FIG.5 is a schematic illustration of the slide scaling process.

[0020] FIG.6 is a schematic illustration of the deep object detector used by the supervised recognition embodiment.

[0021] FIG.7 is a block diagram illustration of the semi-supervised recognition embodiment of the invention.

[0022] FIG.8A shows detection results after lowering the confidence threshold. FIG. 8B shows the differentiation of normal and abnormal cells after reidentification.

[0023] FIG.9 is a block diagram illustration of the weakly-supervised recognition embodiment of the invention.

[0024] FIG.10 is a block diagram illustration of the Box2Mask method for box- supervised instance segmentation.

[0025] FIG.11A shows an original cell image. FIG.11B shows detection results using the weakly-supervised method.

[0026] FIG.12 is a block diagram illustration of the unsupervised recognition embodiment of the invention.

[0027] FIG.13 is a block diagram illustrating multi-cell images being fed into a modulated detector (MDETR) for unsupervised domain adaption for cell detection and segmentation.Attorney Docket: 8350.2023-246WO

[0028] FIG.14 shows detection results using the unsupervised recognition embodiment.

[0029] FIG.15 is an illustration of the decision margins of the Softmax loss function.

[0030] FIG.16 is an illustration of the decision margins of the CovCos loss function.

[0031] FIG.17 is an illustration of the classification boundaries for the Softer-Max loss function. Detailed Description

[0032] For all embodiments disclosed herein a Pap smear dataset is used that contains eight different microscopic slides which were collected and annotated by doctors from the University of Pittsburgh Medical Center (UPMC). These slides have a total of 2,469 cropped multi-cell images that include 3,435 abnormal cells. In this dataset, there are six abnormal cell types (AGC, ASCUS, LSIL, ASC-H, HSIL, EMT), as summarized in Table 1 below. One of the challenges associated with this dataset is the data’s severe imbalance over different classes. FIG.2 shows an example of a Pap smear slide with different multi-cell images. FIG.3. shows examples of multi-cell images with annotations.Attorney Docket: 8350.2023-246WO Cell Type Abnormal Cell Type Number Total Glan Atypical glandular cells (AGC) 289 289 Atypical squamous cells of und. 2381 Low-grade squamous 342 Squa Atypical squamous cells that 335 3113 High-grade squamous 55 Other Extra sign with both types (EMT) 33 33 Total 3435 Table 1

[0033] A supervised approach is first explained as a basis for comparison with the other novel approaches in the multi-supervised framework. The supervised approach is shown in FIG.4. In this approach, a deep framework using deep detection networks that can detect abnormal cells directly from Pap smear microscopic slides is constructed. As shown in FIG.4, the proposed method consists of two main parts. The first part is slide scaling, wherein the gigapixel (whole slide) image is scaled and cropped into multiple smaller-size images (e.g., 1000x1000 pixels). Thereafter, the deep detection model is used for abnormal cell detection.

[0034] To analyze Pap smear slides, they are first converted to whole slide images (WSI). These images are large digital images (e.g., gigapixel images) that contain thousands of cells, generated by scanning pathology slides at a high resolution. As shown in FIG.5, the WSI is divided and cropped into multiple smaller, non-overlapping images or tiles (1000x1000 pixels) that can be used in deep detectors. To do this, a grid-based tiling technique is applied whereinAttorney Docket: 8350.2023-246WO the WSI is divided into a grid of equal-sized, non-overlapping tiles. Grid-based tiling is a straightforward approach for dividing a WSI into smaller, non- overlapping images or tiles. In this method, the WSI is divided into a regular grid of equal-sized tiles, and each tile is extracted one-by-one. This approach ensures the tiles do not overlap and can be processed individually for analysis. The tile size (e.g., 1000x1000 pixels) is based on the application requirements and computational resources. Once the WSI is covered by the grid, the tiles are extracted to iterate through the grid.

[0035] The main goal of this embodiment is to detect abnormal cells from the cropped multi-cell images, as shown in FIG.5. Any deep detection neural network backbone may be used for the supervised recognition task over the Pap smear dataset. The results of different deep detection neural network backbones for the supervised approaches over the dataset were compared. Eight GeForce GTX 1080 GPU units were used to train the models. The training process was applied over 100 epochs to determine where the best validation result occurs. In this stage, the performance of all models was evaluated using a cross-validation approach, where the models are trained on seven slides and validated on the designated slide.

[0036] Different detection architectures were evaluated regarding the abnormal cell detection task. The two-stage detectors were Faster R-CNN, and Cascade R- CNN. The single-stage detectors were YOLOF, RetinaNet, and TOOD with anchor. The anchor-free detectors were TOOD, and RetinaNet (anchor-free).Attorney Docket: 8350.2023-246WO When selecting an object detector for abnormal cervical cell detection, it is essential to consider the advantages and disadvantages of different approaches, such as two-stage, single-stage, and anchor-free detectors. Two- stage detectors are more robust to variations in object size and aspect ratio, which is beneficial when dealing with slides with diverse morphology and more than three abnormal classes. Single-stage detectors performed well with slides with limited abnormal classes. Because anchor-free detectors do not rely on predefined anchor boxes, making them more flexible in handling objects with varying sizes and aspect ratios, they performed better with some specific classes across slides, such as ASCUS. However, they struggle with small objects that appear in groups or when detecting highly deformed objects, such as ASC-H and HSIL. Smaller network backbones tend to capture complex patterns and features as effectively as larger networks, potentially resulting in comparable results when detecting subtle abnormalities in cervical cells. In general, due to the lack of normal cell labels, abnormal cell detectors showed a fairly low mean average precision, over a 0.5 (IoU) threshold.

[0037] Utilizing a newly created dataset, the performance of Pap smear analysis is improved by the design of a supervised deep framework that can automatically detect abnormal cervical cells. As opposed to previous works that focused on abnormal cell detection over small, cropped images with limited annotations, the supervised approach starts with abnormal cellAttorney Docket: 8350.2023-246WO detection directly from smears. However, the high complexity and imbalance due to large variations in abnormal cell scale, shape, and appearance prevented deep detectors from achieving their potential. In general, it is more expensive to annotate all objects in smears. Thus, having a limited amount of labeled data (i.e., a larger amount of unlabeled data) for training makes it hard to learn meaningful representations of abnormal cells.

[0038] Semi-Supervised Recognition

[0039] In a first embodiment of the invention, a deep semi-supervised learning approach is used when a limited amount of labeled data and a larger amount of unlabeled data are available for training. This approach leverages the power of deep neural networks to learn complex features and patterns from both labeled and unlabeled data, which can help improve generalization and reduce overfitting, hence improving the performance of deep neural networks.

[0040] Labeling missing labels in deep object detection tasks is a challenging task that involves identifying and labeling objects missed by the original labeling process. Several techniques have been proposed in the art to address missing label issues in deep object detection datasets. One of the most common approaches is based on the use of human annotation, wherein domain experts are tasked with identifying and labeling the missing objects in the images. However, this approach is time-consuming and expensive, particularly when dealing with large datasets.Attorney Docket: 8350.2023-246WO

[0041] Semi-supervised learning can be especially useful in scenarios where labeled data is difficult or expensive to obtain, such as in medical diagnosis. These methods can reduce the cost and time required for manual labeling while improving the performance of deep neural networks. By using unlabeled data, an algorithm can identify patterns and structures that might not be apparent with only labeled data. This can result in better generalization and more accurate predictions.

[0042] The high complexity and imbalance due to large variations in the scale, shape, and appearance of medical imaging data motivated a semi-supervised approach. In general, it is more expensive to annotate all objects in medical imaging datasets. Thus, having a limited amount of labeled data (i.e., a larger amount of unlabeled data) for training makes it hard to learn meaningful representations of abnormal cells.

[0043] A novel semi-supervised framework for recognizing abnormal cervical cells is disclosed as a first embodiment of the invention. By utilizing deep detectors to identify missing labels and performing abnormal cell recognition using deep classification networks, more accurate and reliable cell feature learning is possible.

[0044] In the supervised learning example just previously discussed, a detector was used that is capable of recognizing only one class of objects. In this case, the one class was positive (i.e., abnormal) cells. Negative (i.e., normal) cells are not detected. In contrast, in the semi-supervised approach, the detector isAttorney Docket: 8350.2023-246WO capable of detecting (i.e., labelling) both positive and negative cells (in bounding boxes), and a classifier is then used to classify the cells as positive (i.e., abnormal) or negative (i.e., normal).

[0045] In this approach, the semi-supervised model constructs a deep framework that can utilize deep object detectors for general cell detection and negative label reidentification. The proposed method is shown in FIG.7 and consists of three main parts. First is the slide scaling part, as previously discussed. Thereafter, a deep detection model for general cell detection is constructed to reidentify missing labels (normal cells). Finally, abnormal cell recognition using is performed feature learning approaches for deep classification networks.

[0046] Deep object detectors typically consist of two main parts: classification and regression, as shown in FIG.7 (Semi-Supervised Detection (Reidentification)) . These components work together to detect and localize objects in images. The classification part estimates the object class probabilities, while the regression part refines the bounding box coordinates for accurate object localization. Both components are crucial for achieving high performance in object detection tasks.

[0047] The classification component of a deep object detector is responsible for predicting the object class probabilities for each potential bounding box or region proposal. It takes the features extracted from the backbone network and processes them to estimate the likelihood of each bounding boxAttorney Docket: 8350.2023-246WO containing an object of a specific class. In single-stage detectors, the classification part directly predicts class probabilities at each spatial location on the feature map using anchor boxes. In two-stage detectors, the classification part operates on region proposals generated in the first stage. A confidence score serves as a metric for evaluating the performance of an object detection algorithm. It represents the likelihood that the algorithm correctly identifies the objects within an image and is expressed as a percentage: 1^^^ ^^^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^(1)

[0048] Inpositive class), ^^ ^^ is false positive (i.e., when the model incorrectly predicts the positive class) and ^^ ^^ is false negative (i.e., when the model incorrectly predicts the negative class).

[0049] The regression component of the deep object detector is responsible for predicting bounding box coordinates and refining the position and size of the bounding boxes to accurately localize the detected objects. It processes the features extracted from the backbone network and outputs adjustments for the bounding box coordinates, either for anchor boxes in single-stage detectors or region proposals in two-stage detectors. The mean average precision (mAP) is calculated at various Intersection over Union (IoU)Attorney Docket: 8350.2023-246WO thresholds to assess the algorithm’s performance. IoU is a measure that quantifies the extent of overlap between the predicted bounding boxes and ground truth bounding boxes. The following formula is used to calculate IoU, where X and Y are the predicted and ground truth bounding boxes: ^^ ^^ ^^^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ∩ ^^(2)

[0050] By analyzing RetinaNet (ResNet-50) for abnormal cell detection, comparable results appear across different IoU thresholds, which indicates a stable performance of the bounding box regression. There is a relationship between the confidence threshold and recall. With a low confidence threshold, the detector can reach 100% average recall with a low precision. Hence, it is able to detect all cells with a high false positive rate. The abnormal cell detector can perform general cell detection and negative label reidentification (missing labels) by matching detection results to ground truth labels. With a low confidence threshold, the detector can reach 100% average recall with aAttorney Docket: 8350.2023-246WO low precision. Hence, it is able to detect all cells with a high false positive rate. As shown in FIG.8A, by utilizing the abnormal cell detector and lowering the threshold in the confidence calculation in Eq. (1), general cell detection can be performed with negative label reidentification by matching detection results to ground truth labels.

[0051] A residual deep neural network (e.g., ResNet-50) was constructed and utilized to deal with the inherent structure of Pap smear data and to classify the detected cells. The DNN is shown in FIG.7 (Feature Relearning) and has been shown to provide good image representation of the data. Therefore, it is used it here to work with cropped cell images. The main goal of this network is to classify the detected cells into binary classes, as shown in FIG. 8B by the different boxes surrounding the cells. In one embodiment, the Softmax loss function, as defined in Eq. (3) may be used. Other novel loss functions are presented later herein, however, Softmax is currently one of the most used classification-based losses. The input sample ^^^∈ ℛௗ, which represents the deep features of the ^^௧^sample, belongs to the ^^^௧^class for the embedding feature dimension ^^. Whereas before the Softmax loss, ^^^denotes the ^^௧^column of the weight ^^ ∈ ℛௗൈ^, and ^^^∈ ℛ^is the bias term. ^^ refers to the batch size, and ^^ represents the number of classes. ே^^^^ ௫^ା^^^^^ 1 ^^ௌ^^௧^^௫ൌ െ^^^ log∑^ ^^^^ೕ ௫^ା^ೕ(3)Attorney Docket: 8350.2023-246WO

[0052] The objective of the training process is to determine the optimal weight values that minimize the loss produced by the loss function (e.g., Softmax) in both detection and classification networks. In both detection and classification networks, pretrained models were primarily used on ImageNet as initial training weights. To address the overfitting issue, several techniques were implemented that address this problem. First, different probabilities of dropout were used in some fully connected layers. Second, by applying different augmentation processes over classes with few samples and subsampling dominant classes in the dataset, the ratio between abnormal data was rebalanced, delaying overfitting and generating better validation and testing results.

[0053] For the feature relearning stage shown in FIG.7, a deep residual neural network (e.g., ResNet-50) is used to perform abnormal cell classification. Four different measures were used to evaluate the validation results: accuracy, F1 score, precision, and recall. The semi-supervised approach achieved a higher precision than the supervised approach.

[0054] In this embodiment, a novel semi-supervised framework for recognizing abnormal cervical cells is disclosed. The semi-supervised framework is motivated by the recent development of deep detection and classification neural networks. By utilizing deep detectors to identify missing labels and then performing abnormal cell classification using deep classification networks, more accurate and reliable cell feature learning is achieved.Attorney Docket: 8350.2023-246WO However, as opposed to natural image annotation, cervical cell bounding boxes contain more diverse information about cell size, shape, and between- cell / between-class relations. Therefore, the noisy bounding boxes prevent deep object detectors from achieving their full potential.

[0055] Weakly-Supervised Recognition

[0056] Deep weakly supervised learning approaches are used when the labeling of training data is incomplete, noisy, or expensive. This approach can help reduce the reliance on labeled data by leveraging weakly supervised data to train deep neural networks. The use of deep weakly supervised learning can help to identify objects of interest without requiring precise annotations for each part of the image. This is especially useful in scenarios where precise annotations are challenging to obtain, such as abnormal cell recognition.

[0057] As opposed to natural image annotation, cervical image cell bounding boxes contain more diverse information about cell size, shape, and between- cell / between-class relations. Therefore, such noisy bounding boxes prevent deep object detectors from achieving their full potential.

[0058] Bounding boxes provide only an approximate size of the object, which can result in inaccurate measurements and diagnoses. In contrast, segmentation masks provide a precise delineation of the object boundary, which enables more accurate and reliable measurements, such as volume, area, and shape analysis. Furthermore, segmentation masks can be used to extract features and information from the object of interest, such as texture, intensity, andAttorney Docket: 8350.2023-246WO morphology. Hence, they can provide valuable insights into the underlying pathology or physiology of the object. In general, abnormal cell annotation boxes in cervical cytology images contain more diverse information about cell size, shape, and between-cell / between-class relations. Transforming bounding box labels into mask segmentation labels is a challenging task, as it requires precise pixel-level annotations for each object in the image.

[0059] Weakly supervised learning can be used to address this challenge by using weak annotations, such as bounding boxes, to generate segmentation mask labels. These methods have shown promising results in various computer vision applications, including object detection and segmentation.

[0060] Disclosed as a second embodiment of the invention is a novel weakly supervised framework for abnormal cervical cell recognition. By utilizing deep weakly supervised detectors to leverage bounding box weak annotations and generate segmentation mask labels, a precise delineation of the cell boundary results, enabling more accurate and reliable cell feature learning. By designing a weakly supervised deep framework that can automatically segment cervical cells, abnormal cell recognition using deep classification networks is improved.

[0061] In the weakly supervised detection approach, the proposed weakly supervised model constructs a deep framework that can utilize box- supervised models for general cell instance segmentation. As shown in FIG.9, the proposed method consists of three main parts. First is the slide scalingAttorney Docket: 8350.2023-246WO part, as previously discussed. After that, a weakly supervised deep model for is used for general cell instance segmentation to reidentify all labels (i.e., both normal and abnormal cells). Finally, abnormal cell classification is performed using feature learning approaches for deep classification networks.

[0062] To perform instance cell segmentation, a weakly supervised approach is used that leverages bounding box annotations for supervision. Box2Mask is a level- set evolution-based method that transforms bounding box supervision into instance segmentation. This method allows for a more efficient and cost- effective training process compared to traditional instance segmentation methods that require pixelwise annotations.

[0063] The Box2Mask method for box-supervised instance segmentation is shown in FIG.10 and consists of four main parts: a backbone, instance-aware decoder (IAD), box-level matching assignment, and level-set evolution module. First, the backbone function acts as a feature encoder, extracting fundamental features from input images. Then, the IAD creates instance-aware mask maps based on the input instances, using kernel learning to embed each instance’s characteristics. The box-level matching assignment is implemented to designate high-quality mask map samples as positive instances. Lastly, the level-set evolution module employs a level-set energy function to allow the network to learn a series of level-set functions, which progress toward precise supervisions’ instance boundaries using only bounding boxAttorney Docket: 8350.2023-246WO annotations. This results in the final instance segmentation, as shown in FIG. 10.

[0064] Instance-aware Decoder (IAD): The IAD creates the instance-aware mask map by learning each instance’s distinct attributes. By utilizing a pixel-wise decoder and kernel learning network, the IAD can separate the output of the instance-aware mask map into unified mask features and their corresponding distinctive kernels.

[0065] Box-level Matching Assignment: The method designates potential mask samples as positives, while the remaining samples are treated as negatives. Here, a CNN-based matching assignment approach assigns category labels and generates positive instance-aware mask maps for the following level-set evolution module.

[0066] Level-set Evolution Within Bounding Box: As a traditional technique, the level-set method frames segmentation as a sequential energy minimization problem. For each instance, the level-set is optimized within the respective bounding box region, allowing for the generation of high-quality mask predictions. The goal is to predict the object boundary curve by evolving a level-set within the bounding box ^^ region in an input image ^^^ ^^, ^^^. Each positive mask map within box ^^ is the level-set ^^^ ^^, ^^^, andcorresponding pixel space of input data ^^^ ^^, ^^^ is denoted as Ω that is, Ω ⊆ ^^. By minimizing the subsequent energy function given by Eq. (4), a sequence ofAttorney Docket: 8350.2023-246WO level sets ^^^ ^^, ^^^ are learned to achieve precise boundaries for each instance. ^^^ ^^^ ^ ^^^ ^^^ ^^^ ^^^^ ^^ ^^ ^^ ^^ (4)

[0067] Here, ^^∗^sigmoid function, which serves as the characteristic function for level-set ^^^ ^^, ^^^, and ^^ is a non-negative weight. The first two terms in Eq. (4) ensure the predicted ^^^ ^^, ^^^ remains uniform both inside region Ω and outside region Ωെ. Eq. (5) shows the definition of ^^^and ^^ଶ, which are the mean values of Ω and Ωെ, respectively. ^∗ஐ^^^^^, ^^^^^^^^^ ^^, ^^^^^^ ^^ ^^ ^^^^^^^∈ఉ^ ^ ൌ^^^ ^^^ ^^^^ ^^ ^^ ^^ ^^(5)Attorney Docket: 8350.2023-246WO

[0068] Feature Relearning: Here, the performance of abnormal cell classification using the weakly supervised approach is shown. FIG.11A shows an original multi-cell image, while FIG.11B shows detection results using the weakly- supervised framework. As shown, the weakly-supervised method exhibits superior precision performance compared to the supervised and semi- supervised approaches. Unlike cell bounding box annotations, which contain more diverse information regarding cell size, shape, and intercell / interclass relations, the weakly-supervised approach enables precise delineation of cell boundaries and fosters more accurate and reliable cell feature learning. Consequently, the presence of such noisy bounding boxes has hindered deep object detectors from realizing their full potential.

[0069] The weakly-supervised approach leverages bounding box weak annotations for supervision and generates segmentation mask labels, thereby providing a precise delineation of the cell boundary. This enables more accurate and reliable cell feature learning. By designing a weakly supervised deep framework that can automatically segment cervical cells, the annotation effort required for training is reduced and deep classification networks can be utilized to improve abnormal cell recognition. However, the high cost of abnormal cell annotations and labeling makes it difficult to obtain good deep detectors for abnormal cell recognition systems. Therefore, a large amount of untapped, unlabeled Pap smear data hinders deep object detectors from reaching their maximum capability.Attorney Docket: 8350.2023-246WO

[0070] Unsupervised Recognition

[0071] Deep unsupervised learning approaches are used in computer vision when labeled data are scarce or expensive to obtain. These approaches leverage unsupervised learning methods to train deep neural networks without relying on labeled data. Deep unsupervised learning can help discover meaningful patterns and representations from large amounts of unlabeled data. These learned representations can be used as a basis for further supervised or semi- supervised learning tasks, which can improve the performance of computer vision systems.

[0072] Unsupervised Domain Adaptation (UDA) is a machine learning technique that improves the performance of a model on a target domain by leveraging the knowledge learned from a source domain without using any labeled data from the target domain. The primary challenge in domain adaptation is the domain shift or distribution discrepancy, which occurs when the data distributions in the source and target domains are different. Here, a model is initially trained on a labeled dataset from the source domain, and the knowledge gained from the source domain is used to adapt the model to the target domain. The adaptation process is done using unsupervised techniques, such as self-supervised learning or zero-shot detection, that rely on the structure and patterns present in the unlabeled target domain data. The primary goal of UDA is to reduce the impact of the domain shift on the model’s performance, enabling the model to generalize better to the targetAttorney Docket: 8350.2023-246WO domain. This approach is particularly useful for cost-effective, time-efficient annotation tasks and when there are limited or no annotations available, as is often the case in abnormal cell recognition tasks.

[0073] Zero-shot learning is a computer vision task in which an object detection model is trained to recognize and localize objects from unseen classes not present during the training phase. This is a challenging task because the model must generalize its knowledge from the seen classes (training classes) to detect and identify objects belonging to unseen classes (test classes) without any labeled examples. The concept of zero-shot detection stems from zero-shot learning, which is a broader area of research that aims to build models capable of handling new or unseen classes without retraining or fine-tuning on labeled examples from those classes. In the context of object detection, zero-shot learning techniques usually rely on the transfer of knowledge between the seen and unseen classes through some form of semantic representation, such as attributes or embeddings. For example, a zero-shot detection model may use class-level attributes (e.g., color, shape, texture, etc.) or semantic embeddings to relate the seen classes to the unseen classes. By learning to associate the visual features of seen objects with their corresponding semantic representations, the model can then infer the visual features of unseen objects based on their semantic representations, enabling the detection and localization of those unseen objects in images.Attorney Docket: 8350.2023-246WO

[0074] Deep detectors can be used to generate bounding boxes and segmentation mask labels from text sentences. The main idea behind this approach is using a deep neural network to learn the correlation between natural language descriptions and bounding box annotations. MDETR (Modulated Detection for End-to-End Multi-Modal Understanding) is a deep learning model designed for tasks that require understanding and alignment between images and textual descriptions. More recently, the Segment Anything Model (SAM) proposed a pioneering framework in the domain of image segmentation through zero-shot learning. SAM introduced a promptable segmentation model trained on the extensive SA-1B dataset, which comprises 1 billion masks across 11 million images. The SA-1B dataset has 400 times more diverse and high-quality masks than any other existing segmentation dataset.

[0075] The high cost of abnormal cell bounding box annotations and labeling makes it difficult to obtain good deep detectors for abnormal cell recognition systems. In general, medical imaging datasets are challenging to collect and even more expensive to annotate, compared to other domains. Therefore, having a large amount of unlabeled data has hindered deep object detectors from reaching their maximum potential.

[0076] Disclosed as a third embodiment of the invention is a novel approach for abnormal cervical cell recognition using an unsupervised framework. The method involves leveraging deep unsupervised methods and raw data to generate cell bounding boxes and segmentation mask labels. Models such asAttorney Docket: 8350.2023-246WO MDETR and SAM are applied for Unsupervised Domain Adaptation (UDA). By adapting these models and incorporating a zero-shot detection technique, the model is helped to learn domain-specific features to make use of the structure and patterns present in the target domain’s unlabeled data. This approach is more cost-efficient for cervical cell recognition. By designing an unsupervised deep framework that can automatically detect and segment cervical cells, deep classification networks can be used for abnormal cell recognition without needing bounding box annotations.

[0077] The proposed unsupervised model constructs a deep framework that can utilize MDETR and SAM models for unsupervised domain adaptation for general cell detection and instance segmentation. As shown in FIG.12, the proposed method consists of three main parts. First is the slide scaling part, as previously described. After that, an unsupervised deep model for general cell detection and instance segmentation is constructed to identify all labels (normal, abnormal cells). Finally, abnormal cell classification using feature learning approaches for deep classification networks is performed.

[0078] To perform general cell detection and segmentation, an unsupervised approach, leveraging modulated detection for modal understanding, is used. MDETR is applied for Unsupervised Domain Adaptation (UDA) by adapting the MDETR model and incorporating a zero-shot detection technique. After pretraining the MDETR model on the source domain (e.g., the COCO dataset), it is used to generate predictions (pseudo labels) for the target domain’sAttorney Docket: 8350.2023-246WO unlabeled data (our dataset). This approach can help the model adapt to cervical cell images without requiring labeled data.

[0079] This approach tackles multi-modal understanding tasks using modulated detection in an end-to-end manner. MDETR is designed to integrate visual and language understanding to improve performance on tasks like the following.

[0080] Phrase grounding: The objective of this task is to generate a collection of bounding boxes for each provided phrase, which may be interrelated.

[0081] Referring expression comprehension: In this task, given an image and plain text referring expression, the goal is to identify the object being referred to by returning a bounding box around it.

[0082] Referring expression segmentation: With an image and plain text referring expression, the task is to identify the object being referred to by returning both a bounding box and a segmentation mask around the detected object. MDETR’s architecture consists of three main components, as shown in FIG. 13:

[0083] Object Detection Backbone: The backbone is responsible for extracting visual features from the input image using a CNN (e.g., ResNet-101). To maintain the spatial information within the representation, 2-D positional embeddings are integrated into the flattened feature vector. The backbone generates a set of object proposals and corresponding features.Attorney Docket: 8350.2023-246WO

[0084] Language Model: The model is responsible for extracting text features from the input text using a text encoder transformer (e.g., RoBERTa).

[0085] Modality Dependent Linear Projection: This technique is employed to map both image and textual features into a shared embedding space. The feature vectors are subsequently concatenated along the sequence dimension, resulting in a unified sequence of image and text features.

[0086] Transformer Encoder-Decoder Detection Module: The transformer module is used for combining visual and textual input features. It consists of an encoder–decoder architecture, where the encoder processes textual input, and the decoder processes the visual features from the object detection backbone. The transformer enables the model to learn the relationships between the textual and visual information.

[0087] Modulated Detection Head: The modulated detection head refines object detections based on the integrated multi-modal context provided by the transformer module. It incorporates a modulation mechanism that adjusts the detection scores according to the textual input. This enables MDETR to focus on specific regions of the image that are relevant to the given textual context, resulting in improved multi-modal understanding.

[0088] Training: There are two loss functions in MDETR that promote alignment between images and text. The first loss function, referred to as the soft token prediction loss, is a non-parametric alignment loss. The second, termed text-Attorney Docket: 8350.2023-246WO query contrastive alignment, is a parametric loss function that enforces similarity between associated object queries and tokens.

[0089] Soft Token Prediction: This is unlike standard detection settings, where we aim to predict a categorical class for each detected object. Here, the goal is to predict the span of tokens from the original text, referring to each matched object. For each predicted box matched to a ground truth box, the model is trained to predict a uniform distribution over all token positions corresponding to the object. Here, multiple words in the text can correspond to the same image object, and vice versa. Any query not matched to a target is trained to predict as “no object” with label ^^.

[0090] Contrastive Alignment: The contrastive alignment loss enforces alignment between the embedded representations of the object at the decoder output and the text representation at the cross-encoder output. This loss ensures the embeddings of an object and its corresponding token are closer in feature space compared to the embeddings of unrelated tokens. The contrastive loss for all objects is as follows: ேି^1 ^^ ^^ ^^൫ ^^^் ^^^⁄ ^^ ൯^^^ൌ ^ ^ െ log^^^ ^^ ^^ ^^^⁄ ^^(6) where ^^^ାaligned, and ^^^ାdenotes the set of objects that should be aligned with a given token ^^^. ^^ is a fixed temperature parameter. The contrastive loss for allAttorney Docket: 8350.2023-246WO tokens, normalized by the number of positive objects for each token, is given by the following: ^ି^^^ ^^ ^^ ^^(7)

[0091] In addition to the soft token prediction loss and contrastive alignment loss, the total loss includes the matching cost, L1, and the GIoU loss between the prediction and the target box, as in DETR.

[0092] The primary innovation of MDETR is its use of modulated detection. This mechanism allows the model to attend to specific image regions based on the textual input, leading to a better understanding of the relationship between visual and textual information and improved performance on multi- modal tasks.

[0093] By leveraging modulated detection in an end-to-end architecture trained on a novel dataset combination of categories and attributes, there are no longer restrictions based on a fixed vocabulary of object classes. Also, anything referred to in free-form text can be detected and an interpretable prediction generated because the training dataset has no training instances with zero boxes aligned to the text (i.e., no negative examples!). Thus, the model will always try to find something (usually salient objects in the image). Therefore, the unsupervised domain adaptation approach is applied to perform general cell detection and instance segmentation.Attorney Docket: 8350.2023-246WO

[0094] Identification: Zero-shot Generalization: To achieve powerful generalization, the Segment Anything Model (SAM) is used. SAM introduces a promptable segmentation model trained on the extensive SA-1B dataset, which comprises 1 billion masks across 11 million images. The SA-1B dataset has 400 times more diverse and high-quality masks than any other existing segmentation dataset. By accepting user interactions as prompts for general segmentation, SAM can transfer robust zero-shot performance to new image distributions and tasks. After pretraining the SAM model on the source domain (SA-1B dataset), we use it to generate predictions for the target domain’s unlabeled data (our dataset). This approach can help the model adapt to cervical cell images without requiring labeled data.

[0095] Feature Relearning: The MDETR model is first pretrained on the source domain (e.g., the COCO dataset) and then used to generate pseudo-labels for the target domain’s unlabeled data (our dataset). The network is pretrained using 1.3 million image–text pairs, which are extracted from preexisting multi-modal datasets possessing explicit alignment between objects in the image and phrases in the text. Without any fine-tuning, the model is tested on many downstream tasks such as phrase grounding, referring expression comprehension, and segmentation. Here, different expressions, such as “body cell,” “body cells in a blood sample,” and “cervical cell” were tried. For some cell images, this approach helped the model adapt perfectly to the target domain without requiring labeled data, as shown in FIG.14. AlthoughAttorney Docket: 8350.2023-246WO this approach performed lower compared to the supervised, semi- supervised, and weakly-supervised approaches, it displayed good potential in using unsupervised approaches for general cell detection and segmentation.

[0096] With an average masks per image of 100 and average image resolution of 1500×2250 pixels, the SAM model is pretrained on the source domain (e.g., the SA-1B dataset) and then used to segment the target domain’s unlabeled data (our dataset). The backbone network is pretrained using 1 billion mask instances. For most cell images, this approach helped the model adapt more better to the target domain comparing to the MDETR model.

[0097] The unsupervised framework method involves leveraging deep unsupervised methods and raw data to generate cell bounding box and segmentation mask labels. This provides more a cost-efficient approach to cervical cell recognition. By designing an unsupervised deep framework that can automatically detect cervical cells, deep classification networks can be used to perform abnormal cell recognition without the need for bounding box annotations. However, the severe imbalance in cervical cell data across different abnormal classes still exists. Therefore, it is difficult to learn robust feature representations from Pap smear images that would tolerate the differences between different abnormal classes as well as the noise associated with Pap smear data.

[0098] Feature RelearningAttorney Docket: 8350.2023-246WO

[0099] Robust object recognition requires a good understanding of supervised classification and representation learning. While deeper CNNs are able to extract clean high-level features, more powerful loss functions can learn better representations. In addressing data imperfections and class imbalances during training, such as in abnormal cell recognition tasks, it is crucial to employ the appropriate loss function for learning features from Pap smear images.

[0100] Because of the severe imbalance in cervical cell data across different abnormal classes, it is difficult to learn robust feature representations from Pap smear images that would tolerate the differences between different abnormal classes as well as the noise associated with Pap smear data.

[0101] Several different feature learning techniques will be disclosed to address the imperfections in the Pap smear dataset and to achieve an optimized training process for deep neural networks.

[0102] First, the Softer-max loss function is disclosed. This loss function creates is effective in enhancing the discriminative power of deeply learned features by minimizing intraclass variations while maintaining separability between different class features. Softer-max is formulated to maximize interclass and minimize intraclass variance utilizing the Softmax loss in addition to the newly proposed term ^^^^^^.

[0103] Second, a Euclidean angular distance-based loss (CovCos) is disclosed. This loss function optimizes deep features along both Euclidean and angularAttorney Docket: 8350.2023-246WO spaces that enhance discriminative power. The CovCos loss function is formulated to maximize interclass variance and minimize intraclass variance through two main parts. First, ^^ ^^ ^^: which learns a covariance for deep features of each class and minimizes the distance between the deep features and their corresponding class covariances in the Euclidean space. Second, ^^ ^^ ^^: which optimizes the cosine distance between the deep features and their corresponding weights in the angular space.

[0104] Third, the ^^ factor, which measures the ratio of the probabilities of an image’s normality and abnormality is disclosed herein and may be used to further enhance classification accuracy. The system may determine to report only the images according to a rule based on the ^^ factor.

[0105] Softer-max: The Softmax loss function, as defined in Eq. (3), is currently considered one of the most used classification-based losses. The Softmax loss provides a discriminative effect on the projected classes without considering the compactness of the classes as well as their distribution. The features of the entire samples for the same class are distinguished from other classes, and the relationship of alike samples is not penalized.

[0106] As shown in FIG.15, there are two-dimensional embeddings of the Softmax loss function on the MNIST training set. Here, the deep features of different classes are distinguished by decision boundaries. However, the deeply learned features are separable but not discriminative enough. Therefore, not optimal for abnormal cell classification.Attorney Docket: 8350.2023-246WO

[0107] The Softer-max loss function enhances the discriminative power of deeply learned features by minimizing intraclass variations while maintaining separability between different class features. Softer-max is formulated to maximize interclass and minimize intraclass variance utilizing ^^^^^^, as defined in Eq. (8). Instead of projecting input features ^^^into weight vector^^, as in Softmax loss, ^ ^^^െ ^^^^ is being projected, where ^^^is the meanvalue of class ^^. As a result, considering ^^^^^^to optimize the classification of the extracted deep features enhances in the classification task for the Euclidean distances, as shown in FIG.17. 1ே^^^^^^ ^௫^ିఓ^^ା^^^(8) (9)

[0108] ^^ ^^ ^^ ^^ ^^ ^^: Todiscriminative power of the deeply learned features by minimizing the intraclass variations while maintaining the separability of the features of different classes, the CovCos loss function is disclosed. CovCos is formulated to maximize interclass variance in the Euclidean space through Cov. The intraclass variance is minimized and the decision margin in the angular space is maximized utilizing Cos.Attorney Docket: 8350.2023-246WO

[0109] ^^ ^^ ^^: In probability and statistics, the covariance matrix addresses the covariance among different random variables within a distribution. However, controlling such a variable of the distribution enables representation of the relation among those random variables in terms of their spread in the Euclidean space. ^^ ^

[0110] In practice, calculating the covariance matrix could be computationally expensive, especially with high-dimensional data. Here, for simplicity, it is assumed that the covariance matrix Σ^is diagonal while assuming ^^ represents a set of random variables, and ^ത^ is their means: Σ ൌ COv^x^ ൌE^^ ^^ െ ^^ ^ത^^^ ^^ െ ^^ ^ത^^்^ ൌ ^ ∑^ ൫ ^^ െ ^^ ൯൫ ்^ ^ୀ^ ^ ^ ^^^ െ ^^^൯ . Conducting thein terms of compactingsamples with a similar set of features. By considering a dataset with ^^ number of classes each consisting of a set of features as the random variables of that class. Minimizing the covariance between the set of random variables of that class addresses the applicability of compacting that distribution’s samples toward its mean. As presented in Eq, (10), ^^ is considered theAttorney Docket: 8350.2023-246WO learnable covariance matrix and Σ the actual calculated covariance matrix from the conducted deep features distribution. The covariance matrix, ^^, is addressed to be minimized toward Σ via updating the covariance matrix while iterating over the training set with ^^ number of classes. However, addendum factors, ^^ and ^^, are proposed for regularizing the optimization of the covariance matrix during the training.

[0111] ^^ ^^ ^^: In addition to optimizing the intraclass compactness and interclass dispersion via optimizing the covariance matrix of a set of random variables in the Euclidean space, improvement can be achieved via performing the optimization in the angular dimension. To generate more constrained classification as well as improve the discrimination among distributions within the space, increasing the angular distance among the latter is proposed. As defined in Eq. (121) below, considering the angular margin between the distribution’s samples by minimizing the cos of the angle for each vector’s direction ^^ can significantly improve the intraclass compactness. ^^ represents the ^^௧^sample, cos ^ ^^௬ೕ, ^^^ is the angular direction of that feature, and ^^ is the the deep features andtheir corresponding weights. Using ^^ଶnormalization, individual weight is fixed as ฮ ^^^ฮ ൌ 1. ^^ ^^^ ൌ ^ฮ ^^ ^ଶ^^^ 2 ^ ^ ^^ ^^൫ ^^௬ , ^^൯ฮAttorney Docket: 8350.2023-246WO (12) ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ‖ ^^‖ ൌ 1, ^^ ^^ ^^൫ ^^௬, ^^൯ ൌ ^^^்^^^(13)

[0112] ^^^^௩^^^is a loss function which is a combination of two terms where both the Euclidean and angular distances are being optimized. This significantly improves the classification of each sample toward its genuine distribution in the space while obtaining several alike distributions in both domains. The proposed loss function, as defined in Eq. (14), can be combined with the Softmax term to apply the classification among the different classes, where ^^ is the number of classes. Both of the ^^^^௩and ^^^^^terms were designed for intraclass optimization. As a result, considering both terms to optimize the classification of the extracted deep features enhances the classification task for the Euclidean and angular distances. ^^^^௩^^^ൌ ^^^^௩^ ^^^^^ൌ^^ ^ଶ2^ฮΣ^ െ ^^^ฮAttorney Docket: 8350.2023-246WO

[0113] The ^^^^௩^^^loss function can be demonstrated with a toy example. To show its power in discriminating different classes, the MNIST dataset was conducted to visualize the 2D embedding features. As shown in FIG.16, the feature embeddings of different loss functions are presented. For the ^^ ^^ ^^ ^^ ^^ ^^, both interclass dispersion and intraclass compactness were improved. The gradients of ^^ ^^ ^^ ^^ ^^ ^^ with respect to ^^ and ^^ are computed as shown in Eq. (15): ^^ ^^^^௩^^^^^ ^^^^௩^^ ^^ ^^^^^

[0114] ^^^^௩^^^can be usedCosFase, ArcFace), and other feature normalization-based loss functions. The loss encourages covariance of samples with value ^^ (a learned parameter). This approach provides informed gradients toward a better minimum, which helps the network satisfy the covariance and cosine constraint.

[0115] Confidence Ratio ( ^^).

[0116] A factor ^^ that measures the ratio of the probabilities of an image’s normality and abnormality may be used to further enhance classification accuracy. The system may determine to report only the images according to a rule based on the ^^ factor. In any given input image ^^, the system uses the detection algorithm to output many segments of interested areas. The system uses theAttorney Docket: 8350.2023-246WO classification neural network to calculate each segmentation’s normal and abnormal probabilities. The two probabilities will always sum up to 100%. The system will then evaluate the segments based on the ratio of the abnormal and normal probabilities. If the ratio is greater than ^^ or smallerthan 1ൗ ^^, the system processes the image to be selected for high accuracyprediction. The system will discard a segment if the ratio falls between ^^ and1ൗ ^^ . In this situation, the difference between the probabilities is too close toa precise prediction. This allows the system to give a more accurate prediction from part of the image’s segments that have more polarized probability. The prediction of the whole image is determined by a voting process for each of the selected segments that falls within the ^^ to1ൗ^^range. If within each image there are more normal than abnormal segments, the whole image is predicted as normal. If there are more abnormal than normal segments, the whole image is predicted as abnormal. ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^: ^^ ൌ ^^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ ^^^ ^^ ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^ ^^^ ^^ െ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^, ^^ ^^ℎ ^^ ^^ ^^ ^^ ^^ ^^Attorney Docket: 8350.2023-246WO

[0117] Disclosed herein are different feature learning techniques to solve the imperfections in the dataset to achieve an optimized training process for deep neural networks. To develop an effective loss function that enhances the discriminative power of the deeply learned features by minimizing the intraclass variations while keeping the features of different classes separable, the Softer-max loss function may be used, which is formulated to maximize interclass and minimize intraclass variance utilizing Softmax loss in addition to the newly proposed term ^^^^^^.

[0118] Additionally, a Euclidean angular distance-based loss ( ^^^^௩^^^) was disclosed to optimize deep features along both Euclidean and angular spaces, which enhances discriminative power.

[0119] A factor ^^, which measures the ratio of the probabilities of an image’s normality and abnormality, may be used to further enhance classification accuracy. The system may determine to report only the images according to a rule based on the k factor.

[0120] The present invention illustrates various techniques and configurations that enable the integration and use of machine learning analysis in a data- driven image evaluation workflow. For example, machine learning analysis (such as trained models of image detection of certain medical conditions) may be performed upon medical imaging procedure data produced as part of a medical imaging study. The medical imaging procedure data may include image data captured by an imaging modality, and order data (such asAttorney Docket: 8350.2023-246WO data indicating a request for a radiological image read), each produced to facilitate a medical imaging evaluation (such as a radiology read to be performed by a radiologist or a diagnostic evaluation by another qualified medical professional).

[0121] For example, the machine learning analysis may receive and process images from medical imaging procedure data, to identify trained structures, conditions, and conditions within images of a particular study. The machine learning analysis may result in the automated detection, indication, or confirmation of certain medical conditions within the images, such as the detection of urgent or life-critical medical conditions, clinically serious abnormalities, and other key findings. Based on the result of the machine learning analysis, the medical evaluation for the images and the associated imaging procedure may be prioritized, or otherwise changed or modified. Further, the detection of the medical conditions may be used to assist the assignment of the medical imaging data to particular evaluators, the evaluation process for the medical imaging data, or implement other actions prior to, or concurrent with, the medical imaging evaluation (or the generation of a data item such as a report from such medical imaging evaluation).

[0122] As further discussed herein, the machine learning analysis may be provided on behalf of any number of machine learning algorithms and trained models, including but not limited to deep learning models (also known asAttorney Docket: 8350.2023-246WO deep machine learning, or hierarchical models) that have been trained to perform image recognition tasks, particularly for certain types of cancer like cell morphology upon medical images of tissue biopsy representations. As used herein, the term "machine learning" is used to refer to the various classes of artificial intelligence algorithms and algorithm-driven approaches that are capable of performing machine-driven (e.g., computer-aided) identification of trained structures, with the term "deep learning" referring to a multiple-level operation of such machine learning algorithms using multiple levels of representation and abstraction. However, it will be apparent that the role of the machine learning algorithms that are applied, used, and configured in the presently described medical imaging evaluation may be supplemented or substituted by any number of other algorithm-based approaches, including variations of artificial neural networks, learning- capable algorithms, trainable object classifications, and other artificial intelligence processing techniques.

[0123] In some of the examples, reference is made to medical imaging procedures and diagnostic evaluation of the images produced from such imaging procedures that would be performed with an image evaluation (e.g., pathological evaluation) by a licensed and credentialed pathologist. It will be understood that the applicability of the presently described techniques and systems will extend to a wide variety of imaging data (and other data representations) produced by various medical procedures and specialties,Attorney Docket: 8350.2023-246WO including those not involving traditional pathology imaging modalities. Such specialties include, but are not limited, to pathology, medical photography, medical data measurements such as electroencephalography (EEG) and electrocardiography (EKG) procedures, cardiology data, neuroscience data, preclinical imaging, and other data collection procedures occurring in connection with telemedicine, telepathology, remote diagnostics, and other applications of medical procedures and medical science. Accordingly, the performance of the data recognition and workflow modification techniques described herein may apply to a variety of medical image data types, settings, and use cases, including captured static images and multi-image (e.g., video) representations.

[0124] The description and the drawings herein sufficiently illustrate specific embodiments to enable those skilled in the art to practice them. Other embodiments may incorporate variations. For example, the specific architectures of the reidentification and classification models for the various embodiments is not key to the invention, and various different implementations may be used. Portions and features of some embodiments may be included in, or substituted for, those of other embodiments.

Claims

Attorney Docket: 8350.2023-246WO Claims:

1. An unsupervised framework for detecƟng and classifying cells comprising: a slide scaling component for segmenƟng large images into a plurality of smaller images; an unsupervised deep object detector capable of idenƟfying and labelling both posiƟve and negaƟve cells in each of the smaller images; a cell segmentaƟon module for generaƟng segmentaƟon mask labels from bounding box labels for each detected cell; and a classifier trained to of classify the cells labelled with a segmentaƟon mask as posiƟve or negaƟve; wherein the detector and classifier are trained on a raw dataset in which no data is associated with a label.

2. The framework of claim 1 wherein the cells are labelled by placing each cell into a bounding box.

3. The framework of claim 2 wherein the semi-supervised detector comprises: a classificaƟon component for esƟmaƟng object class probabiliƟes for each potenƟal bounding box proposal; and a regression component for refining coordinates of the bounding boxes.

4. The framework of claim 3 wherein the deep object detector has a backbone network for extracƟng features from each smaller image.Attorney Docket: 8350.2023-246WO 5. The framework of claim 4 wherein the classificaƟon component processes the extracted features to esƟmate a likelihood that each bonding box contains an object for which the detector is trained to detect.

6. The framework of claim 5 wherein the regression component processes the extracted features to predict coordinates of the bounding boxes and to refine the posiƟon and size of the bounding boxes to accurately localize the detected objects.

7. The framework of claim 6 wherein the regression component uses an intersecƟon over union metric to refine the posiƟon and size of the bounding boxes.

8. The framework of claim 1 wherein the classifier is a deep residual neural network.

9. The framework of claim 8 wherein the deep residual neural network is trained using either a SoŌer-Max loss funcƟon or a CovCos loss funcƟon.

10. The framework of claim 1 wherein the large images are images of Pap smear slides.

11. The framework of claim 1 wherein the cell segmentaƟon module comprises: a backbone; an instance-aware decoder, box-level matching assignment; andAttorney Docket: 8350.2023-246WO a level-set evoluƟon module.

12. The framework of claim 11 wherein: the backbone extracts fundamental features from input images; the instance-aware decoder creates instance-aware mask maps based on the input instances, using kernel learning to embed each instance’s characterisƟcs; box-level matching assignment designates high-quality mask map samples as posiƟve instances; and the level-set evoluƟon module employs a level-set energy funcƟon to allow the network to learn a series of level-set funcƟons, which progress toward precise supervisions’ instance boundaries using only bounding box annotaƟons.

13. The framework of claim 12 wherein the instance-aware decoder creates instance-aware mask maps uƟlizing a pixel-wise decoder and kernel learning network, the IAD can separate the output of the instance-aware mask map into unified mask features and their corresponding disƟncƟve kernels.

14. The framework of claim 12 wherein the box-level matching assignment uses a CNN-based matching assignment approach to assign category labels and generate posiƟve instance-aware mask maps for the level-set evoluƟon module.

15. The framework of claim 12 wherein instance, level-set evoluƟon module predicts an object boundary curve by evolving a level-set within a bounding box region in an input image andAttorney Docket: 8350.2023-246WO opƟmizing the level-set within the respecƟve bounding box region.

16. The framework of claim 1 wherein the unsupervised deep object detector is trained using zero- shot learning.

17. The framework of claim 16 wherein the unsupervised deep object detector learns to detect cells in the smaller images based on an unrelated source domain and text descripƟons of the cells.

18. The framework of claim 1 wherein the unsupervised deep object detector uses mulƟ-model understanding or a segment anything model to learn a target domain comprising the posiƟve and negaƟve cells.

Images