Automated method for assessing zona pellucida binding capacity of sperm in clinically assisted reproduction

Abstract

An automated method for assessing the ZP binding capacity of sperm from morphological characteristics of sperm in assisted reproduction using deep learning is disclosed. The invention also provides a method for predicting fertilization success based on ZP binding capacity of sperm in clinical assisted reproduction.

Description

[0001] Cross Reference to Related Applications

[0002] This application is related to pending U.S. Provisional Application No. US 63 / 511,375, filed June 30, 2023, and U.S. Provisional Application No. US 63 / 567,147, filed March 19, 2024, each of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] An automated method for assessing the zona pellucida (ZP) binding ability of sperm based on their morphological features in assisted reproduction using deep learning is disclosed. A method for predicting the success of fertilization in clinical assisted reproduction based on sperm-ZP binding is also provided. BACKGROUND

[0004] The human fertilization process begins with the binding of a capacitated sperm to the zona pellucida (ZP) of an oocyte. In fertile men, only a limited proportion of motile sperm (<7%) in ejaculated semen are able to bind to and penetrate the ZP. This observation can be attributed to the selectivity of the ZP interaction with a subpopulation of sperm characterized by normal chromatin structure, excellent morphology, and fertility potential.

[0005] Infertility is a global health issue affecting approximately 15% of heterosexual couples of reproductive age, of which about 30% of cases are classified as male factor infertility. Assisted reproductive treatment (ART), including conventional in vitro fertilization (IVF) and intracytoplasmic sperm injection (ICSI), is typically offered to patients with long-standing infertility problems. However, 10% of all couples and 25% of couples with unexplained infertility experience fertilization failure after conventional artificial insemination during IVF. ICSI involves the direct injection of morphologically normal and motile sperm to bypass the binding and subsequent penetration process between gametes. This technique greatly improves the fertilization rate of couples diagnosed with severe male factor infertility compared to conventional artificial insemination during IVF.

[0006] Routine semen analysis according to the World Health Organization (WHO) Laboratory Manual for the Examination and Processing of Human Semen is typically used to determine the fertility profile of a semen sample. Semen analysis results play a crucial role in deciding whether to adopt conventional artificial insemination or ICSI. In the first IVF cycle, patients are typically offered conventional artificial insemination as the method of fertilization unless the semen analysis results indicate severe abnormalities. However, semen parameters such as motility and morphology do not necessarily accurately reflect the fertilization potential of sperm, and therefore, even in men with semen parameters within the normal range, suboptimal fertilization outcomes can still occur after conventional artificial insemination. Moreover, the need for ICSI in subsequent cycles can impose additional financial burden on them.

[0007] Sperm zona pellucida (ZP) binding defects (DSZPB) are a major cause of low fertilization rates in routine insemination during IVF. DSZPB are commonly observed in patients with mild to moderate teratozoospermia, severe teratozoospermia and oligozoospermia. In addition, DSZPB occur in about 13% of men with normal semen parameters, who are usually not detected during the initial semen evaluation due to the lack of direct correlation between semen parameters and sperm ZP binding capacity. Therefore, there is an urgent need to develop a highly robust automated method to assess the fertilization potential of sperm prior to the start of IVF to optimize clinical decision making and overall treatment management.

[0008] Deep learning is an advanced machine learning method involving artificial neural networks. Convolutional neural networks (CNNs) are a specialized class of deep learning that are designed to analyze structural elements of a dataset for subsequent image classification and object recognition. CNNs are highly complex and interwoven multi-layer networks composed of nodes interconnected within layers, similar to the dynamic decision-making processes that occur within the neural networks of the human brain. During the training process, CNNs assign weights to each identifiable feature based on its importance to the final output. CNNs can systematically analyze input data through successive layers of data representation systems to arrive at a final result. Starting with the first layer, the model focuses on low-level features (e.g., edges and corner points) to collect visual information hierarchically from the dataset. As the learning period progresses, the model increases the complexity of the recognition process by extracting higher-level, discriminative features in subsequent layers. A well-trained model can be applied to unseen data collected under the same conditions as the training dataset.

[0009] Routine semen analysis does not provide comprehensive information about the fertilization potential of sperm in a sample. Sperm from men with normal semen parameters can exhibit functional defects and genetic abnormalities, leading to suboptimal fertilization outcomes. Moreover, semen parameters vary greatly between patients, making it difficult to establish thresholds to predict fertility status with absolute certainty. Men with semen parameters below reference values are not necessarily infertile, and vice versa. Therefore, the clinical significance of reference values is controversial. Furthermore, visual assessment of semen parameters relies heavily on the clinical experience and judgment of embryologists, which can lead to significant differences in assessment quality between different embryologists or from different laboratories. There is an urgent need for a highly robust and reliable automated method to assess the fertilization capacity of a semen sample based on its ZP binding capacity.

[0010] Known automated assessment of sperm morphology is described in US patent publication US20210374952A1 (now US patent No. US11926809B2, issued on March 12, 2024), the entire contents of which are incorporated herein by reference. SUMMARY

[0011] The present invention provides a method for assessing the ZP binding ability of spermatozoa through image analysis using deep learning on the morphological features of spermatozoa post Diff-Quick staining. ZP binding ability is a quantitative, independent sperm quality indicator that correlates with fertilization and pregnancy outcomes post conventional intrauterine insemination.

[0012] The present invention allows for the real-time assessment of the fertilization potential of spermatozoa independent of the results obtained through standard semen analysis. The predicted percentage of ZP binding ability generated by the model can identify patients with normal sperm counts as determined post conventional semen analysis, i.e. patients with normal spermatozoa with defective ZP binding ability. Such patients are at a higher risk of experiencing fertilization failure during conventional intrauterine insemination during IVF due to unsuccessful binding between gametes. The additional information provided by the present invention can allow patients to make an informed decision on the method of fertilization (conventional intrauterine insemination versus ICSI), preventing them from suffering psychologically and financially from fertilization failure post conventional intrauterine insemination.

[0013] The present invention is a novel approach that was specifically designed to use an advanced deep learning model to identify spermatozoa with / without ZP binding ability based on their morphological features. During the development phase, a representative database of spermatozoa images post Diff-Quick staining was established, containing spermatozoa with ZP binding ability and spermatozoa with defective ZP binding ability, which is accessible to the public for clinical and research purposes.

[0014] Provided herein is the use of deep learning to predict the morphology of spermatozoa based on the morphological features of ZP-bound and ZP-unbound spermatozoa through image analysis.

[0015] A novel database was also constructed herein, containing two categories of Diff-Quick stained spermatozoa images: 1) laboratory samples of ZP-bound spermatozoa collected using a modified sperm-ZP co-incubation experiment; and 2) clinical samples of spermatozoa with defective ZP binding ability and zero fertilization rate collected in the clinic (considered to be a truly representative class of unbound spermatozoa). Considering the differences in the microenvironment between the laboratory samples and the clinical samples, a CycleGAN was used in the study to translate the specific features of the laboratory samples into the clinical samples to generate a more consistent dataset representing sperm morphology. On this basis, a novel deep learning-based method using a CNN named VGG-13 was constructed to identify shared and unique morphological features within each category (ZP-bound and ZP-unbound spermatozoa) for a reasonable and reliable prediction of ZP binding ability.

[0016] Provided herein is a deep learning-based method for assisted reproduction, comprising:

[0017] a. obtaining at least one sample of sperm bound to zona pellucida (ZP) and at least one sample of sperm not bound to ZP;

[0018] b. obtaining one or more digital images of the ZP-bound sperm and ZP-unbound sperm by one or more imaging modalities; and

[0019] c. processing the one or more digital images according to at least one deep learning model configured to generate a binary prediction of ZP binding ability of sperm using sperm images, wherein the binary prediction is normal or defective ZP binding.

[0020] Also provided is a method of predicting fertilization success after routine artificial insemination during IVF, comprising:

[0021] a. obtaining a sample of sperm from a subject undergoing IVF;

[0022] b. obtaining one or more digital images of the sample of sperm by one or more imaging modalities; and

[0023] c. processing the one or more digital images according to at least one deep learning model to obtain a binary prediction of ZP binding ability of the sample of sperm, wherein normal ZP binding of the sample of sperm predicts fertilization success and defective ZP binding of the sample of sperm predicts fertilization failure.

[0024] Also provided is a method for predicting ZP binding ability of sperm, comprising:

[0025] a. obtaining at least one sample of sperm bound to ZP and at least one sample of sperm not bound to ZP;

[0026] b. obtaining one or more digital images of the ZP-bound sperm and ZP-unbound sperm by one or more imaging modalities; and

[0027] c. processing the one or more digital images according to at least one deep learning model configured to generate a binary prediction of ZP binding ability of sperm using sperm images, wherein the binary prediction is normal or defective ZP binding,

[0028] wherein the deep learning model is VGG-13 that predicts ZP binding ability with an accuracy, specificity, sensitivity, or a combination thereof of at least about 90%. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A schematic showing the procedure for collecting unbound ZP-bound sperm for sperm-ZP sequential co-incubation experiments. After sperm purification, a set of 4 oocytes were placed in a 30 μL droplet of motile capacitated sperm at a concentration of 1 M / mL and incubated together for 6 hours at 37 °C in a 5% CO2 incubator environment. ZP-bound sperm were then removed from the surface of the oocytes by vigorous pipetting or by pipetting manipulation at 2, 4, and 6 hours. The same batch of oocytes was placed back into the incubation droplet after every 2 hours. All the sperm remaining in the incubation droplet after 6 hours were considered unbound sperm. The collected unbound ZP-bound sperm in the droplet were air-dried and subjected to Diff-Quik staining. Images were captured at 1000x magnification under oil immersion.

[0030] Figure 2 A schematic showing the procedure for collecting ZP-bound acrosome-intact sperm for sperm-ZP co-incubation experiments. A set of 4 oocytes were placed in a 30 μL droplet of motile capacitated sperm at a concentration of 2 M / mL and incubated together for 30 minutes to collect ZP-bound acrosome-intact sperm. ZP-bound sperm were removed by vigorous pipetting. The collected ZP-bound sperm were air-dried and subjected to Diff-Quik staining for further evaluation. Images were captured at 1000x magnification under oil immersion.

[0031] Figure 3 A schematic showing the evaluation of DNA integrity of sperm by terminal deoxynucleotidyl transferase-mediated dUTP nick-end labeling (TUNEL) and COMET assays. ZP-bound sperm have significantly lower percentage of DNA strand breakage. (n=5, *p<0.05).

[0032] Figure 4 A schematic showing the evaluation of chromatin integrity of sperm by acridine orange staining. ZP-bound sperm have significantly higher percentage of chromatin integrity. (n=5, *p<0.05).

[0033] Figure 5 A schematic showing the evaluation of protamination of sperm by chromomycin A3 (CMA3) staining. ZP-bound sperm have significantly higher percentage of protamination degree. (n=5, *p<0.05).

[0034] Figure 6 A schematic showing the evaluation of methylation of sperm by immunofluorescence staining of 5-methylcytosine (5-Mc). ZP-bound sperm have significantly higher percentage of methylation level. (n=5, *p<0.05).

[0035] Figure 7Morphological assessment of sperm by manual counting according to WHO criteria is shown. Sperm bound to ZP were found to have a significantly higher percentage of normal morphology. (n=5, *p<0.05).

[0036] Figure 8 Processing and extraction of multiple independent sperm heads in a Diff-Quick stained image is shown. Independent sperm heads are identified by a standard k-means clustering algorithm and cropped to a size of 128x128.

[0037] Figure 9 Design of a hybrid AI model combining CNN and sparse neural network (SNN) is shown. Processed sperm images are input data. Normal and defective ZP binding ability are output labels. Images are fed into an artificial neural network consisting of successive hidden layers for image analysis. At the last layer, the score binding from both artificial neural networks is generated as the final prediction score. The CNN receives transformed images of sperm generated by transformation and segmentation at multiple layers including a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a first sparse fully connected layer, and a second sparse fully connected layer. The multiple layers of the CNN process the transformed images to generate a prediction of normal or defective sperm by the CNN. In parallel, a sparse neural network (SNN) receives the transformed images of sperm generated by transformation and segmentation and processes the transformed images using a feature extraction layer, an encoder dictionary that generates sparse codes, and a connection layer to generate a prediction of normal or defective sperm by handcrafted features. The prediction from the CNN and the prediction from the SNN are combined to generate a final prediction of normal or defective sperm.

[0038] Figure 10 Learning performance of the hybrid AI model for binary classification of sperm bound to ZP (recovered at 30 minutes) and sperm not bound to ZP is shown. Sperm bound to ZP have a significantly higher rate of DNA integrity, normal morphology, protaminization, and methylation compared to sperm not bound to ZP. A) Cost curve: used to evaluate the total error rate generated during training and validation. B) Accuracy curve: used to evaluate the classification accuracy of the model during training. Figure 9 The hybrid AI model in the middle achieved a validation accuracy of 81.6% and a validation cost of 0.480, indicating that morphological features of subpopulations of sperm can be clearly identified by the model.

[0039] Figure 11 Learning performance of the hybrid AI model for binary classification of sperm with normal and defective ZP binding ability is shown. Figure 9 The hybrid model in the middle achieved a validation accuracy of 76.2% and a validation cost of 0.494, indicating that morphological features of subpopulations of sperm can be clearly identified by the model.

[0040] Figure 12 The development process of the deep learning-based method for assessing the ZP binding capacity of human sperm is shown. (A) Images are input into the VGG-13 model consisting of successive hidden layers for image analysis. Considering the limited availability of sperm images and the human intervention involved in the WHO standard in SNN, transfer learning using VGG-13 was chosen as the best technique to develop a deep learning-based method for binary classification. To fine-tune the classifier while taking advantage of the capabilities of the pre-trained VGG-13 model, only the last layer of the network was modified, while the weights of the preceding layers remained unchanged. (B) Accuracy curve: to assess the overall reliability of the model (left). Cost curve: to measure the error rate of the classification (right). (C) ROC curve: to test the discriminant ability of the model as reflected by the AUC value. (D) Confusion matrix: performance measure of the model for binary classification.

[0041] Figure 13 The correlation between ZP binding capacity and fertilization rate after conventional IVF using the newly fine-tuned VGG-13 model is shown. Images of the Diff-Quik staining of three groups of men with fertilization rates of 0-40%, 41-70%, and 71-100% were collected. The percentage of ZP binding capacity for each sample was determined using the VGG-13 model.

[0042] Figure 14 The correlation between ZP binding capacity and fertilization rate after conventional IVF using the original VGG-13 model (not fine-tuned using our database) is shown. Images of the Diff-Quik staining of three groups of men with fertilization rates of 0-40%, 41-70%, and 71-100% were collected. The percentage of ZP binding capacity for each sample was determined using the original VGG-13 model (not fine-tuned using a representative dataset).

[0043] Figure 15 The process of determining the optimal cutoff value for clinical prediction using the newly fine-tuned VGG-13 model is shown. Using the predicted ZP binding capacity percentages of the high and low fertilization groups, an ROC curve for logistic regression was established. At the selected threshold, the specificity was 93.0% and the sensitivity was 91.7%.

[0044] Figure 16A comparison of conventional semen analysis and the clinical relevance of the present application is shown. Clinical data from ten patients undergoing conventional IVF was collected from an IVF clinic to further validate the clinical significance of the present application. All patients recruited in this study were determined by embryologists to have normal semen analysis results and underwent conventional IVF as the primary treatment. However, despite patients 1-4 having normal semen analysis results (good motility and concentration), they failed to achieve high fertilization rates, indicating a lack of correlation between these semen parameters and fertilization outcomes. The residual semen samples of these patients were further examined using the present application to assess the percentage of ZP binding capacity. The ZP binding analysis results determined that patients 1-4 should not be recommended for IVF due to the lack of ZP binding capacity, which also coincides with the lower IVF fertilization rates.

[0045] Figure 17 A-17B shows the image processing and extraction of multiple independent sperm heads, where Figure 17 A shows the identification of independent sperm heads by the standard k-means clustering algorithm and cropping of the independent sperm heads to a pixel size of 128x128 using batch normalization and contrast adjustment, as well as the identification of the minimum region around the sperm head by the k-means clustering algorithm, Figure 17 B shows representative images of processed sperm heads extracted from the original images.

[0046] Figure 18 The development process of the VGG-13 model for assessing the ZP binding capacity of human sperm is shown, where the images are inputted into the VGG-13 model consisting of consecutive hidden layers for binary classification, while the VGG-13 model comprises a feature extractor and a classifier connected by an average pooling layer. The classifier comprises 1 dropout layer and 3 fully connected layers. Figure 18 The accuracy curve for assessing the overall reliability of the model, the cost curve for measuring the error rate of the classification, the receiver operating characteristic (ROC) curve for testing the discriminative ability of the VGG-13 model as reflected by the area under the ROC curve (AUC), and the confusion matrix for measuring the classification performance of the model for binary classification are also shown.

[0047] Figure 19 The evaluation results of the prediction reproducibility generated by the fine-tuned VGG-13 model are shown, where four sets of processed samples were collected and processed using the established protocol described herein, where two independent mounts of each sample were inputted into the VGG-13 model to determine the predicted percentage of ZP binding capacity. The low discrepancy in the predicted values between the mounts indicates that the VGG-13 model is able to generate comparable outputs for different image sets of the same image, indicating high prediction reproducibility.

[0048] Figure 20A-20G shows a saliency map of a representative human sperm from the sperm dataset. The saliency map highlights the relevant features that contribute to the predicted output of the VGG-13 model by the pixel importance shown in orange. The overall percentage of pixel units located in the front region relative to the entire sperm head was quantified using a predetermined image. Figure 20 A-20B shows sperm with high pixel importance percentage that are clustered in the front region. Figure 20 C-20D shows sperm with medium pixel importance percentage that are clustered in the front region. Figure 20 E-20F shows sperm with low pixel importance percentage that are clustered in the front region. Figure 20 G shows a comparison of the overall percentage of pixel importance of sperm that bound ZP and sperm that did not bind ZP in the entire sperm dataset.

[0049] Figure 21 A positive correlation between ZP binding capacity and fertilization rate after conventional intrauterine insemination in IVF using the newly fine-tuned VGG-13 model is shown. Three groups of men (n=165, p<0.001) with fertilization rates of 0-40%, 41-70%, and 71-100% were collected for their Diff-Quick stained images as described herein. The ZP binding capacity percentage for each sample was determined using the fine-tuned VGG-13 model.

[0050] Figure 22 Identification of the optimal cutoff value for clinical prediction using the newly fine-tuned VGG-13 model is shown. The ROC curve of the logistic regression was established using the predicted ZP binding capacity percentage of the high and low fertilization groups (n=113). At the selected threshold of 4.8%, the specificity was 93.8% and the sensitivity was 90.9% for the binary classification of ZP binding capacity.

[0051] Figure 23 A comparison of conventional semen analysis and clinical relevance of the present invention is shown, where clinical data of 30 patients who underwent conventional IVF were collected from an IVF clinic to further validate the clinical significance of the VGG-13 model. All patients recruited in this study were determined by embryologists to have normal semen analysis results and underwent conventional intrauterine insemination as the primary method of fertilization. The patients’ residual sperm samples were further examined using the VGG-13 model to assess the predicted ZP binding capacity percentage.

[0052] Figure 24Collection of ZP-bound sperm using modified sperm-ZP co-incubation experiment, where a set of 4 oocytes were incubated with 2M / mL of motile capacitated sperm for 30 minutes to collect acrosome-intact ZP-bound sperm. The collected ZP-bound sperm were air-dried and subjected to Diff-Quik staining as described herein. Images of sperm were captured under oil immersion at 1000x magnification.

[0053] Figure 25 Representative images of poor quality sperm were shown, and images of sperm heads extracted therefrom were examined to identify and reject images with poor features in the dataset to ensure a comprehensive representation of sperm morphology.

[0054] Figure 26 Comparison of training accuracy generated by the model at different learning rates (LR) was shown, where training accuracy deteriorated as the learning rate was increased to 0.1 and 0.01, respectively, indicating that the model did not converge to an optimal condition during training. A learning rate of 0.001 was selected as the optimal learning rate for the VGG-13 model.

[0055] Figure 27 A-27B shows the development of a CNN-based basic method for binary classification as a VGG-13 model for assessing ZP-binding ability of human sperm, where images were inputted into Figure 27 A shows the CNN-based VGG-13 model consisting of two convolutional layers and two max-pooling layers as consecutive hidden layers for binary classification, and the VGG-13 model includes a feature extractor and a classifier connected through an average pooling layer. The classifier includes 1 dropout layer and 3 fully connected layers. Figure 27 B also shows accuracy curves for assessing the overall reliability of the model, cost curves for measuring the error rate of classification, receiver operating characteristic (ROC) curves for examining the discriminative ability of the VGG-13 model as reflected by the area under the ROC curve (AUC), and confusion matrices for measuring the classification performance of the model for binary classification.

[0056] Figure 28 A-28B shows the comparison and evaluation of the generalization ability of the basic CNN and fine-tuned VGG-13 model, where both models were tested for binary prediction on two independent test datasets of ZP-bound sperm (n = 122) and non-ZP-bound sperm (n = 98). As shown in Figure 28 B, the basic CNN model misclassified most of the sperm in the non-bound sperm dataset as sperm with ZP-binding ability, indicating that the CNN model could not generalize well to unseen data for prediction.

[0057] Figure 29The correlation between ZP binding capacity and fertilization rate after conventional IVF using the original VGG-13 model (not fine-tuned with the dataset) is shown. Three groups of men (n = 101) with fertilization rates of 0-40%, 41-70%, and 71-100% were collected for their Diff-Quik stained images. The predicted percentage of ZP binding capacity for each sample was determined using the original VGG-13 model (not fine-tuned with the representative dataset).

[0058] Figure 30 A schematic diagram of a system according to an embodiment is shown.

[0059] Figure 31 A schematic diagram of a computing device used in an embodiment of the system in Figure 30 is shown.

[0060] Figure 32 A schematic diagram of a module used in an embodiment of the system in Figure 30 is shown.

[0061] Figure 33 A schematic diagram of a classifier according to an alternative embodiment is shown.

[0062] Figure 34 A flowchart of a method of collecting sperm and capturing images of the collected sperm is shown.

[0063] Figure 35 A flowchart of a method of operating the system in Figure 30 is shown. DETAILED DESCRIPTION

[0064] The model performed well in classifying subpopulations of sperm with a high specificity of 96.0% and a high sensitivity of 97.6%. Three groups of men with fertilization rates (FR) of 0-40%, 41-70%, and 71-100% were recruited from one assisted reproductive program to examine the correlation between ZP binding capacity and fertilization rate of IVF. The predictive ability of the model was tested on a total of 100 clinical samples from the aforementioned fertilization groups. The percentage of ZP binding capacity was higher in the high fertilization group (71-100%) than in the low fertilization group (0-40%). The threshold value calculated for distinguishing normal and defective ZP binding capacity of sperm samples was 4.78% (here, specificity: 93.0%, sensitivity: 90.5%), which can be used for clinical prediction of ZP binding capacity in morphological evaluation at the single-cell level.

[0065] The present disclosure aims to evaluate the percentage of ZP binding capacity of sperm samples according to morphological features of Diff-Quik stained sperm samples using deep learning, and patients with a percentage of ZP binding capacity below the threshold value of 4.78% will be recommended to undergo ICSI to improve the fertilization rate.

[0066] The following examples further illustrate the discovery of the present disclosure.

[0067] Example

[0068] ZP-bound spermatozoa were collected through previously modified sperm-ZP co-incubation experiments, as shown in Figure 11 Clinical samples of spermatozoa collected from IVF laboratories with defective ZP binding capacity, as reflected by complete fertilization failure after routine artificial insemination during IVF, were considered as a truly representative class of non-bound spermatozoa. ZP-bound spermatozoa and non-ZP-bound spermatozoa were processed through k-means clustering algorithm for color-based segmentation after DIF fast staining images, to extract independent sperm heads from the background, as shown in Figure 8 All extracted sperm heads were cropped to a pixel size of 128x128 and converted to grayscale for training, as shown in Figure 8

[0069] ​Sperm bound to ZP and sperm not bound to ZP were collected under laboratory and clinical conditions, respectively, which can cause subtle microenvironmental differences that, while not detectable by the naked eye, greatly interfere with the deep learning classifier and generate false interpretations of the dataset during training and validation. To address this issue, a Cycle-Generated Adversarial Network (CycleGAN) was used to translate the prominent features of the laboratory samples to the clinical samples to generate a converted dataset that resembles the microenvironment of the laboratory samples. The present invention was built in Python (3.10.6) using an advanced deep learning architecture named VGG-13. VGG-13 was pre-trained on ImageNet 1000 pre-defined classes and further fine-tuned using a newly established database containing 1,094 Diff-Quick stained images of sperm bound to ZP and sperm not bound to ZP, with 80% of the entire dataset assigned to the training set. 20% of the entire dataset was assigned to the validation set. The VGG-13 model was trained on datasets collected from different conditions (laboratory samples vs. clinical samples). All clinical images were converted using the CycleGAN model before training and testing. To fine-tune the classifier for the specific downstream task of sperm classification while leveraging the capabilities of the pre-trained VGG-13 model, the last layer of the network was modified while the weights of the preceding layers remained unchanged. The number of channels doubled every two layers (ranging from 64 to 512) with a max-pooling layer in between each doubling. During training and validation, the input data (pre-processed, labeled images) was inputted into the VGG-13 network consisting of 10 convolutional layers, which generated independent scores that contribute to the final prediction (sperm bound to ZP or sperm not bound to ZP) over 50 epochs. Forward propagation was performed, whereby the input data was processed via successive hidden layers using 3x3 convolutional filters by multiplying the weight values by the node values to determine the node values in the next layer. More specifically, an interconnected node was activated when the output value of any individual node in the previous layer was above a threshold value. Additional data augmentation, including horizontal flipping and random resizing, was applied to further increase the effective size and quality of the training dataset. Backpropagation was performed simultaneously to adjust the weights on individual nodes such that the error rate was minimized. In the fully connected layer, the combined value generated by the model was the final prediction. The hyperparameters of the model were calibrated as follows: 1) maximum number of epochs: 50 epochs; 2) batch size: 4; and 3) learning rate: 1x10⁻¹.

[0070] The newly fine-tuned VGG-13 model (as shown in FIG. A) proved to successfully classify sperm bound to ZP and sperm not bound to ZP with high accuracy and low cost (a measure of total error rate) during training and validation (as shown in FIG. B). Figure 12 A the newly fine-tuned VGG-13 model (as shown in FIG. A) proved to successfully classify sperm bound to ZP and sperm not bound to ZP with high accuracy and low cost (a measure of total error rate) during training and validation (as shown in FIG. B). Figure 12B). The model also showed high discriminative ability to distinguish between ZP-bound and ZP-unbound sperm according to morphological features of sperm bound to ZP and sperm unbound to ZP, as reflected by the area under the receiver operating characteristic (ROC) curve (AUC = 0.992) (Fig. 4B). Figure 12 C) as reflected by the confusion matrix. The classification performance of the model was assessed by the confusion matrix, with an accuracy of 96.4%, a specificity of 96.0%, and a sensitivity of 97.6% (Fig. 4C). Figure 12 D).

[0071] To test the predictive ability of the model on the fertilization outcome, the Diffast-stained images of three fertilization groups with IVF fertilization rates of 0-40%, 41-70%, and 71-100% were input into the model for testing. The overall percentage of ZP-binding ability of each sample was correlated with the fertilization rate after IVF (Fig. 5A). Figure 13 Further validation was performed to compare the classification efficiency of ZP-binding ability by the present application and the original VGG-13 model (not fine-tuned using the dataset). The results showed that the original VGG-13 model could not effectively distinguish between ZP-bound and ZP-unbound sperm, resulting in a higher percentage of ZP-binding ability in all fertilization groups (Fig. 5B). Figure 14 The overall predictive ability of the new fine-tuned VGG-13 model was superior to the original model. The ROC using logistic regression was used to estimate the AUC (0.094) and P (<0.0001) values as a measure of discrimination between high and low fertilization groups, as shown in Fig. 5C. Figure 15 The best cutoff value (here, specificity = sensitivity) on the ROC curve generated by the data was identified using the Youden index. The cutoff value of 4.78% (here, specificity: 93.0%, sensitivity: 90.5%) was a clinical threshold for distinguishing between normal and defective ZP-binding ability of semen samples.

[0072] The clinical data of ten patients who underwent conventional IVF were selected to test the clinical significance of conventional semen analysis and the present application, as shown in Fig. 6. Figure 16The ten patients were all determined by an embryologist to have normal semen analysis results and underwent conventional IVF as the primary treatment. Consistent with the normal semen analysis results, patients 5-10, who had high rates of fertilization (71-100%) after conventional IVF, had a percentage of ZP binding capacity above the clinical threshold (4.78%) determined by the present application. However, despite having normal semen analysis results, patients 1-4 failed to achieve high rates of fertilization. The present application determined that these patients had a low percentage of ZP binding capacity (possibly due to a defect in ZP binding capacity that caused the IVF treatment to fail), and they should have received ICSI instead of IVF as the method of fertilization to improve the chances of success of fertilization. Conventional semen analysis could not identify in the preliminary screening those patients with low ZP binding capacity of sperm that required a more invasive treatment than the traditional in vitro fertilization treatment. On the other hand, the preliminary results demonstrate that the present application has high prognostic value compared to the use of the results of conventional semen analysis alone. In other words, the present application can be used as part of the semen analysis to provide complementary information about the fertilization potential of a semen sample, especially in cases where the patient appears to have normal semen results but actually has a defective ZP binding capacity.

[0073] Supplemental embodiments

[0074] In routine semen analysis, sperm morphology is manually assessed as one of the surrogate markers of male fertility potential. However, the predictive ability of sperm morphology on in vitro fertilization and pregnancy outcome remains controversial, mainly due to large inter-laboratory or inter-laboratory personnel differences in the human standard of assessment. To address the limitations of traditional methods of assessing sperm morphology, a novel deep learning model was provided for identifying sperm with the ability to bind to the zona pellucida (ZP) of human oocytes, which is the first step of fertilization. The VGG-13 model was fine-tuned to classify 1,083 images of ZP-bound and ZP-unbound sperm with high accuracy (96.4%), specificity (96.0%), and sensitivity (97.6%) according to the distinctive morphological features of ZP-bound and ZP-unbound sperm. The model was validated on over 40,000 sperm images collected from three groups of infertile men (n=165) with a range of 0-40% (low), 41-70% (medium), and 71-100% (high) fertilization rates after in vitro fertilization using routine manual insemination. Using images of sperm from 113 infertile men with high and low fertilization rates from the above groups, a threshold of 4.8% for sperm morphology was established using a receiver operating characteristic curve (area under the curve: 0.970) that could distinguish between samples of sperm with normal and defective ZP binding ability with high specificity (93.8%) and high sensitivity (90.9%). In summary, the fine-tuned VGG-13 model is a highly robust automated method for assessing sperm fertilization ability.

[0075] Sperm and oocyte collection was performed using residual semen samples collected from men. Normal sperm samples were selected according to the following predetermined WHO criteria (5thedition) 6: total volume > 1.5 mL, total motility > 40%, progressive motility > 32%, total number of sperm per ejaculation > 39 x 10 6 , concentration > 15 x 10 6 / mL, viability > 58%, morphology > 4%. Motile and viable sperm were isolated from the seminal plasma using the direct upstream method as previously described 28. All treated sperm were incubated in Earles balanced salt solution (EBSS) with 3% BSA (EBSS / 3% BSA) to induce capacitation for subsequent co-incubation 29. For clinical analysis, clinical semen samples were processed using the density gradient centrifugation method (DCG) by sequential layering of two different densities (40% / 80%) of colloidal medium for centrifugation.

[0076] Matured metaphase II oocytes were obtained from a commercial source (Cryos International, Denmark). Morphologically normal oocytes were stored in a high salt oocyte storage buffer containing 1.5 M MgCl2, 0.1% polyvinylpyrrolidone and 40 mM HEPES, pH 7.2, 4°C until use.

[0077] For the collection of data in the stored data sets, laboratory samples of human sperm binding to ZP were obtained. Sperm with intact acrosomes, binding to ZP, were collected in our modified sperm-ZP co-incubation experiment as shown in Figure 24 Briefly, 4 human oocytes were co-incubated in 30 μL and containing 2 x 10 6 sperm in EBSS / 3% BSA and covered with a mineral oil droplet at 37°C, 5% CO2for 30 min. After incubation, oocytes were washed sequentially in 3 EBSS / BSA-free droplets to remove loosely bound sperm. Then, ZP-bound sperm were removed from the surface of oocytes by vigorous pipetting using a fine-bore glass pipette in a defined area on a sterile glass slide containing 10 μL BSA-free EBSS.

[0078] Clinical samples of human sperm not binding to ZP and patients were assigned to three fertilization rate groups. Normal sperm samples with defective ZP binding capacity as evidenced by complete fertilization failure after routine artificial insemination in IVF and no sperm binding to ZP on oocytes after artificial insemination were considered as truly representative of the class of sperm not binding to ZP. Three groups of samples with fertilization rates (FR) of 0-40%, 41-70% and 71-100% were collected to examine the correlation between ZP binding capacity and fertilization rate after routine artificial insemination in IVF and to identify a clinical threshold that distinguishes normal and defective ZP binding capacity of sperm samples after treatment.

[0079] The procedure for Diff-Quick staining of sperm was such that all collected sperm were Diff-Quick stained. Briefly, air-dried sperm were fixed with a methanol-based fixative for 15 seconds, stained with stain 1 (buffered solution of Eosin Y) for 10 seconds, followed by stain 2 (buffered solution of thiazine dyes) for 10 seconds. Excess solution was allowed to drop from the slide between steps. Diff-Quick stained images of sperm were captured under an optical microscope (Zeiss, Gottingen, Germany) at 1000x magnification using oil immersion. Reference is made to Figure 34 Collection of sperm and oocyte samples and capture of stained sperm images are described.

[0080] The disclosed invention then uses a disclosed deep learning based method with an advanced CNN architecture named VGG-13, pre-trained on VGG-13 on ImageNet 1000 pre-defined classes, and further fine-tuned using a newly established database containing a total of 1,083 Diff-Quik stained images of spermatozoa with and without ZP binding. All Diff-Quik stained images of spermatozoa were processed for color-based segmentation by a known k-means clustering algorithm to extract and identify individual sperm heads from the background, as shown in Figure 17 A. The extracted sperm heads were cropped to a pixel size of 128 x 128, rotated to an upright position, and converted to grayscale before use, as shown in representative images of processed sperm heads extracted from the original images in Figure 17 B. To prevent the model from being misled or recognizing irrelevant information during training, images of poor quality spermatozoa, including images with debris and / or background smears, were manually removed from the dataset, as shown in Figure 25 Clinical conditions can cause subtle microenvironmental differences that are not detectable by the naked eye but greatly interfere with the deep learning classifier and misinterpret the dataset during training and validation. To address this issue, a Cycle-Generative Adversarial Network (CycleGAN) was used to translate the distinctive features of the laboratory samples to the clinical samples to generate a converted dataset that resembles the microenvironment of the laboratory samples. More specifically, all clinical samples used in this study underwent an additional image processing step for image conversion by CycleGAN. 80% of the entire dataset was randomly assigned to the training set, and the remaining 20% of the entire dataset was assigned to the validation set. An additional 220 images were collected as a test set to examine the generalization ability of the fine-tuned model.

[0081] The disclosed VGG-13 model was implemented in PyTorch. The deep learning algorithm was adapted from Sasank Chilamkurthy’s tutorial on transfer learning in PyTorch. In VGG-13, the number of channels doubles every two layers (ranging from 64 to 512) with a 2 x 2 max-pooling layer in between each doubling. During training and validation, the input data was fed into the VGG-13 network consisting of 10 convolutional layers, which generated independent scores that contribute to the final prediction over 50 epochs. The model consists of two parts, a feature extractor and a classifier, connected by an average pooling layer. To fine-tune the classifier for the specific downstream task of sperm classification while leveraging the power of the pre-trained VGG-13 model, only the last layer of the network was modified, while the weights of the preceding layers remained unchanged. In one embodiment, the configuration of the classifier is as shown in Figure 33The classifier consists of three linear layers, with a rectified linear unit (ReLU) layer and a dropout layer sequentially following the first two linear layers. In the last fully connected linear layer, the number of output features is modified from 1,000 to 2 for binary classification. Forward propagation is performed to process the input data via successive hidden layers using 3x3 convolutional filters by multiplying the weight by the node value to determine the node value in the next layer. More specifically, an interconnected node is activated when the output value of any individual node in the previous layer is above a threshold value. Additional data augmentation, including horizontal flipping and random resizing, is applied to further improve the effective size and quality of the training dataset. Simultaneously, stochastic gradient descent (SGD) is performed to adjust the weights on individual nodes to minimize the error rate. Cross-entropy loss is used to measure the difference between the predicted probability and the true class label. The hyperparameters of the model are calibrated as follows: 1) maximum number of epochs: 50 epochs; 2) batch size: 4; and 3) learning rate: 1x10⁻¹. The model is developed and deployed using GOOGLE Colab equipped with NVIDIA Tesla P100 or NVIDIA Tesla V100. Customized code is specific to the computing device used for image processing and binary classification.

[0082] The performance of the model for binary classification is evaluated in terms of accuracy, specificity, sensitivity, precision, and recall using a confusion matrix. The discriminative ability of the model to distinguish between sperm bound to ZP and sperm not bound to ZP based on morphological features of sperm bound to ZP and sperm not bound to ZP is evaluated using the area under the receiver operating characteristic (ROC) curve (AUC). A saliency map is used to identify the pixels most relevant to the classification and to determine the most salient regions of the image. The ROC of logistic regression is used to estimate the AUC and P-value as a measure of discrimination between the high and low fertilization groups of the clinical sample. The Youden index (sensitivity + specificity - 1) is used to determine the optimal cutoff value on the ROC curve generated by the data.

[0083] Statistical software of GraphPad Prism 9.1.0 by GraphPad Software, Inc. of California, USA is used to analyze the clinical data. A two-tailed unpaired t-test is used to test the difference between the three groups of fertilization rates. If the data does not pass the normality test, a Mann-Whitney (non-parametric) test is used for statistical analysis. A probability value <0.05 is considered statistically significant.

[0084] The VGG-13 model was fine-tuned using a database of 1,083 total Diff-Quick stained images containing the following classes: 1) ZP-bound sperm: laboratory samples of sperm that bound to the ZP of oocytes in a modified sperm-ZP co-incubation experiment; and 2) ZP-unbound sperm: clinical samples of sperm with defective ZP binding capacity that failed to fertilize after conventional artificial insemination. Transfer learning allowed the existing VGG-13 model to retain the information previously acquired from ImageNet’s 1,000 pre-defined classes for feature extraction and adapt to the new task by adjusting its parameters on the dataset of Diff-Quick stained sperm. The newly fine-tuned VGG-13 model (as shown in Figure 18 A) demonstrated successful classification of ZP-bound and ZP-unbound sperm with high accuracy and low cost (measured by total error rate) during training, as shown in Figure 18 B) did not show any signs of overfitting over 50 epochs. The model was trained on the dataset using a batch size of 4 with learning rates ranging from 0.0001 to 0.1, as shown in Figure 26 . The results showed that high learning rates had a negative impact on weight adjustments related to loss gradients during training, and therefore, 0.001 was chosen as the best learning rate for the model, as reflected by the high accuracy, as shown in Figure 26 .

[0085] The model distinguished between ZP-bound and ZP-unbound sperm based on morphological features of ZP-bound and ZP-unbound sperm, with an AUC of 0.992 for the ROC curve, as shown in Figure 18 C. The model achieved good classification performance, as determined by the confusion matrix, with high accuracy (96.4%), specificity (96.0%), and sensitivity (97.6%), as shown in Figure 18 D. The observed differences between two independent sets of sample images were small, indicating good predictive reproducibility of the model, as shown in Figure 19 . The saliency maps, such as Figure 20 A-20F, indicated that the model consistently focused on areas corresponding to the sperm head and midpiece in all images, with minimal focus on background noise and debris. Analysis of heatmaps generated using the training dataset showed that the model primarily emphasized the anterior region of the sperm head, as shown in Figure 20 G, which was 43.0% ± 5.0% in both ZP-bound and ZP-unbound sperm. Overall, the saliency maps indicated that the model specifically detected and recognized prominent features of the sperm head, revealing the decision-making process used within the black-box method.

[0086] To further validate the effectiveness of transfer learning for binary classification of human sperm, the classification performance of a simple CNN model trained from scratch was compared to that of the fine-tuned VGG-13 model. The CNN model consisted of 2 convolutional layers and 2 max-pooling layers, as shown in Figure 27 A. The model was trained on the dataset for 50 epochs with a batch size of 4 using a learning rate of 0.001, as shown in Figure 27 B. Although the model was able to distinguish between the subgroups of sperm to some extent, as shown in Figure 27 C, its overall classification performance (accuracy: 80.4%, specificity: 81.0%, sensitivity: 79.7%) was inferior to that of the fine-tuned VGG-13 model, as shown in Figure 27 D. To further test the generalization ability of the CNN and fine-tuned VGG-13 models, both models were tested on an independent test dataset consisting of 122 sperm bound to ZP and 98 sperm not bound to ZP. The CNN model classified 68.4% of the images of sperm not bound to ZP in the test dataset as sperm not bound to ZP, indicating its lack of generalization ability to classify unseen data, as shown in Figure 28 A. In contrast, the fine-tuned VGG-13 model successfully distinguished between the two classes of sperm in approximately 99% of the images based on the prominent features of the two classes of sperm, indicating its excellent generalization ability, as shown in Figure 28 A-28B.

[0087] Three groups of men (n = 165) with fertilization rates of 0-40%, 41-70%, and 71-100% were recruited from an assisted reproduction program to examine the relationship between ZP binding ability using deep learning and fertilization rate in IVF, as shown in Figure 21 A. The predictive ability of the model was tested on over 40,000 images of sperm heads individually extracted from the above fertilization groups. The percentage of ZP binding ability in the high fertilization rate group (71-100%) was significantly higher than that in the low fertilization rate group (0-40%), as shown in Figure 21 B. To further validate the classification efficiency of the model in ZP binding ability, it was compared to the original VGG-13 model (not fine-tuned using the dataset). The results showed that the original VGG-13 model was unable to distinguish between sperm bound to ZP and sperm not bound to ZP, as reflected by the high percentage of ZP binding ability predicted in all fertilization groups (68.2%, 83.9%, and 84.2%), as shown in Figure 29 C. The overall generalization ability of the new fine-tuned VGG-13 model was superior to that of the original model; the AUC for distinguishing between the high fertilization rate group and the low fertilization rate group was 0.97 (in Figure 22In this case, P < 0.0001, n = 113). The optimal threshold was identified on the generated ROC curve using the Youden index. A cut-off of 4.8% (here, specificity of 93.8%, sensitivity of 90.9%) was determined as the clinical threshold to discriminate between normal and defective ZP-binding capacity of treated sperm samples.

[0088] Clinical data of 30 patients undergoing conventional intrauterine insemination in IVF were randomly selected to test the clinical relevance of the model with conventional semen analysis, as Figure 23 shown. All patients underwent conventional IVF according to normal semen analysis results during the Infertility Investigation. Their semen parameters as well as the parameters of DGC-treated sperm (including concentration and progressive motility on the day of insemination) met the predetermined laboratory criteria for conventional intrauterine insemination. The predicted percentage of ZP-binding capacity determined by the model for patients 1-11 was higher than the clinical threshold of 4.8%. Although patients 2, 3, 10 and 12 had a low sperm morphology below the WHO reference value, the above-mentioned patients had a high fertilization rate (71-100%) after conventional intrauterine insemination. On the other hand, the predicted percentage of ZP-binding capacity for patients 12-21 with a low fertilization rate (0-40%) was lower than the threshold of the model. Among them, although patients 16, 17, 19, 20, 21, 23, 24, 26 and 30 had normal semen parameters, they were found to have suboptimal results. These findings demonstrate the prognostic value of the model, highlighting the potential role of ZP-binding capacity as an additional independent measure of sperm fertilizing capacity, complementing conventional semen parameters.

[0089] In one embodiment according to the present application, as Figure 30 shown, the system 3000 includes a deep learning system 3002 having a hardware-based processor 3004, a memory 3006 configured to store instructions and configured to provide the instructions to the hardware-based processor 3004, a communication interface 3008, an input / output device 3010, and a set of modules 3012 configured to implement the instructions provided to the hardware-based processor 3004.

[0090] A sperm sample is obtained from a sperm sample source 3014 and a sperm image of the sperm sample is obtained by an image capture device 3016, as described above. The image capture device 3016 transmits, communicates, or otherwise provides the sperm image to a sperm image source 3018. In one embodiment, the sperm image source 3018 stores the captured sperm image in a memory of the sperm image source 3018. In another embodiment, the sperm image source 3018 transmits, communicates, or otherwise provides the image 3020 to the deep learning system 3002 through the communication interface 3008. For example, the image 3020 is a digital still image. In another example, the image 3020 is a digital still image obtained by video capture of the sperm.

[0091] In yet another embodiment, the sperm image source 3018 formats the image 3020 into a predetermined image format for processing by the deep learning system 3002. Based on the image 3020, the deep learning system 3002 generates a prediction 3022 regarding the percentage of sperm in the sample source 3014 that have ZP binding capability. The prediction 3022 is output from the input / output device 3010.

[0092] In one embodiment according to the present application, the prediction 3022 is a binary classification. For example, the binary prediction 3022 indicates normal or defective ZP binding of the sperm under evaluation. In one embodiment, the prediction 3022 is a message, alert, or notification displayed to a user on a display or monitor of the input / output device 3010. In another embodiment, the prediction 3022 is a message, alert, or notification printed as a hard copy by a physical printer of the input / output device 3010.

[0093] In another embodiment, supplemental data 3024 is transmitted, communicated, or otherwise provided to the deep learning system 3002 through the communication interface 3008. For example, the supplemental data 3024 includes training data used to train a deep learning model as described herein, as described above. In another example, the supplemental data 3024 includes metadata identifying the sperm and the patient from which the sperm was provided.

[0094] In one embodiment, the deep learning system 3002 is operatively connected to data sources such as the sperm image source 3018 and supplemental data through a network. For example, the network is the Internet. In another example, the network is an internal network or intranet of an organization. In yet another example, the network is a heterogeneous or hybrid network including the Internet and an intranet.

[0095] Figure 31A schematic of a computing device 3100 is shown, which includes a processor 3102 having code therein, a memory 3104, and a communication interface 3106. Optionally, the computing device 3100 can include a user interface 3108, such as an input device, an output device, or an input / output device, such as the input / output device 3010. The processor 3102, the memory 3104, the communication interface 3106, and the user interface 3108 are operatively connected to one another via any known connection, such as a system bus, a network, or the like. Figure 30 Any of the components, combinations of components, and modules of the system 3000 in Figure 30 Each of the components 3002-3018 shown can be implemented by Figure 31 the corresponding computing device 3100 shown and described below.

[0096] It is to be understood that the computing device 3100 can include different components. Alternatively, the computing device 3100 can include additional components. In another alternative, portions of or all of the functionality of a given component can be implemented by one or more different components. The computing device 3100 can be implemented by a virtual computing device. Alternatively, the computing device 3100 can be implemented by one or more computing resources in a cloud computing environment. Further, the computing device 3100 can be implemented by a plurality of any known computing devices.

[0097] The processor 3102 can be a hardware-based processor that implements a system, subsystem, or module. The processor 3102 can include one or more general-purpose processors. Alternatively, the processor 3102 can include one or more special-purpose processors. The processor 3102 can be integrated in whole or in part with the memory 3104, the communication interface 3106, and the user interface 3108. In another alternative, the processor 3102 can be implemented by any known hardware-based processing device, such as a controller, an integrated circuit, a microchip, a central processing unit (CPU), a microprocessor, a system on a chip (SoC), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Further, the processor 3102 can include a plurality of processing elements configured to perform parallel processing. In yet another alternative, the processor 3102 can include a plurality of nodes or artificial neurons configured as an artificial neural network. The processor 3102 can be configured to implement any known machine learning (ML)-based device, any known artificial intelligence (AI)-based device, and any known artificial neural network, including a convolutional neural network (CNN).

[0098] The memory 3104 can be implemented as a non-transitory computer readable storage medium such as a hard drive, solid state drive, erasable programmable read only memory (EPROM), universal serial bus (USB) storage, floppy disk, compact disk read only memory (CD-ROM) disc, digital versatile disk (DVD), cloud-based storage, or any known non-volatile storage.

[0099] The code of the processor 3102 can be stored in a memory internal to the processor 3102. The code can be instructions implemented in hardware. Alternatively, the code can be instructions implemented in software. The instructions can be machine language instructions executable by the processor 3102 to cause the computing device 3100 to perform the functions of the computing device 3100 described herein. Alternatively, the instructions can include script instructions executable by a script interpreter configured to cause the processor 3102 and the computing device 3100 to perform the instructions specified in the script instructions. In another alternative, the instructions are executable by the processor 3102 to cause the computing device 3100 to perform an artificial neural network. The processor 3102 can be implemented using hardware or software such as the code. The processor 3102 can implement a system, subsystem, or module as described herein.

[0100] The memory 3104 can store data in any known format such as a database, data structure, data lake, or network parameters of a neural network. The data can be stored in tables, flat files, data in a file system, heap files, B+ trees, hash tables, or hash buckets. The memory 3104 can be implemented by any known memory including random access memory (RAM), cache memory, register memory, or any other known memory device configured to store instructions or data for quick access by the processor 3102, including memory of instructions during execution.

[0101] The communication interface 3106 can be any known device configured to perform the communication interface functions of the computing device 3100 described herein. The communication interface 3106 can enable wired communication between the computing device 3100 and another entity. Alternatively, the communication interface 3106 can enable wireless communication between the computing device 3100 and another entity. The communication interface 3106 can be implemented by an Ethernet, Wi-Fi, Bluetooth, or USB interface. The communication interface 3106 can transmit and receive data to other devices over a network using any known communication link or communication protocol.

[0102] User interface 3108 can be any known device configured to perform user input and output functions. User interface 3108 can be configured to receive input from a user. Alternatively, user interface 3108 can be configured to output information to a user. User interface 3108 can be a computer monitor, television, speaker, computer speaker, or any other known device operatively connected to computing device 3100 and configured to output information to a user. User input can be received through user interface 3108, which is implemented as a keyboard, mouse, or any other known device operatively connected to computing device 3100 to input information from the user. Alternatively, user interface 3108 can be implemented using any known touchscreen. Computing device 3100 can include a server, personal computer, laptop computer, smartphone, or tablet computer.

[0103] like Figure 32 As shown, in Figure 30 In the implementation of system 3000, module 3012 includes at least a deep learning module 3202 configured to operate a deep learning model 3204 such as VGG-13. The deep learning model 3204 is trained and configured as described herein to generate a prediction 3022 from image 3020. Module 3012 also includes an image processing module 3206, a prediction module 3208, and a CycleGAN module 3210. Image processing module 3206 is configured to process the digital image 3020 as described above. Prediction module 3208 is configured to predict successful in vitro fertilization (IVF) based on sperm ZP binding capacity (LAI), such that sperm samples with normal ZP binding capacity predict successful IVF, while sperm samples with defective ZP binding capacity predict unsuccessful IVF. CycleGAN module 3210 is configured to apply CycleGAN processing to the digital image 3020 of sperm that does not bind ZP, using the microenvironment of ZP-bound sperm as a reference. These images 3020 are then fed into the deep learning model 3204.

[0104] In one implementation, the deep learning module 3202 is configured as a CNN including a feature extractor and a classifier, as described above. Figure 9 , 12 As described in 18 and 27. In an alternative implementation, as... Figure 33 As shown, classifier 3300 is configured with a first linear layer 3302, a second linear layer 3304, a rectified linear unit (ReLU) layer 3306, and a deactivation layer 3308. The configuration of classifier 3300 is fine-tuned for the specific downstream task of sperm classification, as described above, while leveraging the capabilities of deep learning models 3204 such as pre-trained VGG-13 models.

[0105] refer to Figure 34 A method 3400 is provided for collecting sperm as... Figure 30 Sperm sample source 3014 was captured and stored in Figure 30 Image 3020 of collected sperm from sperm image source 3018. (Example) Figure 34 As shown, method 3400 includes collecting sperm and oocyte samples in step 3402, selecting normal sperm samples according to predetermined WHO criteria (as described above) in step 3404, and separating motile and viable sperm from seminal plasma in step 3406. Then, method 3400 collects ZP-binding sperm in step 3408 by incubating the treated sperm with a group of human oocytes in a culture medium for 30 minutes to separate ZP-binding sperm, and then collects ZP-binding sperm from clinical samples with defective ZP-binding capacity (as demonstrated by complete fertilization failure after conventional IVF and the absence of ZP-binding sperm on oocytes after artificial insemination) in step 3410. The collected sperm are then stained with Diff's rapid staining in step 3412, as described above, and images 3020 of the stained sperm are captured in step 3414 using one or more imaging modalities. For example, using oil immersion as... Figure 30 Image 3020 is captured at 1000x magnification under an optical microscope (Zeiss AG, Göttingen, Germany) using image capture device 3016, as described above. In another example, the image is captured using any known camera or imaging device that serves as image capture device 3016.

[0106] In one embodiment of the present invention, such as Figure 35 As shown, Figure 30 The operating method 3500 of the system 3000 shown includes the following steps: receiving a training dataset in step 3502, such as training data included in supplementary data 3024, and using the training dataset in step 3504 to train a deep learning model 3204 executed by a deep learning module 3202. The training dataset includes images previously obtained as described above. After training the deep learning model 3204, the method 3500 receives an image 3020 of captured stained sperm in step 3506, applies the captured image 3020 to the trained deep learning model 3204 in step 3508, generates a prediction 3022 about the ZP binding capacity of the sperm represented by image 3020 in step 3510, and outputs the prediction 3022 using input / output device 3010 in step 3512.

[0107] With the trained model, deep learning can be used to analyze sperm head images based on morphological features of the sperm head images to predict the fertilization potential of human sperm. The fine-tuned deep learning model exhibits good generalization ability to classify sperm subpopulations with high discrimination. The prediction threshold established by clinical data can be used to assess the ZP binding capacity of clinical samples to predict the fertilization outcome after conventional artificial insemination.

[0108] Although routine semen analysis is commonly used to examine sperm parameters in male infertility investigations, they are surrogate markers of fertility and do not fully reflect the fertilization potential of sperm. Men with normal semen parameters can be infertile because they can have sperm with functional defects that cannot be detected by surrogate parameters, leading to unexpected fertilization failure in conventional artificial insemination in IVF. On the other hand, men with semen parameters below WHO reference values are not necessarily infertile, as these reference values are derived from fertile men. In addition, semen parameters vary greatly between individuals, making it difficult to establish clear thresholds for predicting fertilization outcomes. Therefore, the clinical significance of WHO's reference values for semen parameters remains controversial. In addition, artificial semen analysis relies heavily on the experience and judgment of technicians, and due to the subjectivity of the assessment method, significant differences in assessment can occur between different laboratories and different laboratory personnel. Despite significant technological advances, fertilization failure still occurs in 5-10% of IVF cycles with conventional artificial insemination. A recent large retrospective cohort study concluded that semen parameters have limited predictive ability for complete fertilization failure after conventional artificial insemination in IVF.

[0109] Deep learning has become a popular method for medical image analysis due to its high versatility and reliability. This method has been used to identify abnormalities in abdominal ultrasound images and risk factors for cardiovascular disease. Recently, there has been a surge of interest in exploring the use of deep learning for sperm morphology assessment. However, these models are trained using datasets annotated by embryologists according to strict WHO criteria, which can introduce bias and subjectivity into the training process. Ideally, deep learning should be able to learn from input data with minimal human intervention.

[0110] Although sperm morphology is an important component of routine semen analysis, the reported results have shown conflicting results regarding the limited predictive ability of sperm morphology on sperm quality and fertilization outcomes in assisted reproductive treatments. In contrast, sperm ZP-binding ability is an independent surrogate marker of sperm quality, closely associated with high acrosome reaction rate and DNA integrity, and positively correlated with the fertilization rate following routine insemination in IVF. The ZP test is initiated by incubating matched aliquots of ZP with samples from fertile men and patients to assess sperm ZP-binding ability for clinical diagnosis. However, the need for micromanipulation to section the ZP and the limited availability of human oocytes limit the widespread use of this method in routine use.

[0111] The present disclosure describes a novel method that uses advanced deep learning models to identify sperm with / without ZP-binding ability based on morphological features of sperm. During the method development phase, a representative database of Diff-Quik stained images of sperm binding ZP and sperm with defective ZP-binding ability was established, which will be made publicly accessible for clinical use in the future. Independent sperm heads up to the neck were extracted from the collected images using the k-means clustering algorithm, which identifies specific objects of interest from the background by grouping images with vectors of similar pixel values to reference images. Compared to morphological features of sperm heads, the clinical significance of morphological features of sperm tails is much more complex, mainly due to the lack of standardization for sperm tail morphology assessment and large differences between independent sperm within a single sample. Studies have shown that the presence of sperm tails within the visual range can cause bias and unreliable prediction results when using deep learning to predict DNA quality based on sperm head morphology.

[0112] Sperm binding ZP were collected by co-incubating sperm processed by upstream methods with oocytes, while non-binding sperm were residual clinical samples after DGC processing for complete fertilization failure following routine insemination in IVF. Considering the potential microenvironment differences between these samples, CycleGAN was used to translate the specific features of processed sperm binding ZP into clinical sperm samples to generate a more consistent dataset representing sperm morphology. CycleGAN is a powerful type of GAN designed to transfer acquired knowledge between two different image domains for image-to-image translation. CycleGAN consists of two generators and two discriminators, which generate reasonable high-quality images through mutual adversarial training. More specifically, the generator creates translated images in one domain, which are evaluated for their authenticity by the discriminator, which is mainly used to determine how realistic the translated images look compared to reference images in the target domain.

[0113] Training deep learning models from scratch that generalize to unseen data requires large amounts of training data and computational power. Transfer learning is commonly used to fine-tune the classifier of a pre-trained model for a new task by retraining the last few layers of the model on a small dataset while preserving all the optimized weights acquired from a large existing dataset. This approach is particularly useful for medical image analysis to address the limitations of sample availability and manual annotation, enabling significant improvements in classification accuracy and training efficiency without overfitting. Recently, it was used to fine-tune two pre-trained deep learning models for sperm segmentation, enabling the automatic identification and labeling of sperm heads, acrosomes, and nuclei with an average agreement of 95% with manually segmented regions. Another study further explored the use of this technique for classifying sperm morphology according to WHO standards with an accuracy of 96.0% and a precision of 96.4%.

[0114] Given the limited availability of ZP-bound and ZP-unbound sperm, the VGG-13 model was used as a feature extractor, which had been previously trained on a different ImageNet database to extract relevant prominent features of the input images that were subsequently used as training data to develop a new classifier during fine-tuning for the specific task of classifying ZP-bound and ZP-unbound sperm with high specificity and sensitivity. A classic, simple CNN model was also trained from scratch to further examine the advantage of transfer learning on limited datasets. Although the CNN model demonstrated reasonable classification performance, it lacked the ability to effectively generalize to unseen clinical data obtained under the same conditions as the training set. One possible explanation is that the model failed to extract unique and prominent features from the small dataset to establish specific representation patterns due to the large morphological differences between independent sperm within each class.

[0115] Deep learning models generally do not provide comprehensive information about the decision-making process due to their "black box" nature that is difficult to interpret. Saliency maps are a method to visualize and conceptually explain the decision-making process by revealing the pixels that are located in specific regions of the image that contribute to the classification process. This technique has been used to validate the effectiveness of deep learning to identify regions of interest such as abnormalities and cell types in medical images. The heat map generated using saliency maps indicates that the model mainly detects the head and midpiece of the sperm, while ignoring background noise and debris. Previous results reported that the ZP-binding capacity of sperm is significantly correlated with a higher rate of normal morphology. However, these studies did not identify the most important morphological features related to the ZP-binding capacity of sperm. The model revealed that the pixels in the anterior region are the main morphological factors that contribute to the learning and classification process, which is consistent with the biological significance of the acrosomal region of sperm for ZP binding. Using transfer learning and data augmentation, relevant features were extracted from the dataset to establish hidden but interrelated patterns for each class by utilizing the trained weights and hyperparameters of the pre-trained VGG-13 model for binary classification.

[0116] To translate these findings into clinical practice, samples of sperm with different rates of fertilization after routine insemination in IVF were collected to validate the discriminative ability of the pre-trained VGG-13 model for the ZP-binding capacity of human sperm. The results demonstrated a significant difference in the percentage of total ZP-binding capacity among the different fertilization rate groups. The percentage of ZP-binding capacity in the high fertilization rate group (71-100%) was significantly higher than that in the low fertilization rate group (0-40%), which is consistent with previous reports on the difference in the number of viable sperm with ZP-binding capacity between fertile and infertile men. Considering the high predictive ability of ZP-binding capacity for the fertilization outcome after routine insemination in IVF, the data generated using the model were used to determine the threshold that distinguishes between normal and defective ZP-binding capacity of sperm. Compared to routine semen results alone, the model provides additional information about the fertilization potential of sperm, independent of sperm parameters. The model is able to identify men who, despite having normal semen parameters in semen analysis, can fail in IVF using routine insemination due to defective ZP-binding capacity. These information can allow patients to make an informed decision about the fertilization method to be used (routine insemination versus ICSI), preventing patients from suffering psychologically and financially due to fertilization failure.

[0117] This model is suitable for DIF-stained images captured at high resolution. However, its compatibility with other staining methods remains to be optimized. It can also be used similarly for non-stained images. A deep learning model has been established for the detection of morphologically abnormal sperm from non-stained and relatively low-resolution images. A known deep learning-based model trained on non-stained sperm of known DNA quality was used for prediction from images taken under brightfield. The disclosed model can be tested on datasets of semen samples collected under different preparation conditions (e.g., differences in sample handling methods and staining protocols) to ensure consistent classification accuracy.

[0118] The disclosed model can serve as a basis for the development of an advanced platform that combines microfluidics and deep learning to simultaneously identify and isolate high-quality and motile sperm cells for ICSI. Microfluidic technology has been used to isolate specific subpopulations of sperm based on their biological parameters and physiological characteristics. Using a high-resolution time-lapse imaging system, this integrated platform can instantly examine and select sperm that move freely based on their morphological features through deep learning. Recent studies have successfully used video recordings to analyze sperm motility using deep learning-based methods, demonstrating the potential of this technology for motile sperm selection.

[0119] The disclosed model can use deep learning to assess the fertilization potential of DIF-stained sperm based on their morphological features. Patients with predicted ZP-binding capacity percentages below 4.8% should be considered for ICSI rather than conventional artificial insemination to improve fertilization outcomes. By increasing the sample size of semen samples captured under various platforms, the classification performance of the model can be further improved, enhancing its sensitivity and specificity under different image quality levels, allowing analysis using low-cost, easily accessible equipment.

[0120] Therefore, the disclosed model that includes fine-tuning of ZP-binding capacity allows: 1) real-time assessment of sperm fertilization potential independent of results obtained through routine semen analysis; and 2) identification of patients with defective ZP-binding capacity through DIF-stained image analysis using deep learning. In particular, the threshold of morphological assessment can be used to identify men whose sperm have poor fertilization capacity, which can lead to a higher likelihood of fertilization failure in conventional artificial insemination in IVF. The development of a reliable and robust method for sperm evaluation allows for a wise decision between ICSI and conventional artificial insemination, which can lead to improved fertilization outcomes. Ultimately, this can contribute to better clinical management of infertile couples undergoing assisted reproductive treatment.

[0121] Exemplary products, systems, and methods are listed in the following clauses:

[0122] 1. A deep learning-based method for assisted reproduction, comprising:

[0123] a. obtaining at least one sample of sperm bound to zona pellucida (ZP) and at least one sample of sperm not bound to ZP;

[0124] b. obtaining one or more digital images of the ZP-bound sperm and ZP-unbound sperm by one or more imaging modalities; and

[0125] c. processing the one or more digital images according to at least one deep learning model configured to generate a binary prediction of ZP binding ability of sperm using sperm images, wherein the binary prediction is normal or defective ZP binding.

[0126] 2. The method of any preceding clause, wherein the at least one deep learning model is transfer learning.

[0127] 3. The method of any preceding clause, wherein the at least one deep learning model is selected from the group consisting of Visual Geometry Group (VGG), LeNet, AlexNet, U-Net, and Residual Neural Network (ResNet).

[0128] 4. The method of any preceding clause, wherein the deep learning model is VGG-13.

[0129] 5. The method of any preceding clause, further comprising the step of processing the digital images by an unsupervised machine learning clustering algorithm.

[0130] 6. The method of any preceding clause, wherein the unsupervised machine learning algorithm is selected from the group consisting of k-means clustering algorithm, Gaussian Mixture Model, hierarchical clustering, spectral clustering, and k-medoids clustering.

[0131] 7. The method of any preceding clause, further comprising the step of staining the at least one sample of ZP-bound sperm and the at least one sample of ZP-unbound sperm prior to step b.

[0132] 8. The method of any preceding clause, further comprising the step of performing a CycleGAN on the digital images of ZP-unbound sperm and ZP-bound sperm prior to step c.

[0133] 9. The method of any preceding clause, wherein the threshold value for distinguishing normal and defective ZP binding ability of sperm is about 4.78%.

[0134] 10. The method of any preceding clause, wherein the non-ZP-bound sperm are obtained from a clinical sample with defective ZP binding capacity, wherein the clinical sample failed routine intrauterine insemination during IVF.

[0135] 11. The method of any preceding clause, wherein the ZP-bound sperm are obtained by ZP-sperm co-incubation, wherein viable sperm are incubated with one or more oocytes for 30 minutes to recover sperm capable of binding to ZP.

[0136] 12. The method of any preceding clause, wherein the deep learning model VGG-13 predicts ZP binding capacity with at least about 90% accuracy, specificity, sensitivity, or a combination thereof.

[0137] 13. The method of any preceding clause, wherein the ZP binding capacity is predicted with at least about 90% specificity or sensitivity.

[0138] 14. The method of any preceding clause, wherein the ZP binding capacity is predicted with at least about 95% specificity or sensitivity.

[0139] 15. The method of any preceding clause, further comprising the step of predicting success of in vitro fertilization (IVF) based on ZP binding capacity of sperm, wherein a sperm sample with normal ZP binding capacity predicts successful IVF, while a sperm sample with defective ZP binding capacity predicts unsuccessful IVF.

[0140] 16. A method of predicting fertilization success after routine intrauterine insemination during IVF, comprising:

[0141] a. obtaining a sperm sample from a subject undergoing IVF;

[0142] b. obtaining one or more digital images of the sperm sample by one or more imaging modalities; and

[0143] c. processing the one or more digital images according to at least one deep learning model to obtain a binary prediction result on ZP binding capacity of the sperm sample according to clause 1, wherein normal ZP binding of the sperm sample predicts fertilization success, while defective ZP binding of the sperm sample predicts fertilization failure.

[0144] 17. The method of any preceding clause, wherein the at least one deep learning model is transfer learning.

[0145] 18. The method of any preceding clause, wherein the at least one deep learning model is selected from the group consisting of Visual Geometry Group (VGG), LeNet, AlexNet, U-Net, and Residual Neural Network (ResNet).

[0146] 19. The method of any preceding clause, wherein the deep learning model is VGG-13.

[0147] 20. The method of any preceding clause, further comprising the step of processing the digital images by an unsupervised machine learning clustering algorithm.

[0148] 21. The method of any preceding clause, wherein the unsupervised machine learning algorithm is selected from the group consisting of k-means clustering algorithm, Gaussian Mixture Model, hierarchical clustering, spectral clustering, and k-medoids clustering.

[0149] 22. The method of any preceding clause, further comprising the step of staining the sperm sample prior to step b.

[0150] 23. The method of any preceding clause, wherein the threshold value to distinguish between normal and defective ZP binding capacity of a test sperm sample is about 4.78%.

[0151] 24. The method of any preceding clause, wherein the VGG-13 model predicts ZP binding capacity with at least 90% accuracy, specificity, sensitivity, or a combination thereof.

[0152] 25. The method of any preceding clause, wherein the ZP binding capacity is predicted with at least about 90% specificity or sensitivity.

[0153] 26. The method of any preceding clause, wherein the ZP binding capacity is predicted with at least about 95% specificity or sensitivity.

[0154] 27. A method for predicting ZP binding capacity of sperm, comprising:

[0155] a. obtaining at least one sample of ZP-bound sperm and at least one sample of ZP-unbound sperm;

[0156] b. obtaining one or more digital images of the ZP-bound sperm and ZP-unbound sperm by one or more imaging modalities; and

[0157] c. processing the one or more digital images according to at least one deep learning model configured to generate a binary prediction of ZP binding capacity of sperm using sperm images, wherein the binary prediction is normal or defective ZP binding.

[0158] wherein the deep learning model is a VGG-13 that predicts ZP binding capacity with an accuracy, specificity, sensitivity, or a combination thereof of at least about 90%.

[0159] 28. A deep learning based system for assisted reproduction comprising:

[0160] a. means for obtaining at least one sample of sperm bound to zona pellucida (ZP) and at least one sample of sperm not bound to ZP;

[0161] b. means for obtaining one or more digital images of the ZP bound sperm and ZP not bound sperm by one or more imaging modalities;

[0162] c. a hardware based processor;

[0163] d. a memory configured to store instructions and configured to provide the instructions to the hardware based processor; and

[0164] e. a set of modules configured to implement the instructions provided to the hardware based processor, the set of modules comprising:

[0165] a deep learning module configured to implement at least one deep learning model to process the one or more digital images and generate a binary prediction of ZP binding capacity of sperm from sperm images, wherein the binary prediction is normal or defective ZP binding.

[0166] 29. The system of any preceding clause, wherein the at least one deep learning model is transfer learning.

[0167] 30. The system of any preceding clause, wherein the at least one deep learning model is selected from the group consisting of Visual Geometry Group (VGG), LeNet, AlexNet, U-Net, and Residual Neural Network (ResNet).

[0168] 31. The system of any preceding clause, wherein the at least one deep learning model is a VGG-13.

[0169] 32. The system of any preceding clause, further comprising a processing module configured to process the digital images by an unsupervised machine learning clustering algorithm.

[0170] 33. The system of any preceding clause, wherein the unsupervised machine learning algorithm is selected from the group consisting of a k-means clustering algorithm, Gaussian Mixture Model, hierarchical clustering, spectral clustering, and k-center clustering.

[0171] 34. The system of any preceding clause, further comprising a device for staining the at least one sample of ZP-bound sperm and the at least one sample of ZP-unbound sperm prior to obtaining the one or more digital images using a device for obtaining one or more digital images.

[0172] 35. The system of any preceding clause, further comprising a CycleGAN module configured to perform a CycleGAN on the digital images of ZP-unbound sperm and ZP-bound sperm prior to implementing the at least one deep learning model.

[0173] 36. The system of any preceding clause, wherein the threshold for distinguishing between normal and defective ZP binding capacity of sperm is about 4.78%.

[0174] 37. The system of any preceding clause, wherein the ZP-unbound sperm are obtained from a clinical sample having defective ZP binding capacity, wherein the clinical sample failed a conventional intrauterine insemination during IVF.

[0175] 38. The system of any preceding clause, wherein the ZP-bound sperm are obtained by ZP-sperm co-incubation, wherein viable sperm are incubated with one or more oocytes for 30 minutes to recover sperm capable of binding to ZP.

[0176] 39. The system of any preceding clause, wherein the deep learning model VGG-13 predicts ZP binding capacity with at least about 90% accuracy, specificity, sensitivity, or a combination thereof.

[0177] 40. The system of any preceding clause, wherein the ZP binding capacity is predicted with at least about 90% specificity or sensitivity.

[0178] 41. The system of any preceding clause, wherein the ZP binding capacity is predicted with at least about 95% specificity or sensitivity.

[0179] 42. The system of any preceding clause, further comprising a prediction module configured to predict in vitro fertilization (IVF) success based on ZP binding capacity of sperm, wherein a sample of sperm having normal ZP binding capacity predicts a successful IVF and a sample of sperm having defective ZP binding capacity predicts an unsuccessful IVF.

[0180] 43. A system for predicting fertilization success after a conventional intrauterine insemination during IVF, comprising:

[0181] a. a device for obtaining a sample of sperm from a subject undergoing IVF;

[0182] b. means for obtaining one or more digital images of the sperm sample by one or more imaging modalities; and

[0183] c. The deep learning-based system of clause 28, configured to obtain a binary prediction of ZP binding capacity of the sperm sample, wherein a normal ZP binding prediction of the sperm sample is predicted to result in fertilization success, and a defective ZP binding prediction of the sperm sample is predicted to result in fertilization failure.

[0184] 44. The system of any preceding clause, wherein the at least one deep learning model is transfer learning.

[0185] 45. The system of any preceding clause, wherein the at least one deep learning model is selected from the group consisting of Visual Geometry Group (VGG), LeNet, AlexNet, U-Net, and Residual Neural Network (ResNet).

[0186] 46. The system of any preceding clause, wherein the deep learning model is VGG-13.

[0187] 47. The system of any preceding clause, further comprising a processing module configured to process the digital images by an unsupervised machine learning clustering algorithm.

[0188] 48. The system of any preceding clause, wherein the unsupervised machine learning algorithm is selected from the group consisting of k-means clustering algorithm, Gaussian Mixture Model, hierarchical clustering, spectral clustering, and k-medoids clustering.

[0189] 49. The system of any preceding clause, further comprising means for staining the sperm sample prior to obtaining the one or more digital images using the means for obtaining one or more digital images.

[0190] 50. The system of any preceding clause, wherein the threshold value for distinguishing between normal and defective ZP binding capacity of a test sperm sample is about 4.78%.

[0191] 51. The method of any preceding clause, wherein the VGG-13 model predicts ZP binding capacity with at least 90% accuracy, specificity, sensitivity, or a combination thereof.

[0192] 52. The system of any preceding clause, wherein the ZP binding capacity is predicted with at least about 90% specificity or sensitivity.

[0193] 53. The system of any preceding clause, wherein the ZP binding capacity is predicted with at least about 95% specificity or sensitivity.

[0194] 54. A system for predicting ZP binding capacity of sperm, comprising:

[0195] a. means for obtaining at least one sample of ZP-bound sperm and at least one sample of ZP-unbound sperm;

[0196] b. means for obtaining one or more digital images of the ZP-bound sperm and ZP-unbound sperm by one or more imaging modalities; and

[0197] c. a hardware-based processor;

[0198] d. a memory configured to store instructions and configured to provide the instructions to the hardware-based processor; and

[0199] e. a set of modules configured to implement the instructions provided to the hardware-based processor, the set of modules comprising:

[0200] a deep learning module configured to implement at least one deep learning model to process the one or more digital images according to the at least one deep learning model, the at least one deep learning model configured to use sperm images and generate a binary prediction of ZP binding capacity of sperm, wherein the binary prediction is normal or defective ZP binding;

[0201] wherein the deep learning model is a VGG-13 that predicts ZP binding capacity with an accuracy, specificity, sensitivity, or a combination thereof of at least about 90%.

[0202] 55. The system of any preceding clause, further comprising:

[0203] a classifier comprising, in order:

[0204] a first linear layer;

[0205] a second linear layer;

[0206] a rectified linear unit (ReLU) layer; and

[0207] a random dropout layer.

[0208] 56. The method of any preceding clause, further comprising the following steps prior to step b:

[0209] selecting a sample of normal sperm from the obtained sample of sperm according to predetermined criteria;

[0210] isolating motile and viable sperm from seminal plasma;

[0211] incubating the isolated sperm in a predetermined solution; and

[0212] The incubated sperm are stained.

[0213] 57. The method of any preceding clause, further comprising, prior to step c:

[0214] receiving a training data set; and

[0215] training the at least one deep learning model using the training data set.

[0216] The foregoing description of specific implementations will so fully reveal the general nature of the disclosure that others can adapt and / or merely apply such descnbed embodiments to various applications and modifications thereof without undue experimentation. Consequently, while the specific embodiments have been shown and described in detail to illustrate the general principles of the disclosure, it will be understood that various modifications can be made to the embodiments described herein without departing from the general nature of the disclosure. Therefore, it is to be understood that the disclosure is not to be limited to the particular examples disclosed as such will only be limited by the appended claims. Needless to say, numerous other modifications, arrangements, and embodiments will be apparent to those skilled in the art upon reading the foregoing description and which are intended to fall within the scope of the present disclosure. Thus, it is intended that the disclosure cover any and all adaptations of the various embodiments.

[0217] While various embodiments of the present disclosure have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made therein without departing from the spirit and scope of the disclosure. Thus, the disclosure should not be limited by any of the above-described exemplary embodiments, but should only be defined in accordance with the following claims and their equivalents.

[0218] All references cited herein are incorporated by reference in their entirety as if each individual publication or patent or patent application was specifically and individually indicated to be incorporated by reference in its entirety for all purposes.

[0219] Implementation of the techniques, blocks, steps and means described above can be done by various means. For example, these techniques, blocks, steps and means can be implemented in hardware, software, or a combination thereof. For a hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described above and / or a combination thereof.

[0220] Furthermore, embodiments can be described as a process that is depicted as a flow diagram, flowchart, data flow diagram, structure diagram, or a block diagram. Although a flow diagram can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in a figure. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

[0221] Furthermore, embodiments can be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting language, and / or microcode, the program code or code segments to perform the necessary tasks can be stored in a machine readable medium such as a storage medium. A code segment or machine-executable instruction can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or any combination of instructions, data structures, and / or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0222] For firmware and / or software implementations, these methods can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. Any machine readable medium tangibly embodying instructions can be used in implementing the techniques described herein. For example, software codes can be stored in memory. Memory can be implemented within the processor or external to the processor. As used herein, the term "memory" refers to any type of long-term, short-term, volatile, nonvolatile, or other memory and is not to be limited to any particular type of memory or number of memories, or type of media upon which memory is stored.

[0223] Furthermore, as disclosed herein, the term "storage media" can represent one or more memories for storing data including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and / or other machine readable mediums for storing information. The term "computer-readable medium" and "machine-readable medium" includes, but is not limited to portable or fixed storage devices, optical storage devices, wireless channels, and various other storage mediums capable of storing that can store data including information and / or instructions that can be accessed by a computer or machines. It should be understood that "computer-readable medium" or "machine-readable medium" can include a single medium or multiple media (e.g., central, northbridge, southbridge, and / or bus connected and / or peripheral storage devices) depending on what is desired to be achieved by a particular application. The application should not be limited necessarily by any of the various embodiments given above.

[0224] References

[0225] 1. The Annual Scientific Meeting of Hong Kong Society of Endocrinology, Metabolism and Reproduction, Hong Kong. (Oral presentation and poster presentation)

[0226] 2. Deep Learning for the classification of human spermatozoa.

Claims

1. A deep learning-based method for assisted reproduction, comprising: a. Obtain at least one sperm sample bound to the zona pellucida (ZP) and at least one sperm sample not bound to the ZP; b. Obtain one or more digital images of the ZP-bound sperm and the unbound sperm using one or more imaging modes; as well as c. Processing the one or more digital images according to at least one deep learning model, the at least one deep learning model being configured to generate a binary prediction of the sperm’s ZP binding capacity using the sperm image, wherein the binary prediction is normal or defective ZP binding.

2. The method according to any of the preceding claims, wherein the at least one deep learning model is transfer learning.

3. The method according to any of the preceding claims, wherein the at least one deep learning model is selected from the group consisting of Visual Geometry Group (VGG), LeNet, AlexNet, U Net, and Residual Neural Network (ResNet).

4. The method according to any of the preceding claims, wherein the deep learning model is VGG-13.

5. The method according to any of the preceding claims further includes the step of processing the digital image using an unsupervised machine learning clustering algorithm.

6. The method according to any of the preceding claims, wherein the unsupervised machine learning algorithm is selected from the group consisting of k-means clustering, Gaussian mixture model, hierarchical clustering, spectral clustering, and k-centroid clustering.

7. The method according to any of the preceding claims further comprises, prior to step b, staining the at least one sample of sperm bound to ZP and the at least one sample of sperm not bound to ZP.

8. The method according to any of the preceding claims further comprises, prior to step c, performing a CycleGAN (Recurrent Generative Adversarial Network) step on the digital images of sperm unbound and sperm bound with ZP.

9. The method according to any of the preceding claims, wherein the threshold for distinguishing between normal and defective sperm and their ZP binding capacity is approximately 4.78%.

10. The method according to any of the preceding claims, wherein the unbound ZP sperm is obtained from a clinical sample with defective ZP binding capacity, wherein the clinical sample has failed conventional artificial insemination during IVF.

11. The method according to any of the preceding claims, wherein the ZP-bound sperm is obtained by ZP-sperm co-incubation, wherein viable sperm are incubated with one or more oocytes for 30 minutes to recover sperm capable of binding to ZP.

12. The method according to any of the preceding claims, wherein the deep learning model VGG-13 predicts ZP binding ability with at least about 90% accuracy, specificity, sensitivity, or a combination thereof.

13. The method according to any of the preceding claims, wherein the ZP binding capacity is predicted with at least about 90% specificity or sensitivity.

14. The method according to any of the preceding claims, wherein the ZP binding capacity is predicted with at least about 95% specificity or sensitivity.

15. The method according to any of the preceding claims further comprises predicting in vitro fertilization (IVF) success based on the ZP binding capacity of sperm, wherein sperm samples with normal ZP binding capacity predict successful IVF, while sperm samples with defective ZP binding capacity predict unsuccessful IVF.

16. A method for predicting fertilization success after routine artificial insemination during IVF, comprising: a. Obtain sperm samples from subjects undergoing IVF; b. Obtain one or more digital images of the sperm sample using one or more imaging modes; as well as c. The method of processing the one or more digital images according to at least one deep learning model as described in claim 1 to obtain a binary prediction result regarding the ZP binding capacity of the sperm sample, wherein normal ZP binding of the sperm sample predicts successful fertilization, while defective ZP binding of the sperm sample predicts failed fertilization.

17. The method according to any of the preceding claims, wherein the at least one deep learning model is transfer learning.

18. The method according to any of the preceding claims, wherein the at least one deep learning model is selected from the group consisting of Visual Geometry Group (VGG), LeNet, AlexNet, UNet, and Residual Neural Network (ResNet).

19. The method according to any of the preceding claims, wherein the deep learning model is VGG-13.

20. The method according to any preceding claim further comprises the step of processing the digital image using an unsupervised machine learning clustering algorithm.

21. The method according to any of the preceding claims, wherein the unsupervised machine learning algorithm is selected from the group consisting of k-means clustering, Gaussian mixture model, hierarchical clustering, spectral clustering, and k-centroid clustering.

22. The method according to any of the preceding claims further comprises a step of staining the sperm sample prior to step b.

23. The method according to any of the preceding claims, wherein the threshold for distinguishing the binding capacity of normal and defective ZP in the tested sperm sample is approximately 4.78%.

24. The method according to any of the preceding claims, wherein the VGG-13 model predicts ZP binding capacity with at least 90% accuracy, specificity, sensitivity, or a combination thereof.

25. The method according to any of the preceding claims, wherein the ZP binding capacity is predicted with at least about 90% specificity or sensitivity.

26. The method according to any of the preceding claims, wherein the ZP binding capacity is predicted with at least about 95% specificity or sensitivity.

27. A method for predicting ZP binding capacity of sperm, comprising: a. Obtain at least one sperm sample bound to ZP and at least one sperm sample not bound to ZP; b. Obtain one or more digital images of the ZP-bound sperm and the unbound sperm using one or more imaging modes; as well as c. Processing the one or more digital images according to at least one deep learning model, the at least one deep learning model being configured to generate a binary prediction of the sperm’s ZP binding capacity using the sperm image, wherein the binary prediction is normal or defective ZP binding; The deep learning model mentioned is VGG-13, which predicts ZP binding ability with at least about 90% accuracy, specificity, sensitivity, or a combination thereof.

28. A deep learning-based system for assisted reproduction, comprising: a. An apparatus for obtaining at least one sperm sample bound to the zona pellucida (ZP) and at least one sperm sample unbound to the ZP; b. An apparatus for obtaining one or more digital images of ZP-bound sperm and ZP-unbound sperm through one or more imaging modes; c. Hardware-based processors; d. A memory configured to store instructions and configured to provide the instructions to the hardware-based processor; as well as e. A set of modules configured to implement the instructions provided to the hardware-based processor, the set of modules comprising: A deep learning module is configured to implement at least one deep learning model to process the one or more digital images and to generate a binary prediction of the sperm’s ZP binding capacity using the sperm images, wherein the binary prediction is normal or defective ZP binding.

29. The system according to any of the preceding claims, wherein the at least one deep learning model is transfer learning.

30. The system according to any of the preceding claims, wherein the at least one deep learning model is selected from the group consisting of Visual Geometry Group (VGG), LeNet, AlexNet, UNet, and Residual Neural Network (ResNet).

31. The system according to any of the preceding claims, wherein the at least one deep learning model is VGG-13.

32. The system according to any of the preceding claims further includes a processing module configured to process the digital image using an unsupervised machine learning clustering algorithm.

33. The system according to any of the preceding claims, wherein the unsupervised machine learning algorithm is selected from the group consisting of k-means clustering, Gaussian mixture model, hierarchical clustering, spectral clustering and k-centroid clustering.

34. The system according to any of the preceding claims further includes means for staining a sample of the at least one ZP-bound sperm and a sample of the at least one ZP-unbound sperm before acquiring the one or more digital images using means for acquiring one or more digital images.

35. The system according to any of the preceding claims further includes a recurrent generative adversarial network (CycleGAN) module configured to perform CycleGAN on the digital images of sperm without ZP and sperm with ZP prior to implementing the at least one deep learning model.

36. The system according to any of the preceding claims, wherein the threshold for distinguishing between normal and defective sperm ZP binding capacity is approximately 4.78%.

37. The system according to any of the preceding claims, wherein the unbound ZP sperm is obtained from a clinical sample with defective ZP binding capacity, wherein the clinical sample has failed conventional artificial insemination during IVF.

38. The system according to any of the preceding claims, wherein the ZP-bound sperm is obtained by ZP-sperm co-incubation, wherein viable sperm are incubated with one or more oocytes for 30 minutes to recover sperm capable of binding to ZP.

39. The system according to any of the preceding claims, wherein the deep learning model VGG-13 predicts ZP binding ability with at least about 90% accuracy, specificity, sensitivity, or a combination thereof.

40. The system according to any of the preceding claims, wherein the ZP binding capacity is predicted with at least about 90% specificity or sensitivity.

41. The system according to any of the preceding claims, wherein the ZP binding capacity is predicted with at least about 95% specificity or sensitivity.

42. The system according to any of the preceding claims further includes a prediction module configured to predict whether in vitro fertilization (IVF) is successful based on the ZP binding capacity of sperm, wherein sperm samples with normal ZP binding capacity predict successful IVF, while sperm samples with defective ZP binding capacity predict unsuccessful IVF.

43. A system for predicting fertilization success after routine artificial insemination during IVF, comprising: a. A device for obtaining sperm samples from subjects undergoing IVF; b. A device for acquiring one or more digital images of the sperm sample through one or more imaging modes; as well as c. The deep learning-based system of claim 28, configured to obtain a binary prediction result regarding the ZP binding capacity of the sperm sample, wherein normal ZP binding of the sperm sample predicts successful fertilization, while defective ZP binding of the sperm sample predicts failed fertilization.

44. The system according to any of the preceding claims, wherein the at least one deep learning model is transfer learning.

45. The system according to any of the preceding claims, wherein the at least one deep learning model is selected from the group consisting of Visual Geometry Group (VGG), LeNet, AlexNet, UNet, and Residual Neural Network (ResNet).

46. ​​The system according to any of the preceding claims, wherein the deep learning model is VGG-13.

47. The system according to any of the preceding claims further includes a processing module configured to process the digital image using an unsupervised machine learning clustering algorithm.

48. The system according to any of the preceding claims, wherein the unsupervised machine learning algorithm is selected from the group consisting of k-means clustering, Gaussian mixture model, hierarchical clustering, spectral clustering and k-centroid clustering.

49. The system according to any of the preceding claims further includes means for staining the sperm sample prior to acquiring the one or more digital images using means for acquiring one or more digital images.

50. The system according to any of the preceding claims, wherein the threshold for distinguishing the binding capacity of normal and defective ZP in a sperm sample is approximately 4.78%.

51. The method according to any of the preceding claims, wherein the VGG-13 model predicts ZP binding capacity with at least 90% accuracy, specificity, sensitivity, or a combination thereof.

52. The system according to any of the preceding claims, wherein the ZP binding capacity is predicted with at least about 90% specificity or sensitivity.

53. The system according to any of the preceding claims, wherein the ZP binding capacity is predicted with at least about 95% specificity or sensitivity.

54. A system for predicting ZP binding capacity of sperm, comprising: a. An apparatus for obtaining at least one sperm sample bound to ZP and at least one sperm sample unbound to ZP; b. An apparatus for obtaining one or more digital images of ZP-bound sperm and ZP-unbound sperm through one or more imaging modes; as well as c. Hardware-based processors; d. A memory configured to store instructions and configured to provide the instructions to the hardware-based processor; as well as e. A set of modules configured to implement instructions provided to the hardware-based processor, the set of modules comprising: A deep learning module configured to implement at least one deep learning model to process the one or more digital images according to the at least one deep learning model, the at least one deep learning model being configured to use sperm images and generate a binary prediction about the ZP binding capacity of sperm, wherein the binary prediction is normal or defective ZP binding. The deep learning model mentioned is VGG-13, which predicts ZP binding ability with at least about 90% accuracy, specificity, sensitivity, or a combination thereof.

55. The system according to any of the preceding claims, further comprising: Classifiers, in order, include: First linear layer; Second linear layer; Modified Linear Unit (ReLU) layers; and Random deactivation layer.

56. The method according to any of the preceding claims, prior to step b, further comprises the following step: Normal sperm samples were selected from the obtained sperm samples according to predetermined criteria; Separate motile and viable sperm from seminal plasma; The separated sperm were incubated in a predetermined solution; and The incubated sperm were stained.

57. The method according to any of the preceding claims, further comprising, before step c: Receive the training dataset; as well as The at least one deep learning model is trained using the training dataset.

Images