A deep learning-based sperm morphology analysis method

Abstract

The application relates to the technical field of sperm morphology analysis, and discloses a sperm morphology analysis method based on deep learning, which comprises the following operation steps: S1, preparing and dividing a data set, collecting sperm data of 85 male participants, for each participant, parameters of standard semen analysis, a video of living sperm, sperm fatty acid, a fatty acid composition of serum phospholipid, demographic data and WHO analysis data can be obtained, 60% of the data set is used for training, 20% is used as a verification set, and 20% is used as a test set; a 3-fold verification method is used; and the data set is expanded to prevent overfitting. The sperm morphology analysis method based on deep learning directly extracts time information and space information of sperm video by using a 3D convolutional neural network, realizes high-precision end-to-end sperm activity prediction, secondly, a deep separable convolution is used to replace ordinary convolution, the parameters of the convolutional network are reduced, and the calculation efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sperm morphology analysis, in particular to a sperm morphology analysis method based on deep learning. BACKGROUND

[0002] In recent years, with the development of computer technology, people began to study how to use existing technology to change the time-consuming and laborious detection in the medical field, improve the efficiency and quality of detection, and sperm detection technology has important significance for reproductive health and the success rate of test-tube babies. In the early days, the detection of sperm status was purely completed by manual work, which was not accurate and consumed human energy. With the development of technology, people began to use computer-aided examination, but the high cost and cumbersome process made this technology not widely promoted. With the maturity of convolutional neural network and the development of target detection field, people began to use computer vision to detect sperm cells. However, target detection can only detect the position of the cell, and the subsequent motion state and morphology of the sperm cell still need to use traditional image analysis technology, which cannot achieve end-to-end detection.

[0003] Since the 1980s, attempts have been made to develop automated systems for semen analysis, the CASA system that can capture and process digital video was born, but computer-aided semen analysis (CASA) is done using expensive heavy equipment that tests real-time physical samples to obtain sperm activity and morphological indicators, in tests that cannot afford these machines, samples are manually inspected under a microscope, the current CASA method does indeed give quite accurate results, but this method still has some problems, the test results depend on the physical consistency of the sample, which changes over time, the test results will be different if the sample is not stored properly, the time to obtain the results is relatively long, it usually takes five to six minutes to generate a report, if we consider the time from preparing the sample to the report, it takes about ten to twelve minutes, and when other particles and cells float in the semen, it affects the accuracy of the test results, running these programs also requires a lot of training and preparation, despite the long history of CASA, due to the above reasons, the CASA system is less used in clinical testing, with the development of computer vision, someone proposed a fully automatic multi-sperm tracking system that can track hundreds of sperm at the same time, the working principle of this method is to use the JPDA data association algorithm to realize the association of the point track and the track of two uniform motion targets, and the modified version of the algorithm is applied to microscopic semen detection to track a single sperm, but they only tested the algorithm on two sperm, so the universality of their method cannot be determined, someone also tested various machine learning and deep learning methods on the VISEM dataset to predict sperm movement, such as linear and logistic regression, they also used deep learning methods such as convolutional neural networks, but there is a problem with their method, they use two-dimensional CNN to process the video frame by frame, in this way, although the spatial information is retained during the training process, the time sequence information in the video cannot be well learned by the network, therefore, the information of the video cannot be used to the maximum extent in the index detection, and thus a sperm morphology analysis method based on deep learning is proposed to solve the above problems. SUMMARY

[0004] (I) Technical problems solved

[0005] The sperm morphology analysis method based on deep learning is proposed to solve the above problems.

[0006] (II) Technical solutions

[0007] The technical solutions of the sperm morphology analysis method based on deep learning to solve the above technical problems are as follows:

[0008] A sperm morphology analysis method based on deep learning, comprising the following operation steps:

[0009] S1, prepare and divide the data set, collect sperm data of 85 male participants, for each participant, the parameters of standard semen analysis, video of live sperm, sperm fatty acid, fatty acid composition of serum phospholipid, demographic data and WHO analysis data can be obtained, 60% of the data set is used for training, 20% is used as a validation set, 20% is used as a test set, 3-fold cross-validation is used, and the data set is expanded to prevent overfitting;

[0010] S2, in the training process, the training set is preprocessed first, and the connected frames are intercepted as input, after multiple experiments, 16 consecutive frames are used as input, the input image is cropped and randomly inverted to simulate the shaking of manual detection and the diversity of samples, then the improved 3D convolution network is used to extract information from the input consecutive image frames, the loss function is calculated by forward propagation, the gradient descent method is used for backward propagation to adjust the parameters of the convolutional neural network, so that the result of the loss function decreases, when the calculation results of the loss function on the training set and the validation set tend to be stable, the learning rate is fine-tuned, the convolutional network is continuously trained, the test set is used to test the result of network training, and evaluation indexes for evaluating the convolutional neural network are given.

[0011] S3, sample inference, package the trained deep learning network and the trained weight parameters, use the streamlit framework in python to build an app application for predicting sperm activity indexes, load any semen sample video, call the trained convolutional neural network, and display the predicted activity indexes on the page.

[0012] On the basis of the above technical scheme, the application can also be improved as follows.

[0013] Preferably, the ages of the 85 men in step S1 are all above 18 years old.

[0014] Preferably, the data set in step S1 contains more than 35GB of video, each video lasts for two to seven minutes, the frame rate is 50 frames per second, the data set contains six CSV files in total, a description file and a video folder, each video file is identified by ID, video capture date, the data set is made and arranged by a professional medical institution, and is valid.

[0015] Preferably, when the convolutional network is continuously trained in step S2, the loss function is calculated by forward propagation first, then the gradient descent method is used for backward propagation to adjust the parameters of the convolutional neural network, so that the result of the loss function decreases, until the loss of the training set and the validation set tends to be stable.

[0016] Preferably, the parameter setting of the training process, the video image input of the training set is preprocessed, then the input sperm image is randomly reversed and cropped to 16*112*112 to realize spatial and temporal jitter, SGD is selected as the optimizer, the initial learning rate is 0.001, when the loss function result no longer decreases, the learning rate is divided by 2, pre-training is carried out on the UCF101 classic action classification data set, then fine-tuning is carried out on the Visem data set, after the video image input of the training set is preprocessed, the video image is adjusted to 128*171 size, and the continuous 16 frames of images are taken as the input of the 3D feature extraction network.

[0017] Preferably, the parameter setting of the convolutional neural network, the network comprises 5 convolutional layers, 5 pooling layers, 2 fully connected layers and a softmax classification layer to predict the activity index of the mirror, the filter numbers of the convolutional layers are 64, 128, 256, 256 and 256 in turn, the pooling kernel size of the first pooling layer is 1*2*2, the pooling kernel sizes of the remaining pooling layers are 2*2*2, the steps are all 1, the last two fully connected layers have 1024-dimensional output, the last is connected with a softmax layer to output three indexes, the convolutional layer is composed of a deep convolution with a convolution kernel size of 3*3*3 and a standard convolution with a size of 1*1*1, which is used to replace the 3*3*3 convolution, so that the parameters of the convolutional network are reduced, overfitting is prevented, and the inference speed is faster.

[0018] (III) Beneficial effects

[0019] Compared with the prior art, the technical scheme of the application has the following beneficial technical effects:

[0020] The sperm video is directly sent into the deep neural network after preprocessing, the time information and the spatial information of the video are extracted through the 3D deep convolutional network, the network convolutional network parameters are trained, the last layer of the convolutional network is replaced with a regression layer, then any sperm video can be used to predict the morphological index of the sperm in the semen through forward inference, the video classification method in the deep learning field is applied to the sperm activity analysis, the 3D convolutional neural network is used to directly extract the time information and the spatial information of the sperm video, high-precision end-to-end sperm abnormal morphology prediction is realized, secondly, the deep separable convolution is used to replace the ordinary convolution, the parameters of the convolutional network are reduced, and the calculation efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 It is a final network structure schematic diagram of the sperm morphological analysis method based on deep learning;

[0022] Figure 2 It is a whole flow chart of the sperm morphological analysis method based on deep learning;

[0023] Figure 3 The loss function of forward propagation calculation of the sperm morphology analysis method based on deep learning is given.

[0024] Figure 4 The evaluation index of the evaluation convolutional neural network is finally given. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0026] In the embodiments, the sperm morphology analysis method based on deep learning is given, which comprises the following operation steps: Figures 1-4 The sperm morphology analysis method based on deep learning is given, which comprises the following operation steps:

[0027] S1, preparing and dividing a data set, collecting sperm data of 85 male participants, the ages of the 85 males are all above 18 years old, for each participant, the parameters of standard semen analysis, the video of live sperm, the fatty acid of sperm, the fatty acid composition of serum phospholipid, demographic data and WHO analysis data can be obtained, the data set contains more than 35GB of video, each video lasts two to seven minutes, the frame rate is 50 frames per second, the data set contains six CSV files in total, a description file and a video folder, each video file is labeled with ID, video capture date, the data set is made and arranged by a professional medical institution, is valid, 60% of the data set is used for training, 20% is used as a validation set, 20% is used as a test set, the 3-fold validation method is used, the data set is the VISEM data set, and the data set is expanded to prevent overfitting.

[0028] S2. During training, the training set is preprocessed by extracting consecutive frames as input. After multiple experiments, 16 consecutive frames are used as input. The input images are cropped and randomly reversed to simulate the jitter of manual detection and the diversity of samples. Then, an improved 3D convolutional network is used to extract information from the consecutive input image frames. The loss function is calculated during forward propagation, and the parameters of the convolutional neural network are adjusted using gradient descent during backpropagation. The loss function is calculated during forward propagation and then adjusted using gradient descent during backpropagation to reduce the loss function until the loss on the training and validation sets tends to stabilize. When the loss function on the training and validation sets tends to stabilize, the learning rate is fine-tuned, and the convolutional network is trained repeatedly. The network training results are tested using a test set, and an evaluation metric for the convolutional neural network is provided.

[0029] S3. Sample reasoning: Package the trained deep learning network and the trained weight parameters, use the Streamlit framework in Python to build an app for predicting sperm motility indicators, load any semen sample video, call the trained convolutional neural network, and display the predicted motility indicators on the page.

[0030] The training process involves setting parameters, preprocessing the input video images in the training set to adjust the video images to 128*171 size, taking 16 consecutive frames as input to the 3D feature extraction network, then randomly inverting and cropping the input sperm images to 16×112×112 to achieve spatial and temporal jitter, using SGD as the optimizer, with an initial learning rate of 0.001, and dividing the learning rate by 2 when the loss function result no longer decreases. The training was pre-trained on the UCF101 classic action classification dataset and then fine-tuned on the Visem dataset.

[0031] The convolutional neural network (CNN) parameters are set as follows: the network consists of 5 convolutional layers, 5 pooling layers, 2 fully connected layers, and 1 softmax classification layer to predict the activity index of the mirror. The convolutional layers are composed of depthwise convolutions with a kernel size of 3×3×3 superimposed with standard convolutions of 1×1×1, which replaces the 3×3×3 convolutions, reducing the parameters of the CNN, preventing overfitting, and also making the inference speed faster in later stages. The number of filters in the convolutional layers are 64, 128, 256, 256, and 256 respectively. The first pooling layer has a pooling kernel size of 1×2×2, and the pooling kernel size of the remaining pooling layers is 2×2×2 with a stride of 1. The last two fully connected layers have 1024-dimensional outputs, and the final fully connected layer outputs 3 morphological indicators.

[0032] Expected Results: This algorithm applies video classification methods from the field of deep learning to sperm motility analysis. It directly extracts temporal and spatial information from sperm video pairs using a 3D convolutional neural network, achieving high-precision end-to-end sperm motility prediction. The trained convolutional neural network is visualized using the Streamlit framework in Python, providing a complete and clear display of the sperm motility detection results.

[0033] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0034] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A sperm morphology analysis method based on deep learning, characterized in that, The following steps are included: S1. Prepare and divide the dataset. Collect sperm data from 85 male participants. For each participant, obtain the parameters for standard semen analysis, videos of live sperm, sperm fatty acids, fatty acid composition of serum phospholipids, demographic data, and WHO analysis data. Use 60% of the dataset for training, 20% as a validation set, and 20% as a test set. Use the 3-fold validation method to expand the dataset to prevent overfitting. S2. During the training process, the training set is first preprocessed, and consecutive frames are extracted as input. After multiple experiments, 16 consecutive frames are used as input. The input images are cropped and randomly reversed to simulate the jitter of manual detection and the diversity of samples. Then, an improved 3D convolutional network is used to extract information from the consecutive input image frames. The loss function is calculated in forward propagation, and the parameters of the convolutional neural network are adjusted in backpropagation using gradient descent to reduce the loss function. When the calculation results of the loss function on the training set and validation set tend to stabilize, the learning rate is fine-tuned, and the convolutional network is trained again. The network training results are tested using the test set, and evaluation metrics for the convolutional neural network are given. S3. Sample reasoning: Package the trained deep learning network and the trained weight parameters, use the Streamlit framework in Python to build an app for predicting sperm motility indicators, load any semen sample video, call the trained convolutional neural network, and display the predicted motility indicators on the page. The training process involves preprocessing the input video images for the training set, then randomly inverting and cropping the input sperm images to 16×112×112 to achieve spatial and temporal jitter. SGD is used as the optimizer, with an initial learning rate of 0.

001. When the loss function no longer decreases, the learning rate is divided by 2. The training is pre-trained on the UCF101 classic action classification dataset and then fine-tuned on the Visem dataset. After preprocessing the input video images for the training set, the video images are adjusted to 128*171 size, and 16 consecutive frames are used as input to the 3D feature extraction network. The convolutional neural network (CNN) parameters are set, and the network includes 5 convolutional layers, 5 pooling layers, 2 fully connected layers, and 1 softmax classification layer to predict the activity index of the mirror. The number of filters in the convolutional layers are 64, 128, 256, 256, and 256 respectively. The first pooling layer has a pooling kernel size of 1×2×2, and the remaining pooling layers have a pooling kernel size of 2×2×2 with a stride of 1. The last two fully connected layers have 1024-dimensional outputs, and finally, a softmax layer outputs 3 indicators. The convolutional layers are composed of depthwise convolutions with a kernel size of 3×3×3 superimposed with standard convolutions of 1×1×1, which replaces the 3×3×3 convolutions, reducing the parameters of the CNN, preventing overfitting, and also making the subsequent inference speed faster.

2. The sperm morphology analysis method based on deep learning according to claim 1, characterized in that, In step S1, all 85 men must be 18 years of age or older.

3. The sperm morphology analysis method based on deep learning according to claim 1, characterized in that, In step S1, the dataset contains more than 35 GB of videos, each lasting two to seven minutes, with a frame rate of 50 frames per second. The dataset contains a total of six CSV files, a description file, and a video folder. Each video file is labeled with its ID and the video capture date. The dataset was created and organized by a professional medical institution.

4. The sperm morphology analysis method based on deep learning according to claim 1, characterized in that, In step S2, when the convolutional network is repeatedly trained, the loss function is calculated during forward propagation, and the parameters of the convolutional neural network are adjusted using gradient descent during backward propagation to reduce the loss function until the loss of the training set and the validation set tends to stabilize.

Images