Method for detection, classification and athletic ability assessment of sperm cells in images or videos

Abstract

The present invention discloses means for accurately and quantitatively assessing the motility of sperm based on a plurality of movement and morphological parameters measured using various image processing methods. The quantitative evaluation thus achieved is objective and multi-dimensional, such that detailed and repeatable evaluation of the sample is allowed. Also disclosed are deep learning methods for the detection and classification of sperm cells (or other types of cells or particles) using neural networks adapted to work under good and poor imaging conditions, such as low magnification and resolution. Particular methods for supervised and unsupervised deep learning methods are described. In the non-limiting disclosure, the method is particularly suited for the treatment of azoospermia in which a very low number of sperm cells (most of which do not swim) and a large number of fragments are present in the imaging field of view.

Description

Technical Field

[0001] The present invention relates generally to the field of automated visual inspection devices for the detection and classification of spermatozoa, including hardware and methods for analysis of motility. Background Art

[0002] Thus, motility is the ability of sperm to move correctly through the female reproductive tract to reach the egg. Since sperm motility is often correlated with successful fertilization rates, methods for assessing sperm motility often attempt to indirectly assess the reproductive capacity of the sperm population. Several different instruments and methods have been developed for automated or semi-automated assessment of sperm motility, including traditional spermatogenesis charts, time-lapse microscopy, frame-by-frame video microscopy, spectrophotometry, stroboscopy, and various computerized analysis methods.

[0003] Computerized motility analysis can provide objective measures of sperm motility characteristics taken from a large number of sperm trajectories. Such measures include the percentage of sperm that are motile, the percentage of sperm that are progressively motile (i.e., above preset cutoffs for movement velocity and curvature, associated with traditional manual assessment), the amplitude of lateral head displacement during forward motion, and measures of linear and curvilinear velocity.

[0004] Traditional spermatogenesis charts assess the fraction of motile sperm and a more fine-grained "motility grade" from grade A (rapid forward movement - swimming rapidly forward in a roughly straight line) to grade D (for non-motile sperm). Although "rapidity" here is a subjective assessment, the motility of rapidly forward moving sperm is generally considered the most reliable sperm motility criterion for predicting the fertilizing capacity of a semen sample. Traditionally, the assessment has usually been performed manually by a trained technician using a microscope equipped with phase contrast optics and a warming stage.

[0005] Such assessments of sperm motility typically include a total sperm motility fraction (the percentage of sperm exhibiting any form of motility), a forward-moving sperm motility fraction (the percentage of sperm exhibiting rapid linear movement), and sperm velocity on an arbitrary scale of 0 [non-motile] to 4 [rapid motility]. For example, a motility of 75 / 70 (4) indicates that 75% of the sperm are motile, while 70% are progressively motile and thus move "rapidly" across the microscopic field.

[0006] However, these methods all suffer from various disadvantages, including limited spatial and temporal resolution, crude assessments of measured motion that are often limited to mean linear velocity (if measured at all), and the requirement for varying degrees of human intervention and analysis. Summary of the invention

[0007] The present invention comprises systems and methods suitable for quantitatively and reproducibly assessing sperm motility, providing substantial improvements over the state of the art in terms of the resolution of the measurements obtained and in the nature of the motility parameters that can be assessed.

[0008] Specifically, the present invention provides firstly an apparatus and method for achieving super-resolution to assess position and motion at the sub-pixel level, and secondly, for modeling the motion of motile spermatozoa by fitting multiple motion parameters to the observed motion. The latter allows for better characterization of spermatozoa motion by introducing quantitative measures of linearity and curvature as well as "higher order" motion, as will be discussed in the detailed description.

[0009] Disclosed are additional methods for automatically detecting sperm cells (or other types of cells or particles) using a neural network that is adapted to work under both good and poor imaging conditions, such as low magnification and resolution. Detection is a necessary first step to further automated analysis, including motility assessment. For this purpose, a training phase is used that can use automatically generated training data as well as more standard human annotated training data, and once this phase is complete, the network can, in some cases, automatically detect sperm cells (stationary or dynamic) in a given field of view in real time. The training phase can typically be trained in batches using a stochastic approach, using methods such as back-propagation.

[0010] The method is particularly suitable for dealing with cases of azoospermia, where there are very low numbers of sperm cells (mostly non-motile) and a large amount of debris in the imaging field. The method allows for the automatic digital removal of most of this debris, thereby facilitating subsequent analysis.

[0011] Additional methods are disclosed for determining neural network latent space or hidden features from static images (morphological features) and / or videos (dynamic features) for classification purposes, as well as unsupervised methods that generate such features and allow training on unlabeled data. The use of latent space features allows for a new standardization of image and video analysis of sperm by indicating what the most important static and dynamic features for sperm analysis are.

[0012] The method of the present invention can be used to automatically grade the quality of sperm cells, and can also be used to automatically detect sperm cells in images or videos and distinguish them from debris.

[0013] The above embodiments of the present invention have been described and illustrated in conjunction with the systems and methods of the present invention, which are merely illustrative and not restrictive. In addition, just as each specific reference may embody a specific method / system, it is not required that such a specific embodiment be used, but ultimately such teachings apply to all expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Embodiments and features of the present invention are described herein in conjunction with the following drawings:

[0015] Figure 1 A method for parameterization of the sperm's path is shown.

[0016] Figure 2 A graph of athletic performance versus beat frequency is shown.

[0017] Figure 3 A sperm sample and a pipette are shown.

[0018] Figure 4 A possible implementation of CNN and RNN suitable for motion prediction in video sequences is shown. DETAILED DESCRIPTION

[0019] The present invention will be understood from the detailed description of the following preferred embodiments, which are intended to be illustrative rather than limiting. For the sake of brevity, some well-known features, methods, systems, processes, components, circuits, etc. are not described in detail.

[0020] Hardware Setup

[0021] The present invention can use a standard biological cell imaging setup (which may, for example, include one or more phase contrast microscopes or other types of microscopes, a sample stage with xy and possibly z-axis control, an optional heated sample bed, and a video camera). In principle, a cell phone with appropriate magnification can be used instead of a dedicated imaging setup. Alternatively, images or videos taken using such methods can be analyzed by the software and other devices of the present invention.

[0022] Super pixel resolution

[0023] The present invention provides apparatus and methods for achieving super-resolution to estimate position and movement at a sub-pixel level, allowing the subsequent image processing steps described below to be performed with greater accuracy than would otherwise be possible.

[0024] The present invention uses methods for super-resolution, which are generally techniques that use multiple lower resolution images together to achieve a resolution higher than the camera hardware. This can be achieved by one or more devices. Below we mention methods for increasing the number of images acquired in a given time, and then discuss some methods for combining the information obtained from these images to achieve super-pixel resolution.

[0025] Sample stage movement

[0026] To handle motion of the sample stage (e.g., when the stage is moved to analyze a new area of ​​the sample), collective motion cancellation can be used. This can be done using a continuous estimate of the motion vectors for the entire FOV. Relative to this motion, the corresponding motion vectors of the individually tracked moving objects can be subtracted to find the "absolute motion" of these objects. This feature allows continuous tracking of sperm cells and continuous accurate motion assessment even when the operator is moving the microscope stage. The motion vector can be calculated as the highest amplitude percentile motion vector detected within the sampling space within the FOV, where the FOV is divided into equal rectangles of, for example, 1 / 10 of the width of the FOV × 1 / 10 of the length (or any other ratio of the FOV to the size of each individual unit). It is also within the provisions of the present invention to use a fixed pattern on the sample plate (e.g., a grid, ruler or other easily identifiable and tracked object), on the basis of which the basic motion vectors of the FOV are estimated. Such subtraction should be conditioned on a minimum threshold energy of the "frequent" average motion vector percentile.

[0027] An alternative implementation is to average the motion of the tracked object and, in the case where such motion is found to have sufficient energy (e.g., measured in terms of kinetic energy averaged over a certain time frame), calculate the average motion of the slide from this value. This approach may have limitations in the case where the motion of the sperm cells is spatially constrained (e.g., constrained by the edges of the aqueous droplet in which the sperm are swimming). In addition, such calculations may also be biased in the case of significant effects of chemotaxis or thermal axes, i.e., in the case of chemical or thermal stimulation of individual sperm cells, in which case the motion vector of the FOV may not be correctly estimated. However, since chemotaxis and / or thermal axes may be relevant parameters for healthy sperm, these cases may have additional utility. Therefore, it is within the provisions of the present invention to provide chemical and / or thermal gradients in the sample tray and observe sperm motion under the influence of these gradients. By observing motion with and without the gradient, the preferential motion under the gradient can be determined. Therefore, the sample tray can be initially neutral, motion is observed, then a thermal or chemical gradient is introduced, and new motion is observed, allowing the difference in the gradient to be discovered, and then the contribution of the gradient to be discovered.

[0028] Another approach is to use an xy (and possibly z-) stage that is motorized and equipped with shaft encoders, linear encoders, or other means for position readout, or both. In this case, the stage motions are known from the shaft encoders, motor actuation, or both and can be accounted for in the above algorithm. In the case of superpixel resolution, the measured / known motions can be used as initial input to an algorithm adapted to further refine these initial estimates of stage motions.

[0029] For this purpose, additional methods we considered were the eight-parameter projective motion model, block matching, and Horn-Schunck optical flow estimation (see "Subpixel Motion Estimation for Super-Resolution Image Sequence Enhancement" in Journal of Visual Communication and Image Representation, Vol. 9, No. 1, March 1998, pp. 38-50).

[0030] Registration and background removal

[0031] Registration and / or background subtraction methods can be used as a first step before superpixel and other subsequent operations.

[0032] If the camera is set up to be largely stationary, then various types of background removal can also be implemented to improve resolution. The background image can be evaluated by using a moving average over history, or for example a "median image" can be utilized, where the background image is calculated for each pixel as the median (not the average) of that pixel over some history. This will tend to remove motion artifacts due to (for example) the movement of sperm in the image more completely than taking an average image. However, the background can be calculated and then subtracted from a given frame to more clearly show only the moving objects. Since not all elements of the moving object must move from frame to frame (or equivalently, if movement occurs between areas with the same pixel value), moving pixels may not be marked simply from frame to frame, but for example any pixel that has moved within the last N frames may be marked as showing movement. Segmentation methods can be employed to identify the entire moving object (in this case typically the moving sperm), for example based on conditional random fields, neural networks or other segmentation methods.

[0033] If the camera is not fixed relative to the sample stage, various image registration methods can be used to register consecutive images and produce a situation where consecutive frames "match" with respect to the background, largely eliminating the effects of relative movement between the camera and the sample stage. For example, registration can be achieved by using image features such as SURF or SIFT features (as will be familiar to those skilled in the art of computer vision) in the two frames to be registered, and from these features, computing a homography using features common to both frames to transform the coordinate system between the images.

[0034] Filter irrelevant objects

[0035] In order to avoid tracking objects that appear to be moving in the incoming frame but are not relevant to the measurement being made, several techniques can be used.

[0036] Objects may be eliminated based on their size, depending on the magnification. For example, a micropipette, which has a surface area significantly larger than a typical sperm cell head, may be eliminated from the list of tracked moving objects. This is to prevent expensive computational resources from being "wasted" on processing the motion of irrelevant objects. Thus, segmentation techniques (e.g., based on conditional random fields, neural networks, or other segmentation means) may be used to segment out these objects and remove them from the frames in which they appear. Figure 3 An example of an image including sperm and a pipette is shown; in this case, the larger pipette as described above may not be considered further. Figure 3 In the figure, sperm cells 301 and capillary tip 302 can be seen.

[0037] It is also possible to eliminate objects based on their morphology. For example, tissue fragments can be effectively removed even if they are on the same scale as sperm cells. This can be done with morphology-based means, such as curvature and moment-based measurements, or neural network approaches, because they do not have a "tail" (as mature sperm cells are expected to have), so they are morphologically different from sperm cells and can be excluded from being tracked on this basis. Figure 3 An example of an image including sperm and a pipette is shown; in this case, the morphologically distinct pipette as described above may not be further considered.

[0038] Another method used to eliminate objects is based on their estimated average speed, i.e. if this speed exceeds or falls below a certain threshold. This is also to prevent expensive computing resources from being "wasted" on irrelevant objects. For example, an operator may (manually) move the pipette at a speed far exceeding the maximum speed of sperm cells, in which case this can be used as an indication of an irrelevant target.

[0039] Adaptive noise reduction thresholding (e.g. using the CFAR - "Constant False Alarm Rate" method) and global vibration cancellation may also be used, which is performed using the following steps:

[0040] 1. Perform spatial (i.e. grid) calculations on the “noise” statistics within each grid region,

[0041] 2. Define a "low pass filter" criterion that determines the accumulation of historical values ​​for each pixel in order to increase the sensitivity of the detection mechanism in case of low "duty cycle" of motion.

[0042] 3. Define the “threshold percentile” above which the value will be considered a threshold crossing.

[0043] 4. Define N / M “persistence criteria” so that only pixels exceeding a “threshold percentile” value in their corresponding grid will be considered “persistent” objects rather than spurious noise.

[0044] All of these methods can be enhanced by the background subtraction methods mentioned above, such as using the median image as an indicator of static background to be removed.

[0045] Brightness Calibration

[0046] A calibration step can be performed early in the process (e.g., before or after registration / background removal) to remove the effects of variable illumination. Fluorescent, LED, or other ambient illumination that leaks into the image can, for example, have periodic peak-and-valley effects, also known as "flapping," which will cause some frames to be brighter and others to be lower; power supply variations in microscope illumination can also produce such effects. To handle this, a step of automatic brightness equalization can be performed, in which, for example, a "reference frame" is calculated as an average over multiple frames (possibly a moving average, slowly varying over time), and the brightness (or brightness, contrast, histogram, white balance, entropy, or other parameter) of subsequent frames is adjusted to match the "reference frame." The goal of such automatic calibration is to equalize the brightness differences between incoming frames before they are combined in a subsequent step. Similarly, automatic detection and removal of hot pixels and dead columns can be performed to remove the effects of CCD defects.

[0047] Increased number of images

[0048] In order to obtain a greater number of frames with minimal motion in the observed image, the camera frame rate can be operated as high as possible, such as in a "burst mode" available on some cameras, or by using a particularly high frame rate camera. It will be appreciated that a higher frame rate may itself be traded off with greater noise unless the system utilizes a larger aperture and / or brighter illumination to achieve the same image brightness as that achieved at a lower frame rate.

[0049] Secondly, if color information is not of interest (and in the case of sperm motion tracking, it may actually be redundant), each of the R, G, B planes of a given color image can be used alone as an intensity image to provide three images for each frame acquired. Such an approach may allow for an increase in the SNR in the target image.

[0050] Combined images

[0051] Now that we have a collection or "stack" of registered and calibrated images, various means for combining these images can be employed.

[0052] average

[0053] The first approach for superpixel resolution uses the average of a collection of images to achieve superpixel resolution. This is the simplest approach and uses the average of all pixels in the stack calculated for each pixel.

[0054] Median

[0055] The second method uses the median (also used for background subtraction). This method uses the median value of the pixels in the set calculated for each pixel of the image.

[0056] Maximum

[0057] In this method, the maximum value of all pixels in the stack is calculated for each pixel. This is useful for debugging purposes to visualize all defects of all calibration images.

[0058] Kappa-Sigma clipping

[0059] This method is used to iteratively remove offset pixels and utilizes two parameters: the number of iterations and the standard deviation multiplier (Kappa). For each iteration, the mean and standard deviation (Sigma) of the pixels in the stack are calculated. Each pixel with a value that is farther away from the mean than κ*σ is removed. The mean of the remaining pixels in the stack is calculated for each pixel.

[0060] Median Kappa-Sigma Trimming

[0061] This method is similar to Kappa-Sigma pruning, but instead of removing outlying pixel values, they are replaced by the median value.

[0062] Adaptive Weighted Average

[0063] The method computes a robust average obtained by iteratively weighting each pixel in terms of its deviation from the average, as a fraction of the standard deviation (see Least Squares Methods and Star Photometry Techniques Using CCDs (The Techniques of Least Squares and Stellar Photometry with CCDs) -Peter B. Stetson 1989)

[0064] Entropy Weighted Average (High Dynamic Range)

[0065] This method is based on the German method of Jenkin and Lesperince (see Entropy-Based Image Merging image merging)-2005 ) and is used to stack a collection of images into a final picture while maintaining the best dynamic range for each pixel.

[0066] Another approach for achieving super-resolution is to use bursts of CFA raw images with small offsets. As described in Handheld Multi-Frame Super-Resolution ( https: / / arxiv.org / abs / 1905.03277)The frames are then aligned and merged to form a single image with red, green, and blue values ​​at each pixel location, increasing image resolution and enhancing signal-to-noise ratio.

[0067] The new image produced by any of the above methods may be implemented as a higher bit depth image with the same resolution (e.g., by summing rather than averaging), or may have the same bit depth but with lower noise due to averaging, or may have a higher spatial resolution at the same bit depth.

[0068] By using such superpixel methods as listed above, the method can account for situations where (for example) the typical position difference of sperm cells from frame to frame is, for example, 0.3 pixels (in the original unenhanced image), so that on average over 60% of the "unused" frames will not indicate any motion. However, assuming that the motion detection process subtracts the "motion tail" of the 10 most recent frames, in this case even an average motion of about 0.1 pixels per frame is identifiable.

[0069] Move parameterization

[0070] The present invention provides for simulating the movement of motile sperm by fitting curves based on multiple movement parameters to the observed movement. The latter allows for better characterization of sperm movement by introducing quantitative measures of linearity and curvilinearity as well as "higher order" movement, as will be discussed in the detailed description. These methods can allow for estimation of trajectory parameters as well as handling of sperm cell "collisions" via movement characterization.

[0071] In one approach, the best estimator can be selected from among several estimators over several motion models. The concept of "best" here can be quantified, for example, by using a cost function on the estimation error for each particular fitted model, for example, such a cost function can use the average rms distance between the fitted position and the observed position over a certain number of frames or over a certain time frame.

[0072] Motion Estimator

[0073] Kalman filtering can be used as a motion estimator. For example, the position to be estimated can be the center of the sperm cell head. The position can be forward extrapolated based on a polynomial curve fit, a sinusoidal curve fit, or any other function fit suitable for the observed motion. The polynomial or other estimator can be calculated by using linear and rotational velocities and accelerations, and slalom-like trajectories. Alternatively, the motion of the sperm can be simulated as a sinusoid, a cardioid, an Nth order polynomial, etc. Regardless of the type of motion used for simulation, a Kalman filter can be used to estimate the parameters of the motion to estimate the subsequent or previous position of the sperm cell.

[0074] Figure 1The model shown shows three types of trajectories, one superimposed on the other. The thickest trajectory 101 is circular and has a radius R 0 , angular velocity ω 0 and the starting angle θ 0 .

[0075] The intermediate trajectory 102 is a sinusoidal curve superimposed on the above-mentioned 0th order circular motion and has an amplitude R 1 , angular velocity ω 1 and the starting angle θ 1 .

[0076] The thinnest track 103 is also a sinusoidal curve, which is superimposed on the above-mentioned 0th and 1st order tracks and has an amplitude R 2 (shown as "B" in the image), the angular rate ω 2 and the starting angle θ 2 .

[0077] In general, the polar coordinate representation of the position of an individual sperm cell can be approximated by the following equation:

[0078] R(t)=R 0 +R 1 *sin(ω 1 *t+θ 1 )+R 2 *Sin(ω 2 *t+θ 2 )

[0079] θ(t)=θ(t)+ω 0 *t

[0080] Kalman filtering on at least the third degree of freedom (i.e., position, velocity, acceleration, derivative of acceleration) can be used to estimate various parameters, such as R in the above parameterization. 0 , R 1 , R 2 ,θ 0 ,θ 1 ,θ 2 ,ω 0 ,ω 1 ,ω 2 , or in the parameterized R 0 , V and a.

[0081] R(t)=R 0 +V*t+1 / 2a 2 *t

[0082] Once the motion is fitted, the motion capability function can be calculated from the fitted movement parameters. For example, for the circular and sine parameterizations, the motion capability function can be

[0083] M=A / R 0 +B|R 1,阈值 -R 1 |+C|ω 1,阈值 -ω 1 |+D|R 2,阈值 -R 2 |+E|ω 2,阈值 -ω 2 |

[0084] And similarly, for the second velocity and acceleration parameterization,

[0085] M=A / R 0 +B|V 阈值 -V|+C|a 阈值 -a|

[0086] can be used as a motility function. In both cases, for lower values ​​of M, the motility function defined here is better (describes healthier sperm). Alternatively, a motility function that increases for healthier sperm can be defined, bounded or unbounded. Furthermore, the use of a multi-valued motility function is also within the scope of the present invention. For example, two values

[0087] M R =A / R 0 +B|R 1,阈值 -R 1 |+D|R 2,阈值- R 2 |

[0088] as well as

[0089] M ω =C|ω 1,阈值 -ω 1 |+E|ω 2,阈值 -ω 2 |

[0090] can be calculated to give the path curvature M R and the movement speed M ω different estimates of health.

[0091] Overcoming sperm "collisions" (i.e. sperm trajectory interception)

[0092] To track individual sperm, the method outlined above is based on various possible parameterized empirical motion representations, i.e., feeding forward the position of each individual sperm cell head, which is performed via a Kalman filter. As mentioned, these may include linear and rotational velocities and accelerations, as well as slalom-like or sinusoidal trajectories, and using a Kalman filter to feed forward or estimate the subsequent position of the sperm to keep track of the individual sperm cell heads.

[0093] An alternative approach is to use a temporal neural network, such as a recurrent neural network (RNN) suitable for motion prediction purposes. The input to such an RNN may be a series of images, and the output may be the expected next frame or frames. Alternatively, the input may be a series of images and labels for each sperm, and the output may be a set of next expected positions for each labeled sperm. A more detailed description of these methods is discussed below.

[0094] Determination of length scale

[0095] Automatic scaling can be used by introducing a known spatial pattern, such as a grid or ruler. Moreover, automatic scaling can be employed by, for example, measuring the average sperm cell tail length, which is known to be approximately 50 μm, or by measuring a standard object introduced into the field of view, such as a pipette of known shape and size. If the object magnification and sample distance are known, the length scale can be calculated directly from this information. Another "natural ruler" can also simply be the known average size of the sperm itself - because this size is known (e.g., for a given donor population or individual donors), the length scale can be determined to a certain level of accuracy based solely on the size of the sperm, the accuracy of the length scale increasing with the number of sperm visible.

[0096] Pattern recognition for TESA / TESE

[0097] Pattern recognition methods for identifying immotile sperm cells in TESE / TESA tissue samples are now discussed. The identification of viable but immobile sperm cells in TESE / TESA tissue samples can be accomplished using sperm tail beat patterns. Typically, sperm cells in TESE / TESA tissue samples move their tails, but they do not appear to be swimming because they may be trapped in tissue debris or simply lack energy. Applying a temporal Fourier transform to individual pixels, or alternatively to individual spatial cells in a grid in the FOV (e.g., a 10×10 or 50×50 grid), can be used to identify areas with a large amount of energy content in the typical tail beat frequency of sperm cells (e.g., a frequency range of 6-14 Hz). The system can then use the presence of suprathreshold (energy-wise) signals in specific spatial cells in the grid of the FOV to provide a visual cue to the operator of these specific spatial cells, so that the operator can position the stage in the center of these cells and switch the microscope to a higher magnification to examine whether there are motile sperm cells arranged therein. According to the invention, this stage centering operation is automatically performed by means of a computer driven xy (and possibly z-) stage. The tail beat frequency of both moving and stationary cells can be determined by means of a movement parameterization such as described above, which is particularly suitable for the purpose of tail beat frequency, for example by means of a preliminary step of subtracting the sperm centroid movement, in order to obtain a stationary but still tail beating sample.

[0098] According to the provisions of the present invention, a scoring mechanism is implemented to distinguish sperm cells with different tail beat frequencies as an indication of the quality of each individual sperm cell. For example, Figure 2 Indicates the correlation between beat frequency and athletic ability.

[0099] By using a deep neural network (e.g., a convolutional neural network, CNN) to detect sperm cells (or other types of cells or particles), it is possible to automatically detect moving, stationary tail-beating, and completely stationary sperm cells in an image or video. A specific implementation of the method is as follows Figure 5 shown.

[0100] The method may use any of the preprocessing steps described above, but even without such preprocessing, the method is suitable for working even under poor imaging conditions, such as those of low magnification, low resolution, variable illumination, noisy environments (as encountered in TESE / TESA samples), etc.

[0101] During training, the neural network receives images of verified sperm cells ("labeled examples") under identical imaging conditions, and learns the weights of the network. These labeled examples used for training may be obtained by manual labeling or by automatic labeling, as described above. For example, the use of conventional computer vision tracking algorithms allows multiple position frames to be derived from a single initial frame. The above-described motion subtraction algorithms may be further used, such as subtraction of the median image, since in most samples only sperm will be moving, and thus the only objects left after motion subtraction will be sperm cells, which may be automatically labeled using, for example, conventional computer vision morphological operations, and then manually inspected.

[0102] Figure 5 An example of the training phase is shown in flowchart 610 of . The initial input 601 includes images or videos of swimming sperm cells, which may be under poor imaging conditions. If these sperm are swimming, then automatic detection, tracking, and labeling are easily performed with the help of a traditional computer vision (CV) motion detector / tracker 602. Motion detection and tracking can include algorithms such as dense optical flow, which estimates the motion vector of each pixel in the video frame; sparse optical flow or Kanade-Lucas-Tomashi (KLT) feature trackers, which track the positions of several feature points in the image; Kalman filtering, which can be used to predict the position of moving objects based on previous motion information; Meanshift and Camshift (continuous adaptive mean shift) to locate the maximum value of the density function; and any other object detectors and trackers that can be used in this context. A single object tracker can be combined to perform multiple object tracking by re-identification, as is the case with multiple algorithms listed above.

[0103] These detections produce labeled sperm images or videos 603 that are suitable for training a deep neural network 603 suitable for detecting sperm. The detections may be verified by other means such as manual verification. Once a suitable number of labeled image or video sequences have been generated, they may be used to train a neural network 604, such as a deep convolutional neural network.

[0104] Then, after the training phase is complete, an inference phase is possible. In inference, because the network has learned the characteristics of sperm cell images under various expected conditions (low magnification, low resolution, variable illumination, etc.), the network can automatically detect static or dynamic sperm cells over the imaging field of view.

[0105] exist Figure 5An example of the inference phase is shown in flowchart 620 of FIG. Here, an unlabeled image or video sequence 605 (of the type used in the training step of flowchart 610) is input into a previously generated trained deep neural network 606. The network outputs inferences 607 representing the locations of resting (or non-resting) sperm cells. The output may be in terms of bounding boxes, pixel-level segmentations, etc. determined by the nature of the training data.

[0106] Inference on any network of the present invention may be performed in real time (eg, by dedicated hardware).

[0107] This method can be used in the case of azoospermia in TESE / TESA samples, where sperm cells need to be found at low magnification (in order to see the whole sample) and there are very low numbers of sperm cells, most of which are not swimming. In addition, there is often a lot of debris in the imaged field of view. In this case, for training, we can take images of swimming sperm cells (which can be detected automatically) from another healthy sample (without azoospermia) so that the trained network can then, during inference, detect all sperm cells in the azoospermia sample.

[0108] Viscous media test

[0109] According to the present invention, motility parameters measured in the same cells in aqueous medium are evaluated by the following steps, wherein the sperm sample is placed in a more viscous medium:

[0110] 1. Obtaining the velocity threshold of human sperm cells in aqueous media and indicating their motility quality score

[0111] 2. Obtaining the Viscosity Velocity Curve of Human Sperm Cells

[0112] 3. Obtaining the concentration-viscosity relationship of a specific sperm cell suspension medium (e.g., PVP)

[0113] 4. Obtain specific types of IVF media and their concentrations (relative to water) per operator session.

[0114] 5. Applying the concentration viscosity and subsequent viscosity velocity transformation to each measured velocity of an individual human sperm cell in the medium, thereby estimating the instantaneous equivalent velocity of the human sperm cells in the aqueous medium.

[0115] 6. Compare the transformation velocity of an individual sperm cell with the threshold value to obtain a motility quality score for that particular sperm cell.

[0116] Specifically, the relationship between sperm velocity and water velocity in polyvinylpyrrolidone (PVP) solutions of various commonly used concentrations is provided by the following approximate relationship:

[0117] ■For PVP7% V_who = V_pvp * 1.9, or

[0118] ■For PVP 10% V_who = V_pvp * 2.4, or

[0119] ■For water, V_who = V_pvp

[0120] Where V_who is the speed in water, and V_pvp is the speed in various PVP solutions.

[0121] flow chart

[0122] Figure 6 A flow chart of an exemplary embodiment is shown in simplified form. Optional steps are shown in boxes with dashed outlines, while required steps are shown in boxes with solid outlines.

[0123] First, an image sequence is obtained 501. This input may include one or more images; in principle, the pipeline can work on an image-by-image or sequence-by-sequence basis (e.g., it can be combined with an RNN for tracking as described below). It can be found that other information can be used as input, including system magnification and illumination, sample parameters such as temperature, pH, viscosity, PVP concentration, etc.; and perhaps other information that can provide useful context for subsequent analysis.

[0124] As described above, one may now optionally proceed to the steps of sample stage motion removal 502, registration 503, background removal 504, length scale determination 505, brightness and camera calibration 506, and image combination for superpixel resolution 507. The order of these operations is not necessarily as listed in this example, and a different order may be found to be more useful in different scenarios.

[0125] The sperm detection step 508 is now ideally performed in an "instance-aware" manner, so that each sperm is distinguishable from the others and can be tracked frame by frame. Motion parameterization 509 of individual and / or combined motions may now be performed in a variety of ways as described above; generally, a set of measures related to the motility of individual sperm are calculated. An optional step of non-motile sperm detection and parameterization 510 may also now be performed (perhaps using images that have not been subjected to background subtraction 504, which may tend to eliminate stationary portions of the sperm of interest).

[0126] The individual sperm calculations obtained can now be combined with sample level calculations 511, for example including 2 An overall rating of the sample in terms of the number of motile sperm or an average of motility measurements, or more detailed information such as a histogram or other measure of population size versus motility level.

[0127] The measurements obtained from the sample may now be presented 512 to a user in some form, such as by means of a suitable GUI or other web interface that may be remotely managed over a network. The sequence may now be repeated as new images or sequences of images are acquired 501. In some implementations, sample-level calculations and presentations may be performed only once every N iterations of the loop.

[0128] Pattern Recognition - Supervised Learning for Detection, Tracking, Classification and Regression

[0129] The above methods can be combined with machine learning, computer vision, and artificial intelligence to better determine the various parameters of interest.

[0130] For example, the systems and methods of the present invention can use images or videos that have been preprocessed such as described above or used directly. One method of the present invention for detecting sperm cells (or other types of cells or particles) includes using deep neural networks (e.g., convolutional neural networks, CNNs) and / or various types of recurrent neural networks (RNNs) for detection, tracking, regression, and classification. In addition to training with images or videos having such poor imaging conditions, with the aid of preprocessing as described above, these methods can be adapted to work even under poor imaging conditions, such as low magnification and resolution.

[0131] Analysis can be performed on detection (e.g., generation of bounding boxes around each detected example or pixel-level segmentation), classification (e.g., good / medium / bad or motility rating 1 to 5), regression (e.g., motility rating on a scale from 1-100), tracking (determining paths over time, ideally in an "instance-aware" implementation), and any other output of the neural network model.

[0132] A training phase is employed during which one or more neural networks receive images of validated sperm cells under a certain set of imaging conditions along with class labels and / or continuous variables such as velocity ("labeled examples") and learn the weights of the network. Training will be performed using labeled examples of the desired output type.

[0133] Training can be achieved with the help of back-propagation of errors to gradually change the weights of the neural network in a direction that tends to minimize the loss function, which can be achieved using cross entropy (for class labels), mean squared error (for continuous variables), etc. The loss can be calculated in mini-batches using so-called stochastic methods to speed up learning by avoiding having to use the entire training cohort for each training step.

[0134] Labeled examples for supervised learning situations can be obtained by manual labeling or with the aid of automatic or semi-automatic labeling. For example, given a video of moving sperm cells with no other moving elements (an "easy to detect" sample), methods such as background subtraction or traditional motion detection based on computer vision can be used to detect the moving elements, leaving the moving sperm cells that can be used as training examples. Labeling can be achieved in terms of bounding boxes, pixel-level segmentation, or other means. Bootstrap tagging, where some training examples are used to produce a network with low accuracy, which is used to label more images, which are then corrected and used to train a slightly better network, the output of which is used to train a third generation, etc., can also be used in this context.

[0135] Methods for (automatically) generating sperm cell images for training deep learning networks can be used to further train neural network classifiers. These methods include using parametric 3D models, using generative adversary models based on realistic collections of 3D simulated images, and using any other methods that may be deemed useful.

[0136] After the training phase is complete, the network can be used for inference for automatic detection, classification, and parameter estimation (regression) of sperm cells in static or dynamic environments. In the case where training data is provided under suboptimal conditions (of poor resolution, lighting, jitter, etc.), inference (trained network operation) can also be expected to successfully process such images or videos under similar conditions.

[0137] Inference of the network can be implemented in real time (e.g., via dedicated hardware) or offline, possibly occurring over an online connection (whereby, for example, images / video are sent to a server, analyzed, and the results are sent back to a user, for example, with the aid of an application programming interface (API) that defines a protocol for requesting and receiving such inferences).

[0138] It is also within the provisions of the present invention to propose hybrid operating modes, wherein, within the scope of the so-called "operating" mode of such a system, the operator can indicate to the system where he / she identifies a suspicious sperm cell, in which case the system will add it to its training set, either locally, i.e., in a way that only affects this particular operator, or on a workstation basis, i.e., for this particular system, or on a field basis, i.e., for a plurality of such systems set up in a given laboratory, and finally, globally, i.e., so that all similar systems are similarly updated.

[0139] Tracking can be accomplished in several ways: the simplest approach involves the use of a system adapted for still frame localization as described above. More sophisticated approaches utilize convolutional neural networks adapted to take multiple frames as input, and / or convolutional nets in conjunction with recurrent neural networks (RNNs). Any of these approaches allow the system to process video for the purpose of tracking. Ideally, tracking is performed in an instance-aware manner so that a given sperm is followed even in the presence of occlusions, interferences, and overlaps. Training in the case of video can be accomplished using labeled or annotated video sequences, which themselves can be semi-automatically generated using the still frame recognition / localization apparatus as described above together with instance labeling, which in most cases can also be automatic (e.g.) human-corrected in the presence of obstructions, interferences, or overlaps.

[0140] It is now possible to measure the dynamic characteristics of sperm using a network capable of processing sequences of images, either in terms of classification (giving a "class grade" output such as no motility, low motility, normal motility and high motility) or in terms of regression (giving a continuous fraction of motility, for example in arbitrary units or units of velocity, or measuring a physical characteristic such as power output, morphology or beat frequency). In the latter case of measuring motility, velocity or other dynamic characteristics, the output of the still frame positioning method described above can also be used in conjunction with knowledge of the time difference between frames (and possible sample stage movement) to calculate velocity or other motility parameters. Other motility parameters that may be of interest include straightness, swimming efficiency, forward velocity compared to "swing" velocity, velocity in various media and as a function of chemical or other (e.g., temperature or pressure) gradients, etc. In accordance with the provisions of the present invention, output is provided in terms of any static or dynamic parameter of sperm behavior and morphology.

[0141] Similar classifications can be made with respect to the morphology of the cells; for example, into "ideal," "fairly good," "poor," and "abnormal" groups. Combined classification / regression with several outputs is also within the scope of the invention, e.g., a single network is trained to give outputs in terms of morphology class and velocity.

[0142] Unsupervised methods

[0143] In accordance with the provisions of the present invention, unsupervised methods of machine learning are also utilized for clustering or determining effective latent representations, which in turn can be used for classification. For example, an autoencoder or another method for unsupervised feature learning can be used. In the case of an autoencoder, convolutional layers can be used to reduce the information content of the input image or sperm video sequence to a bottleneck where the network attempts to reproduce the original input. The key point of this exercise is that the "bottleneck layer" now contains a compact or "latent space" representation of the input, which can then be used to compare various inputs and show clusters of similar inputs, which may more faithfully reflect the actual situation than a set of more or less arbitrary categories that are then manually annotated or otherwise determined by human permission.

[0144] For example, if an autoencoder is trained on static images or video sequences of sperm, after training it may be found that the images of sperm naturally cluster into three types that appear upon visual inspection as a normal type and two different types of abnormal morphology or motility. Similarly, autoencoding of video sequences may reveal several different types of swimmers, a finding that may be useful in both clinical and research settings. Once such a network is generated, the network may be used for classification using clustering-based means such as measuring the distance to the center of a cluster or similar means as will be apparent to those skilled in the art. Similarly, to overcome situations with sparse training data, a compact representation of such a network may be fine-tuned with annotated data.

[0145] Specific Applications - Azoospermia

[0146] In the case of azoospermia, the number of spermatozoa is very low (most are non-motile). Therefore, a wide field of view and low magnification are usually used (to allow potentially more cells to be seen at one time). Since in azoospermia, the biological sample is obtained via a semi-surgical testicular sperm extraction procedure called TESE or TESA, in which some volume of testicular tissue is removed or aspirated and then turned into a cell suspension, it turns out that most of the non-fluid content of such samples is non-sperm cells and cellular debris surrounding sperm cells, which in most cases are non-motile or, although motile, are stationary because they are trapped in the tissue debris.

[0147] In this case, for the supervised learning case, we can take images of, for example, swimming sperm from a healthy sample (which, as will be clear to one skilled in the art, can be detected automatically by using motion detection) so that a trained network can be trained. During inference, such a trained network will be able to detect all sperm in an azoospermic sample, since the network has learned the morphology of the sperm even under poor imaging conditions.

[0148] Other methods

[0149] Here, we describe a method for automatic detection and classification of sperm cells based on traditional and latent features detected by neural networks. These features can be extracted from still images (morphological features) and / or videos (dynamic features).

[0150] During training, the deep neural network uses training images (or videos) with labeled sperm cells (e.g., "normal" vs. "abnormal," etc., and / or continuous variables for regression) and learns features that characterize these inputs by calibrating the network weights.

[0151] For training, this labeling can be done artificially (e.g., using the ingenuity of one or more embryologists) by video object motion detection methods, by imaging sperm through sperm enrichment assays (such as swimming assays, microfluidics assays, DNA fragmentation assays, hyaluronan binding assays), or even by collecting sperm cells that naturally reach the egg area in the female body (and can therefore be assumed to be "good" sperm). As mentioned above, classification is only one possibility, and regression is another possible method for the network output (in which case the input would be an image not labeled with a class label, but rather one or more continuous values, such as sperm mass, swimming speed, etc. on a scale).

[0152] The network encodes each sperm cell into a latent space vector representing all the features (conventional and hidden) of the examined sperm, for example with the help of a suitably trained autoencoder.

[0153] During inference, the network receives new sperm cell images or videos and gives an appropriate output (e.g., classifies them, or in the case of a network trained for regression, gives one or more continuous outputs).

[0154] The differences between sperm cells can now be quantified by the mathematical distance between their latent space vectors. The latent space can be, for example, the penultimate layer of a neural network before the regression or class labeling layer.

[0155] The process can be used to automatically grade the quality of sperm cells, or to automatically detect sperm cells in images or videos and distinguish them from debris.

[0156] To check what specific features the network of good cells detects, we can use a generative network that takes a latent space vector and uses it "backwards" to generate images of good sperm cells. Then, by changing the values ​​in the latent space vector, we can visually check what features are changing in the generated images.

[0157] This could lead to a new standardization of sperm image analysis and video analysis, indicating what are the most important static and dynamic features for sperm analysis. A similar approach is to find the highest correlation between the latent space elements and the network output.

[0158] Overcoming sperm "collisions" (i.e. sperm trajectory interception)

[0159] As mentioned above, one approach to allow instance-aware tracking (which largely overcomes common problems with tracking such as occlusion) is to use a temporal neural network such as a recurrent neural network (RNN) suitable for motion prediction purposes. The input to such an RNN (or LSTM, GRU, or other network suitable for such purposes) can be a sequence of images, and the output is the position of the set of (uniquely identified) detected objects on each frame. Figure 4 As shown, this input can pass through the preliminary CNN stage. An alternative input can be a series of images and labels (e.g., instance labels, category labels and / or continuous variable measurements for regression) for each sperm, and a set of next expected positions and labels for each labeled sperm are output. Another possibility is to give an output in terms of predicted motion vectors, where each sperm cell is detected by the network and assigned an expected motion vector. The motion vector can be, for example, a vector that tends to produce an expected position in the next frame of the video sequence. High-order motion can also be extracted from such a network, with, for example, position, velocity, and acceleration outputs for each instance detection. In the case where sperm cell motion is parameterized (e.g., by considering the roughly sinusoidal shape of cell position over time, as produced by the swimming motion of healthy sperm cells), the network can be trained to output such parameterized coefficients.

[0160] The above description and illustration of embodiments of the present invention have been given for the purpose of illustration. It is not intended to be exhaustive or to limit the present invention in any way to the above description.

[0161] Any terms defined above and used in the claims should be interpreted in accordance with this definition.

[0162] Reference signs in the claims are not part of the claims but are used to facilitate reading thereof. These reference signs should not be construed as limiting the claims in any way.

[0163] All features disclosed in the specification (including claims, abstract and drawings), and all steps in any disclosed method or process may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in the specification (including claims, abstract and drawings) may be replaced by an alternative feature serving the same, equivalent or similar purpose.

Claims

1. A method for measuring sperm motility, characterized in that: The method consists of the following steps: a. Capture the image sequence of sperm sample; b. Tracking the position of each sperm in the image; c. fitting a function to the trajectory of each sperm; d. calculating one or more athletic performance functions based on the function; One or more quantitative mobility parameters are thereby objectively measured.

2. The method according to claim 1, characterized in that The step of capturing a sequence of images is accomplished with the aid of a video camera viewing the field of view through the microscope.

3. The method according to claim 1, characterized in that The step of tracking position is performed using sub-pixel positioning accuracy.

4. The method according to claim 1, characterized in that: The method uses a convolutional neural network to perform the tracking.

5. The method according to claim 4, characterized in that The method also uses a recurrent neural network to perform the tracking.

6. The method according to claim 4, characterized in that The method trains the neural network with artificial data, wherein the artificial data is generated by a means selected from the group consisting of: GAN, 2D model, 3D model.

7. The method according to claim 3, characterized in that The method uses median image subtraction to perform the tracking.

8. The method according to claim 1, characterized in that The fitting function is the following system of equations R(t)=R0+R1*sin(ω1*t+θ1)+R2*Sin(ω2*t+θ2). ; θ(t)=θ(t)+ω0*t 9. The method according to claim 1, characterized in that: The fitting function is R(t)=R0+V*t+ 1 / 2a2*t 10. The method according to claim 4, characterized in that The athletic ability function is defined by: M=A / R0+B|R 1,阈值 -R1|+C|ω 1,阈值 -ω1|+D|R 2,阈值 -R2|++E|ω 2,阈值 -ω2| 11. The method according to claim 5, characterized in that The athletic ability function is defined by: M=A / R0+B|V 阈值 -V|+C|a 阈值 -a| 12. The method according to claim 1, characterized in that The fitting step is done using a minimization method.

13. The method according to claim 1, characterized in that The fitting step is done using a Kalman filter.

14. The method according to claim 1, characterized in that The method further eliminates the effects of sample stage motion by means selected from the group consisting of: determining the sample stage motion using a position encoder; The sample stage motion is determined using an average particle velocity; the sample stage motion is determined using a known fixed pattern on the sample stage.

15. The method according to claim 1, characterized in that The method also estimates the velocity of the sperm in water by measuring the velocity of the sperm in a solution of polyvinyl pyrrolidone using the following relationship: V_who=k(c)*V_pvp Where V_who is the velocity of the sperm in water, V_pvp is the velocity of the sperm in the PVP solution, and k(c) is a constant that depends on the concentration of the polyvinylpyrrolidone solution.

16. A method for analyzing sperm, characterized in that The method consists of the following steps: a. Obtain training data consisting of a set of labeled images or image sequences; b. training a neural network using said labels by means of back propagation; c. Use the trained neural network to predict labels for incoming images.

17. The method according to claim 16, characterized in that The neural network comprises a convolutional neural network fed into a recurrent neural network, and wherein the labeled image sequence comprises future instance labels and positions of sperm head centroids.

18. The method according to claim 16, characterized in that The step of obtaining training data comprises: a. obtaining a video sequence including sperm; b. perform an optional step of background removal; c. a step of performing motion detection to generate a bounding box around the moving sperm; Training data for the neural network are thereby automatically generated from the video sequence.

19. The method for sperm cell identification according to claim 16, characterized in that: The method also enables hybrid identification of potential sperm cells in a given image, combining neural network reasoning with optional human reasoning and operator visual selection of such cells, in which case the system enriches its training dataset to include the cells selected by the operator.

20. The method according to claim 19, characterized in that The neural network is retrained after manual reasoning.

21. The method according to claim 19, characterized in that The retraining occurs on an individual operator basis, on a work system basis, on a laboratory basis (ie, all systems belonging to the same laboratory), or on a regional or global basis.

22. The method according to claim 18, characterized in that The artificial reasoning includes localization, recognition, classification and regression.

23. A method for generating sperm images or video sequences from upscaled latent vectors of a trained network.

24. The method according to claim 23, characterized in that The method implements a GAN to generate the image or video sequence.

25. The method according to claim 16, characterized in that The step of obtaining the training data is completed by using GAN to generate the training data.

26. A method for analyzing a sperm image sequence using unsupervised learning and a set of training data image sequences, characterized in that The method comprises the following steps: a. training an autoencoder having a bottleneck layer on the training data image sequence, wherein the output of the bottleneck layer can be used as a potential representation of the image sequence; b. identifying clusters of said latent representations on said training data image sequence in terms of a discrete number of population clusters; c. Identifying the image sequence to be analyzed in terms of its membership in one or more of the clusters.

27. A method for sperm analysis, characterized in that The sperm analysis may be used in the case of immobile sperm cells, such as those encountered in azoospermia, the method comprising the steps of: a. In the initial stage: i. Obtaining a video sequence of sperm motility; ii. labeling moving objects in the video sequence using a conventional motion detection algorithm; iii. training a convolutional neural network using the labeled objects as training data; b. In the inference phase, the positions of both stationary sperm cells and moving sperm cells are inferred using the convolutional neural network; Widely available training data of motile sperm cells are utilized to train a network suitable for detecting sparse, immobile sperm cells that may be encountered in the case of azoospermia samples.

28. The method according to claim 27, characterized in that The step of obtaining a video sequence of motile sperm uses normal, non-azoospermic ejaculate.

29. The method according to claim 27, characterized in that Superpixel resolution is also obtained by averaging multiple frames of the video sequence and using the averaged frames as input to the convolutional neural network in both the training step and the inference step.

Images