Automatic Semen Analysis Method with Occlusion Awareness Based on Multi-Frame Information Fusion

Abstract

The present invention provides an automatic semen analysis method based on multi-frame information fusion with occlusion awareness. The present invention first proposes the combination of an edge-sensitive U-Net model and an occlusion-aware tracker based on joint probabilistic data association to achieve accurate and stable sperm detection and tracking. Thanks to the pixel-level contour segmentation and the overlap inference based on multi-frame target contours of the present invention, the present invention can accurately distinguish two or more spermatozoa that are close but not overlapping, and provide accurate head positioning; can accurately predict and judge the occurrence, progress, and end of target overlap; and can effectively match spermatozoa before and after overlap to ensure the consistency of tracking. The present invention can be conveniently integrated into existing computer-aided sperm analysis systems to provide semen analysis data with better accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of semen analysis methods, and in particular to an automatic semen analysis method based on multi-frame information fusion with occlusion perception. Background Art

[0002] According to the World Health Organization, semen analysis is the most important method for evaluating male reproductive function. Sperm concentration and motility are the most commonly used evaluation indicators by embryologists in semen analysis. These two indicators have been proven to be highly correlated with the pregnancy rate. Traditionally, semen analysis needs to be performed manually by experienced embryologists. The introduction of computer-aided sperm analysis (CASA) has greatly reduced the workload of embryologists in the above examinations. Nevertheless, professional CASA equipment still needs to be correctly used after rigorous professional training and usually has a high equipment cost. Moreover, the accuracy of the results of the CASA system has been questioned by some clinical embryologists.

[0003] The evaluation of sperm concentration and sperm motility by the CASA system is based on the accurate detection and tracking of sperm. However, the accuracy of sperm detection and tracking by the CASA system is greatly affected by the sample. For example, for samples with a high sperm concentration, overlap and collision between sperm often occur. Traditional tracking algorithms really cannot handle this situation well, which usually leads to target loss and exchange, resulting in a large variance in the evaluation of sample concentration and motility, thus affecting the diagnostic judgment of embryologists.

[0004] Generally speaking, the existing technologies mainly have the following defects:

[0005] (1) Currently, the sperm detection methods are basically achieved by the way of selecting the target with a rectangular box. However, since the head of the sperm is not a regular rectangle, the rectangular box detection often leads to the deviation of the sperm head positioning. Moreover, the rectangular box detection causes the loss of sperm contour information and is not conducive to the subsequent utilization of this feature information. In addition, due to the use of non-maximum suppression (NMS) in the rectangular box detection, two or more sperms that are close but not overlapping are often selected in the same box, resulting in a high missed detection rate.

[0006] (2) Currently, the sperm tracking methods basically do not effectively handle or optimize target overlap. For a kind of trackers based on the state equation, after the target overlap occurs, especially when the number of overlapping frames is large, the model parameters are often contaminated. Even after the overlap ends, the contaminated model cannot match it with the original target, resulting in target loss. Therefore, it is also impossible to directly handle high sperm concentration samples with frequent overlap phenomena.

[0007] Therefore, in view of the above problems, it is necessary to propose a further solution. Summary of the Invention

[0008] The object of the present invention is to provide an automatic semen analysis method based on multi-frame information fusion with occlusion perception to overcome the deficiencies existing in the prior art.

[0009] To achieve the above object of the invention, the present invention provides an automatic semen analysis method based on multi-frame information fusion with occlusion perception, which comprises the following steps:

[0010] S1. Read the video data of the semen sample and preprocess the video data;

[0011] S2. Obtain the contours of sperm or sperm groups through a trained edge-sensitive U-net model;

[0012] S3. Continuously judge the overlapping situation and track and update the status of the target;

[0013] S4. Calculate the motility parameters and sperm concentration of sperm in the semen sample according to the final tracking result.

[0014] As an improvement of the automatic semen analysis method based on multi-frame information fusion with occlusion perception of the present invention, in the step S1, the preprocessing of the video data includes:

[0015] Automatically crop the central effective area of the video data according to the resolution information of the video data, and enlarge it to the required pixel value by means of downsampling and interpolation.

[0016] As an improvement of the automatic semen analysis method based on multi-frame information fusion with occlusion perception of the present invention, the U-Net model is trained by using a loss function that can strengthen the boundaries between closely adjacent objects.

[0017] As an improvement of the automatic semen analysis method based on multi-frame information fusion with occlusion perception of the present invention, the training method includes:

[0018] S21. Intercept the video data of the semen sample and crop it to obtain video screenshots with the required pixels;

[0019] S22. Mark the contours of the sperm heads along the edges of the sperm heads, and the categories are all 0 to construct a data set for model training;

[0020] S23. For the overlapping situation where single sperm can be distinguished, mark the contours of the heads belonging to different sperm respectively, and leave at least 2 pixels of spacing between the contours; for the overlapping situation where single sperm cannot be distinguished, no annotation is performed.

[0021] As an improvement of the automatic semen analysis method based on multi-frame information fusion with occlusion perception of the present invention, the step S3 includes:

[0022] S31. Identify newly emerged targets and targets that have swam out of the field of view, add the newly emerged targets as objects, and delete the targets that have swam out of the field of view;

[0023] S32. Predict the overlap possibility between each target in the next frame based on the existing target information;

[0024] S33. Estimate whether the contour is a multi-sperm overlap based on multi-frame contour information;

[0025] S34. Based on the overlap information, match the measurement values with the targets and update the status.

[0026] As an improvement to the automatic semen analysis method based on multi-frame information fusion with occlusion perception of the present invention, the step S31 includes:

[0027] When the measurement value coordinates do not match any existing targets, a new target is established with the measurement value as the starting point; when there is no measurement value within the target search threshold or the upper limit of the threshold, the target is deleted.

[0028] As an improvement to the automatic semen analysis method based on multi-frame information fusion with occlusion perception of the present invention, the step S32 includes:

[0029] Predict the overlap possibility between each target in the next frame by calculating the integral probability of the threshold overlap area of each target.

[0030] As an improvement to the automatic semen analysis method based on multi-frame information fusion with occlusion perception of the present invention, the step S33 includes:

[0031] According to the area with a relatively high overlap probability predicted in step S32, combined with the contours of the previous two frames of the targets predicted to overlap or the targets already marked as overlapping, calculate the difference in the area of a single contour under the logarithm, and judge the start and end of the multi-sperm overlap.

[0032] As an improvement to the automatic semen analysis method based on multi-frame information fusion with occlusion perception of the present invention, the step S34 includes:

[0033] For the area with a relatively high overlap probability predicted in step S32, when it is confirmed that the overlap has started, match a single measurement value to multiple corresponding targets, and store the update matrix of each target;

[0034] After it is confirmed that the overlap has ended, cancel the target overlap mark and take out the update matrix of each target;

[0035] If the overlap has not started or the target has not been marked, match a single measurement value to a single target;

[0036] Based on the new measurement values, calculate the probability of each feasible event combination, and use probability weighting to estimate the position of the target measurement value in the next frame and determine the search range.

[0037] As an improvement to the automatic semen analysis method with occlusion perception based on multi-frame information fusion of the present invention, in step S4, the sperm motility parameters include average straight-line velocity, average curvilinear velocity, average path velocity, linearity, wobble, straightness, amplitude of lateral head displacement, and average angular displacement; the concentration of sperm will be obtained by averaging the number of tracked targets in each frame.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention first proposes the combination of an edge-sensitive U-Net model and an occlusion perception tracker based on joint probabilistic data association to achieve accurate and stable sperm detection and tracking. Thanks to the pixel-level contour segmentation and the overlap inference based on multi-frame target contours of the present invention, the present invention can accurately distinguish two or more sperm that are close but not overlapping, and provide accurate head positioning; it can accurately predict and judge the occurrence, progress, and end of target overlap; it can effectively match sperm before and after overlap to ensure the consistency of tracking. The present invention can be conveniently integrated into existing computer-aided sperm analysis systems to provide semen analysis data with better accuracy and robustness. Description of the Drawings

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments recorded in the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0040] Figure 1 It is a flowchart of an automatic semen analysis method with occlusion perception based on multi-frame information fusion provided by an embodiment of the present invention.

[0041] Figure 2 It is a sperm detection-tracking flowchart of an automatic semen analysis method with occlusion perception based on multi-frame information fusion provided by an embodiment of the present invention.

[0042] Figure 3 It is a preprocessed sperm sample image provided by an embodiment of the present invention.

[0043] Figure 4 It is a result comparison schematic diagram of whether to use an edge-sensitive loss function to train the sperm target image segmented by U-Net provided by an embodiment of the present invention.

[0044] Figure 5 It is a schematic diagram of a method for predicting the overlap possibility between each target in the next frame based on existing target information provided by an embodiment of the present invention.

[0045] Figure 6 Schematic diagram of a method for estimating whether a contour is a multiple sperm overlap based on multi-frame contour information provided by an embodiment of the present invention.

[0046] Figure 7 Schematic diagram of the comprehensive tracking effect provided by an embodiment of the present invention. Specific embodiments

[0047] The present invention will be described in detail below in conjunction with each embodiment. However, it should be noted that these embodiments are not intended to limit the present invention. Any equivalent transformation or substitution in terms of function, method, or structure made by those of ordinary skill in the art based on these embodiments shall fall within the protection scope of the present invention.

[0048] As Figure 1 shown, an embodiment of the present invention provides an automatic semen analysis method based on multi-frame information fusion with occlusion awareness.

[0049] As Figure 2 shown, the automatic semen analysis method of this embodiment specifically includes the following steps:

[0050] S1. Read the video data of the semen sample and preprocess the video data.

[0051] In the specific implementation process, the preprocessing of the video data of the semen sample includes:

[0052] According to the video resolution information, automatically crop the central effective area of the video and downsample / upsample and magnify it to 1024 pixels × 1024 pixels, as shown in Figure 3 the result after preprocessing.

[0053] S2. Obtain the contours of sperm or sperm groups through a trained edge-sensitive U-net model.

[0054] In step S2, U-Net is an image semantic segmentation model for pixel-level classification, which is widely used in the semantic segmentation and extraction of medical images. U-Net adopts a structure called a symmetric encoder-decoder. In the encoder part, a downsampling module is composed of two 3×3 convolutional layers (ReLU) plus a 2×2 max pooling layer. In this embodiment, the encoder is 4 identical downsampling modules used in an overlay manner. In the decoder part, it is repeatedly composed of an upsampling convolutional layer, the feature splicing provided by the corresponding downsampling module, and two 3×3 convolutional layers (ReLU). Using U-Net to obtain the contours of sperm or sperm groups is beneficial to retaining as much effective information as possible and improving the sperm detection quality under poor imaging conditions.

[0055] As Figure 4As shown in FIG. 1 , it is a schematic diagram showing the comparison of the results of the sperm target image segmented by using and not using the edge-sensitive loss function to train the U-Net. The edge-sensitive U-Net model is trained using a loss function that can strengthen the boundaries between closely adjacent objects. The loss function is based on a position-weighted cross entropy loss function, and pixels close to the boundary points in the image have higher weights.

[0056] The above loss function is specifically implemented as:

[0057] Formula (1).

[0058] in is the classic cross entropy loss function, is the weight of the pixel, the purpose is to give higher weight to the pixels close to the boundary points in the image, which is specifically implemented as follows:

[0059] Formula (2).

[0060] in is the weight constant, is the distance from a pixel point x to the nearest contour. It is the distance from the pixel point x to the second closest contour. and is the constant value obtained by statistical fitting.

[0061] In step S2, the training method used includes:

[0062] S21, intercepting the semen sample video data, and cutting it to obtain a video screenshot of required pixels;

[0063] S22, mark the outline of the sperm head along the edge of the sperm head, with all categories set to 0, to construct a data set for model training;

[0064] S23. For the overlap of single sperm that can be distinguished, the head contours of different sperm are marked respectively, and a spacing of at least 2 pixels is left between each contour; for the overlap of single sperm that cannot be distinguished, no marking is performed.

[0065] S3. Continuously judge the overlapping situation, track the target and update the status.

[0066] In step S3, continuous judgment of the overlapping situation is performed based on joint probability data association. Among them, joint probability data association (JPDA) is a data association method that can achieve relatively accurate target tracking in the case of a large target density. It uses the measurements within a certain range around the target tracking trajectory to jointly update the trajectory to avoid large problems in trajectory tracking caused by clutter or measurement errors. Its basic assumption is that a measurement can represent at most one target, and a target can generate at most one measurement. It sets up a validation matrix that shows all possible sources of each measurement and calculates all feasible joint association events based on the above assumptions. The measurements generated by the target are assumed to follow a Gaussian probability distribution, the false measurements are assumed to follow a uniform random distribution, and the number of false measurements follows a Poisson prior.

[0067] The specific steps of S3 include:

[0068] S31. Identify newly emerging targets and targets that swim out of the field of view, add the newly emerging targets as objects, and delete the targets that swim out of the field of view.

[0069] The specific steps of step S31 include:

[0070] When the measurement coordinate does not match any existing target, a new target is established with this measurement as the starting point. When there is no measurement within the target search gate, the search gate is continuously expanded. When no measurement falls within the gate even after expanding to the upper limit of the gate threshold, the target is deleted.

[0071] The calculation and update method of the target search gate is:

[0072] Equation (3).

[0073] Among them, PN is the process noise of the filter, MN is the measurement noise of the filter, is the first eigenvalue of the A matrix, and GT is the upper threshold of the artificially set gate.

[0074] S32. Predict the overlapping possibility between each target in the next frame based on the existing target information.

[0075] As Figure 5 shown, the specific steps of step S32 include: predicting the overlapping possibility between each target in the next frame by calculating the integral probability of the overlapping area of each target gate. Its specific implementation is:

[0076] Equation (4).

[0077] Among them, is the target probability of overlap occurring, and are respectively the target and credibility of the predicted position, and are respectively probability density of the threshold, is the target and region of overlap with the threshold.

[0078] S33. Estimate whether the contour is polyspermy overlap based on multi-frame contour information.

[0079] As Figure 6 shown, the step S33 specifically includes: For the region with a relatively high predicted overlap probability, combine the contours of the previous two frames of the target that is predicted to overlap or has been marked as overlapping, calculate the difference in the area of a single contour under the logarithm, and judge the start and end of polyspermy overlap. Its specific implementation is:

[0080] , Equation (5).

[0081] where w is a constant coefficient set manually.

[0082] S34. Based on the overlap information, match the measurement value with the target and update the state.

[0083] The step S34 specifically includes:

[0084] For the target marked as having a high probability of overlap, confirm whether the overlap starts or ends through local multi-frame mask information. After confirming that the overlap has started, the tracker matches a single measurement value to multiple corresponding targets and stores the update matrix of each target. After confirming that the overlap has ended, cancel the target overlap mark and take out the update matrix of each target. If the overlap has not started or the target is not marked, the tracker matches a single measurement value to a single target. The tracker calculates the probability of each feasible event combination based on the new measurement value and uses probability weighting to estimate the position of the target measurement value in the next frame and determine the search range.

[0085] S4. Calculate the motility parameters and sperm concentration of sperm in the semen sample based on the final tracking results.

[0086] As Figure 7 shown, in step S4, the sperm motility parameters include: average curvilinear velocity (VCL), average straight-line velocity (VSL), average path velocity (VAP), linearity (LIN), wobble (WOB), straightness (STR), amplitude of lateral head displacement (ALH), and average angular displacement (MAD).

[0087] The calculation method of the average curvilinear velocity (VCL) is: calculated by dividing the straight-line distance between the coordinate points of the first frame and the last frame of the target tracking result by the time difference between the two frames.

[0088] The calculation method of the average straight-line velocity (VSL) is: calculated by summing the distances traveled by the target from the first frame to the last frame of the target tracking result and dividing by the time difference between the first and last frames.

[0089] The calculation method of the average path velocity (VAP) is: determine the average movement direction of the sperm according to the target tracking result, and then statistically calculate the velocity of the sperm's movement accordingly.

[0090] Linearity (LIN) is the linear degree of the sperm's curvilinear path, and its calculation formula is: LIN = VSL / VCL.

[0091] Wobble (WOB) is the degree of oscillation of the sperm's actual path around the average path, and its calculation formula is: WOB = VAP / VCL.

[0092] Straightness (STR) is the degree of linearity of the average path, and its calculation formula is STR = VSL / VAP.

[0093] The amplitude of lateral head displacement (ALH) is the average distance between the sperm head and the average movement direction of the sperm during the tracking process. The calculation method is: take the positions of the sperm head in 5 equally spaced frames, subtract the average movement direction, and then average the obtained values.

[0094] Average angular displacement (MAD) is the curvature of the sperm's movement trajectory.

[0095] The sperm concentration is calculated as follows: First, calibrate the ratio of video pixels to the actual length. The sample cup for holding the semen sample has a determined depth. Therefore, after determining the average number of tracked targets in each frame, the sperm concentration of the semen sample can be obtained through coefficient conversion.

[0096] In summary, the present invention first proposes the combination of an edge-sensitive U-Net model and an occlusion-aware tracker based on joint probabilistic data association to achieve accurate and stable sperm detection and tracking. Benefiting from the pixel-level contour segmentation and the overlap inference based on multi-frame object contours of the present invention, the present invention can accurately distinguish two or more sperms that are close but not overlapping, and provide accurate head positioning; can accurately predict and judge the occurrence, progress, and end of the overlap of the target; and can effectively match the sperms before and after the overlap to ensure the consistency of tracking. The present invention can be conveniently integrated into the existing computer-aided sperm analysis system to provide semen analysis data with better accuracy and robustness.

[0097] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention.

[0098] In addition, it should be understood that although this specification is described according to embodiments, not every embodiment only contains an independent technical solution. This narrative way of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An automatic semen analysis method based on multi-frame information fusion with occlusion perception, characterized in that, the automatic semen analysis method comprises the following steps: S1. Read the video data of the semen sample and preprocess the video data; S2. Obtain the contours of sperm or sperm groups through a trained edge-sensitive U-net model; S3. Continuously judge the overlapping situation and track the target and update its status; The step S3 includes: S31. Identify newly emerged targets and targets that swim out of the field of view, add the newly emerged targets as objects, and delete the targets that swim out of the field of view; The step S31 includes: When the measured value coordinates do not match any existing targets, a new target is established with the measured value as the starting point; when there is no measured value within the target search threshold or the upper threshold, the target is deleted; S32. Predict the overlapping possibility between each target in the next frame based on the existing target information; The step S32 includes: Predict the overlapping possibility between each target in the next frame by calculating the integral probability of the threshold overlapping area of each target; S33. Estimate whether the contour is a multi-sperm overlap based on multi-frame contour information; The step S33 includes: According to the area with a relatively high overlapping probability predicted in step S32, combined with the contours of the previous two frames of the target with a predicted overlap or a marked overlap, calculate the difference in the area of a single contour logarithmically to judge the start and end of the multi-sperm overlap; S34. Based on the overlapping information, match the measured value with the target and update the status; The step S34 includes: For the area with a relatively high overlapping probability predicted in step S32, when it is confirmed that the overlap has started, match a single measured value to multiple corresponding targets and store the update matrix of each target; After it is confirmed that the overlap has ended, cancel the target overlap mark and take out the update matrix of each target; When the overlap has not started or the target has not been marked, match a single measured value to a single target; Based on the new measured value, calculate the probability of each feasible event combination, and use probability weighting to estimate the position of the target measured value in the next frame and determine the search range; S4. Calculate the motility parameters and sperm concentration of sperm in the semen sample according to the final tracking result.

2. The automatic semen analysis method based on multi-frame information fusion with occlusion perception according to claim 1, characterized in that, in the step S1, the preprocessing of the video data includes: Automatically crop the central effective area of the video data according to the resolution information of the video data, and enlarge it to the required pixel value by means of downsampling and interpolation.

3. The automatic semen analysis method based on multi-frame information fusion with occlusion perception according to claim 1, characterized in that, The U-Net model is trained using a loss function that can strengthen the boundaries between closely adjacent objects.

4. The automatic semen analysis method based on multi-frame information fusion with occlusion perception according to claim 3, characterized in that, The training method includes: S21. Intercept the video data of the semen sample and crop it to obtain a video screenshot with the required pixels; S22. Mark the contour of the sperm head along the edge of the sperm head, with all categories being 0, to construct a data set for model training; S23. For the overlapping situation where single sperm can be distinguished, mark the head contours belonging to different sperm respectively, and leave at least 2 pixels of spacing between each contour; for the overlapping situation where single sperm cannot be distinguished, no annotation is performed.

5. The automatic semen analysis method based on multi-frame information fusion with occlusion perception according to claim 1, characterized in that in the step S4, the sperm motility parameters include average straight-line velocity, average curvilinear velocity, average path velocity, linearity, wobble, straightness, amplitude of lateral head displacement, and average angular displacement; the concentration of the sperm will be obtained by averaging the number of tracking targets in each frame.

Images