Sperm motility detection method, device, computer device, storage medium

Abstract

The present application relates to a sperm motility detection method, device, computer device, storage medium, and computer program product. The method includes: obtaining a plurality of consecutive frames of sperm images, performing target detection on each frame of sperm image, and determining the position information of the same sperm in multiple frames of sperm images according to the obtained target detection results; generating a sperm feature matrix corresponding to the sperm according to the position information of the sperm in multiple frames of sperm images, performing feature extraction on the sperm feature matrix to obtain a first sperm feature; classifying and identifying the first sperm feature to obtain the motility category of the sperm; and generating a detection result of the sperm motility according to the motility category of the sperm. By using this method, the workflow of sperm motility detection can be greatly simplified, and the efficiency of sperm motility detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular, to a sperm motility detection method, device, computer device, storage medium, and computer program product. Background Art

[0002] Sperm motility is an essential indicator for evaluating sperm quality. Sperm motility is generally divided into four levels, including rapid forward movement, slow forward movement, non-forward movement, and immotility. If the proportion of motile sperm (including rapidly forward-moving sperm, slowly forward-moving sperm, and non-forward-moving sperm) is less than a certain value, or the proportion of forward-moving sperm (including rapidly forward-moving sperm and slowly forward-moving sperm) is less than a certain value, it is determined that the sperm lacks motility.

[0003] In traditional techniques, sperm motility detection can be achieved based on methods such as computer vision technology, machine learning theory, and deep learning. Taking the deep learning method as an example, multiple sperm images can be detected through a deep learning model to obtain the position coordinates of sperm in each sperm image. Then, logical operations are performed on the position coordinates of sperm in multiple sperm images to obtain the sperm movement trajectory, and further, the sperm motility detection result is determined based on the sperm movement trajectory. However, with the continuous development of deep learning technology and computer devices, there is an urgent need for a more efficient sperm motility detection method. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a sperm motility detection method, device, computer device, computer-readable storage medium, and computer program product that can more efficiently and accurately identify sperm motility.

[0005] In a first aspect, the present application provides a sperm motility detection method. The method includes:

[0006] Obtain a series of consecutive sperm images, perform object detection on each sperm image, and determine the position information of the same sperm in multiple sperm images according to the obtained object detection results;

[0007] Generate a sperm feature matrix corresponding to the sperm according to the position information of the sperm in multiple sperm images, perform feature extraction on the sperm feature matrix, and obtain a first sperm feature;

[0008] Perform classification and recognition on the first sperm feature to obtain the motility category of the sperm;

[0009] Generate a detection result of sperm motility according to the motility category of the sperm.

[0010] In one embodiment, generating a sperm feature matrix corresponding to the sperm according to the position information of the sperm in multiple frames of the sperm images includes:

[0011] Arranging the position information of the sperm in multiple frames of the sperm images according to the sorting of the multiple frames of the sperm images to generate a position matrix;

[0012] Performing a matrix transformation on the position matrix, and generating the sperm feature matrix of the sperm according to the position matrix and the position matrix after the matrix transformation.

[0013] In one embodiment, the performing a matrix transformation on the position matrix, and generating the sperm feature matrix of the sperm according to the position matrix and the position matrix after the matrix transformation includes:

[0014] Performing a flipping process on the position matrix, and splicing the position matrix and the position matrix after the flipping process in the row direction of the matrix to obtain an intermediate feature matrix;

[0015] Performing an arithmetic process on each position information in the (i + 1)-th row and each position information in the i-th row and at the same column in the intermediate feature matrix, and using the position matrix after the arithmetic process as the sperm feature matrix, where i is a positive integer.

[0016] In one embodiment, the extracting features from the sperm feature matrix to obtain a first sperm feature includes:

[0017] Inputting the sperm feature matrix into a frame convolution model, where the frame convolution model includes a plurality of two-dimensional convolutional kernels and a pooling layer, and the first dimension of the plurality of two-dimensional convolutional kernels represents the number of frames of the multiple frames of the sperm images, and the second dimension represents the position information dimension corresponding to the sperm images;

[0018] Performing a convolution process on the sperm feature matrix through each two-dimensional convolutional kernel to obtain a convolution result corresponding to each two-dimensional convolutional kernel;

[0019] Performing a process on the convolution result corresponding to each two-dimensional convolutional kernel through the pooling layer, and splicing the multiple convolution results after the pooling process to obtain the first sperm feature.

[0020] In one embodiment, the method further includes:

[0021] Performing a feature engineering process on the position information of the sperm in multiple frames of the sperm images to generate a second sperm feature;

[0022] The classifying and recognizing the first sperm feature to obtain the motility category of the sperm includes:

[0023] Classify and identify the first sperm feature and the second sperm feature to obtain the motility category of the sperm.

[0024] In one embodiment, the feature engineering process on the position information of the sperm in multiple frames of the sperm images to generate the second sperm feature includes:

[0025] Process the position information of the sperm in multiple frames of the sperm images through at least one of the following processing methods to generate the second sperm feature:

[0026] Obtain the difference between the position information of the sperm in the first frame of sperm image and the position information in the last frame of sperm image;

[0027] Obtain the standard deviation of the position information of the sperm in multiple frames of the sperm images;

[0028] Obtain the difference between the position information of the sperm in every two adjacent frames of sperm images, and obtain the mean value of the obtained differences;

[0029] Obtain the angular difference between the position information of the sperm in every two adjacent frames of sperm images, and obtain the mean value of the angular differences;

[0030] Obtain the first distance between the position information of the sperm in the first frame of sperm image and the position information in the last frame of sperm image, and obtain the second distance between the position information of the sperm in every two adjacent frames of sperm images, and obtain the ratio between the first distance and the second distance;

[0031] Obtain the ratio between the first number of frames in which the moving distance of the sperm is greater than a preset distance and the number of frames of multiple frames of the sperm images;

[0032] Obtain the ratio between the moving distance of the sperm in each frame of the sperm images and the number of frames of multiple frames of the sperm images.

[0033] In one embodiment, the classification and identification of the first sperm feature and the second sperm feature to obtain the motility category of the sperm includes:

[0034] Input the first sperm feature and the second sperm feature into a motility classification model, and the motility classification model includes a splicing layer and a fully connected layer;

[0035] Splice the first sperm feature and the second sperm feature through the splicing layer to obtain the target sperm feature of the sperm;

[0036] Process the target sperm feature through the fully connected layer and output the motility category of the sperm.

[0037] In a second aspect, the present application also provides a sperm motility detection device. The device includes:

[0038] A position detection module, configured to obtain a plurality of consecutive frames of sperm images, perform target detection on each frame of the sperm images, and determine the position information of the same sperm in the plurality of frames of sperm images according to the obtained target detection results;

[0039] A feature generation module, configured to generate a sperm feature matrix corresponding to the sperm according to the position information of the sperm in the plurality of frames of sperm images, perform feature extraction on the sperm feature matrix, and obtain a first sperm feature;

[0040] A classification module, configured to perform classification and recognition on the first sperm feature to obtain the motility category of the sperm;

[0041] A motility result generation module, configured to generate a detection result of sperm motility according to the motility category of the sperm.

[0042] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the sperm motility detection method according to any one of the embodiments in the first aspect above.

[0043] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the sperm motility detection method according to any one of the embodiments in the first aspect above.

[0044] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the sperm motility detection method according to any one of the embodiments in the first aspect above.

[0045] The above sperm motility detection method, device, computer device, storage medium, and computer program product perform object detection on multiple acquired sperm images, and determine the position information of the same sperm in multiple frames of sperm images according to the obtained object detection results. Then, according to the position information of the sperm in multiple frames of sperm images, a sperm feature matrix corresponding to the sperm is generated, feature extraction is performed on the sperm feature matrix, and the first sperm feature is obtained. Finally, classification and recognition are performed on the first sperm feature to obtain the motility category of the sperm. According to the motility category of the sperm, the detection result of the sperm motility is generated. By using the position information of each sperm in multiple sperm images to describe the characteristics of each sperm and based on a deep learning model to identify the sperm characteristics, the motility category of the sperm can be directly obtained, thus greatly simplifying the workflow of sperm motility detection and improving the detection efficiency of sperm motility. In addition, using a deep learning model with sufficient detection ability to detect sperm images can also improve the accuracy of sperm motility detection. Description of the Drawings

[0046] Figure 1 It is a schematic flowchart of the sperm motility detection method in an embodiment;

[0047] Figure 2 It is a schematic flowchart of the feature extraction step for the sperm feature matrix in an embodiment;

[0048] Figure 3 It is a schematic structural diagram of a frame convolution model in an embodiment;

[0049] Figure 4 It is a schematic flowchart of the sperm motility detection method in another embodiment;

[0050] Figure 5 It is a schematic structural diagram of obtaining the motility category of the sperm in an embodiment;

[0051] Figure 6 It is a structural block diagram of the sperm motility detection device in an embodiment;

[0052] Figure 7 It is an internal structural diagram of a computer device in an embodiment. Detailed Embodiments

[0053] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] The sperm motility detection method provided by the embodiments of this application can be applied to computer devices such as terminals and servers, or in a system including a terminal and a server. The following content will be described by taking the application to a terminal as an example. At least one pre-trained deep learning model is pre-deployed in the terminal. The deep learning model can be pre-trained using this terminal or other computer devices outside this terminal. Specifically, the terminal acquires multiple consecutive frames of sperm images, performs object detection on each frame of sperm image, and determines the position information of the same sperm in multiple frames of sperm images according to the obtained object detection results. According to the position information of the sperm in multiple frames of sperm images, a sperm feature matrix corresponding to the sperm is generated, feature extraction is performed on the sperm feature matrix, and the first sperm feature is obtained. Classification and recognition are performed on the first sperm feature to obtain the motility category of the sperm. According to the motility category of the sperm, the detection result of the sperm motility is generated.

[0055] Among them, the terminal can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0056] In one embodiment, as Figure 1 shown, a sperm motility detection method is provided. Taking the application of this method to a terminal as an example, the method includes the following steps:

[0057] Step S110, acquire multiple consecutive frames of sperm images, perform object detection on each frame of sperm image, and determine the position information of the same sperm in multiple frames of sperm images according to the obtained object detection results.

[0058] Among them, the sperm image can be an image extracted from a sperm video. The sperm video can refer to a video obtained by collecting a fresh semen sample. The fresh semen sample can be an unstained or stained sample. For example, a fresh semen drop is dripped onto a glass slide, and the glass slide is collected using an image acquisition device (such as an optical microscope). The sperm video can be a real-time collected video, and the terminal can obtain the collected video from the image acquisition device in real time. The sperm video can also be a video pre-collected and stored in a local database or a server database, and then the terminal can obtain the sperm video from the local database or the server database. After the terminal acquires the sperm video, multiple consecutive frames of sperm images are extracted from the sperm video.

[0059] In some embodiments, the multi-frame sperm images can also be images obtained after a series of preprocessing on the images extracted from a sperm video. The preprocessing can be, but is not limited to, size processing, image enhancement processing, etc.

[0060] In other embodiments, the number of frames of the multi-frame sperm images depends on actual needs. For example, it can be 10 consecutive frames extracted from any time period.

[0061] Specifically, the terminal detects each frame of sperm image through a trained object detection model. The object detection model is a model with at least object detection capabilities, which can be achieved through an end-to-end model or through a combination of multiple independent models. The terminal inputs each frame of sperm image into the object detection model. When it is determined through the object detection model that there are sperm in each frame of sperm image, the position information of a single sperm in each frame of sperm image is obtained. There may be multiple sperm in each frame of sperm image. In this case, the terminal obtains the position information of each sperm.

[0062] When there is one sperm in each frame of sperm image, the sperm in the multi-frame sperm images can be regarded as the same sperm. When there are multiple sperm in at least one frame of sperm image, starting from the first frame of sperm image, according to the position information of each sperm in the current frame of sperm image and the next frame of the current frame, the sperm that is the same as each sperm in the current frame is determined from the multiple sperm in the next frame. Until all the multi-frame sperm images are processed, the same sperm in the multi-frame sperm images is determined, and then the position information of the same sperm in the multi-frame sperm images is obtained.

[0063] In one embodiment, the same sperm in each frame of sperm image can be represented by the same unique sperm identifier for subsequent use.

[0064] In another embodiment, each frame of sperm image can be represented by a unique image identifier. After obtaining the position information of the sperm in each frame of sperm image, the terminal can establish a correspondence relationship among the image identifier, the sperm identifier, and the position information of the sperm for subsequent use.

[0065] Step S120: Generate a sperm feature matrix corresponding to the sperm according to the position information of the sperm in the multi-frame sperm images, and perform feature extraction on the sperm feature matrix to obtain the first sperm feature.

[0066] Specifically, the terminal arranges the position information of the sperm in the multi-frame sperm images according to a preset arrangement method to generate a sperm feature matrix corresponding to each sperm. The feature extraction model performs feature extraction on the sperm feature matrix to obtain the first sperm feature of each sperm. Among them, the feature extraction model can be any one of a convolutional neural network, a recurrent neural network, etc.

[0067] In one embodiment, the terminal may use the position information of sperm in each frame of sperm images as matrix rows to generate sperm feature matrices corresponding to each sperm. In another embodiment, the terminal may use the position information of sperm in each frame of sperm images as matrix columns to generate sperm feature matrices corresponding to each sperm.

[0068] Step S130: Classify and identify the first sperm feature to obtain the motility category of the sperm.

[0069] Among them, the motility category may be, but is not limited to, one of rapid forward movement, slow forward movement, non-forward movement, and immotility.

[0070] Specifically, the terminal classifies and identifies the first sperm feature of each sperm through a trained motility classification model. The motility classification model is a model with at least the ability to identify motility categories, which can be achieved through an end-to-end model or a combination of multiple independent models. The terminal inputs the first sperm feature of each sperm into the motility classification model to obtain the motility category of each sperm.

[0071] Step S140: Generate a detection result of sperm motility according to the motility category of the sperm.

[0072] Specifically, usually, if the proportion of motile sperm (including sperm with categories of rapid forward movement, slow forward movement, and non-forward movement) among all sperm is less than a first threshold (such as 40%), or the proportion of forward-moving sperm (including sperm with categories of rapid forward movement and slow forward movement) among all sperm is less than a second threshold (such as 32%), then the sperm motility is abnormal (it is asthenospermia). Otherwise, the sperm motility is normal. The terminal can obtain the actual proportion of motile sperm or forward-motile sperm among all sperm according to the motility category of each sperm, and generate a detection result of normal or abnormal sperm motility according to the actual proportion.

[0073] In the above sperm motility detection method, target detection is performed on multiple acquired sperm images. According to the obtained target detection results, the position information of the same sperm in multiple frames of sperm images is determined. Then, according to the position information of the sperm in multiple frames of sperm images, a sperm feature matrix corresponding to the sperm is generated, feature extraction is performed on the sperm feature matrix, and the first sperm feature is obtained. Finally, classification and recognition are performed on the first sperm feature to obtain the motility category of the sperm. According to the motility category of the sperm, the detection result of the sperm motility is generated. By using the position information of each sperm in multiple sperm images to describe the characteristics of each sperm and based on the deep learning model to identify the sperm features, the motility category of the sperm can be directly obtained, thus greatly simplifying the workflow of sperm motility detection and improving the detection efficiency of sperm motility. In addition, using a deep learning model with sufficient detection ability to detect sperm images can also improve the accuracy of sperm motility detection.

[0074] In one embodiment, in step S120, according to the position information of the sperm in multiple frames of sperm images, generating a sperm feature matrix corresponding to the sperm includes: arranging the position information of the sperm in multiple frames of sperm images according to the sorting of the multiple frames of sperm images to generate a position matrix; performing matrix transformation on the position matrix, and generating a sperm feature matrix of the sperm according to the position matrix and the position matrix after matrix transformation.

[0075] Among them, the way of matrix transformation can be any one of translation, flipping, scaling, etc., depending on actual needs.

[0076] Specifically, the terminal uses the position information of the sperm in each frame of sperm image as a matrix row. The i-th row is the position information of the sperm in the i-th frame of sperm image, and the values belonging to the same attribute in the position information are used as the same column to generate a position matrix. Where i = 1, 2... n, and n is the number of frames of multiple frames of sperm images. In one example, there are 5 frames of sperm images, and Table 1 shows the position information of a certain sperm in 5 frames of sperm images:

[0077] Table 1:

[0078]

[0079] The terminal performs transformation on the position matrix of each sperm according to the preset matrix transformation method to obtain the position matrix after transformation. Then, according to the position matrix and the position matrix after transformation, the first sperm feature corresponding to each sperm is obtained.

[0080] In one example, the position matrix and the position matrix after transformation can be spliced in the matrix row direction or the matrix column direction to obtain the sperm feature matrix. In another example, the position matrix and the position matrix after transformation can be fused (such as calculating the matrix product) to obtain the sperm feature matrix.

[0081] In this embodiment, by performing matrix transformation on the position matrix of each sperm and generating the sperm feature matrix of each sperm according to the position matrix before and after transformation, the position characteristics of the sperm can be described from different aspects, thereby improving the comprehensiveness of sperm characteristics.

[0082] In one embodiment, an implementation manner of generating a sperm feature matrix based on matrix transformation is described. Specifically, after the terminal obtains the position matrix of each sperm (assumed to be P (n×m) , where n represents the number of rows of the position matrix and the number of frames of multiple-frame sperm images, and m represents the number of columns of the position matrix), the position matrix is flipped according to a preset flipping method to obtain the flipped position matrix (which is P' (n×m) ), and the position matrix and the flipped position matrix are concatenated in the row direction of the matrix to obtain an intermediate feature matrix (assumed to be M (n×2m) , where n represents the number of rows of the intermediate feature matrix and 2m represents the number of columns of the intermediate feature matrix). Among them, the method of flipping the position matrix can be but is not limited to any one of horizontal flipping, vertical flipping, diagonal flipping, etc. The terminal performs arithmetic processing on each position information in the (i + 1)-th row and each position information in the i-th row and the same column in the intermediate feature matrix, and uses the position matrix after the arithmetic processing as the sperm feature matrix. Among them, the method of arithmetic processing can be but is not limited to any one of addition, subtraction, multiplication, division, etc.

[0083] In one embodiment, the flipping method is horizontal flipping. The arithmetic processing method for the intermediate feature matrix is subtraction.

[0084] Continuing with the position information shown in Table 1 as an example for illustration. Generate the position matrix P according to Table 1 (n×m) :

[0085]

[0086] If the flipping method is horizontal flipping, then the flipped P' can be obtained (n×m) :

[0087]

[0088] The position matrix P (n×m) and the flipped position matrix P' (n×m) are concatenated in the row direction of the matrix to obtain the intermediate feature matrix M (n×2m) :

[0089]

[0090] If the operation processing method is subtraction, then for the intermediate feature matrix, subtract the position information at each position in the (i + 1)-th row from the position information at the same column in the i-th row (take the absolute value) to obtain the sperm feature matrix G ((n-1)×2m) :

[0091]

[0092] In this embodiment, by sequentially performing flipping processing and operation processing on the position matrix of each sperm, a sperm feature matrix is obtained, enabling the sperm feature matrix to describe the position change information of sperm in multiple frames of sperm images, thereby improving the classification accuracy of sperm motility categories.

[0093] In one embodiment, as Figure 2 shown, in step S120, feature extraction is performed on the sperm feature matrix to obtain the first sperm feature, including:

[0094] Step S210, input the sperm feature matrix into the frame convolution model.

[0095] Among them, Figure 3 exemplarily shows a feature extraction model (frame convolution model), and the frame convolution model includes multiple two-dimensional convolution kernels( Figure 3 shows 3 two-dimensional convolution kernels), and a pooling layer. The first dimension of the multiple two-dimensional convolution kernels represents the number of frames of multiple frames of sperm images, and the second dimension represents the dimension of the position information corresponding to the sperm image (such as the X-axis coordinate, Y-axis coordinate). The first dimension sizes of different two-dimensional convolution kernels are different, and the number of channels is the same, so that convolution can be performed on different degrees of continuous frame numbers, and different results can be extracted from the movement changes of the frame numbers. The second dimensions of different two-dimensional convolution kernels are the same, so that all position information in a single frame can be completely covered during each convolution.

[0096] Step S220, perform convolution processing on the sperm feature matrix through each two-dimensional convolution kernel to obtain a convolution result corresponding to each two-dimensional convolution kernel.

[0097] Step S230, process the convolution result corresponding to each two-dimensional convolution kernel through the pooling layer, and splice the multiple convolution results after pooling processing to obtain the first sperm feature.

[0098] Specifically, the terminal inputs the sperm feature matrix into the frame convolution model, and performs convolution processing on the sperm feature matrix through each two-dimensional convolution kernel respectively to obtain the convolution results output by each two-dimensional convolution kernel. The convolution results output by each two-dimensional convolution kernel are processed through a rectified linear unit (ReLU) to eliminate a certain amount of redundant information. The convolution results after eliminating the redundant information are respectively input into the pooling layer, and the pooling layer performs pooling processing on the convolution results corresponding to each two-dimensional convolution kernel. The multiple convolution results after the pooling processing are concatenated and unfolded to obtain the first sperm feature.

[0099] In this embodiment, by using Figure 3 the shown frame convolution model to process the sperm feature matrix, it can ensure that the position information of sperm in each frame of sperm images can be covered, and can also capture the movement conditions between different consecutive image frames, which helps to distinguish the sperm jittering in place and the immotile sperm, and further improves the recognition accuracy of sperm motility categories.

[0100] In one embodiment, after determining the position information of the same sperm in multiple frames of sperm images, the method further includes: performing feature engineering processing on the position information of each sperm in multiple frames of sperm images to generate the second sperm feature.

[0101] Among them, feature engineering refers to obtaining better data features from the original data in a series of engineering ways to improve the model's ability. Feature engineering can include, but is not limited to, data preprocessing, feature selection, dimensionality reduction, feature construction, etc. In this embodiment, feature engineering refers to constructing the second sperm feature of each sperm according to at least one preset feature construction method and the position information of each sperm in multiple frames of sperm images.

[0102] Further, in this embodiment, after obtaining the second sperm feature, the terminal inputs the first sperm feature and the second sperm feature as input data into the motility classification model to obtain the motility category of the sperm.

[0103] In one embodiment, the motility classification model includes a concatenation layer and a fully connected layer. When there are the first sperm feature and the second sperm feature, the terminal inputs the first sperm feature and the second sperm feature into the motility classification model, concatenates the first sperm feature and the second sperm feature through the concatenation layer to obtain the target sperm feature of the sperm. The target sperm feature is processed through the fully connected layer to output the motility category of the sperm.

[0104] In this embodiment, by performing feature engineering processing on the position information of sperm in multiple frames of sperm images to construct more sperm features, the model can obtain more position feature information, which helps to improve the recognition ability of the model.

[0105] In one embodiment, the terminal can perform feature engineering on the position information of sperm in multiple frames of sperm images through at least one of the following processing methods to generate second sperm features. In one example, there are 5 frames of sperm images, and the position information of each frame of sperm image of a certain sperm is the center coordinates of the bounding box output by the target detection model, including two coordinate values. Then the position information of a certain sperm can be expressed as [(x 1 , y 1 ), (x 2 , y 2 ), (x 3 , y 3 ), (x 4 , y 4 ), (x 5 , y 5 )].

[0106] (1) Obtain the difference between the position information of the sperm in the first frame of sperm image and the position information in the last frame of sperm image: (x 5 - x 1 , y 5 - y 1 )

[0107] (2) Obtain the standard deviation of the position information of the sperm in multiple frames of sperm images:

[0108] (std(x 1 , x 2 , x 3 , x 4 , x 5 ), std(y 1 , y 2 , y 3 , y 4 , y 5 ))

[0109] (3) Obtain the difference between the position information of the sperm in each adjacent two frames of sperm images, and obtain the mean value of the obtained differences:

[0110] (mean(x 2 - x 1 , x 3 - x 2 , x 4 - x 3 , x 5 - x 4 ), mean(y 2 - y 1 , y 3 - y 2 , y 4 - y 3 , y5 -y 4 ))

[0111] (4) Obtain the angular difference between the position information of the sperm in each two adjacent frames of sperm images, and obtain the average value of the angular differences:

[0112]

[0113] (5) Obtain the first distance between the position information of the sperm in the first frame of sperm image and the position information in the last frame of sperm image, and obtain the second distance between the position information of the sperm in each two adjacent frames of sperm images, and obtain the ratio between the first distance and the second distance:

[0114]

[0115] (6) Obtain the ratio between the number of the first frames in which the moving distance of the sperm is greater than a preset distance and the number of frames of multiple frames of sperm images:

[0116]

[0117] Wherein, when x > 2, m(x) = 1; otherwise m(x) = 0.

[0118] (7) Obtain the ratio between the moving distance of the sperm in each frame of sperm image and the number of frames of multiple frames of sperm images:

[0119]

[0120] Wherein, the moving distance can be characterized by the Euclidean distance. In this case, the moving distance L2(m,n) between the sperm in the m-th frame and the n-th frame is:

[0121]

[0122] In one embodiment, as Figure 4 shown, a specific sperm motility detection method is provided. This method is applied to the flowchart as Figure 5 shown, and includes the following steps:

[0123] Step S402, obtain the sperm video to be detected, and extract a series of consecutive frames of original sperm images from the sperm video to be detected. Preprocess each frame of the original sperm image to obtain the corresponding sperm image.

[0124] Among them, the sperm video to be detected can be obtained by imaging and photographing a glass slide with semen dropped thereon under a 40x objective lens of an optical microscope equipped with a digital camera. The image size (pixel size) of the extracted original sperm image is 1024*1536. The preprocessing includes pixel value normalization processing and size standardization processing. That is, each pixel value of the obtained original sperm image is divided by 255 and normalized, and the size of each frame of the original sperm image after normalization is scaled to 640*640 to obtain the corresponding sperm image.

[0125] Step S404, perform object detection on each frame of sperm image through the object detection model. When a sperm head exists in the detected sperm image, obtain the center position coordinates of the rectangular frame where the sperm head is located as the position information of the sperm.

[0126] The following describes a training method of the object detection model:

[0127] The object detection model adopts the YOLO model of version 5. First, obtain a number of sperm head image samples and the data labels corresponding to each sperm head image sample. The image size of the sperm head image sample is 1024*1536. From the left and right sides of the width direction of each sperm head image sample, images of size 1024*1024 are cropped respectively. Pixel value normalization processing is performed on the cropped images, and the processed images are scaled to a size of 640*640 to obtain training image samples. The same cropping processing and scaling processing are performed on the data labels to obtain training labels corresponding to each training image sample. The training image samples are input into the initial YOLO model. The predicted head results are output through the initial YOLO model. A regression loss function is used to calculate the loss value between the predicted head results and the training labels. The Adam optimizer is used to adjust the model parameters of the initial YOLO model. Repeat the above process until the loss value reaches the preset threshold or the number of iterations reaches the preset number to generate the finally used YOLO model.

[0128] Step S406, determine the position information of the same sperm in multiple frames of sperm images according to the position information of the sperm corresponding to each frame of sperm image.

[0129] Specifically, the terminal obtains the first position information of each sperm in the current frame of sperm image (i.e., the image currently being processed by the terminal), and the second position information of each sperm in the next frame of sperm image after the current frame. For each sperm in the current frame of sperm image, the terminal calculates the distance between the first position information of the sperm and the second position information of each sperm in the next frame of sperm image. Among them, the distance can be obtained through the following calculation formula:

[0130] D = Dist - 10 * IoU

[0131] Among them, D represents distance; Dist represents the distance between the center points of the head detection frames of each sperm in the current frame and the center points of the head detection frames of each sperm in the next frame; IoU represents the IoU (Intersection over Union) value between the head detection frames of each sperm in the current frame and the head detection frames of each sperm in the next frame.

[0132] The terminal obtains the distance with the smallest value from multiple distances. Determine the first position information and the second position information corresponding to the distance with the smallest value, and regard the sperm located at the first position information and the second position information as the same sperm, thereby determining the same sperm in multiple frames of sperm images, and obtaining the position information of the same sperm in multiple frames of sperm images.

[0133] Step S408: Arrange the position information of the sperm in multiple frames of sperm images according to the sorting of the multiple frames of sperm images to generate a position matrix, and perform a flipping process on the position matrix. According to the position matrix and the flipped position matrix, generate a sperm feature matrix of the sperm. The specific generation method of the sperm feature matrix can refer to the above embodiments and will not be specifically described here.

[0134] Step S410: Input the sperm feature matrix into the frame convolution model, and perform feature extraction on the sperm feature matrix through the frame convolution model to obtain the first sperm feature.

[0135] Step S412: Perform feature engineering on the position information of the same sperm in multiple frames of sperm images to generate a second feature matrix.

[0136] Step S414: Input the first sperm feature and the second sperm feature into the motility classification model to obtain the motility category of the sperm. Repeat steps S406 - S414 to obtain the motility category of each sperm in the semen sample.

[0137] The following describes a training method of the motility classification model:

[0138] First, use a pre-trained object detection model to identify several consecutive multi-frame sperm image samples, and obtain the position information of the same sperm in the multi-frame sperm image samples. Generate a first sperm feature sample and a second sperm feature sample according to the position information of the sperm in the multi-frame sperm image samples. Among them, the generation method of the first sperm feature sample can refer to the generation method of the above-mentioned first sperm feature, and the generation method of the second sperm feature sample can refer to the generation method of the above-mentioned second sperm feature, which will not be specifically described here. Input the first sperm feature sample and the second sperm feature sample into the initial motility classification model, and output the predicted motility category through the initial motility classification model. Use the cross-entropy loss cost function to calculate the loss value between the predicted motility category and the training label. Use the Adam optimizer to adjust the model parameters of the initial motility classification model. Repeat the above process until the loss value reaches the preset threshold or the number of iterations reaches the preset number, and generate the finally used motility classification model.

[0139] Step S416, generate the detection result of sperm motility according to the motility category of each sperm in the semen sample.

[0140] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0141] Based on the same inventive concept, an embodiment of the present application also provides a sperm motility detection device for implementing the sperm motility detection method involved above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the sperm motility detection device provided below can refer to the limitations on the sperm motility detection method in the above text, and will not be repeated here.

[0142] In one embodiment, as Figure 6 shown, a sperm motility detection device 600 is provided, including: a position detection module 602, a feature generation module 604, a classification module 606, and a motility result generation module 608, where:

[0143] A position detection module 602, configured to obtain multiple consecutive frames of sperm images, perform object detection on each frame of sperm image, and determine the position information of the same sperm in multiple frames of sperm images according to the obtained object detection results; a feature generation module 604, configured to generate a sperm feature matrix corresponding to the sperm according to the position information of the sperm in multiple frames of sperm images, perform feature extraction on the sperm feature matrix to obtain a first sperm feature; a classification module 606, configured to perform classification and recognition on the first sperm feature to obtain the motility category of the sperm; a motility result generation module 608, configured to generate a detection result of the sperm motility according to the motility category of the sperm.

[0144] In one embodiment, the feature generation module 604 includes: a matrix generation unit, configured to arrange the position information of the sperm in multiple frames of sperm images according to the sorting of the multiple frames of sperm images to generate a position matrix; perform matrix transformation on the position matrix, and generate a sperm feature matrix of the sperm according to the position matrix and the position matrix after matrix transformation.

[0145] In one embodiment, the feature generation module 604 includes: a matrix transformation unit, configured to perform a flipping process on the position matrix, splice the position matrix and the position matrix after the flipping process in the row direction of the matrix to obtain an intermediate feature matrix; an operation unit, configured to perform an operation process on each position information in the (i + 1)-th row and each position information in the i-th row and the same column in the intermediate feature matrix, and use the position matrix after the operation process as the sperm feature matrix, where i is a positive integer.

[0146] In one embodiment, the feature generation module 604 includes: a first input unit, configured to input the sperm feature matrix into a frame convolution model, where the frame convolution model includes a plurality of two-dimensional convolution kernels and a pooling layer, the first dimension of the plurality of two-dimensional convolution kernels represents the number of frames of multiple frames of sperm images, and the second dimension represents the position information dimension corresponding to the sperm image; a convolution unit, configured to perform convolution processing on the sperm feature matrix through each two-dimensional convolution kernel to obtain a convolution result corresponding to each two-dimensional convolution kernel; a pooling and splicing unit, configured to

[0147] process the convolution result corresponding to each two-dimensional convolution kernel through the pooling layer, and splice the multiple convolution results after the pooling process to obtain a first sperm feature.

[0148] In one embodiment, the apparatus 600 further includes: a feature engineering processing module, configured to perform feature engineering processing on the position information of the sperm in multiple frames of sperm images to generate a second sperm feature; in this embodiment, the classification module 606 is configured to perform classification and recognition on the first sperm feature and the second sperm feature to obtain the motility category of the sperm.

[0149] In one embodiment, a feature engineering processing module is configured to process the position information of sperm in multiple frames of sperm images through at least one of the following processing methods to generate second sperm features: obtaining the difference between the position information of sperm in the first frame of sperm image and the position information in the last frame of sperm image; obtaining the standard deviation of the position information of sperm in multiple frames of sperm images; obtaining the difference between the position information of sperm in each adjacent two frames of sperm images and obtaining the mean value of the obtained differences; obtaining the angular difference between the position information of sperm in each adjacent two frames of sperm images and obtaining the mean value of the angular differences; obtaining the first distance between the position information of sperm in the first frame of sperm image and the position information in the last frame of sperm image, and obtaining the second distance between the position information of sperm in each adjacent two frames of sperm images, and obtaining the ratio between the first distance and the second distance; obtaining the ratio between the number of first frames in which the moving distance of sperm is greater than a preset distance and the number of frames of multiple frames of sperm images; obtaining the ratio between the moving distance of sperm in each frame of sperm image and the number of frames of multiple frames of sperm images.

[0150] In one embodiment, the classification module 606 includes: a second input unit configured to input the first sperm feature and the second sperm feature into a motility classification model, the motility classification model including a splicing layer and a fully connected layer; a splicing unit configured to splice the first sperm feature and the second sperm feature through the splicing layer to obtain a target sperm feature of the sperm; a classification unit configured to process the target sperm feature through the fully connected layer and output the motility category of the sperm.

[0151] Each module in the above sperm motility detection device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0152] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as Figure 7As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a sperm motility detection method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad set on the computer device housing, or an external keyboard, touchpad, or mouse, etc.

[0153] Those skilled in the art can understand that Figure 7 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0154] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, it implements the sperm motility detection method described in any one of the above embodiments.

[0155] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, it implements the sperm motility detection method described in any one of the above embodiments.

[0156] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, it implements the sperm motility detection method described in any one of the above embodiments.

[0157] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0158] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0159] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0160] The above-described embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for detecting sperm motility, characterized in that, the method includes: Obtaining a series of consecutive sperm images, performing object detection on each sperm image, and determining the position information of the same sperm in multiple sperm images according to the obtained object detection results; Arranging the position information of the sperm in multiple sperm images according to the sorting of the multiple sperm images to generate a position matrix; performing a flipping process on the position matrix, splicing the position matrix and the flipped position matrix in the row direction of the matrix to obtain an intermediate feature matrix; performing an arithmetic process on the position information at the (i + 1)-th row and the position information at the i-th row in the same column in the intermediate feature matrix, and using the position matrix after the arithmetic process as the sperm feature matrix, where i is a positive integer; Performing feature extraction on the sperm feature matrix through a feature extraction model to obtain a first sperm feature; Performing classification and recognition on the first sperm feature through a trained motility classification model to obtain the motility category of the sperm; Generating a detection result of sperm motility according to the motility category of the sperm.

2. The method according to claim 1, characterized in that, the performing feature extraction on the sperm feature matrix to obtain a first sperm feature includes: Inputting the sperm feature matrix into a frame convolution model, where the frame convolution model includes multiple two-dimensional convolution kernels and a pooling layer, and the first dimension of the multiple two-dimensional convolution kernels represents the number of frames of the multiple sperm images, and the second dimension represents the dimension of the position information corresponding to the sperm images; Performing convolution processing on the sperm feature matrix through each two-dimensional convolution kernel to obtain a convolution result corresponding to each two-dimensional convolution kernel; Performing processing on the convolution result corresponding to each two-dimensional convolution kernel through the pooling layer, and splicing the multiple convolution results after pooling processing to obtain the first sperm feature.

3. The method according to claim 1 or 2, characterized in that, the method further includes: Performing feature engineering processing on the position information of the sperm in multiple sperm images to generate a second sperm feature; the performing classification and recognition on the first sperm feature to obtain the motility category of the sperm includes: Performing classification and recognition on the first sperm feature and the second sperm feature to obtain the motility category of the sperm.

4. The method according to claim 3, characterized in that, the performing feature engineering processing on the position information of the sperm in multiple sperm images to generate a second sperm feature includes: Performing processing on the position information of the sperm in multiple sperm images through at least one of the following processing methods to generate the second sperm feature: Obtaining the difference between the position information of the sperm in the first sperm image and the position information in the last sperm image; Obtaining the standard deviation of the position information of the sperm in multiple sperm images; Obtaining the difference between the position information of the sperm in each adjacent two sperm images, and obtaining the mean value of the obtained differences. Obtain the angular difference between the position information of the sperm in every two adjacent frames of sperm images, and obtain the average value of the angular differences; Obtain the first distance between the position information of the sperm in the first frame of sperm image and the position information in the last frame of sperm image, and obtain the second distance between the position information of the sperm in every two adjacent frames of sperm images, and obtain the ratio between the first distance and the second distance; Obtain the ratio between the number of the first frames in which the moving distance of the sperm is greater than a preset distance and the number of frames of multiple frames of the sperm images; Obtain the ratio between the moving distance of the sperm in each frame of the sperm images and the number of frames of multiple frames of the sperm images.

5. The method according to claim 4, wherein, the classifying and recognizing the first sperm feature and the second sperm feature to obtain the motility category of the sperm includes: inputting the first sperm feature and the second sperm feature into a motility classification model, the motility classification model including a splicing layer and a fully connected layer; splicing the first sperm feature and the second sperm feature through the splicing layer to obtain a target sperm feature of the sperm; processing the target sperm feature through the fully connected layer to output the motility category of the sperm.

6. A sperm motility detection device, wherein, the device includes: a position detection module, configured to obtain a plurality of consecutive frames of sperm images, perform target detection on each frame of the sperm images, and determine the position information of the same sperm in the plurality of frames of sperm images according to the obtained target detection results; a feature generation module, configured to arrange the position information of the sperm in the plurality of frames of sperm images according to the sorting of the plurality of frames of sperm images to generate a position matrix; perform a flipping process on the position matrix, splice the position matrix and the flipped position matrix in the row direction of the matrix to obtain an intermediate feature matrix; perform an arithmetic process on each position information in the (i + 1)-th row and each position information in the i-th row and the same column in the intermediate feature matrix, and use the position matrix after the arithmetic process as the sperm feature matrix, where i is a positive integer; perform feature extraction on the sperm feature matrix through a feature extraction model to obtain a first sperm feature; a classification module, configured to perform classification and recognition on the first sperm feature through a trained motility classification model to obtain the motility category of the sperm; a motility result generation module, configured to generate a detection result of the sperm motility according to the motility category of the sperm.

7. A computer device, including a memory and a processor, the memory storing a computer program, wherein, when the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product, including a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images