Multi-sperm image segmentation detection system based on yov8 framework

Abstract

The invention belongs to the technical field of sperm image segmentation detection, and particularly relates to a multi-sperm image segmentation detection system based on a yov8 framework. The multi-sperm image segmentation detection system is based on a yolov8 network, an HAT network and a RevColV1 network are fused, a multi-sperm segmentation model yolov8-HAT and a single-sperm detection model yolov8-RevColV1 are designed, and the two models are cascaded to form the multi-sperm image segmentation detection system; the YOLOv8-HAT model is obtained by fusing a hybrid attention converter (HAT) on the basis of a YOLOv8 network framework, and is used for segmenting a multi-sperm image; the yolov8-RevColV1 is formed by fusing a reversible column network (RevColV1) on the basis of a yolov8 network framework, and is used for detecting a single sperm; according to the method, the label corresponding to each sperm can be displayed on the multi-sperm image, the problem that it is difficult for a doctor to effectively count and judge whether the sperms are normal when the sperms in the multi-sperm image are dense is solved, and the sperm image segmentation efficiency is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of sperm image segmentation detection, and in particular relates to a polysperm image segmentation detection system. Background Art

[0002] Currently, about 8% to 12% of couples worldwide are suffering from infertility. According to existing statistics, about 40% to 50% of infertility cases are related to male infertility. In order to achieve effective diagnosis and treatment of infertility, the assessment of male fertility is indispensable. In this process, rapid and accurate morphological analysis of the semen submitted for examination is particularly critical.

[0003] Sperm morphology analysis aims to determine the ratio of normal sperm to abnormal sperm within the pathological range by observing the morphological characteristics of sperm. When performing this operation, hospitals usually follow the following steps: First, collect the patient's semen sample, then dilute, stain, and prepare it into a slide specimen. Next, the doctor places the slide specimen under a microscope for observation. During the observation process, the doctor needs to constantly adjust the field of view of the microscope to count the number of normal and abnormal sperm. This process continues until all sperm are observed. Finally, the doctor calculates the ratio of normal to abnormal sperm based on the observation data and writes a detailed sperm test report.

[0004] However, the traditional sperm morphology analysis method has obvious defects. This method relies on manual observation. Doctors need to observe and count manually, which is inefficient and easily affected by subjective factors. Different doctors may make different judgments and conclusions on the same sperm sample. In addition, frequent adjustments to the microscope field of view may lead to repeated counting errors, making the judgment of the ratio of the two types of sperm only a fuzzy interval close to the true result, seriously affecting the authenticity and accuracy of medical testing.

[0005] Currently, most studies focus on the head morphology detection of single sperm images after staining, ignoring the statistical problems of polysperm image detection faced by doctors in actual operations. Therefore, these studies cannot provide practical help for clinical practice.

[0006] In view of this, the key problem that researchers urgently need to solve is how to develop a segmentation detection method for polyspermia images. This method should be able to resolve the differences between different doctors in traditional sperm image processing and assist doctors in quickly and accurately calculating the statistics and proportions of normal and abnormal sperm. This can not only improve the accuracy and efficiency of diagnosis, but also provide a more solid basis for the treatment of infertility. Summary of the invention

[0007] The purpose of the present invention is to provide a polyspermia image segmentation detection system that is independent of manual observation and has high accuracy, so as to solve the problem that sperm morphology detection has long been affected by personal subjective factors and the detection results are inaccurate.

[0008] In the field of computer vision, small target detection has always been a challenging problem. Since small targets account for a small proportion of pixels in an image and are easily disturbed by the surrounding background, traditional target detection algorithms often perform poorly when dealing with small targets. Polyspermia image detection is a type of small target detection; when dealing with polyspermia image detection tasks, it is particularly important to improve the accuracy of small target detection, which is the key to the present invention. YOLOv8 was open sourced by ultralytics on January 10, 2023. As an updated version of YOLOv5, it currently supports image classification, object detection, and instance segmentation tasks. The network consists of three main parts: a backbone network backbone based on CSP (compact and separate), a feature enhancement network neck, and a detection head head. The present invention is based on the yolov8 network architecture and improves it. Specifically, the HAT and RevColV1 networks are integrated to design two models: the polysperm segmentation model yolov8-HAT and the single sperm detection model yolov8-RevColV1. The two models are cascaded to form a polyspermia image segmentation detection system, as follows:

[0009] (I) The YOLOv8-HAT model is used for the segmentation of polysperm images; YOLOv8-HAT is based on the YOLOv8 network framework and integrates the Hybrid Attention Transformer (HAT). HAT is a deep learning network for image super-resolution reconstruction. It activates more input pixels by combining channel attention and window self-attention mechanisms, thereby improving the quality and accuracy of the reconstructed image. The input image is first passed through a convolutional layer for shallow feature extraction, and the extracted shallow features are further input into multiple Residual Hybrid Attention Groups (RHAGs) for deep feature extraction. Each RHAG contains multiple Hybrid Attention Blocks (HABs) and an Overlapping Cross-Attention Block (OCAB). HAB combines channel attention and window self-attention, while OCAB enhances the interaction between adjacent window features through an overlapping window division mechanism. The high-level features after deep feature extraction are converted into high-resolution images through the reconstruction module. The innovation of HAT lies in its hybrid attention mechanism, which combines channel attention and self-attention, as well as the introduction of overlapping cross-attention modules. These designs enable HAT to effectively activate more input pixels and enhance the interaction of cross-window information. Specifically, a HAT network is added to each scale of the feature pyramid network (FPN) in YOLOv8 to enhance the expression of multi-scale features. The image data is reconstructed through high resolution by the HAT network to generate a high-resolution feature map, which is combined with the original feature map to further enhance the detection accuracy of small targets. By inputting the training set and validation set consisting of preprocessed images and annotation information into the model for training optimization, an efficient polysperm segmentation model can be obtained.

[0010] (ii) The yolov8-RevColV1 model is used for the detection of single sperm; yolov8-RevColV1 is formed by integrating the reversible column network (RevColV1) on the basis of the yolov8 network framework. Specifically, on the basis of the yolov8 model, RevColV1 is used to replace the backbone network (backbone), and the data input to the model is extracted by the RevColV1 network and then passed to the feature enhancement network (neck) and the detection head (head). RevColV1 is a new type of neural network, which is composed of multiple sub-networks (columns) through multi-level reversible connections. This design allows information not to be lost during the forward propagation process. Information is feature decoupled in the forward process, keeping the total information without compression or discarding. Feature decoupling means that in each sub-network (column) of the RevColV1 network, features are transmitted through reversible connections, and are processed and learned independently at the same time. In this way, each column can maintain the integrity of the input information, and will not compress or discard information when passing between layers like traditional deep networks. As information progresses through the columns, the correlations between features gradually weaken (decouple), allowing the network to capture and emphasize important features in more detail, which helps improve the model's performance and generalization ability on complex tasks.

[0011] In the image input yolov8-RevColV1 model, the input image is first divided into non-overlapping patches through the RevColV1 backbone network and a patch embedding module. After that, these patches are fed into each subnetwork (i.e., column). In each subnetwork, four-level feature maps are extracted to propagate information between subnetworks. For classification tasks, only the last level feature maps of the last subnetwork are used because they contain rich semantic information. For other downstream tasks, such as object detection and semantic segmentation, all four levels of feature maps of the last subnetwork are used because they contain both low-level and semantic information. The feature decoupling mechanism of the RevColV1 network allows different levels of information to be gradually decoupled during the propagation process between columns. Some feature maps become more semantic, while some remain as low-level features. This design provides flexibility for downstream tasks that rely on high- and low-level features. At the same time, this architecture is particularly effective in cases where the data is rich and complex, and can be flexibly applied to different types of neural network models. It is very suitable for target detection tasks with large data sets, and the more data sets there are, the better the performance. The single sperm images saved after segmentation by the segmentation system are annotated with two types of label information, normal and abnormal, and input into the model as training sets and validation sets. After training and optimization, a single sperm detection model can be obtained.

[0012] (III) Polyspermia image segmentation and detection system (cascade system): In this advanced system, a cascade architecture is used to cleverly integrate multiple efficient and complementary models. Specifically, the system contains two key models: the yolov8-HAT model, which is used for polyspermia image segmentation tasks; and the yolov8-RevColV1 model, which is used for single sperm detection.

[0013] In the polysperm image segmentation stage, the polysperm image to be predicted is first input into the segmentation model yolov8-HAT, all sperm and segmentation coordinates are identified, and each sperm image is segmented according to the segmentation coordinates. After being processed by the segmentation model yolov8-HAT, the system can accurately obtain the total number of sperm in the image and the detailed annotation information of each sperm. These single sperm images that have undergone preliminary segmentation processing are then passed to the single sperm detection model yolov8-RevColV1 for more in-depth analysis and detection.

[0014] In the single sperm detection stage, the yolov8-RevColV1 model performs two-category recognition on each input single sperm image, identifies abnormal sperm and normal sperm in the image, and counts the number of these two types of sperm. At the same time, the model also provides corresponding predicted label information for each sperm, and passes the label and quantity information to the system. Finally, the system overlays the label information provided by the second-stage single sperm detection model on the label provided by the first-stage segmentation model, thereby directly marking the labels of normal and abnormal sperm on the initial polyspermia image.

[0015] In addition, after obtaining the information transmitted by the two models, the system can organize and calculate all the information through a statistical module to obtain the ratio of the two types of sperm, providing doctors with important auxiliary information to help them complete the sperm morphology detection report more accurately. Through this cascade method, doctors can diagnose and analyze more efficiently and accurately, thereby significantly improving the overall efficiency and accuracy of medical diagnosis. This system not only improves the accuracy of diagnosis, but also greatly reduces the workload of doctors, allowing them to devote more energy to other critical medical decisions. In short, the polysperm image segmentation detection system (cascade system) provides strong support for auxiliary medical diagnosis through its innovative architecture and efficient workflow.

[0016] The technical features and functional advantages of the present invention are as follows:

[0017] Traditional methods often require manual observation through a microscope, which is time-consuming and laborious, and is easily interfered by subjective factors, affecting the stability of the diagnosis. The present invention adopts deep learning technology to achieve rapid and accurate analysis of sperm morphology through high-precision polysperm image segmentation detection, which not only improves the analysis efficiency, but also greatly improves the accuracy of the analysis results, while significantly reducing the workload of doctors.

[0018] In addition, the system design of the present invention is simple and intuitive, and is easy for clinicians to learn and apply. Its promotion will help improve the application level of sperm morphology analysis in clinical diagnosis and provide strong support for the improvement of assisted reproductive technology. It has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a diagram to summarize the present invention.

[0020] Figure 2 This is a flowchart of the model training part of the present invention.

[0021] Figure 3 It is a diagrammatic representation of the structure of the yolov8-HAT model of the present invention.

[0022] Figure 4 It is a diagram of the HAT network structure of the present invention.

[0023] Figure 5 It is a diagrammatic representation of the structure of the yolov8-RevColV1 model of the present invention.

[0024] Figure 6 It is a diagram of the RevColV1 network structure of the present invention.

[0025] Figure 7 It is a flowchart diagram of the use of the system of the present invention.

[0026] Figure 8 It is a diagram of the user interface of the system of the present invention.

[0027] Fig. 9 It is a diagram showing the confidence adjustment options of the system of the present invention.

[0028] Fig.10 It is a diagrammatic representation of the polysperm image segmentation of the system of the present invention.

[0029] Fig.11 This is a diagram showing the normal sperm prediction of the system of the present invention.

[0030] Fig.12 The figure shows the ratio of two types of sperm predicted by the system of the present invention.

[0031] Fig.13 It is a flowchart of the training and use of the model of the present invention. DETAILED DESCRIPTION

[0032] In order to enable technicians and doctors in this field to have a deeper understanding of the technical solution of the present invention, the present invention will be further elaborated in detail below in combination with the accompanying drawings and specific work processes. In this way, relevant personnel can better grasp the core ideas and implementation details of the present invention, so as to achieve the best effect in practical applications. This not only helps to improve work efficiency, but also ensures the accuracy and reliability of technical implementation.

[0033] like Figure 1 As shown, the workflow of the present invention is mainly divided into two parts.

[0034] The first part focuses on the training of the segmentation model and the sperm detection model. This stage is crucial and involves the initialization and parameter optimization of the model so that sperm segmentation and sperm detection can be performed accurately.

[0035] Specifically, the segmentation model is first trained to identify and segment specific regions in the image, such as where sperm are located. Next, the sperm detection model is trained to further identify and classify sperm in these segmented regions.

[0036] The second part focuses on the practical application of the segmentation detection system. The model trained in the first part is used to process actual image data to detect and analyze sperm. The purpose is to verify the performance and accuracy of the model in practical applications and ensure its stable operation under various conditions.

[0037] It is worth noting that the training work in the first part is indispensable when there are no pre-trained gradient parameters available for the segmentation model and the detection model.

[0038] This is because a model without pre-trained parameters is equivalent to a blank sheet of paper, and it requires a large amount of data and training to learn how to perform effective sperm segmentation and sperm detection.

[0039] Therefore, the training work in the first part must be carried out before the actual application in the second part to ensure that the model has sufficient capabilities and accuracy to process actual image data.

[0040] This is the only way to ensure the reliability and effectiveness of the segmentation detection system in actual use.

[0041] like Figure 2 As shown in the figure, the model training process can be divided into two stages in detail: first, the training of the polysperm segmentation model, and second, the training of the single sperm detection model.

[0042] Before training begins, it is necessary to first photograph the sperm in the sperm slide under a microscope to obtain the original polyspermia images. These images will serve as the basic data for subsequent training.

[0043] Next, these images are preprocessed to ensure that the image quality meets the training requirements.

[0044] During the preprocessing process, the areas with too dense sperm were removed, and those images with low sperm overlap were selected for annotation. This is because too dense sperm images will increase the difficulty of annotation and affect the training effect of the model.

[0045] To label each sperm in the image, you can use the labelimg tool to label each sperm in the image. In this process, only one type of information needs to be labeled, that is, the sperm itself.

[0046] However, it should be noted that all sperm images need to be annotated without omission to avoid affecting the model training results. The annotation box should be close to the edge of the sperm and reduce the intersection between different annotation boxes in order to obtain better training results.

[0047] Through these steps, the quality of training data can be ensured and the training effect of the model can be improved. Through these two stages of training, a model that can accurately segment and detect single sperm can be obtained.

[0048] Then, the image data with labeled information and the images themselves are integrated into a complete dataset, which is divided into two parts: a training set and a validation set.

[0049] The data set is sent to the yolov8-HAT model for deep training, such as Figure 3 As shown in the figure, the image data first passes through the HAT network. The image is subjected to shallow and deep feature extraction and super-resolution reconstruction under the action of the HAT network, and then fused with the original image and enters the yolov8 network for further training.

[0050] In this process, the batch size and learning rate are continuously adjusted to achieve fine optimization of the model performance, and ultimately obtain the best gradient information to create a highly efficient polysperm segmentation model.

[0051] Then, the original polysperm image is input into the mature segmentation model to obtain the predicted annotation information of all sperm, the corresponding labels and the segmented image. This information will provide key data support for subsequent single sperm detection.

[0052] Use the labelimg tool to perform detailed annotations on the segmented single sperm images, where it is necessary to distinguish and mark the two types of normal sperm and abnormal sperm.

[0053] Then, the labeled image data and related information are used as a new data set and input into the yolov8-RevColV1 model for training, such as Figure 5 As shown. The graphic data enters the backbone network replaced by the RevColV1 network. After passing through the RevColV1 network structure, more feature information is extracted and the integrity of the information is maintained. Then it enters the neck and head of the yolov8 model for training, and a model that can accurately detect single sperm is trained. Through this model, accurate detection of segmented single sperm images can be achieved.

[0054] Finally, the gradient parameters of the segmentation model and detection model obtained during the training process are summarized and integrated, and these optimized parameters are embedded in the entire system. The system can use these optimized model parameters to perform efficient segmentation of polysperm images and high-precision detection of single sperm.

[0055] System usage process see Figure 7 shown.

[0056] For the display of the system interface, see Figure 8 shown.

[0057] First, in the system interface, the doctor uploads the polyspermia image to be predicted.

[0058] Given that the image quality generated by different imaging devices may vary significantly, and the multiple steps in the slide production process may also introduce various variables, these factors together affect the quality of sperm images.

[0059] In order to effectively deal with these challenges, the system provides optional model confidence options for doctors to choose according to their actual needs, such as Fig. 9 shown.

[0060] After selecting the appropriate model confidence, the doctor needs to perform the segmentation operation and click the segmentation button. Then, the system will instantly present a polyspermia image with an accurate sperm detection annotation box, such as Fig.10 Clear display. This not only makes it easier for doctors to visually review the test results, but also greatly improves the convenience of operation.

[0061] In addition, the system interface will also synchronously display the total sperm count information and clearly mark the folder name where the segmented single sperm images are saved.

[0062] This folder is specifically used to store all generated segmented single sperm images to ensure orderly storage and convenient management of data.

[0063] Doctors can continuously optimize the detection effect by cyclically adjusting the model confidence until each sperm in the polyspermia image displayed by the system is accurately assigned a detection label box.

[0064] Then after clicking the detection option, the system will automatically call the cut single sperm image to input the detection model to obtain the detection information of each sperm and the number of normal and abnormal sperm.

[0065] By matching the sperm detection information (two categories) obtained by the detection model with the sperm prediction label information in the segmentation model, the system can display polyspermia images with two types of sperm prediction labels, such as Fig.11 shown.

[0066] At the same time, if Fig.12 As shown in the figure, the system can also display the ratio data of the two types of sperm in detail. Doctors can write sperm morphology examination reports based on these detailed information provided by the system.

[0067] With these data, doctors can more accurately assess the health of sperm, thereby providing strong support and basis for diagnosis and treatment.

[0068] In this way, doctors can more effectively analyze sperm morphology and provide patients with more accurate diagnosis and treatment plans. This not only improves the accuracy of diagnosis, but also helps doctors develop more personalized treatment plans, ultimately achieving the goal of improving patients' fertility.

[0069] References:

[0070] [1]Liu, R., Wang, M., Wang, M., Yin, J., Yuan, Y., & Liu, J. (2021). Automatic Microscopy Analysis with Transfer Learning for Classification of Human Sperm.

[0071] Applied Sciences,11(12),Article 5369.

[0072] [2] Soroush Javadi, Seyed Abolghasem Mirroshandel. A novel deep learning method for automatic assessment of human sperm images. Computers in Biology and Medicine, 109 (2019), 182–194.

[0073] [3]Shahzad,S.;Ilyas,M.;Lali,M.I.U.;Rauf,H.T.;Kadry,S.;Nasr,E.A.SpermAbnormality Detection Using Sequential Deep NeuralNetwork.Mathematics 2023,11,515.

[0075] [4]Sapkota N,Zhang Y,Li S,et al.Shmc-Net:A Mask-Guided FeatureFusionNetwork for Sperm Head Morphology Classification[C] / / 2024 IEEEInternationalSymposium on Biomedical Imaging(ISBI).0[2024-09-14].

[0076] [5]Yuxuan Cai,Yizhuang Zhou,Qi Han,Jianjian Sun,Xiangwen Kong,Jun Li,Xiangyu Zhang.REVERSIBLE COLUMN NETWORKS.Published as a conference paperatICLR 2023.MEGVII Technology,Beijing Academy of Artificial Intelligence.[6]Xiangyu Chen,Xintao Wang,Jiantao Zhou,Yu Qiao,Chao Dong.Activating MorePixelsin Image Super-Resolution Transformer.arXiv:2205.04437v3[eess.IV],19 Mar2023。

Claims

1. A polyspermia image segmentation detection system based on yolov8 framework, characterized in that: Based on the yolov8 network architecture, it is improved; specifically, the HAT and RevColV1 networks are integrated to design two models, the polysperm segmentation model yolov8-HAT and the single sperm detection model yolov8-RevColV1, which are cascaded to form a polysperm image segmentation detection system; Among them: (i) The YOLOv8-HAT model is used for the segmentation of polyspermia images; YOLOv8-HAT is obtained by integrating the hybrid attention transformer (HAT) on the basis of the YOLOv8 network framework; HAT is a deep learning model for image super-resolution, which activates more input pixels by combining channel attention and window self-attention mechanisms, thereby improving the quality and accuracy of the reconstructed image; the input image is first passed through a convolutional layer for shallow feature extraction, and the extracted shallow features are further input into multiple residual hybrid attention groups (RHAGs) for deep feature extraction; each RHAG contains multiple hybrid attention blocks (HABs) and an overlapping cross attention module (OCAB) ; HAB combines channel attention and window self-attention, while OCAB enhances the interaction between adjacent window features through an overlapping window division mechanism; the high-level features after deep feature extraction are converted into high-resolution images through a reconstruction module; specifically, a HAT network is added to each scale of the feature pyramid network (FPN) in YOLOv8 to enhance the expression ability of multi-scale features; image data is reconstructed through high-resolution by the HAT network to generate a high-resolution feature map, which is combined with the original feature map to further enhance the detection accuracy of small targets; an optimized polysperm image segmentation model is obtained by inputting the training set and validation set consisting of preprocessed images and annotation information into the model for training; (ii) The yolov8-RevColV1 model is used for single sperm detection; yolov8-RevColV1 is formed by integrating the reversible column network (RevColV1) on the basis of the yolov8 network framework; specifically, on the basis of the yolov8 model, RevColV1 is used to replace the backbone network (backbone), and the data input to the model is extracted by the RevColV1 network and then passed into the feature enhancement network (neck) and the detection head (head); RevColV1 is a neural network, which is composed of multiple sub-networks (columns) through multi-level reversible connections, and the information is feature decoupled in the forward process to keep the total information without compression or discarding; feature decoupling means that in each sub-network (column) of the RevColV1 network, the features are transmitted through reversible connections, and are processed and learned independently at the same time; as the information advances in the column, the correlation between the features gradually weakens, that is, decoupling, so that the network can capture and emphasize important features more carefully; The image is input into the yolov8-RevColV1 model. It first passes through the RevColV1 backbone network and a patch embedding module to segment the input image into non-overlapping patches. After that, these patches are sent to each sub-network, namely the column. In each sub-network, four-level feature maps are extracted to propagate information between sub-networks. For classification tasks, only the last level feature map of the last sub-network is used. For other downstream tasks, including target detection and semantic segmentation, all four levels of feature maps of the last sub-network are used. The feature decoupling mechanism of the RevColV1 network allows different levels of information to be gradually decoupled during the propagation process between columns. Some feature maps become more semantic, while some remain as low-level features. The single sperm images saved after segmentation by the segmentation system are annotated with two types of label information, normal and abnormal, and input into the model as training sets and validation sets. After training, an optimized single sperm detection model is obtained. (iii) the polyspermia image segmentation detection system combines the yolov8-HAT model and the yolov8-RevColV1 model in a cascade manner; In the polyspermia image segmentation stage, the polyspermia image to be predicted is first input into the segmentation model yolov8-HAT, all sperm and segmentation coordinates are identified, and each sperm image is segmented according to the segmentation coordinates; after being processed by the segmentation model yolov8-HAT, the total number of sperm in the image and the detailed labeling information of each sperm are accurately obtained; these single sperm images after preliminary segmentation are then passed to the single sperm detection model yolov8-RevColV1; In the single sperm detection stage, the yolov8-RevColV1 model performs two-category recognition on each input single sperm image, identifies abnormal sperm and normal sperm in the image, and counts the number of these two types of sperm; at the same time, the model also provides corresponding predicted label information for each sperm, and passes the label and quantity information to the system; finally, the system overwrites the label information provided by the second-stage single sperm detection model on the label provided by the first-stage segmentation model, thereby directly marking the labels of normal and abnormal sperm on the initial polysperm image.

2. The polyspermia image segmentation detection system based on the yolov8 framework according to claim 1, characterized in that: Before training the system, sperm images need to be acquired, preprocessed, and labeled; specifically: Sperm in sperm slides were photographed under a microscope to obtain original polyspermia images; Preprocess these images, including removing areas where sperm are too dense and selecting images with low sperm overlap for labeling. Specifically, use the labelimg tool to label each sperm in the image. The labeling box is close to the edge of the sperm and the intersection between different labeling boxes is reduced. Then, the image data with labeled information and the images themselves are integrated into a complete dataset, which is divided into two parts: a training set and a validation set.

3. The polyspermia image segmentation detection system based on the yolov8 framework according to claim 2 is characterized in that: After obtaining the information transmitted by the two models, all the information is sorted and calculated through a statistical module to obtain the ratio of normal and abnormal sperm.

Images