Method, apparatus, medium and program product for segmenting and classifying image

Abstract

The invention provides a method and device for carrying out segmentation processing on an image, a medium and a program product. The method comprises the following steps: acquiring a target sperm image to be processed; performing image segmentation processing on each sperm target instance contained in the target sperm image by using a segmentation model combined with SAM; performing skeleton extraction and tail breaking processing on the sperm tail data by using a skeleton extraction algorithm to obtain tail segments corresponding to each sperm tail; carrying out tail splicing processing on the tail segments by using a broken tail splicing algorithm to obtain spliced sperm tail data; based on the obtained sperm head data and the spliced sperm tail data, performing matching and splicing processing on the head and the tail of each sperm to obtain segmentation result information of each sperm target instance contained in the target sperm image; and performing classification processing on the segmentation result information of the target sperm image based on a preset classification rule.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, computer-readable medium and computer-readable product for segmenting and classifying an image. Background Art

[0002] Based on the image segmentation scheme of the prior art, in the scene where multiple overlapping structures in the image need to be segmented, for example, the scene of segmenting sperm photos, blood vessel images or nerve images, the segmentation effect of such overlapping slender structures is poor.

[0003] For the scenario of sperm testing, sperm morphology analysis is an important part of the clinical diagnosis process of men's department in hospitals, and is one of the important indicators for evaluating sperm quality and male fertility. At present, this work still relies on the hospital's men's department experts to invest 4 to 6 hours a day for pure naked eye identification, which not only consumes the doctor's precious time, but also makes it difficult to meet the needs of large-scale and high efficiency. Moreover, sperm morphology analysis has problems such as strong subjectivity and poor mutual recognition of experimental time and results, which has always been the focus and difficulty in the field of reproductive testing. Therefore, the realization of a system that combines artificial intelligence technology to automatically detect sperm abnormalities is a rigid demand for clinical diagnosis of men's department today. If the automatic detection system can be embedded directly into the microscope, the automation and intelligence of this subject will be completely completed, which will have a great significance for promoting automation and intelligence in the medical field to a certain extent.

[0004] However, sperm micrographs usually have the characteristics of easy overlap, poor recognizability, and many interfering color blocks. Therefore, sperm in micrographs are usually mixed together, which is not conducive to the segmentation of existing image segmentation models. At the same time, sperm morphology is complex and the images are intertwined, which makes it extremely difficult to train supervised segmentation models through large-scale labeled data.

[0005] Image detection technology is already commonly used in the field of medical image segmentation, but existing solutions are mostly supervised training models that rely on a large amount of labeled data. The generalization ability of the model is weak and it is difficult to transplant it to the task of sperm segmentation. Another major difficulty in sperm segmentation is that the tail of the sperm is very easy to overlap, which makes the segmentation effect of the existing segmentation model on the overlapping area with similar color and texture generally poor. Summary of the invention

[0006] Multiple aspects of the present application provide a method, an apparatus, a medium and a program product for segmenting and classifying an image.

[0007] In one aspect of the present application, a method for segmenting an image is provided, wherein the method comprises:

[0008] Acquire a target sperm image to be processed;

[0009] Performing image segmentation processing on each sperm target instance contained in the target sperm image by using a segmentation model combined with SAM to obtain sperm head data and sperm tail data contained in the target sperm image;

[0010] The skeleton extraction algorithm is used to extract the skeleton and break the tail of the sperm tail data to obtain the tail fragments corresponding to each sperm tail;

[0011] Using a tail-breaking splicing algorithm to perform tail splicing processing on the tail fragments to obtain spliced ​​sperm tail data;

[0012] Based on the obtained sperm head data and the spliced ​​sperm tail data, the heads and tails of each sperm are matched and spliced ​​to obtain the segmentation result information of each sperm target instance contained in the target sperm image;

[0013] The segmentation result information of the target sperm image is classified based on preset classification rules.

[0014] In one aspect of the present application, a device for segmenting and classifying an image is provided, wherein the device comprises:

[0015] means for acquiring an image of a target sperm to be processed;

[0016] A device for performing image segmentation processing on each sperm target instance contained in the target sperm image by using a segmentation model combined with SAM to obtain sperm head data and sperm tail data contained in the target sperm image;

[0017] A device for performing skeleton extraction and tail breaking processing on sperm tail data by using a skeleton extraction algorithm to obtain tail fragments corresponding to each sperm tail;

[0018] A device for performing tail splicing processing on the tail fragments using a tail-breaking splicing algorithm to obtain spliced ​​sperm tail data;

[0019] A device for matching and splicing the heads and tails of each sperm based on the obtained sperm head data and the spliced ​​sperm tail data to obtain segmentation result information of each sperm target instance contained in the target sperm image;

[0020] A device for classifying the segmentation result information of the target sperm image based on preset classification rules.

[0021] Another aspect of the present application provides an electronic device, the electronic device comprising:

[0022] at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the embodiment of the present application.

[0025] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0026] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0027] The solution provided in the embodiment of the present application combines the SAM segmentation model to perform image segmentation processing on sperm images such as sperm micrographs in the scenario of sperm detection, and combines the tail-breaking splicing algorithm to accurately identify and segment each sperm instance contained in the sperm image. The entire process does not require human intervention, and automatic sperm segmentation is achieved, which greatly improves the efficiency of sperm segmentation in micrographs and provides great convenience for sperm morphology detection. The method of the embodiment of the present application can be combined with the annotation data of medical experts to classify sperm morphology as normal or abnormal, and provide a convenient and efficient operation process and accurate and reliable results for clinical diagnosis of male sperm morphology analysis, which is expected to completely change the current dilemma of sperm morphology relying on manual analysis, and play an important role in clinical diagnosis of andrology. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0029] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0030] Figure 1 A schematic diagram of a process for segmenting and classifying an image provided by an embodiment of the present application is shown;

[0031] Figure 2 A schematic diagram of the structure of a device for segmenting and classifying images provided in an embodiment of the present application is shown;

[0032] Figure 3 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0033] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0036] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0037] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0038] Figure 1 A schematic flow chart of a method for segmenting an image provided in an embodiment of the present application is shown. The method at least includes steps S101 to S106.

[0039] In actual scenarios, the execution subject of the method can be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0040] In the scenario of sperm detection, the execution subject of the method can be applied to the equipment for sperm morphology detection. According to the method of the embodiment of the present application, it is possible to accurately identify and instance segment a large number of overlapping sperm in sperm staining micrographs, and based on the segmentation results, perform unsupervised anomaly detection based on morphological rules on the head, neck, and tail of the sperm, and finally classify the segmentation results.

[0041] In addition to the sperm detection described above, the image segmentation method according to this embodiment can also be applied to similar scenarios where multiple overlapping structures in an image need to be segmented. For example, for vascular images, neural images, street images, etc., since these images all have the problem of overlapping a large number of slender structures, the image segmentation method based on the existing technical solution is difficult to achieve the image segmentation effect, while the method based on the embodiment of the present application can achieve a better image segmentation effect.

[0042] Reference Figure 1 , in step S101, a target sperm image to be processed is obtained.

[0043] The target sperm image includes a plurality of sperm target instances overlapping each other, for example, a sperm micrograph in a sperm detection scene.

[0044] Optionally, sperm micrographs are collected and preliminarily screened to exclude azoospermia images, images with severe color cast, and the sperm micrographs after the preliminarily screening are used as target sperm images.

[0045] According to one embodiment, the method further includes step S107 before step S102.

[0046] In step S107, image preprocessing is performed on the acquired target sperm image.

[0047] The image preprocessing includes but is not limited to at least one of the following operations:

[0048] 1) Standardizing the size of the target image;

[0049] 2) performing image optimization processing on the target image; for example, using Python to increase the image saturation, image contrast, and / or image green area vividness of the target image;

[0050] 3) Performing clarity enhancement processing on the target image; for example, using functions such as Color, Filters, and Morphology in the Skimage library to simulate the dehazing operation in PS to increase the clarity of key information of the target image;

[0051] 4) Change the background and small impurities in the target image into pure white.

[0052] Optionally, before performing image preprocessing, the method performs super-resolution processing on the target sperm image.

[0053] The super-resolution processing improves the clarity of the target sperm image by using super-resolution technology. For example, using Real-ESRGAN super-resolution technology, the process is to use Real-ESRGAN to perform 2 times super-resolution on the image to make the image details clearer, thereby improving the effect of subsequent pre-processing.

[0054] Continue to refer to Figure 1 To illustrate, in step S102, image segmentation processing is performed on each sperm target instance contained in the target sperm image by using a segmentation model combined with SAM to obtain sperm head data and sperm tail data contained in the target sperm image.

[0055] The segmentation model combined with SAM includes various models that can be used for image segmentation processing.

[0056] Optionally, the segmentation model combined with SAM includes but is not limited to improved variants of algorithms such as the SegmentAnything Model (SAM), the Broken Line Splicing (BLS), and the Head-Tail Splicing algorithm. The segmentation model combined with SAM can be various variant models based on the SAM model, such as the SAM-Med2D model or the TinySAM model, etc.

[0057] The Broken Tail Splicing Algorithm (BLS) includes various algorithms that can be used to process the splicing of broken skeletonized tails.

[0058] Specifically, the step S102 further includes steps S1021 to S1023.

[0059] In step S1021, the target sperm image is segmented by using a segmentation model combined with SAM to obtain preliminary sperm head data.

[0060] Specifically, a segmentation model combined with SAM is used to perform a preliminary recognition and segmentation on sperm target instances contained in the target sperm image, and each layer mask of the obtained segmentation result is stored.

[0061] In step S1022, the impurity color blocks contained in the preliminary sperm head data are screened out based on a predetermined screening rule to obtain the sperm head data contained in the target sperm image.

[0062] Optionally, the method may screen out the impurity color blocks contained in the preliminary sperm head data based on a predetermined screening rule in a manner including but not limited to at least any of the following:

[0063] 1) Screening based on shape ratio; the shape ratio is the ratio of the actual area of ​​each mask layer to the minimum circumscribed circle area of ​​the mask. For example, the method calculates the ratio SI of the actual area Sm of each mask layer to the minimum circumscribed circle area Sc of the mask, SI = Sm / Sc, and compares the ratio SI with a preset shape ratio threshold, and regards those that are greater than the shape ratio threshold as sperm heads or impurity blocks.

[0064] 2) Screening based on color ratio; the color ratio is the ratio of the HSV color area of ​​each mask layer to the overall mask area. For example, the method is based on the head mask file, sets the purple HSV color value range corresponding to the sperm head, calculates the ratio CI of the purple area in each mask layer to the overall mask area, and compares the ratio CI with a preset color ratio threshold, and treats the color ratio greater than the threshold as a sperm head or impurity block.

[0065] The method filters out the impurity color blocks contained in the preliminary sperm head data based on a predetermined screening rule, and then stores the obtained sperm head data in a new npy file.

[0066] In step S1023, based on the obtained sperm head data and the screened impurity blocks, the area corresponding to the sperm head and the impurity blocks is removed from the target sperm image to obtain the sperm tail data.

[0067] Continue to refer to Figure 1 To illustrate, in step S103, a skeleton extraction (Skeletonize) algorithm is used to perform skeleton extraction and tail cutting processing on the sperm tail data to obtain tail fragments corresponding to each sperm tail.

[0068] The skeleton extraction process refers to extracting lines connecting all local extreme points from the binary image as the "skeleton" of the object.

[0069] Specifically, for the sperm tail set, each tail is skeletonized, and then the intersection of each skeletonized line is found and disconnected to facilitate subsequent splicing.

[0070] In step S104, a Broken Line Splicing (BLS) algorithm is used to perform tail splicing processing on the tail fragments to obtain spliced ​​sperm tail data.

[0071] Specifically, the method saves each individual curve segment, i.e., the tail segment, as a layer of an npy file. For each layer, the tails are traversed to connect, and each tail segment is spliced ​​and connected according to the tangent slope k1 of the segment, the least squares fitting slope k2, and the distance d between the endpoint of the curve segment and the endpoint of another segment, and the process is iterated in sequence until all tails that meet the threshold conditions are successfully spliced.

[0072] In step S105, based on the obtained sperm head data and the spliced ​​sperm tail data, the head and tail of each sperm are matched and spliced ​​to obtain the segmentation result information of each sperm target instance contained in the target sperm image.

[0073] The segmentation result includes but is not limited to the quantity information, contour information and position information of all sperms obtained through segmentation processing in the target image.

[0074] The segmentation result information includes but is not limited to the number of sperms contained in the target sperm image, the outline and size of each sperm contained therein, and the position information in the target image, etc.

[0075] Specifically, each tail is spliced ​​with the corresponding head previously retained, similar to the method based on the BLS algorithm in step S104, and the endpoints and slopes of the long axis of the sperm head (the longest line segment in the head mask) are calculated, and the tangent slopes and endpoint positions at the end of the tail curve segment are matched and spliced ​​to obtain a complete sperm mask.

[0076] In step S106, the segmentation result information of the target sperm image is classified based on a preset classification rule.

[0077] The classification process divides each sperm into two types: "normal" and "abnormal".

[0078] Optionally, the method sets classification rules for binary classification based on the segmentation result information of multiple sperm images and corresponding annotation data, and the binary classification refers to the classification of sperm into "normal" and "abnormal".

[0079] Optionally, the method screens out sperm belonging to the "abnormal" type by setting a classification rule, and the classification rule is set based on at least any one of the following:

[0080] 1) The size of the acrosome area of ​​the sperm head; for example, grayscale the image and convert all HSV values ​​into the style of (0, 0, N), where N is directly the grayscale value, which is easy to match (0<=N<=255). Set the grayscale value interval of the acrosome color for the grayscale image. Next, calculate the ratio of the acrosome area to the total area in each segmented head, and compare the calculated ratio with the preset threshold. If the calculated ratio is greater than the threshold, it is normal, otherwise it is abnormal. Among them, in order to avoid the influence of brightness and color difference between each image, the average grayscale value of the head position is calculated for correction, and the threshold is set = baseline interval + (grayscale of the image - baseline grayscale).

[0081] 2) The shape of the sperm head; for example, use the `fitEllipse` function of OpenCV to fit an ellipse on the mask. Then, calculate the intersection-over-union (IOU) of the area of ​​the mask and the fitted ellipse, and set a threshold for screening. By comparing the calculated IOU value with a preset threshold, if the calculated IOU value is greater than the threshold, it is normal, otherwise it is abnormal.

[0082] Optionally, the method calculates the ratio of the major axis to the minor axis for the fitted ellipse, and sets a threshold value of the ratio of the major axis to the minor axis for being round or flat, thereby treating sperm that are round or flat as abnormal types of sperm.

[0083] Optionally, the method calculates the pixel value size of the sperm head, sets a threshold to filter out sperm with a head that is too large or too small and classifies them as abnormal.

[0084] 3) Sperm tail width: skeletonize the tail mask, find the outline of the mask, draw a perpendicular line from each point on the outline to the skeleton as the width of that point, record the width of each point, exclude outliers, calculate the variance of the width, and set a threshold to screen sperm with abnormal tail width as abnormal sperm.

[0085] 4) The bending of the sperm tail; for example, whether the bending of the sperm tail is abnormal can be determined by judging the curvature value at that location. Specifically, the curvature value of each point is obtained by the following method: extend the point to both sides by 10 pixels, if it is less than 10 pixels, take the maximum pixel; fit the two ends of the straight line, and the angle between the two straight lines is used as the angle at the pixel point. By setting a threshold for the angle, sperm with an angle less than the threshold is regarded as an abnormal type of sperm.

[0086] 5) The length and width of the sperm tail: Specifically, the mask is skeletonized, and the tail length is obtained by calculating the pixel value of the skeletonized result (the skeleton is a line with a width of 1 pixel). Then, the average width is calculated and the width threshold is set to classify sperm with a tail width that is too thin as abnormal.

[0087] Optionally, the method classifies all the segmented sperms to determine whether they are "normal" or "abnormal." Next, the ratio of normal sperm to all segmented sperms is calculated, and if it is greater than 4%, the patient is judged to be normal, and if it is less than 4%, the patient is judged to be abnormal.

[0088] According to the method of the embodiment of the present application, by combining the SAM segmentation model to perform image segmentation processing on sperm images such as sperm micrographs in the sperm detection scenario, and combining the tail-breaking splicing algorithm, the various sperm instances contained in the sperm image can be accurately identified and segmented. The entire process does not require human participation, and automatic sperm segmentation is achieved, which greatly improves the efficiency of sperm segmentation in micrographs and provides great convenience for sperm morphology detection. The method of the embodiment of the present application can be combined with the annotation data of medical experts to classify sperm morphology as normal or abnormal, and provide a convenient and efficient operation process and accurate and reliable results for the clinical diagnosis of male sperm morphology analysis, which is expected to completely change the current dilemma of sperm morphology relying on manual analysis, and play an important role in clinical diagnosis of andrology.

[0089] In addition, the present application also provides a device for segmenting and classifying an image. The structure of the device is as follows: Figure 3 shown.

[0090] The device comprises: a device for acquiring a target sperm image to be processed (hereinafter referred to as "image acquisition device 101"); a device for performing image segmentation processing on each sperm target instance contained in the target sperm image by using a segmentation model combined with SAM to obtain sperm head data and sperm tail data contained in the target sperm image (hereinafter referred to as "segmentation processing device 102"); a device for performing skeleton extraction and tail cutting processing on the sperm tail data by using a skeleton extraction algorithm to obtain tail fragments corresponding to each sperm tail (hereinafter referred to as "tail cutting processing device 103"); and a device for performing image segmentation processing on each sperm target instance contained in the target sperm image by using a segmentation model combined with SAM to obtain sperm head data and sperm tail data contained in the target sperm image. A device for performing tail splicing processing on the tail fragments using a tail-breaking splicing algorithm to obtain spliced ​​sperm tail data (hereinafter referred to as "tail splicing device 104"); a device for matching and splicing the head and tail of each sperm based on the obtained sperm head data and the spliced ​​sperm tail data to obtain segmentation result information of each sperm target instance contained in the target sperm image (hereinafter referred to as "head and tail splicing device 105"); a device for classifying the segmentation result information of the target sperm image based on a preset classification rule (hereinafter referred to as "classification processing device 106").

[0091] Reference Figure 3 , the image acquisition device 101 acquires the target sperm image to be processed.

[0092] The target sperm image includes a plurality of sperm target instances overlapping each other, for example, a sperm micrograph in a sperm detection scene.

[0093] Optionally, sperm micrographs are collected and preliminarily screened to exclude azoospermia images, images with severe color cast, and the sperm micrographs after the preliminarily screening are used as target sperm images.

[0094] According to one embodiment, the device further comprises a pre-processing device.

[0095] The preprocessing device performs image preprocessing on the acquired target sperm image.

[0096] The image preprocessing includes but is not limited to at least one of the following operations:

[0097] 1) Standardizing the size of the target image;

[0098] 2) performing image optimization processing on the target image; for example, using Python to increase the image saturation, image contrast, and / or image green area vividness of the target image;

[0099] 3) Performing clarity enhancement processing on the target image; for example, using functions such as Color, Filters, and Morphology in the Skimage library to simulate the dehazing operation in PS to increase the clarity of key information of the target image;

[0100] 4) Change the background and small impurities in the target image into pure white.

[0101] Optionally, before performing image preprocessing, the method performs super-resolution processing on the target sperm image.

[0102] The super-resolution processing improves the clarity of the target sperm image by using super-resolution technology. For example, using Real-ESRGAN super-resolution technology, the process is to use Real-ESRGAN to perform 2 times super-resolution on the image to make the image details clearer, thereby improving the effect of subsequent pre-processing.

[0103] Continue to refer to Figure 3 To illustrate, the segmentation processing device 102 performs image segmentation processing on each sperm target instance contained in the target sperm image by using a segmentation model combined with SAM, so as to obtain sperm head data and sperm tail data contained in the target sperm image.

[0104] The segmentation model combined with SAM includes various models that can be used for image segmentation processing.

[0105] Optionally, the segmentation model combined with SAM includes but is not limited to improved variants of algorithms such as the SegmentAnything Model (SAM), the Broken Line Splicing (BLS), and the Head-Tail Splicing algorithm. The segmentation model combined with SAM can be various variant models based on the SAM model, such as the SAM-Med2D model or the TinySAM model, etc.

[0106] The Broken Tail Splicing Algorithm (BLS) includes various algorithms that can be used to process the splicing of broken skeletonized tails.

[0107] Specifically, the segmentation processing device 102 further includes a preliminary recognition device, a head screening device and a removal processing device.

[0108] The preliminary recognition device performs image segmentation processing on the target sperm image by using a segmentation model combined with SAM to obtain the recognized preliminary sperm head data.

[0109] Specifically, a segmentation model combined with SAM is used to perform a preliminary recognition and segmentation on sperm target instances contained in the target sperm image, and each layer mask of the obtained segmentation result is stored.

[0110] The head screening device screens out the impurity color blocks contained in the preliminary sperm head data based on a predetermined screening rule to obtain the sperm head data contained in the target sperm image.

[0111] Optionally, the head screening device screens out the impurity color blocks contained in the preliminary sperm head data according to the predetermined screening rules, including but not limited to at least any one of the following:

[0112] 1) Screening based on shape ratio; the shape ratio is the ratio of the actual area of ​​each mask layer to the minimum circumscribed circle area of ​​the mask. For example, the method calculates the ratio SI of the actual area Sm of each mask layer to the minimum circumscribed circle area Sc of the mask, SI = Sm / Sc, and compares the ratio SI with a preset shape ratio threshold, and regards those that are greater than the shape ratio threshold as sperm heads or impurity blocks.

[0113] 2) Screening based on color ratio; the color ratio is the ratio of the HSV color area of ​​each mask layer to the overall mask area. For example, the method is based on the head mask file, sets the purple HSV color value range corresponding to the sperm head, calculates the ratio CI of the purple area in each mask layer to the overall mask area, and compares the ratio CI with a preset color ratio threshold, and treats the color ratio greater than the threshold as a sperm head or impurity block.

[0114] The head screening device screens out the impurity color blocks contained in the preliminary sperm head data based on a predetermined screening rule, and then stores the obtained sperm head data in a new npy file.

[0115] The removal processing device removes the sperm head and the area corresponding to the impurity block from the target sperm image based on the obtained sperm head data and the screened impurity block to obtain the sperm tail data.

[0116] Continue to refer to Figure 3 To explain, the tail cutting processing device 103 uses a skeleton extraction (Skeletonize) algorithm to perform skeleton extraction and tail cutting processing on the sperm tail data to obtain tail fragments corresponding to each sperm tail.

[0117] The skeleton extraction process refers to extracting lines connecting all local extreme points from the binary image as the "skeleton" of the object.

[0118] Specifically, for the sperm tail set, the tail breaking processing device 103 performs skeletonization processing on each tail, and then finds the intersection of each skeletonized line and breaks it to facilitate subsequent splicing.

[0119] The tail splicing device 104 uses a Broken Line Splicing (BLS) algorithm to perform tail splicing processing on the tail fragments to obtain spliced ​​sperm tail data.

[0120] Specifically, the tail splicing device 104 saves each individual curve segment, i.e., the tail segment, as a layer of an npy file, and for each layer, traverses to connect the tails, and splices and connects each tail segment according to the tangent slope k1 of the line segment, the least squares fitting slope k2, and the distance d between the endpoint of the curve segment and the endpoint of another line segment, and iterates in sequence until all tails that meet the threshold conditions are successfully spliced.

[0121] The head-tail splicing device 105 matches and splices the head and tail of each sperm based on the obtained sperm head data and the spliced ​​sperm tail data, and obtains the segmentation result information of each sperm target instance contained in the target sperm image.

[0122] The segmentation result includes but is not limited to the quantity information, contour information and position information of all sperms obtained through segmentation processing in the target image.

[0123] The segmentation result information includes but is not limited to the number of sperms contained in the target sperm image, the outline and size of each sperm contained therein, and the position information in the target image, etc.

[0124] Specifically, the head and tail splicing device 105 splices each tail with the corresponding head previously retained, similar to the operation of the tail splicing device 104 based on the BLS algorithm, calculates the endpoint and slope of the long axis of the sperm head (the longest line segment in the head mask), and matches and splices the tangent slope and endpoint positions of the end of the tail curve segment to obtain a complete sperm mask.

[0125] The classification processing device 106 performs classification processing on the segmentation result information of the target sperm image based on a preset classification rule.

[0126] The classification process divides each sperm into two types: "normal" and "abnormal".

[0127] Optionally, the classification processing device 106 sets classification rules for binary classification based on the segmentation result information of multiple sperm images and corresponding annotation data, and the binary classification refers to the classification of sperm into "normal" and "abnormal".

[0128] Optionally, the classification processing device 106 selects sperm belonging to the "abnormal" type by setting a classification rule, and the classification rule is set based on at least any one of the following:

[0129] 1) The size of the acrosome area of ​​the sperm head; for example, grayscale the image and convert all HSV values ​​into the style of (0, 0, N), where N is directly the grayscale value, which is easy to match (0<=N<=255). Set the grayscale value interval of the acrosome color for the grayscale image. Next, calculate the ratio of the acrosome area to the total area in each segmented head, and compare the calculated ratio with the preset threshold. If the calculated ratio is greater than the threshold, it is normal, otherwise it is abnormal. Among them, in order to avoid the influence of brightness and color difference between each image, the average grayscale value of the head position is calculated for correction, and the threshold is set = baseline interval + (grayscale of the image - baseline grayscale).

[0130] 2) The shape of the sperm head; for example, use the `fitEllipse` function of OpenCV to fit an ellipse on the mask. Then, calculate the intersection-over-union (IOU) of the area of ​​the mask and the fitted ellipse, and set a threshold for screening. By comparing the calculated IOU value with a preset threshold, if the calculated IOU value is greater than the threshold, it is normal, otherwise it is abnormal.

[0131] Optionally, the ratio of the major axis to the minor axis is calculated for the fitted ellipse, and a threshold of the ratio of the major axis to the minor axis is set for being round or flat, thereby treating sperm that are round or flat as abnormal types of sperm.

[0132] Optionally, the pixel value size of the sperm head is calculated, and a threshold is set to filter out sperm with a head that is too large or too small and classify them as abnormal.

[0133] 3) Sperm tail width: skeletonize the tail mask, find the outline of the mask, draw a perpendicular line from each point on the outline to the skeleton as the width of that point, record the width of each point, exclude outliers, calculate the variance of the width, and set a threshold to screen sperm with abnormal tail width as abnormal sperm.

[0134] 4) The bending of the sperm tail; for example, whether the bending of the sperm tail is abnormal can be determined by judging the curvature value at that location. Specifically, the curvature value of each point is obtained by the following method: extend the point to both sides by 10 pixels, if it is less than 10 pixels, take the maximum pixel; fit the two ends of the straight line, and the angle between the two straight lines is used as the angle at the pixel point. By setting a threshold for the angle, sperm with an angle less than the threshold is regarded as an abnormal type of sperm.

[0135] 5) The length and width of the sperm tail: Specifically, the mask is skeletonized, and the tail length is obtained by calculating the pixel value of the skeletonized result (the skeleton is a line with a width of 1 pixel). Then, the average width is calculated and the width threshold is set to classify sperm with a tail width that is too thin as abnormal.

[0136] Optionally, the classification processing device 106 classifies all the segmented sperms to determine whether the sperms are "normal" or "abnormal". Then, the proportion of normal sperms to all segmented sperms is calculated. If it is greater than 4%, the patient is judged to be normal, and if it is less than 4%, the patient is judged to be abnormal.

[0137] According to the device of the embodiment of the present application, by combining the SAM segmentation model to perform image segmentation processing on sperm images such as sperm micrographs in the sperm detection scenario, and combining the tail-breaking splicing algorithm, the various sperm instances contained in the sperm image can be accurately identified and segmented. The entire process does not require human intervention, and automatic sperm segmentation is achieved, which greatly improves the efficiency of sperm segmentation in micrographs and provides great convenience for sperm morphology detection. The method of the embodiment of the present application can be combined with the annotation data of medical experts to classify sperm morphology as normal or abnormal, and provide a convenient and efficient operation process and accurate and reliable results for the clinical diagnosis of male sperm morphology analysis, which is expected to completely change the current dilemma of sperm morphology relying on manual analysis, and play an important role in andrological clinical diagnosis.

[0138] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method for segmenting and classifying images in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0139] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0140] Figure 3 The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0141] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0142] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0143] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0144] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0145] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0146] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0147] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0150] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0153] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0155] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for segmenting and classifying an image, wherein: The method comprises: Acquire a target sperm image to be processed; Performing image segmentation processing on each sperm target instance contained in the target sperm image by using a segmentation model combined with SAM to obtain sperm head data and sperm tail data contained in the target sperm image; The skeleton extraction algorithm is used to extract the skeleton and break the tail of the sperm tail data to obtain the tail fragments corresponding to each sperm tail; Using a tail-breaking splicing algorithm to perform tail splicing processing on the tail fragments to obtain spliced ​​sperm tail data; Based on the obtained sperm head data and the spliced ​​sperm tail data, the heads and tails of each sperm are matched and spliced ​​to obtain the segmentation result information of each sperm target instance contained in the target sperm image; The segmentation result information of the target sperm image is classified based on preset classification rules.

2. The method according to claim 1, wherein: Performing image segmentation processing on each sperm target instance contained in the target sperm image by using the segmentation model combined with SAM includes: Performing image segmentation processing on the target sperm image by using a segmentation model combined with SAM to obtain preliminary sperm head data identified; Based on a predetermined screening rule, the impurity color blocks contained in the preliminary sperm head data are screened out to obtain the sperm head data contained in the target sperm image; Based on the obtained sperm head data and the screened impurity blocks, the area corresponding to the sperm head and the impurity blocks is removed from the target sperm image to obtain the sperm tail data.

3. The method according to claim 1 or 2, further comprising: Based on the segmentation result information of multiple sperm images and the corresponding annotation data, a classification rule for binary classification is set, and the binary classification refers to the classification of sperm into "normal" and "abnormal".

4. The method according to claim 3, wherein: The method screens out sperm belonging to the "abnormal" type by setting a classification rule, and the classification rule is set based on at least any one of the following: The size of the acrosomal area of ​​the sperm head; The shape of the sperm head; Sperm tail width; The bending of the sperm tail; The length and width of the sperm tail.

5. The method according to claim 1 or 2, wherein: The method further comprises: Perform image preprocessing on the acquired target sperm image.

6. The method according to claim 5, wherein: The method further comprises: During the image preprocessing, the target sperm image is subjected to super-resolution processing, wherein the super-resolution processing improves the clarity of the target sperm image by using super-resolution technology.

7. A device for segmenting and classifying an image, wherein: The device comprises: means for acquiring an image of a target sperm to be processed; A device for performing image segmentation processing on each sperm target instance contained in the target sperm image by using a segmentation model combined with SAM to obtain sperm head data and sperm tail data contained in the target sperm image; A device for performing skeleton extraction and tail breaking processing on sperm tail data by using a skeleton extraction algorithm to obtain tail fragments corresponding to each sperm tail; A device for performing tail splicing processing on the tail fragments using a tail-breaking splicing algorithm to obtain spliced ​​sperm tail data; A device for matching and splicing the heads and tails of each sperm based on the obtained sperm head data and the spliced ​​sperm tail data to obtain segmentation result information of each sperm target instance contained in the target sperm image; A device for classifying the segmentation result information of the target sperm image based on preset classification rules.

8. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be used by a processor to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images