Method, device and computer readable medium for segmenting an image

Abstract

The present application provides a method, device and computer-readable medium for performing segmentation processing on an image. The method according to the present application includes: obtaining a target image to be processed, the target image containing multiple target objects; performing segmentation processing on each target object contained in the target image by using a target segmentation model and a target clustering algorithm; and generating segmentation result information of the target image. The present application performs image segmentation processing on objects with overlapping slender structures in an image by combining a segmentation model and a clustering algorithm, and achieves excellent image segmentation effects; and according to the method of an embodiment of the present application, by combining a segmentation model and a clustering algorithm, image segmentation processing is performed on sperm micrographs in a sperm detection scenario, and the overlapping sperm tails are accurately identified and segmented. The entire process does not require human intervention, and automatic sperm segmentation is achieved, which greatly improves the efficiency of sperm segmentation in micrographs.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and computer-readable medium for segmenting an image. Background Art

[0002] Based on the image segmentation scheme of the prior art, in the scene where multiple overlapping structures in the image need to be segmented, for example, the scene of segmenting sperm photos, blood vessel images or nerve images, the segmentation effect of such overlapping slender structures is poor.

[0003] For the scenario of sperm testing, sperm morphology analysis is an important part of the clinical diagnosis process of men's department in hospitals, and is one of the important indicators for evaluating sperm quality and male fertility. At present, this work still relies on the hospital's men's department experts to invest 4 to 6 hours a day for pure naked eye identification, which not only consumes the doctor's precious time, but also makes it difficult to meet the needs of large-scale and high efficiency. Moreover, sperm morphology analysis has problems such as strong subjectivity and poor mutual recognition of experimental time and results, which has always been the focus and difficulty in the field of reproductive testing. Therefore, the realization of a system that combines artificial intelligence technology to automatically detect sperm abnormalities is a rigid demand for clinical diagnosis of men's department today. If the automatic detection system can be embedded directly into the microscope, the automation and intelligence of this subject will be completely completed, which will have a great significance for promoting automation and intelligence in the medical field to a certain extent.

[0004] However, sperm micrographs usually have the characteristics of easy overlap, poor recognizability, and many interfering color blocks. Therefore, sperm in micrographs are usually mixed together, which is not conducive to the segmentation of existing image segmentation models. At the same time, sperm morphology is complex and the images are intertwined, which makes it extremely difficult to train supervised segmentation models through large-scale labeled data.

[0005] Image detection technology is already commonly used in the field of medical image segmentation, but existing solutions are mostly supervised training models that rely on a large amount of labeled data. The generalization ability of the model is weak and it is difficult to transplant it to the task of sperm segmentation. Another major difficulty in sperm segmentation is that the tail of the sperm is very easy to overlap, which makes the segmentation effect of the existing segmentation model on the overlapping area with similar color and texture generally poor. Summary of the invention

[0006] Multiple aspects of the present application provide a method, an apparatus, and a computer-readable medium for performing image segmentation processing.

[0007] In one aspect of the present application, a method for segmenting an image is provided, wherein the method comprises:

[0008] Acquire a target image to be processed, wherein the target image contains a plurality of target objects;

[0009] By using the target segmentation model and the target clustering algorithm, each target object contained in the target image is segmented;

[0010] Generate segmentation result information of the target image.

[0011] In one aspect of the present application, a device for performing segmentation processing on an image is provided, wherein the device comprises:

[0012] A device for acquiring a target image to be processed, wherein the target image includes a plurality of target objects;

[0013] A device for performing segmentation processing on each target object contained in the target image by using a target segmentation model and a target clustering algorithm;

[0014] A device for generating segmentation result information of the target image.

[0015] Another aspect of the present application provides an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the embodiment of the application.

[0019] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the application.

[0020] The solution provided in the embodiment of the present application combines a segmentation model and a clustering algorithm to perform image segmentation processing on objects with overlapping slender structures in an image, and achieves excellent image segmentation effect.

[0021] Furthermore, according to the method of the embodiment of the present application, by combining the segmentation model and the clustering algorithm, image segmentation processing is performed on sperm micrographs in the sperm detection scenario, and the overlapping sperm tails are accurately identified and segmented. The entire process does not require human intervention, thereby achieving automatic sperm segmentation, greatly improving the efficiency of sperm segmentation in micrographs, and providing great convenience for sperm morphology detection. The method of the embodiment of the present application can be combined with the annotated data of medical experts to judge whether the sperm morphology is normal or abnormal, and obtain corresponding test results, providing a convenient and efficient operation process and accurate and reliable results for the clinical diagnosis of male sperm morphology analysis, which is expected to completely change the current dilemma of sperm morphology relying on manual analysis, and play an important role in andrological clinical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0024] Figure 1 A schematic flow chart of a method for segmenting an image provided by an embodiment of the present application is shown;

[0025] Figure 2 A flowchart of a method for segmenting a sperm image according to an embodiment of the present application is shown.

[0026] Figure 3 A schematic diagram of the structure of a device for segmenting an image provided by an embodiment of the present application is shown;

[0027] Figure 4 A schematic diagram of the structure of a device for segmenting sperm images provided in an embodiment of the present application is shown;

[0028] Figure 5 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0029] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0032] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0033] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0034] Figure 1 A schematic flow chart of a method for segmenting an image provided by an embodiment of the present application is shown. The method at least includes step S101, step S102 and step S103.

[0035] In actual scenarios, the execution subject of the method can be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0036] In the scenario of sperm detection, the execution subject of the method can be applied to the device for sperm morphology detection. According to the method of the embodiment of the present application, the sperm in the sperm staining micrograph can be segmented, especially the segmentation in the scenario of a large number of sperm overlapping, so as to detect sperm abnormality based on the segmented image.

[0037] In addition to the sperm detection described above, the image segmentation method according to this embodiment can also be applied to similar scenarios where multiple overlapping structures in an image need to be segmented. For example, for vascular images, neural images, street images, etc., since these images all have the problem of overlapping a large number of slender structures, the image segmentation method based on the existing technical solution is difficult to achieve the image segmentation effect, while the method based on the embodiment of the present application can achieve a better image segmentation effect.

[0038] Reference Figure 1 In step S101, a target image to be processed is obtained, and the target image contains multiple target objects.

[0039] Wherein, the target object is an object with a slender structure.

[0040] Optionally, the target image includes a plurality of target objects that overlap with each other, such as a sperm image, a blood vessel image, a nerve image in the detection scene, or a street image in the scene, etc.

[0041] According to one embodiment, the method further includes step S104 before step S102.

[0042] In step S104, image preprocessing is performed on the acquired target image.

[0043] The image preprocessing includes at least one of the following operations:

[0044] 1) Standardizing the size of the target image;

[0045] 2) performing image optimization processing on the target image; for example, using Python to increase the image saturation, image contrast, and / or image green area vividness of the target image;

[0046] 3) Performing clarity enhancement processing on the target image; for example, using functions such as Color, Filters, and Morphology in the Skimage library to simulate the dehazing operation in PS to increase the clarity of key information of the target image;

[0047] 4) Change the background and small impurities in the target image into pure white.

[0048] Continue to refer to Figure 1 To illustrate, in step S102, each target object included in the target image is identified and segmented by using a target segmentation model and a target clustering algorithm.

[0049] The target segmentation model includes various models that can be used for image segmentation processing.

[0050] Optionally, the target segmentation model is a Segment Anything Model (SAM). The target model may be a variant model based on the SAM model, such as a SAM-Med2D model or a TinySAM model.

[0051] The target clustering algorithm includes various clustering algorithms that can be used to process cross-structures. Optionally, the target clustering algorithm is a spectral clustering on multiple manifolds (SMMC), which is a clustering algorithm mainly used for cross-manifold structures.

[0052] Specifically, the step S102 includes steps S1021 to S1026.

[0053] In step S1021, the target segmentation model is used to iteratively identify the target objects contained in the target image until all target objects are identified.

[0054] In step S1022, the mask corresponding to the target object identified in each round is stored in the target file.

[0055] In step S1023, a skeleton extraction (Skeletonize) algorithm is used to perform skeleton extraction processing on the mask corresponding to the identified target object.

[0056] The skeleton extraction process refers to extracting lines connecting all local extreme points from the binary image as the "skeleton" of the object. In step S1023, by performing the skeleton extraction process, all lines connecting all local extreme points in each identified target object are obtained as the "skeleton" of the target object.

[0057] In step S1024, based on each layer of mask obtained by the skeleton extraction process, the number of endpoints is counted and the number of target objects included in the mask is determined based on a predetermined endpoint matching rule.

[0058] In step S1025, a target clustering algorithm is used to segment the multiple skeletonized lines obtained by the skeleton extraction process.

[0059] Specifically, pixel sampling is performed on a plurality of skeletonized lines obtained by skeleton extraction processing to convert them into a dot matrix structure supported by the target clustering algorithm; then, the target clustering algorithm is called to segment the plurality of skeletonized lines by setting appropriate parameters and random seeds.

[0060] In step S1026, the segmented skeletonized lines are used as reference lines to assign corresponding categories to each pixel point of the original mask.

[0061] By assigning a corresponding category to each pixel of the original mask, the pixel-level classification of the original mask is achieved, thereby achieving the segmentation of each target object in the original mask.

[0062] Continue to refer to Figure 1 To explain, in step S103, segmentation result information of the target object corresponding to the target image is generated.

[0063] The segmentation result includes but is not limited to the quantity information, contour information and position information of all target objects obtained through segmentation processing in the target image.

[0064] According to the method of the embodiment of the present application, the image segmentation processing is performed on the objects with overlapping slender structures in the image by combining the segmentation model and the clustering algorithm, and an excellent image segmentation effect is achieved.

[0065] Figure 2 A flow chart of a method for segmenting a sperm image according to an embodiment of the present application is shown. The method includes steps S201 to S206.

[0066] In step S201, a target sperm image to be processed is obtained.

[0067] The sperm image data includes a plurality of sperms overlapping each other, for example, a sperm micrograph in a sperm detection scene.

[0068] Optionally, sperm micrographs are collected and preliminarily screened to exclude azoospermia images, images with severe color cast, and the sperm micrographs after the preliminarily screening are used as target sperm images.

[0069] Optionally, before step S202 , the method further includes step 207 .

[0070] In step S207, the acquired target sperm image is preprocessed, wherein the image preprocessing method is similar to the aforementioned step S104 and will not be described in detail here.

[0071] In step S202, the target sperm image is subjected to a first segmentation process by using a target segmentation model to obtain sperm head data.

[0072] The target segmentation model is a SAM model. The target segmentation model is the same or similar to the target segmentation model described in step 102, and will not be described in detail here.

[0073] Specifically, the step S202 includes step S2021 and step S202.

[0074] In step S2021, the sperm heads contained in the target sperm image are identified by using the target segmentation model to obtain head masks corresponding to all identified sperm heads.

[0075] In step S2022, based on the head mask file, valid sperm heads are screened out according to predetermined screening conditions.

[0076] The screening condition is used to determine whether the identified sperm head is a valid sperm head.

[0077] According to one embodiment, the screening condition is set based on the intersection over union (IoU) of the sperm head region and the overall mask. Specifically, in step S2022, based on the head mask file, the color value range of the sperm head corresponding to the color of the sperm head is set; the IoU of the sperm head corresponding color region in each mask layer and the overall mask is calculated; and by comparing with a preset IoU threshold, the effective sperm head in the head mask is obtained.

[0078] Optionally, the step S202 also includes step S2023.

[0079] In step S2023, the head mask corresponding to the effective sperm head is stored.

[0080] Continue to refer to Figure 2To illustrate, in step S203, a head subtraction process is performed on the target sperm image based on the sperm head data to obtain target tail data for performing a second segmentation process.

[0081] According to the first example of the present application, it is assumed that the dyeing method used in the first example dyes the sperm head purple. For the sperm micrographs that have undergone image preprocessing, a predetermined SAM model is used to perform a first segmentation process to obtain a mask file for all first segmentation processes. Next, for the generated mask, a purple HSV color value range is set, and the intersection-and-union ratio of the purple area in each layer of the mask to the overall mask is calculated. Next, based on the preset intersection-and-union ratio threshold, it is determined whether it is a valid sperm head, and the valid sperm head is filtered into the header file for storage, and the area corresponding to the sperm head is cut out in the original image for subsequent tail segmentation.

[0082] Optionally, the target sperm image after the head subtraction process is subjected to impurity removal processing, so that the image data after the impurities are removed is used as the target tail data for the second segmentation process.

[0083] Continuing to explain the first example above, after deducting the head, there are still many dye blocks that interfere with model recognition. The following rules are set to remove these dye blocks: for each layer of mask, find the two pixels a and b that are farthest apart from each other in the mask, calculate the distance d between a and b, and calculate S=d^2, where S represents the theoretical maximum area containing points a and b, and s represents the actual area, and calculate q=s / S. The larger the q value, the closer the mask layer is to a mass structure rather than an elongated sperm tail. After setting a suitable q value threshold to screen out mass impurities and sperm tails that do not meet the requirements, a pure tail set is obtained for fine tail segmentation.

[0084] Continue to refer to Figure 2 To illustrate, in step S204, the target tail data is subjected to a second segmentation process by using the target model and the target clustering algorithm to obtain sperm tail data.

[0085] Optionally, the target clustering algorithm is SMMC. The target clustering algorithm is the same as or similar to the target clustering algorithm described in step 102, and will not be described in detail here.

[0086] Specifically, the step S204 includes step S2041 to step S2046.

[0087] In step S2041, the target segmentation model is used to iteratively identify the sperm tails contained in the target tail data until all the sperm tails are identified.

[0088] In step S2042, the tail mask image corresponding to the sperm tail identified in each round is stored.

[0089] In step S2043, a skeleton extraction algorithm is used to perform skeleton extraction processing on the tail mask image.

[0090] In step S2044, based on each layer of mask obtained by the skeleton extraction process, the number of endpoints is counted and the number of sperm contained in the mask is determined based on a predetermined endpoint matching rule.

[0091] In step S2045, a target clustering algorithm is used to segment the multiple skeletonized lines obtained by the skeleton extraction process.

[0092] Specifically, pixel sampling is performed on a plurality of skeletonized lines obtained by skeleton extraction processing to convert them into a dot matrix structure supported by the target clustering algorithm; then, the target clustering algorithm is called to segment the plurality of skeletonized lines by setting appropriate parameters and random seeds.

[0093] In step S2046, the skeletonized line after segmentation is used as a reference line to assign a corresponding category to each pixel point of the original mask, so as to achieve the segmentation of a single sperm tail in the original mask.

[0094] Continuing to explain the first example mentioned above, based on the pure tail set, the SAM model is first used to iteratively identify all the sperm tails in the image and save them in a mask format. In the result obtained in this step, many tails overlap, and the separation of individual sperm tails is not achieved. After each iteration, the position corresponding to the mask obtained in this step is erased in the original image, where the pixel points at the corresponding position of the mask are set to the background color white, and the edge residual image is removed. Subsequently, the next round of SAM model processing is performed until all the sperm tails in the image are identified, and then the tail masks identified by all rounds of SAM models are summarized and stored in a tail file for the next step of single tail segmentation.

[0095] Next, the tail mask is subjected to skeleton extraction using the Skeletonize algorithm. Subsequently, the number of endpoints of each layer of the skeletonized mask is determined, and masks with endpoints greater than 2 (i.e., masks greater than one tail) are selected, and the number of sperm contained in the mask is inferred according to the number of endpoints. The predetermined pairing rule is: endpoint number 3, 4 - number of tails 2; endpoint number 5, 6 - number of tails 3, and so on. Subsequently, the skeletonized result is pixel-sampled and converted into a lattice structure supported by the clustering algorithm. Appropriate parameters and random seeds are set, and the clustering algorithm is called to segment the skeletonized lines. Next, the segmented skeletonized lines are used as the baseline to assign categories to each pixel of the original mask, thereby achieving pixel-level classification of the original mask, that is, achieving the splitting of a single sperm tail.

[0096] Continue to refer to Figure 2 To illustrate, in step S205, based on the obtained sperm head data and sperm tail data, the head and tail of each sperm are matched and spliced ​​to obtain the segmentation result of each sperm.

[0097] Specifically, based on the sperm head data obtained by the first segmentation process, the long axis of each sperm head is obtained; then, based on the sperm tail data obtained by the second segmentation process, and in accordance with the preset front and rear end determination rules, the front end and rear end of each sperm tail are obtained; then, according to the preset geometric rules, the front end skeletonized straight line of the sperm tail is matched with the long axis of the sperm head to obtain matching sperm heads and tails; then, the matching sperm heads and tails are spliced ​​to obtain the segmentation results of each sperm.

[0098] Continuing the description of the first example, the front and rear ends of each sperm tail segmented in the previous step are determined by the color change trend and the mask width, and the front skeleton line is straightened. Then the long axis of the sperm head segmented previously is found, and the tail skeleton straight line is matched with the head long axis according to the geometric rules in terms of distance and angle, and the head and tail are paired, and then the corresponding head and tail are spliced ​​to obtain the complete sperm segmentation result.

[0099] In step S206, segmentation result information of the target sperm image is generated based on the segmentation results of each sperm.

[0100] The segmentation result information includes but is not limited to the number of sperms contained in the target sperm image, the outline and size of each sperm contained therein, and the position information in the target image, etc.

[0101] According to one embodiment, the method further includes step S208.

[0102] In step S208, a classification model for sperm morphology classification is trained based on the segmentation result information of multiple sperm images and the corresponding annotation data.

[0103] Specifically, data annotations corresponding to sperm morphology classification problems are obtained from medical experts. Then, deep learning methods such as semi-supervised learning are used to train the classification model to achieve a binary classification of qualified / unqualified sperm morphology after segmentation.

[0104] According to one embodiment, the method further includes step S209.

[0105] In step S209, the trained classification model is used to perform sperm eligibility analysis on the patient's sperm micrographs.

[0106] For example, the trained classification model is used to perform eligibility analysis on each sperm after segmentation, and the number of sperm with normal morphology and sperm with abnormal morphology in multiple sperm micrographs of a patient is counted. Based on the counted number, the sperm morphology analysis result of the patient is calculated.

[0107] According to the method of the embodiment of the present application, by combining the segmentation model and the clustering algorithm to perform image segmentation processing on the sperm micrographs in the sperm detection scenario, the overlapping sperm tails are accurately identified and segmented, and the entire process does not require human intervention, thereby realizing automatic sperm segmentation, greatly improving the efficiency of sperm segmentation in micrographs, and providing great convenience for sperm morphology detection; the method of the embodiment of the present application can be combined with the annotated data of medical experts to judge whether the sperm morphology is normal or abnormal, and obtain corresponding test results, providing a convenient and efficient operation process and accurate and reliable results for the clinical diagnosis of male sperm morphology analysis, which is expected to completely change the current dilemma of sperm morphology relying on manual analysis, and play an important role in clinical diagnosis of andrology.

[0108] In addition, the present application also provides a device for segmenting an image. The structure of the device is as follows: Figure 3 shown.

[0109] The device includes: a device for acquiring a target image to be processed (hereinafter referred to as "image acquisition device 101"); a device for performing segmentation processing on each target object contained in the target image by using a target segmentation model and a target clustering algorithm (hereinafter referred to as "segmentation processing device 102"); and a device for generating segmentation result information of the target image (hereinafter referred to as "result generation device 103").

[0110] Reference Figure 3 , the image acquisition device 101 acquires a target image to be processed, and the target image contains multiple target objects.

[0111] Wherein, the target object is an object with a slender structure.

[0112] Optionally, the target image includes a plurality of target objects that overlap with each other, such as a sperm image, a blood vessel image, a nerve image in the detection scene, or a street image in the scene, etc.

[0113] According to one embodiment, the apparatus further comprises an image preprocessing device, and the operation of the image preprocessing device is performed before the operation of the segmentation processing device 102 .

[0114] The image preprocessing device performs image preprocessing on the acquired target image.

[0115] The image preprocessing includes at least one of the following operations:

[0116] 1) Standardizing the size of the target image;

[0117] 2) performing image optimization processing on the target image; for example, using Python to increase the image saturation, image contrast, and / or image green area vividness of the target image;

[0118] 3) Performing clarity enhancement processing on the target image; for example, using functions such as Color, Filters, and Morphology in the Skimage library to simulate the dehazing operation in PS to increase the clarity of key information of the target image;

[0119] 4) Change the background and small impurities in the target image into pure white.

[0120] Continue to refer to Figure 3 To explain, the segmentation processing device 102 recognizes and segments each target object included in the target image by using the target segmentation model and the target clustering algorithm.

[0121] The target segmentation model includes various models that can be used for image segmentation processing.

[0122] Optionally, the target segmentation model is a Segment Anything Model (SAM). The target model may be a variant model based on the SAM model, such as a SAM-Med2D model or a TinySAM model.

[0123] The target clustering algorithm includes various clustering algorithms that can be used to process cross-structures. Optionally, the target clustering algorithm is a spectral clustering on multiple manifolds (SMMC), which is a clustering algorithm mainly used for cross-manifold structures.

[0124] Specifically, the segmentation processing device 102 includes an object recognition device, a mask storage device, a skeleton extraction device, a quantity counting device, a line segmentation device and a category assignment device.

[0125] The object recognition device uses the target segmentation model to iteratively recognize the target objects contained in the target image until all the target objects are recognized.

[0126] The mask storage device stores the mask corresponding to the target object identified in each round into the target file.

[0127] The skeleton extraction device uses a skeleton extraction (Skeletonize) algorithm to perform skeleton extraction processing on the mask corresponding to the identified target object.

[0128] The skeleton extraction process refers to extracting lines connecting all local extreme points from the binary image as the "skeleton" of the object. The skeleton extraction device obtains all lines connecting all local extreme points in each identified target object as the "skeleton" of the target object by performing the skeleton extraction process.

[0129] The quantity counting device counts the number of endpoints based on each layer of mask obtained by skeleton extraction processing and determines the number of target objects contained in the mask based on a predetermined endpoint matching rule.

[0130] The line segmentation device uses a target clustering algorithm to segment the multiple skeletonized lines obtained by the skeleton extraction process.

[0131] Specifically, pixel sampling is performed on a plurality of skeletonized lines obtained by skeleton extraction processing to convert them into a dot matrix structure supported by the target clustering algorithm; then, the target clustering algorithm is called to segment the plurality of skeletonized lines by setting appropriate parameters and random seeds.

[0132] The category assignment device uses the segmented skeletonized lines as reference lines to assign corresponding categories to each pixel point of the original mask.

[0133] By assigning a corresponding category to each pixel of the original mask, the pixel-level classification of the original mask is achieved, thereby achieving the segmentation of each target object in the original mask.

[0134] Continue to refer to Figure 3To explain, the result generating means 103 generates segmentation result information of the target object corresponding to the target image.

[0135] The segmentation result includes but is not limited to the quantity information, contour information and position information of all target objects obtained through segmentation processing in the target image.

[0136] According to the device of the embodiment of the present application, the image segmentation processing is performed on the objects with overlapping slender structures in the image by combining the segmentation model and the clustering algorithm, and an excellent image segmentation effect is achieved.

[0137] Figure 4 A schematic diagram of the structure of a device for segmenting sperm images provided in an embodiment of the present application is shown.

[0138] The device includes: a device for acquiring a target sperm image to be processed (hereinafter referred to as "sperm image acquisition device 201"); a device for performing a first segmentation process on the target sperm image by using a target segmentation model to obtain sperm head data (hereinafter referred to as "first segmentation processing device 202"); a device for performing a head subtraction process on the target sperm image based on the sperm head data to obtain target tail data for second segmentation processing (hereinafter referred to as "head subtraction processing device 203"); a device for performing a second segmentation process on the target tail data by using the target model and a target clustering algorithm to obtain sperm tail data (hereinafter referred to as "second segmentation processing device 204"); based on the obtained sperm head data and sperm tail data, matching and splicing the head and tail of each sperm to obtain the segmentation result of each sperm (hereinafter referred to as "splicing processing device 205"); a device for generating segmentation result information of the target sperm image based on the segmentation result of each sperm (hereinafter referred to as "segmentation result generating device 206").

[0139] Reference Figure 4 , the sperm image acquisition device 201 acquires the target sperm image to be processed.

[0140] The sperm image data includes a plurality of sperms overlapping each other, for example, a sperm micrograph in a sperm detection scene.

[0141] Optionally, the sperm image acquisition device 201 collects sperm microphotographs and performs preliminary screening to exclude azoospermia images and images with severe color cast, and uses the sperm microphotographs after the preliminary screening as target sperm images.

[0142] Optionally, the device further comprises a sperm image preprocessing device.

[0143] The sperm image preprocessing device performs image preprocessing on the acquired target sperm image. The operation of the sperm image preprocessing device is similar to the operation of the aforementioned image preprocessing device, and will not be described in detail here.

[0144] The first segmentation processing means 202 performs a first segmentation process on the target sperm image by using a target segmentation model to obtain sperm head data.

[0145] The target segmentation model is a SAM model. The target segmentation model is the same or similar to the target segmentation model described in the segmentation processing device 102, and will not be described in detail here.

[0146] Specifically, the first segmentation processing device 202 includes a head recognition device and a head screening device.

[0147] The head recognition device recognizes the sperm heads contained in the target sperm image by using the target segmentation model to obtain head masks corresponding to all recognized sperm heads.

[0148] The head screening device screens out effective sperm heads based on the head mask file and according to predetermined screening conditions.

[0149] The screening condition is used to determine whether the identified sperm head is a valid sperm head.

[0150] According to one embodiment, the screening condition is set based on the intersection over union (IoU) of the sperm head region and the overall mask. Specifically, in step S2022, based on the head mask file, the color value range of the sperm head corresponding to the color of the sperm head is set; the IoU of the sperm head corresponding color region in each mask layer and the overall mask is calculated; and by comparing with a preset IoU threshold, the effective sperm head in the head mask is obtained.

[0151] Optionally, the first segmentation processing device 202 further includes a head mask storage device.

[0152] The head mask storage device stores the head mask corresponding to the effective sperm head.

[0153] Continue to refer to Figure 4 To explain, the head subtraction processing device 203 performs head subtraction processing on the target sperm image based on the sperm head data to obtain target tail data for performing the second segmentation processing.

[0154] According to the first example of the present application, it is assumed that the dyeing method used in the first example dyes the sperm head purple. For the sperm micrograph that has undergone image preprocessing, the first segmentation processing device 202 uses a predetermined SAM model to perform a first segmentation process to obtain a mask file of all first segmentation processes. Then, for the generated mask, the head screening device sets the purple HSV color value range and calculates the intersection-and-union ratio of the purple area in each layer of the mask to the overall mask. Then, based on the preset intersection-and-union ratio threshold, it is determined whether it is a valid sperm head, and the head mask storage device screens the valid sperm head into the header file for storage. The head subtraction processing device 203 cuts out the area corresponding to the sperm head in the original image to perform subsequent tail segmentation.

[0155] Optionally, the device further comprises an impurity processing device.

[0156] The impurity processing device performs impurity removal processing on the target sperm image after the head subtraction processing, so as to use the image data after the impurities are removed as the target tail data for the second segmentation processing.

[0157] Continuing to explain the first example above, after deducting the head, there are still many dye blocks that interfere with model recognition. The following rules are set to remove these dye blocks by the impurity processing device: For each layer of mask, find the two pixels a and b that are farthest apart in the mask, calculate the distance d between a and b, and calculate S=d^2, where S represents the theoretical maximum area containing points a and b, and s represents the actual area, and calculate q=s / S. The larger the q value, the closer it is to the mass structure rather than the slender sperm tail. After setting a suitable q value threshold to screen out the mass impurities and sperm tails that do not meet the requirements, a pure tail set is obtained for fine tail segmentation.

[0158] Continue to refer to Figure 4 To explain, the second segmentation processing means 204 performs the second segmentation processing on the target tail data by using the target model and the target clustering algorithm to obtain the sperm tail data.

[0159] Optionally, the target clustering algorithm is SMMC. The target clustering algorithm is the same as or similar to the target clustering algorithm described in step 102, and will not be described in detail here.

[0160] Specifically, the second segmentation processing device 204 includes a tail recognition device, a tail storage device, a tail skeleton extraction device, a sperm number counting device, a tail line segmentation device, and a tail category assignment device.

[0161] The tail recognition device uses the target segmentation model to iteratively recognize the sperm tails contained in the target tail data until all the sperm tails are recognized.

[0162] The tail storage device stores the tail mask image corresponding to the sperm tail identified in each round.

[0163] The tail skeleton extraction device uses a skeleton extraction algorithm to perform skeleton extraction processing on the tail mask image.

[0164] The sperm number counting device counts the number of endpoints based on each mask layer obtained by skeleton extraction processing and determines the number of sperm contained in the mask based on a predetermined endpoint matching rule.

[0165] The tail line segmentation device uses a target clustering algorithm to segment the multiple skeletonized lines obtained by the skeleton extraction process.

[0166] Specifically, the tail line segmentation device samples the pixels of the multiple skeletonized lines obtained by the skeleton extraction process and converts them into a dot matrix structure supported by the target clustering algorithm; then, by setting appropriate parameters and random seeds, the target clustering algorithm is called to segment the multiple skeletonized lines.

[0167] The tail category assignment device uses the segmented skeletonized lines as the reference line and assigns a corresponding category to each pixel point of the original mask, so as to achieve the segmentation of a single sperm tail in the original mask.

[0168] Continuing to explain the first example mentioned above, based on the pure tail set, the tail recognition device uses the SAM model to iteratively identify all the sperm tails in the image and save them in a mask format. In the result obtained in this step, many tails overlap, and the separation of individual sperm tails is not achieved. After each iteration, the position corresponding to the mask obtained in this step is erased in the original image, where the pixel points at the corresponding position of the mask are set to the background color white, and the edge afterimage is removed. Subsequently, the tail recognition device performs the next round of SAM model processing until all the sperm tails in the image are identified, and then the tail storage device summarizes the tail masks identified by the SAM model in all rounds and stores them in a tail file for the next step of single tail segmentation.

[0169] Next, the tail skeleton extraction device uses the Skeletonize algorithm to perform skeleton extraction on the tail mask. Subsequently, the sperm number counting device determines the number of endpoints of each layer of the skeletonized mask, selects the mask with a number of endpoints greater than 2 (i.e., a mask greater than one tail), and infers the number of sperm contained in the mask according to the number of endpoints. The predetermined pairing rule is: endpoint number 3, 4 - tail number 2; endpoint number 5, 6 - tail number 3, and so on. Subsequently, the tail line segmentation device samples the pixel points of the skeletonized result, converts it into a lattice structure supported by the clustering algorithm, sets appropriate parameters and random seeds, and calls the clustering algorithm to segment the skeletonized lines. The tail category assignment device uses the segmented skeletonized lines as the baseline to assign categories to each pixel of the original mask, thereby realizing the pixel-level classification of the original mask, that is, realizing the splitting of a single sperm tail.

[0170] Continue to refer to Figure 4 To explain, the splicing processing device 205 matches and splices the head and tail of each sperm based on the obtained sperm head data and sperm tail data to obtain the segmentation result of each sperm.

[0171] Specifically, the splicing processing device 205 obtains the long axis of each sperm head based on the sperm head data obtained by the first segmentation processing; then, based on the sperm tail data obtained by the second segmentation processing, and in accordance with the preset front and rear end determination rules, obtains the front end and rear end of each sperm tail; then, according to the preset geometric rules, matches the front end skeletonized straight line of the sperm tail with the long axis of the sperm head to obtain matching sperm heads and tails; then, splices the matching sperm heads and tails to obtain the segmentation results of each sperm.

[0172] Continuing to explain the first example, the splicing processing device 205 determines the front and rear ends of each sperm tail segmented in the previous step by the color change trend and the mask width, and straightens the front skeleton line. Then find the long axis of the sperm head segmented before, match the tail skeleton straight line with the head long axis according to the geometric rules in terms of distance and angle, pair the head and tail, and then splice the corresponding head and tail to obtain a complete sperm segmentation result.

[0173] The segmentation result generating means 206 generates segmentation result information of the target sperm image based on the segmentation results of each sperm.

[0174] The segmentation result information includes but is not limited to the number of sperms contained in the target sperm image, the outline and size of each sperm contained therein, and the position information in the target image, etc.

[0175] According to one embodiment, the device further comprises a model training device.

[0176] The model training device trains a classification model for sperm morphology classification based on the segmentation result information of multiple sperm images and corresponding annotation data.

[0177] Specifically, data annotations corresponding to the sperm morphology classification problem are obtained from medical experts; then, the model training device uses deep learning methods such as semi-supervised learning to train the classification model to achieve a binary classification of qualified / unqualified sperm morphology after segmentation.

[0178] According to one embodiment, the device further comprises a detection and analysis device.

[0179] The detection and analysis device uses the trained classification model to perform sperm eligibility analysis on the patient's sperm micrographs.

[0180] For example, the detection and analysis device uses the trained classification model to perform eligibility analysis on each sperm after segmentation, and counts the number of sperm with normal morphology and sperm with abnormal morphology in multiple sperm micrographs of a patient. Based on the counted number, the sperm morphology analysis result of the patient is calculated.

[0181] According to the device of the embodiment of the present application, by combining the segmentation model and the clustering algorithm, image segmentation processing is performed on sperm micrographs in the sperm detection scenario, and the overlapping sperm tails are accurately identified and segmented. The entire process does not require human intervention, and automatic sperm segmentation is achieved, which greatly improves the efficiency of sperm segmentation in micrographs and provides great convenience for sperm morphology detection. The method of the embodiment of the present application can be combined with the annotation data of medical experts to judge whether the sperm morphology is normal or abnormal, and obtain corresponding test results, providing a convenient and efficient operation process and accurate and reliable results for the clinical diagnosis of male sperm morphology analysis, which is expected to completely change the current dilemma of sperm morphology relying on manual analysis, and play an important role in clinical diagnosis of andrology.

[0182] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method for segmenting an image in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0183] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0184] Figure 5 The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0185] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0186] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0187] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0188] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0189] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0190] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0191] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0192] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0194] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0195] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0197] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0199] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for segmenting a sperm image, the method comprising: Acquire a target sperm image to be processed; Performing a first segmentation process on the target sperm image by using a target segmentation model to obtain sperm head data; Performing head subtraction processing on the target sperm image based on the sperm head data to obtain target tail data for performing a second segmentation processing; Performing a second segmentation process on the target tail data by using the target segmentation model and the target clustering algorithm to obtain sperm tail data; Based on the obtained sperm head data and sperm tail data, the head and tail of each sperm are matched and spliced ​​to obtain the segmentation results of each sperm; Based on the segmentation results of each sperm, generating segmentation result information of the target sperm image; Wherein, performing a second segmentation process on the target tail data by using the target segmentation model and the target clustering algorithm to obtain the sperm tail data includes: Using the target segmentation model to iteratively identify the sperm tails contained in the target tail data until all the sperm tails are identified; The tail mask image corresponding to the sperm tail identified in each round is stored; Performing skeleton extraction processing on the tail mask image using a skeleton extraction algorithm; Based on each layer of mask obtained by skeleton extraction processing, the number of endpoints is counted and the number of sperm contained in the mask is determined based on a predetermined endpoint matching rule; Using a target clustering algorithm to segment multiple skeletonized lines obtained by skeleton extraction; The skeletonized lines after segmentation are used as the baseline to assign corresponding categories to each pixel of the original mask, so as to realize the segmentation of the single sperm tail in the original mask.

2. The method according to claim 1, wherein: The step of performing a first segmentation process on the target sperm image by using the target segmentation model to obtain sperm head data comprises: Using the target segmentation model to identify the sperm heads contained in the target sperm image, and obtaining head masks corresponding to all identified sperm heads; Based on the head mask file, the effective sperm heads are screened out according to the predetermined screening conditions.

3. The method according to claim 2, wherein: The method of screening out effective sperm heads according to predetermined screening conditions includes: Based on the head mask file, set the color value range of the sperm head; Calculate the intersection-over-union ratio of the color area corresponding to the sperm head in each mask layer and the overall mask; By comparing with a preset intersection-over-union ratio threshold, the effective sperm head in the head mask is obtained.

4. The method according to claim 1, wherein: The matching and splicing processing of the heads and tails of each sperm based on the obtained sperm head data and sperm tail data includes: Based on the sperm head data obtained by the first segmentation process, obtaining the long axis of each sperm head; Based on the sperm tail data obtained by the second segmentation process, and in accordance with a preset front and rear end determination rule, the front end and rear end of each sperm tail are obtained; According to the preset geometric rules, the front skeleton straight line of the sperm tail is matched with the long axis of the sperm head to obtain the matching heads and tails of each sperm; The matching heads and tails of each sperm are spliced ​​together to obtain the segmentation results of each sperm.

5. The method according to any one of claims 1 to 4, further comprising: Based on the segmentation result information of multiple sperm images and the corresponding annotation data, a classification model for sperm morphology classification is trained.

6. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

7. A computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be used by a processor to execute the method according to any one of claims 1 to 5.

Images