Cell image segmentation method and device, electronic equipment and storage medium
By acquiring and superimposing multiple microscope images, combining key point detection and watershed segmentation strategies, the problem of unclear image edges of blood tumor cells is solved, the accuracy of diagnosis is improved, and clear image support is provided for individualized treatment.
Patent Information
- Application Number
- CN202410995552.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-12-10
AI Technical Summary
There are insufficient accurate diagnosis of hematologic tumors and monitoring of micro-residual diseases, resulting in unclear cell image edges and inaccurate positioning and identification results.
By acquiring multiple microscope images at the same shooting angle, image overlay and segmentation are performed, and the cell images are segmented to improve edge clarity by combining key point detection and watershed segmentation strategies.
It improves the problem of unclear edges of cell images, improves the accuracy of cell positioning and recognition, and provides a clear image basis for individualized treatment plans.
Smart Images
Figure CN119107324A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cell detection technology, and in particular to a cell image segmentation method, device, electronic equipment and storage medium. Background Art
[0002] Hematologic malignancies are one of the top ten most common malignancies in my country. They are usually caused by the abnormal proliferation of leukemia cells, mainly due to the malignant cloning of hematopoietic stem cells, uncontrolled proliferation, differentiation disorders, and blocked apoptosis. They proliferate and accumulate in large quantities in the bone marrow and other hematopoietic tissues, and infiltrate other non-hematopoietic tissues and organs, while inhibiting normal hematopoietic function. At present, the treatment methods for hematologic malignancies are advanced, including hematopoietic stem cell transplantation and CAR-T cell therapy, and the clinical treatment effect has been significantly improved. However, there are still many deficiencies in the accurate diagnosis of hematologic malignancies and the monitoring of minimal residual disease, which brings confusion to the selection of individualized treatment plans and directly affects the cure rate and survival rate of patients.
[0003] For the experimental diagnosis of hematological tumors, cell morphology, histochemical staining and fluorescence in situ hybridization (FISH) are commonly used detection methods, each with its own advantages. Cell morphology mainly determines the benign or malignant nature of diseased cells by heterogeneous changes in cell morphology, histochemical staining analyzes the immune phenotype of cells by antigen expression of abnormal cells, and FISH technology determines the displacement or deletion of chromosomes by in situ hybridization. However, in many cases, such as when the number of tumor cells is small and the immune phenotype is abnormal and unclear, the diagnostic directions of these three technologies are contradictory, which brings great confusion or interference to clinical diagnosis.
[0004] Therefore, it is possible to accurately locate abnormal cells by positioning through multiple images and superimposing various positioning images. However, in practical applications, the superimposed cell images usually have unclear edges, resulting in inaccurate positioning and recognition results. Summary of the invention
[0005] In view of this, the purpose of the embodiments of the present application is to provide a cell image segmentation method, device, electronic device and storage medium, which can improve the problem that superimposed cell images usually have unclear edges, resulting in inaccurate positioning and recognition results.
[0006] In order to achieve the above technical objectives, the technical solutions adopted in this application are as follows:
[0007] In a first aspect, an embodiment of the present application provides a cell image segmentation method, the method comprising:
[0008] Acquiring multiple initial images, wherein the initial images represent microscopic images of the same FISH sample taken under different microscopes at the same shooting angle;
[0009] Superimposing a plurality of the initial images to obtain an image to be inspected;
[0010] Performing image segmentation on the image to be inspected to obtain a segmentation result image, wherein the segmentation result image includes the contour of the cell to be inspected;
[0011] Performing key point detection on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected;
[0012] According to the segmentation result image and the positioning image, determining that a contour area in the segmentation result image, in which the number of the center points is greater than 1, is a target area;
[0013] According to the number and distribution of the center points of the target area in the positioning image, the target area in the segmentation result image is segmented into multiple sub-areas by a preset watershed segmentation strategy, each of which includes a cell to be tested that only contains one center point, thereby obtaining a target image.
[0014] In combination with the first aspect, in some optional implementations, performing image segmentation on the image to be inspected to obtain the segmentation result image includes:
[0015] Performing a first convolution operation on the image to be inspected to obtain a first feature map;
[0016] Performing four downsampling operations on the first feature map in sequence to extract semantic information in the image to be inspected, and obtaining a semantic feature map corresponding to each downsampling operation;
[0017] The semantic feature map obtained by the last downsampling operation is used as the first target feature map, and the first target feature map is sequentially upsampled four times to extract spatial information in the image to be inspected, so as to obtain a spatial feature map corresponding to each upsampling operation;
[0018] The spatial feature map obtained by the last upsampling operation is used as the second target feature map, and a second convolution operation is performed on the second target feature map to obtain the segmentation result image.
[0019] In combination with the first aspect, in some optional implementations, four downsampling operations are sequentially performed on the first feature map to extract semantic information in the image to be inspected, and a semantic feature map corresponding to each downsampling operation is obtained, including:
[0020] A1321, taking the first feature map as the feature map to be pooled, and performing a pooling operation on the feature map to be pooled to obtain a feature map of size Figure 1 Half the downsampled feature map;
[0021] A1322, performing a first convolution operation on the downsampled feature map to extract the semantic information in the downsampled feature map to obtain a second feature map;
[0022] A1323. Perform batch normalization on the second feature map to obtain a first normalization result, and output the result through a first activation function;
[0023] A1324. Perform a second convolution operation on the first normalized result output by the first activation function to obtain a third feature map;
[0024] A1325. Perform batch normalization on the third feature map to obtain a second normalization result;
[0025] A1326, splicing the first normalized result and the second normalized result in a channel dimension to fuse the semantic information respectively extracted from the first convolution operation and the second convolution operation, and outputting the spliced first normalized result and the second normalized result via a second activation function to obtain the semantic feature map;
[0026] A1327. Use the semantic feature map as a new feature map to be pooled, and loop the above steps A1321 to A1326 three times to obtain the semantic feature map corresponding to each loop.
[0027] In combination with the first aspect, in some optional implementations, the semantic feature map obtained by the last downsampling operation is used as the first target feature map, and the first target feature map is sequentially upsampled four times to extract spatial information in the image to be inspected, and a spatial feature map corresponding to each upsampling operation is obtained, including:
[0028] A1331, using the first target feature map as a first feature map to be convolved, and performing a deconvolution operation on the first feature map to be convolved to obtain an upsampled feature map whose size is twice that of the first target feature map;
[0029] A1332, determining an attention weight according to a current up-sampled feature map and the semantic feature map having the same size as the current up-sampled feature map;
[0030] A1333, weighting the semantic feature map having the same size as the current up-sampled feature map by the attention weight to obtain a weighted feature map, and concatenating the weighted feature map with the current up-sampled feature map in terms of channel dimension to obtain a second feature map to be convolved;
[0031] A1334, continuously performing two convolution operations on the second feature map to be convolved to extract spatial information in the second feature map to be convolved to obtain the spatial feature map;
[0032] A1335. Use the spatial feature map as the new first feature map to be convolved, and loop the above steps A1331 to A1334 three times to obtain the spatial feature map corresponding to each loop.
[0033] In combination with the first aspect, in some optional implementations, determining the attention weight according to the current upsampled feature map and the semantic feature map having the same size as the current upsampled feature map includes:
[0034] According to the current up-sampled feature map and the semantic feature map of the same size as the current up-sampled feature map, the attention weight is determined by the following formula:
[0035]
[0036] In the formula, represents the attention weight, σ 1 represents the ReLU activation function, σ 2 represents the Sigmoid activation function, b 1 、b 2 represents the bias term, W, W x , W g represents the linear transformation parameters, represents a semantic feature map of the same size as the current upsampled feature map, Represents the current upsampled feature map.
[0037] In combination with the first aspect, in some optional implementations, key point detection is performed on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected, including:
[0038] By using a preset key point detection model, key point detection is performed on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected.
[0039] In combination with the first aspect, in some optional implementations, the preset key point detection model is an hourglass network.
[0040] In a second aspect, an embodiment of the present application further provides a cell image segmentation device, the device comprising:
[0041] An acquisition unit, used to acquire a plurality of initial images, wherein the initial images represent microscopic images obtained by photographing the same FISH sample under different microscopes at the same shooting angle;
[0042] A superposition unit, used for superimposing a plurality of the initial images to obtain an image to be inspected;
[0043] A first segmentation unit, used for performing image segmentation on the image to be inspected to obtain a segmentation result image, wherein the segmentation result image includes the contour of the cell to be inspected;
[0044] A key point positioning unit, used for performing key point detection on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected;
[0045] a determining unit, configured to determine, according to the segmentation result image and the positioning image, a contour region in which the number of the center points is greater than 1 in the segmentation result image as a target region;
[0046] The second segmentation unit is used to segment the target area in the segmentation result image into multiple sub-areas according to the number and distribution of the center points of the target area in the positioning image by a preset watershed segmentation strategy, each of which includes a cell to be tested that only contains one center point, so as to obtain a target image.
[0047] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory coupled to each other, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the electronic device executes the above method.
[0048] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed on a computer, the computer executes the above method.
[0049] The invention adopting the above technical solution has the following advantages:
[0050] In the technical solution provided by the present application, a plurality of microscopic images obtained by shooting the same FISH sample under different microscopes at the same shooting angle are first obtained as the initial image, and then the plurality of initial images are superimposed to obtain the image to be inspected, and the image to be inspected is segmented to obtain the segmentation result image, and then the key point detection is performed on the segmentation result image to obtain a positioning image representing the center point of the cell to be inspected, and based on the segmentation result image and the positioning image, the contour area with the number of center points greater than 1 in the segmentation result image is determined as the target area, and finally, based on the number and distribution of the center points of the target area in the positioning image, the target area in the segmentation result image is segmented into a plurality of sub-areas by a preset watershed segmentation strategy, each of which includes a cell to be inspected containing only one center point, to obtain the target image. In this way, the problem that the superimposed cell images usually have unclear edges leading to inaccurate positioning and recognition results can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] The present application may be further described by the non-limiting embodiments given in the accompanying drawings. It should be understood that the following drawings only illustrate certain embodiments of the present application and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings may be obtained based on these drawings without creative effort.
[0052] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0053] Figure 2 A schematic diagram of the process of cell image segmentation provided in an embodiment of the present application.
[0054] Figure 3 A schematic diagram of the structure of the image segmentation model provided in an embodiment of the present application.
[0055] Figure 4 A schematic diagram of the downsampling operation flow provided in an embodiment of the present application.
[0056] Figure 5 This is a structural block diagram of the cell image segmentation device provided in an embodiment of the present application.
[0057] Icon: 100 - electronic device; 101 - processor; 102 - memory; 200 - cell image segmentation device; 210 - acquisition unit; 220 - superposition unit; 230 - first segmentation unit; 240 - key point positioning unit; 250 - determination unit; 260 - second segmentation unit. DETAILED DESCRIPTION
[0058] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that in the drawings or descriptions, similar or identical parts use the same figure numbers, and the implementation methods not shown or described in the drawings are forms known to ordinary technicians in the relevant technical field. In the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0059] Please refer to Figure 1 , an embodiment of the present application provides an electronic device 100 that may include a processor 101 and a memory 102. The memory 102 stores a computer program, and when the computer program is executed by the processor 101, the electronic device 100 can perform the corresponding steps in the following cell image segmentation method.
[0060] In this embodiment, the electronic device 100 can be a personal computer, a laptop computer, a cloud server, etc. It is used to obtain multiple initial images, and then superimpose the multiple initial images to obtain an image to be inspected, and perform image segmentation on the image to be inspected to obtain a segmentation result image. Then, key point detection is performed on the segmentation result image to obtain a positioning image representing the center point of the cell to be inspected, and according to the segmentation result image and the positioning image, the contour area with the number of center points greater than 1 in the segmentation result image is determined as the target area. Finally, by using a preset watershed segmentation strategy, the target area in the segmentation result image is segmented into multiple sub-areas, each of which includes a cell to be inspected that contains only one center point, to obtain a target image.
[0061] In this way, a clear division of cell edges is achieved in the cell microscopic image (i.e., the initial image), providing a clear image basis for further cell identification and pathological judgment, and improving the problem of inaccurate positioning and identification results caused by unclear edges in superimposed cell images.
[0062] Please refer to Figure 2 The present application also provides a cell image segmentation method, which can be applied to the above electronic device 100, and each step in the method is executed or implemented by the electronic device 100. The cell image segmentation method can include the following steps:
[0063] Step 110, acquiring a plurality of initial images, wherein the initial images represent microscopic images of the same FISH sample taken under different microscopes at the same shooting angle;
[0064] Step 120, superimposing the multiple initial images to obtain an image to be inspected;
[0065] Step 130, performing image segmentation on the image to be inspected to obtain a segmentation result image, wherein the segmentation result image includes the contour of the cell to be inspected;
[0066] Step 140, performing key point detection on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected;
[0067] Step 150, determining, according to the segmentation result image and the positioning image, a contour region in the segmentation result image where the number of the center points is greater than 1 as a target region;
[0068] Step 160, according to the number and distribution of the center points of the target area in the positioning image, by using a preset watershed segmentation strategy, the target area in the segmentation result image is segmented into multiple sub-areas, each of which includes a cell to be tested that only contains one center point, to obtain a target image.
[0069] In the above-mentioned implementation, a plurality of microscopic images of the same FISH sample taken under different microscopes at the same shooting angle are first obtained as the initial image, and then the plurality of initial images are superimposed to obtain the image to be inspected, and the image to be inspected is segmented to obtain the segmentation result image, and then the segmentation result image is subjected to key point detection to obtain a positioning image representing the center point of the cell to be inspected, and based on the segmentation result image and the positioning image, the contour area in the segmentation result image with the number of center points greater than 1 is determined as the target area, and finally, based on the number and distribution of the center points of the target area in the positioning image, the target area in the segmentation result image is segmented into a plurality of sub-areas by a preset watershed segmentation strategy, each of which includes a cell to be inspected that contains only one center point, to obtain the target image. In this way, the problem that the superimposed cell images usually have unclear edges leading to inaccurate positioning and recognition results can be improved.
[0070] The following is a detailed description of the various steps of the cell image segmentation method, as follows:
[0071] In step 110, the preparation of FISH samples can be performed by first labeling the bone marrow cells with immunohistochemical antibodies or immunofluorescent antibodies, and then breaking the membrane of the labeled bone marrow cells, fixing them, and staining them with FISH labeling antibodies to obtain FISH samples.
[0072] The FISH samples are then photographed and imaged using a conventional microscope and a fluorescence microscope, respectively, so that the collected images can be superimposed and located for identification, ultimately achieving the purpose of determining whether there are genetic abnormalities in the cell source.
[0073] In this embodiment, dual localization is performed by combining immunohistochemical antibodies or immunofluorescent antibodies with FISH technology to more accurately locate specific chromosomal abnormalities in cells.
[0074] In step 120, after the microscopic images corresponding to the FISH samples under different microscopes are collected in step 110, a plurality of microscopic images are used as initial images, and the initial images are physically superimposed to obtain an image to be inspected.
[0075] In step 130, performing image segmentation on the image to be inspected to obtain the segmentation result image may include:
[0076] Performing a first convolution operation on the image to be inspected to obtain a first feature map;
[0077] Performing four downsampling operations on the first feature map in sequence to extract semantic information in the image to be inspected, and obtaining a semantic feature map corresponding to each downsampling operation;
[0078] The semantic feature map obtained by the last downsampling operation is used as the first target feature map, and the first target feature map is sequentially upsampled four times to extract spatial information in the image to be inspected, so as to obtain a spatial feature map corresponding to each upsampling operation;
[0079] The spatial feature map obtained by the last upsampling operation is used as the second target feature map, and a second convolution operation is performed on the second target feature map to obtain the segmentation result image.
[0080] In this embodiment, the first convolution operation and the second convolution operation may both be 3×3 convolutions (convolution operations with a convolution kernel of 3×3).
[0081] In this embodiment, the image to be inspected is segmented to obtain a segmentation result image containing the contour of the cell to be inspected. The segmentation of the image to be inspected can be achieved by an image segmentation model. Figure 3 , the structure of the image segmentation model can be as follows Figure 3 As shown, on the basis of the traditional U-net model, the splicing of the channel dimension is added in the downsampling stage of the encoder, and the attention weight is added to the original splicing path of the encoder and decoder in the U-net, so as to adjust the output characteristics of the encoder, enhance the difference between the cells to be detected and the background in the image segmentation process, weaken the influence of factors such as the shape of the cells to be detected and pixel blur on the image segmentation effect, and improve the segmentation effect of the contour of the cells to be detected. The specific execution steps of the image segmentation model are shown in step 130 above, which will not be repeated here.
[0082] In this embodiment, four downsampling operations are sequentially performed on the first feature map to extract semantic information in the image to be inspected, and obtaining a semantic feature map corresponding to each downsampling operation may include:
[0083] A1321, taking the first feature map as the feature map to be pooled, and performing a pooling operation on the feature map to be pooled to obtain a feature map of size Figure 1 Half the downsampled feature map;
[0084] A1322, performing a first convolution operation on the downsampled feature map to extract the semantic information in the downsampled feature map to obtain a second feature map;
[0085] A1323. Perform batch normalization on the second feature map to obtain a first normalization result, and output the result through a first activation function;
[0086] A1324. Perform a second convolution operation on the first normalized result output by the first activation function to obtain a third feature map;
[0087] A1325. Perform batch normalization on the third feature map to obtain a second normalization result;
[0088] A1326, splicing the first normalized result and the second normalized result in a channel dimension to fuse the semantic information respectively extracted from the first convolution operation and the second convolution operation, and outputting the spliced first normalized result and the second normalized result via a second activation function to obtain the semantic feature map;
[0089] A1327. Use the semantic feature map as a new feature map to be pooled, and loop the above steps A1321 to A1326 three times to obtain the semantic feature map corresponding to each loop.
[0090] In this embodiment, the first convolution operation and the second convolution operation may both be 3×3 convolutions, and the first activation function and the second activation function may both be ReLU (recti-fied linear unit).
[0091] Specifically, this embodiment performs four downsampling operations on the first feature map, and uses the feature map obtained after each downsampling operation as the semantic feature map. The execution logic of each downsampling operation is the same, and the difference is only that the input and output results are different. Taking the first downsampling operation as an example, the first feature map is used as the image to be pooled, and the pooling operation (which can be the maximum pooling) is performed on the image to be pooled to obtain a feature map of size 1000. Figure 1 Then refer to Figure 4, perform 3×3 convolution on the downsampled feature map to extract the semantic information and obtain the second feature map. Then perform batch normalization on the second feature map to obtain the first normalization result, and then perform a second 3×3 convolution on the result after passing the ReLU activation function to obtain the third feature map. Then perform batch normalization on the third feature map again to obtain the second normalization result, concatenate the first normalization result and the second normalization result in the channel dimension, and output the concatenated result through the ReLU activation function to obtain the semantic feature map corresponding to the first downsampling operation.
[0092] For the subsequent three downsampling operations, it is only necessary to use the semantic feature map obtained in the previous downsampling operation as the new image to be pooled and repeat the above steps.
[0093] In this way, through two convolutions and splicing, the features between the upper and lower convolutional layers in the model are fused, which reduces the gradient disappearance in the model, improves the reuse rate of features, and is conducive to extracting more feature information of the cells to be tested.
[0094] In this embodiment, the semantic feature map obtained by the last downsampling operation is used as the first target feature map, and the first target feature map is sequentially upsampled four times to extract the spatial information in the image to be inspected, and the spatial feature map corresponding to each upsampling operation is obtained, which may include:
[0095] A1331, using the first target feature map as a first feature map to be convolved, and performing a deconvolution operation on the first feature map to be convolved to obtain an upsampled feature map whose size is twice that of the first target feature map;
[0096] A1332, determining an attention weight according to a current up-sampled feature map and the semantic feature map having the same size as the current up-sampled feature map;
[0097] A1333, weighting the semantic feature map having the same size as the current up-sampled feature map by the attention weight to obtain a weighted feature map, and concatenating the weighted feature map with the current up-sampled feature map in terms of channel dimension to obtain a second feature map to be convolved;
[0098] A1334, continuously performing two convolution operations on the second feature map to be convolved to extract spatial information in the second feature map to be convolved to obtain the spatial feature map;
[0099] A1335. Use the spatial feature map as the new first feature map to be convolved, and loop the above steps A1331 to A1334 three times to obtain the spatial feature map corresponding to each loop.
[0100] In this embodiment, this embodiment performs four upsampling operations on the first target feature map, and uses the feature map obtained after each upsampling operation as a spatial feature map. Among them, the execution logic of each upsampling operation is the same, and the difference is only that the input and output results are different. Taking the first upsampling operation as an example, the first target feature map is used as the feature map to be convolved, and the deconvolution operation is performed on the feature map to be convolved to obtain an upsampled feature map twice the size of the first target feature map. Then, according to the upsampled feature map and the semantic feature map of the same size as the upsampled feature map, the attention weight is calculated. After obtaining the attention weight, the semantic feature map of the same size as the current upsampled feature map is weighted by the attention weight, and the weighted result is spliced with the current upsampled feature map in the channel dimension to obtain the second feature map to be convolved. Then, the second feature map to be convolved is continuously subjected to two 3×3 convolution operations to obtain the spatial feature map corresponding to the first upsampling operation. Then, the spatial feature map corresponding to the first upsampling operation is used as the new first feature map to be convolved, and the above steps are repeated three times to complete the four upsampling operations.
[0101] In this way, on the basis of traditional U-net splicing, by weighting the semantic feature map in the encoder and splicing the weighted result with the upsampled feature map in the decoder, the output features of the encoder are adjusted, the difference between the cells to be inspected and the background in the image segmentation process is enhanced, the influence of factors such as the shape of the cells to be inspected and pixel blur on the image segmentation effect is weakened, and the segmentation effect of the contour of the cells to be inspected is improved.
[0102] In this embodiment, determining the attention weight according to the current up-sampled feature map and the semantic feature map having the same size as the current up-sampled feature map may include:
[0103] According to the current up-sampled feature map and the semantic feature map of the same size as the current up-sampled feature map, the attention weight is determined by the following formula:
[0104]
[0105] In the formula, represents the attention weight, σ 1 represents the ReLU activation function, σ 2 represents the Sigmoid activation function, b 1 、b 2 represents the bias term, W, W x , W g represents the linear transformation parameters, represents a semantic feature map of the same size as the current upsampled feature map, Represents the current upsampled feature map.
[0106] In step 140, key point detection is performed on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected, which may include:
[0107] By using a preset key point detection model, key point detection is performed on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected.
[0108] In this embodiment, the preset key point detection model can be an hourglass network. By detecting the center point of the cell to be detected in the segmentation result image through the hourglass network, the annotation workload in the model training process can be reduced, and a more efficient model learning solution can be provided. In addition, since the model only needs to focus on the key points during the training and detection process, and does not need to focus on all the information and features of the entire image or region, it has stronger fault tolerance and higher accuracy.
[0109] In step 150, after the contours and center points of the cells to be tested are obtained by segmentation, each closed contour area can be regarded as a connected domain, and then the connected domain containing more than 1 center points can be regarded as the target area, so as to screen out the cells to be tested that are stacked, overlapped, or have unclear edges, so as to facilitate subsequent secondary segmentation.
[0110] In step 160, the number and distribution of center points in the target area relocation image are used as prior information, and the segmentation result image is segmented by a watershed segmentation algorithm, thereby dividing the target area into multiple sub-areas divided by edge lines, each sub-area contains only one cell to be tested, and the cell to be tested contains only one center point.
[0111] In this way, accurate segmentation of cell images is achieved, providing clear image references for abnormalities and pathological judgments of cell origin.
[0112] Please refer to Figure 5 The present application also provides a cell image segmentation device 200, which includes at least one software function module that can be stored in the memory 102 in the form of software or firmware or fixed in the operating system (OS) of the electronic device 100. The processor 101 is used to execute the executable modules stored in the memory 102, such as the software function modules and computer programs included in the cell image segmentation device 200.
[0113] The cell image segmentation device 200 includes an acquisition unit 210, a superposition unit 220, a first segmentation unit 230, a key point positioning unit 240, a determination unit 250, and a second segmentation unit 260. The functions of each unit may be as follows:
[0114] An acquisition unit 210 is used to acquire a plurality of initial images, wherein the initial images represent microscopic images obtained by photographing the same FISH sample under different microscopes at the same shooting angle;
[0115] A superposition unit 220, used for superimposing a plurality of the initial images to obtain an image to be inspected;
[0116] A first segmentation unit 230 is used to perform image segmentation on the image to be inspected to obtain a segmentation result image, wherein the segmentation result image includes the contour of the cell to be inspected;
[0117] A key point positioning unit 240 is used to perform key point detection on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected;
[0118] A determination unit 250 is used to determine, according to the segmentation result image and the positioning image, that a contour region in which the number of the center points is greater than 1 in the segmentation result image is a target region;
[0119] The second segmentation unit 260 is used to segment the target area in the segmentation result image into multiple sub-areas according to the number and distribution of the center points of the target area in the positioning image by a preset watershed segmentation strategy, each of which includes a cell to be tested that only contains one center point, to obtain a target image.
[0120] Optionally, the first segmentation unit 230 is further configured to:
[0121] Performing a first convolution operation on the image to be inspected to obtain a first feature map;
[0122] Performing four downsampling operations on the first feature map in sequence to extract semantic information in the image to be inspected, and obtaining a semantic feature map corresponding to each downsampling operation;
[0123] The semantic feature map obtained by the last downsampling operation is used as the first target feature map, and the first target feature map is sequentially upsampled four times to extract spatial information in the image to be inspected, so as to obtain a spatial feature map corresponding to each upsampling operation;
[0124] The spatial feature map obtained by the last upsampling operation is used as the second target feature map, and a second convolution operation is performed on the second target feature map to obtain the segmentation result image.
[0125] Optionally, the first segmentation unit 230 is further configured to:
[0126] A1321, taking the first feature map as the feature map to be pooled, and performing a pooling operation on the feature map to be pooled to obtain a feature map of size Figure 1 Half the downsampled feature map;
[0127] A1322, performing a first convolution operation on the downsampled feature map to extract the semantic information in the downsampled feature map to obtain a second feature map;
[0128] A1323. Perform batch normalization on the second feature map to obtain a first normalization result, and output the result through a first activation function;
[0129] A1324. Perform a second convolution operation on the first normalized result output by the first activation function to obtain a third feature map;
[0130] A1325. Perform batch normalization on the third feature map to obtain a second normalization result;
[0131] A1326, splicing the first normalized result and the second normalized result in a channel dimension to fuse the semantic information respectively extracted from the first convolution operation and the second convolution operation, and outputting the spliced first normalized result and the second normalized result via a second activation function to obtain the semantic feature map;
[0132] A1327. Use the semantic feature map as a new feature map to be pooled, and loop the above steps A1321 to A1326 three times to obtain the semantic feature map corresponding to each loop.
[0133] Optionally, the first segmentation unit 230 is further configured to:
[0134] A1331, using the first target feature map as a first feature map to be convolved, and performing a deconvolution operation on the first feature map to be convolved to obtain an upsampled feature map whose size is twice that of the first target feature map;
[0135] A1332, determining an attention weight according to a current up-sampled feature map and the semantic feature map having the same size as the current up-sampled feature map;
[0136] A1333, weighting the semantic feature map having the same size as the current up-sampled feature map by the attention weight to obtain a weighted feature map, and concatenating the weighted feature map with the current up-sampled feature map in terms of channel dimension to obtain a second feature map to be convolved;
[0137] A1334, continuously performing two convolution operations on the second feature map to be convolved to extract spatial information in the second feature map to be convolved to obtain the spatial feature map;
[0138] A1335. Use the spatial feature map as the new first feature map to be convolved, and loop the above steps A1331 to A1334 three times to obtain the spatial feature map corresponding to each loop.
[0139] Optionally, the first segmentation unit 230 is further configured to:
[0140] According to the current up-sampled feature map and the semantic feature map of the same size as the current up-sampled feature map, the attention weight is determined by the following formula:
[0141]
[0142]
[0143] In the formula, represents the attention weight, σ 1 represents the ReLU activation function, σ 2 represents the Sigmoid activation function, b 1 , b 2 represents the bias term, W, W x , W g represents the linear transformation parameters, represents a semantic feature map of the same size as the current upsampled feature map, Represents the current upsampled feature map.
[0144] Optionally, the key point detection unit is further used to:
[0145] By using a preset key point detection model, key point detection is performed on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected.
[0146] Optionally, the preset key point detection model is an hourglass network.
[0147] In this embodiment, the processor 101 may be an integrated circuit chip having signal processing capabilities. The processor 101 may be a general-purpose processor. For example, the processor 101 may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and may implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application.
[0148] The memory 102 may be, but is not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc. In this embodiment, the memory 102 may be used to store an initial image, an image to be inspected, a segmentation result image, a positioning image, a target image, etc. Of course, the memory 102 may also be used to store a program, and the processor 101 executes the program after receiving an execution instruction.
[0149] Understandably, Figure 1 The structure of the electronic device 100 shown in FIG. 1 is only a schematic diagram of a structure. The electronic device 100 may also include Figure 1 More components are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0150] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device 100 described above can refer to the corresponding process of each step in the aforementioned method, and will not be elaborated herein.
[0151] The embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is run on a computer, the computer executes the cell image segmentation method described in the above embodiment.
[0152] Through the description of the above implementation methods, technical personnel in this field can clearly understand that the present application can be implemented by hardware, and can also be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.
[0153] In summary, the embodiments of the present application provide a cell image segmentation method, device, electronic device 100 and storage medium. In the present technical solution, firstly, a plurality of microscopic images obtained by shooting the same FISH sample under different microscopes at the same shooting angle are obtained as the initial image, and then the plurality of initial images are superimposed to obtain the image to be inspected, and the image to be inspected is segmented to obtain the segmentation result image, and then the segmentation result image is subjected to key point detection to obtain a positioning image representing the center point of the cell to be inspected, and according to the segmentation result image and the positioning image, the contour area in the segmentation result image with the number of center points greater than 1 is determined as the target area, and finally, according to the number and distribution of the center points of the target area in the positioning image, the target area in the segmentation result image is segmented into a plurality of sub-areas by a preset watershed segmentation strategy, each sub-area includes a cell to be inspected containing only one center point, and the target image is obtained. In this way, the problem that the superimposed cell images usually have unclear edges leading to inaccurate positioning and recognition results can be improved.
[0154] In the embodiments provided in the present application, it should be understood that the disclosed device, system and method can also be implemented in other ways. The above-described device, system and method embodiments are merely schematic, for example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.
[0155] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A cell image segmentation method, characterized in that: The method comprises: Acquiring multiple initial images, wherein the initial images represent microscopic images of the same FISH sample taken under different microscopes at the same shooting angle; Superimposing a plurality of the initial images to obtain an image to be inspected; Performing image segmentation on the image to be inspected to obtain a segmentation result image, wherein the segmentation result image includes the contour of the cell to be inspected; Performing key point detection on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected; According to the segmentation result image and the positioning image, determining that a contour area in the segmentation result image, in which the number of the center points is greater than 1, is a target area; According to the number and distribution of the center points of the target area in the positioning image, the target area in the segmentation result image is segmented into multiple sub-areas by a preset watershed segmentation strategy, each of which includes a cell to be tested that only contains one center point, thereby obtaining a target image.
2. The method according to claim 1, characterized in that Performing image segmentation on the image to be inspected to obtain the segmentation result image includes: Performing a first convolution operation on the image to be inspected to obtain a first feature map; Performing four downsampling operations on the first feature map in sequence to extract semantic information in the image to be inspected, and obtaining a semantic feature map corresponding to each downsampling operation; The semantic feature map obtained by the last downsampling operation is used as the first target feature map, and the first target feature map is sequentially upsampled four times to extract spatial information in the image to be inspected, so as to obtain a spatial feature map corresponding to each upsampling operation; The spatial feature map obtained by the last upsampling operation is used as the second target feature map, and a second convolution operation is performed on the second target feature map to obtain the segmentation result image.
3. The method according to claim 2, characterized in that The first feature map is sequentially subjected to four downsampling operations to extract semantic information in the image to be inspected, and a semantic feature map corresponding to each downsampling operation is obtained, including: A1321. Use the first feature map as a feature map to be pooled, and perform a pooling operation on the feature map to be pooled to obtain a downsampled feature map whose size is half of the feature map to be pooled; A1322, performing a first convolution operation on the downsampled feature map to extract the semantic information in the downsampled feature map to obtain a second feature map; A1323. Perform batch normalization on the second feature map to obtain a first normalization result, and output the result through a first activation function; A1324. Perform a second convolution operation on the first normalized result output by the first activation function to obtain a third feature map; A1325. Perform batch normalization on the third feature map to obtain a second normalization result; A1326, splicing the first normalized result and the second normalized result in a channel dimension to fuse the semantic information respectively extracted from the first convolution operation and the second convolution operation, and outputting the spliced first normalized result and the second normalized result via a second activation function to obtain the semantic feature map; A1327. Use the semantic feature map as a new feature map to be pooled, and loop the above steps A1321 to A1326 three times to obtain the semantic feature map corresponding to each loop.
4. The method according to claim 2, characterized in that: The semantic feature map obtained by the last downsampling operation is used as the first target feature map, and the first target feature map is sequentially upsampled four times to extract spatial information in the image to be inspected, and a spatial feature map corresponding to each upsampling operation is obtained, including: A1331, using the first target feature map as a first feature map to be convolved, and performing a deconvolution operation on the first feature map to be convolved to obtain an upsampled feature map whose size is twice that of the first target feature map; A1332, determining an attention weight according to a current up-sampled feature map and the semantic feature map having the same size as the current up-sampled feature map; A1333, weighting the semantic feature map having the same size as the current up-sampled feature map by the attention weight to obtain a weighted feature map, and concatenating the weighted feature map with the current up-sampled feature map in terms of channel dimension to obtain a second feature map to be convolved; A1334, continuously performing two convolution operations on the second feature map to be convolved to extract spatial information in the second feature map to be convolved to obtain the spatial feature map; A1335. Use the spatial feature map as the new first feature map to be convolved, and loop the above steps A1331 to A1334 three times to obtain the spatial feature map corresponding to each loop.
5. The method according to claim 4, characterized in that Determining an attention weight according to a current up-sampled feature map and the semantic feature map having the same size as the current up-sampled feature map comprises: According to the current up-sampled feature map and the semantic feature map of the same size as the current up-sampled feature map, the attention weight is determined by the following formula: In the formula, represents the attention weight, σ1 represents the ReLU activation function, σ2 represents the Sigmoid activation function, b1 and b2 represent the bias terms, W and W x , W g represents the linear transformation parameters, represents a semantic feature map of the same size as the current upsampled feature map, Represents the current upsampled feature map.
6. The method according to claim 1, characterized in that Performing key point detection on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected, including: By using a preset key point detection model, key point detection is performed on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected.
7. The method according to claim 6, characterized in that The preset key point detection model is an hourglass network.
8. A cell image segmentation device, characterized in that: The device comprises: An acquisition unit, used to acquire a plurality of initial images, wherein the initial images represent microscopic images obtained by photographing the same FISH sample under different microscopes at the same shooting angle; A superposition unit, used for superimposing a plurality of the initial images to obtain an image to be inspected; A first segmentation unit, used for performing image segmentation on the image to be inspected to obtain a segmentation result image, wherein the segmentation result image includes the contour of the cell to be inspected; A key point positioning unit, used for performing key point detection on the segmentation result image to obtain a positioning image representing the center point of the cell to be detected; a determining unit, configured to determine, according to the segmentation result image and the positioning image, a contour region in which the number of the center points is greater than 1 in the segmentation result image as a target region; The second segmentation unit is used to segment the target area in the segmentation result image into multiple sub-areas according to the number and distribution of the center points of the target area in the positioning image by a preset watershed segmentation strategy, each of which includes a cell to be tested that only contains one center point, so as to obtain a target image.
9. An electronic device, characterized in that: The electronic device comprises a processor and a memory coupled to each other, wherein the memory stores a computer program. When the computer program is executed by the processor, the electronic device executes the method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 7.