A cell segmentation method and related apparatus

Abstract

The application discloses a cell segmentation method and a related device, and relates to the technical field of image processing. The method comprises the following steps: acquiring a biological tissue image; performing feature extraction on the biological tissue image to obtain tissue features; determining convolution kernel information corresponding to the tissue features, score information of pixels in the tissue features, and detection boxes of the pixels according to the tissue features; determining respective corresponding score masks according to the score information of the pixels, and determining respective corresponding convolution positions according to the detection boxes of the pixels; performing modulated deformable convolution processing on the tissue features according to the score masks and the convolution positions to obtain focused features; and determining a segmentation result of segmenting cells in the biological tissue image according to the convolution kernel information and the focused features. The method can improve processing efficiency and the accuracy of the segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a cell segmentation method and related device. BACKGROUND

[0002] Cell segmentation is a very important step in biological and medical research, which can separate cells in the image in order to further analyze and study.

[0003] At present, medical institutions can detect cells in biological samples through artificial microscopy, and then segment the cells, but the segmentation result determined by this way is more dependent on the experience of the staff, and is mixed with the subjective factors of the staff, lacks objectivity, and further leads to poor accuracy of the segmentation result. SUMMARY

[0004] The present application provides a cell segmentation method and related device, which can improve the accuracy of the segmentation result.

[0005] In a first aspect, the present application provides a cell segmentation method, comprising:

[0006] obtaining a biological tissue image;

[0007] performing feature extraction on the biological tissue image to obtain tissue features;

[0008] determining, according to the tissue features, convolution kernel information corresponding to the tissue features, score information of pixels in the tissue features, and detection boxes of the pixels;

[0009] determining respective score masks according to the score information of each pixel, and determining respective convolution positions according to the detection boxes of each pixel;

[0010] performing modulated deformable convolution processing on the tissue features according to the score masks and the convolution positions to obtain focused features;

[0011] determining a segmentation result of segmenting cells in the biological tissue image according to the convolution kernel information and the focused features.

[0012] Optionally, the modulated deformable convolution processing on the tissue features according to the score masks and the convolution positions to obtain focused features comprises:

[0013] performing modulated deformable convolution processing on the tissue features according to the score masks and the convolution positions to obtain sub-focused features;

[0014] fusing the sub-focused features and the tissue features to obtain focused features.

[0015] Optionally, determining the corresponding score mask based on the score information of each pixel includes:

[0016] Based on the preset first activation function and the score information of each pixel, the corresponding score mask is determined.

[0017] Optionally, the score mask can be determined using the following formula:

[0018] Δm p =sigmoid(score) p )

[0019] Where, Δm p Let be the score mask corresponding to the p-th pixel, sigmoid() be the first activation function, and score p This represents the score information for the p-th pixel.

[0020] Optionally, determining the corresponding convolution position based on the detection bounding box of each pixel includes:

[0021] Based on the preset step size, the center position of the detection box of each pixel, the length of the detection box of each pixel, the width of the detection box of each pixel, the size of the convolution kernel corresponding to each pixel, and the position of the convolution kernel, the corresponding convolution position is determined.

[0022] Optionally, the convolution position can be determined using the following formula:

[0023]

[0024] in, Let S be the convolution position corresponding to the k-th position of the convolution kernel on the p-th pixel, and let S be the preset stride. p ,y′ p Let be the center position of the detection box at the p-th pixel, and size be... K w is the kernel size. p h is the width of the detection box at the p-th pixel. p Let p be the length of the detection box at the p-th pixel. k It represents the k-th position on the convolution kernel corresponding to the p-th pixel.

[0025] Optionally, the focusing feature can be obtained using the following formula:

[0026]

[0027] Where PFM(p) is the feature vector corresponding to the p-th pixel, f is the feature vector of the p-th pixel, and Q is the feature vector of the p-th pixel. k The weights corresponding to the k-th position of the convolution kernel; Let be the feature vector at the convolution position, and D be the focused feature.

[0028] Optionally, the organizational features include at least a first organizational feature and a second organizational feature, wherein the first organizational feature and the second organizational feature have different scales;

[0029] The step of performing modulated deformable convolution processing on the tissue features based on the score mask and the convolution position to obtain focused features includes:

[0030] Based on the score mask and convolution position corresponding to the first tissue feature, the first tissue feature is subjected to modulated deformable convolution processing to obtain the first focused feature;

[0031] Based on the score mask and convolution position corresponding to the second tissue feature, the second tissue feature is subjected to modulated deformable convolution processing to obtain the second focused feature;

[0032] The first focusing feature and the second focusing feature are fused together to obtain the focusing feature.

[0033] Optionally, fusing the first focusing feature with the second focusing feature includes:

[0034] The first focusing feature is fused with the second focusing feature based on channel attention and / or spatial attention.

[0035] Secondly, this application provides a cell segmentation device, comprising:

[0036] The acquisition module is used to acquire images of biological tissues;

[0037] The feature extraction module is used to extract features from the biological tissue image to obtain tissue features;

[0038] The detection module is used to determine the convolution kernel information corresponding to the tissue feature, the score information of the pixels in the tissue feature, and the detection box of the pixel based on the tissue feature.

[0039] The focusing module is used to determine the corresponding score mask for each pixel based on the score information of each pixel, and to determine the corresponding convolution position for each pixel based on the detection box of each pixel; based on the score mask and the convolution position, the tissue features are subjected to modulated deformable convolution processing to obtain the focusing features;

[0040] The segmentation module is used to determine the segmentation result of the cells in the biological tissue image based on the convolution kernel information and the focusing features.

[0041] Optionally, the focusing module is specifically used to perform modulated deformable convolution processing on the tissue features according to the score mask and the convolution position to obtain sub-focusing features; and to fuse the sub-focusing features with the tissue features to obtain focusing features.

[0042] Optionally, the focusing module is specifically used to determine the corresponding score mask based on the preset first activation function and the score information of each pixel.

[0043] Optionally, the focusing module is specifically used to determine the score mask using the following formula:

[0044] Δm p =sigmoid(score) p )

[0045] Where, Δm p Let be the score mask corresponding to the p-th pixel, sigmoid() be the first activation function, and score p This represents the score information for the p-th pixel.

[0046] Optionally, the focusing module is specifically used to determine the corresponding convolution position based on a preset stride, the center position of the detection box of each pixel, the length of the detection box of each pixel, the width of the detection box of each pixel, the size of the convolution kernel corresponding to each pixel, and the positions of the convolution kernel.

[0047] Optionally, the focusing module is specifically used to determine the convolution position using the following formula:

[0048]

[0049] in, Let S be the convolution position corresponding to the k-th position of the convolution kernel on the p-th pixel, and let S be the preset stride (x... ′ p ,y ′ p Let be the center position of the detection box at the p-th pixel, and size be... K w is the kernel size. p h is the width of the detection box at the p-th pixel. p Let p be the length of the detection box at the p-th pixel. k It represents the k-th position on the convolution kernel corresponding to the p-th pixel.

[0050] Optionally, the focusing module is specifically used to obtain the focusing feature using the following formula:

[0051]

[0052] Where PFM(p) is the feature vector corresponding to the p-th pixel, f is the feature vector of the p-th pixel, and Q is the feature vector of the p-th pixel. k The weights corresponding to the k-th position of the convolution kernel; Let be the feature vector at the convolution position, and D be the focused feature.

[0053] Optionally, the organizational features include at least a first organizational feature and a second organizational feature, wherein the first organizational feature and the second organizational feature have different scales; the focusing module is specifically used to perform modulated deformable convolution processing on the first organizational feature according to the score mask and convolution position corresponding to the first organizational feature to obtain a first focusing feature; to perform modulated deformable convolution processing on the second organizational feature according to the score mask and convolution position corresponding to the second organizational feature to obtain a second focusing feature; and to fuse the first focusing feature and the second focusing feature to obtain the focusing feature.

[0054] Optionally, the focusing module is specifically used to fuse the first focusing feature with the second focusing feature based on channel attention and / or spatial attention.

[0055] Thirdly, this application provides an apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the apparatus performs any of the optional methods described in the first aspect.

[0056] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the optional methods described in the first aspect.

[0057] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform any of the optional methods described in the first aspect.

[0058] Therefore, this application has the following beneficial effects:

[0059] This application provides a cell segmentation method. The method includes acquiring a biological tissue image, extracting features from the biological tissue image to obtain tissue features, then determining corresponding convolution kernel information, pixel score information, and pixel bounding boxes based on these tissue features. Next, based on the score information of each pixel, a corresponding score mask is determined. Based on the bounding boxes of each pixel, the corresponding convolution position is determined. Based on the score mask and convolution position, the tissue features are subjected to modulated deformable convolution processing to obtain focused features. This method reuses the score information and bounding boxes output by the detection head, thus eliminating the need for machine learning to obtain the score mask and convolution position, shortening processing time and improving processing efficiency. Finally, based on the convolution kernel information and focused features, the segmentation result of the cells in the biological tissue image is determined. Therefore, compared with traditional manual microscopic examination, this method not only improves processing efficiency but also enhances the accuracy of the segmentation results. Attached Figure Description

[0060] To more clearly illustrate the technical methods of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below.

[0061] Figure 1 A flowchart illustrating a cell segmentation method provided in this application embodiment;

[0062] Figure 2 A schematic diagram of a network architecture provided in an embodiment of this application;

[0063] Figure 3 A schematic diagram of a detection head provided in an embodiment of this application;

[0064] Figure 4 A schematic diagram illustrating the acquisition of a focusing feature according to an embodiment of this application;

[0065] Figure 5 A flowchart illustrating yet another cell segmentation method provided in this application embodiment;

[0066] Figure 6 A schematic diagram of yet another network structure provided in the embodiments of this application;

[0067] Figure 7 A schematic diagram illustrating a feature fusion process provided in an embodiment of this application;

[0068] Figure 8 A flowchart of a training method provided in an embodiment of this application;

[0069] Figure 9 This is a schematic diagram illustrating a multi-slice fusion technique provided in an embodiment of this application.

[0070] Figure 10 This is a schematic diagram of a cell segmentation device provided in an embodiment of this application. Detailed Implementation

[0071] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0072] The following is a description of the technical terms used in this application.

[0073] Biological tissue images refer to images of biological tissues acquired through techniques such as microscopy, X-rays, and magnetic resonance imaging (MRI). Generally, biological tissues are composed of multiple cells. Analyzing and studying biological tissue images provides information about the structure, function, and pathological changes of biological tissues, and has wide applications in medical diagnosis, drug development, and biological research. Biological tissue images can be analyzed and processed using digital image processing and computer vision techniques to improve image quality. Common biological tissue images include cell images, tissue images, organ images, and biological specimen images.

[0074] Cells are the basic structural and functional units of living organisms, and cell localization and segmentation are fundamental to image-based cell research. Typically, cell images exhibit the following characteristics:

[0075] The cells exhibit significant morphological differences; for example, BV-2 cells appear as small spherical shapes, while blastoma cells appear as nerve-like structures. The image features are complex, with cells often adhering to each other, making them difficult to distinguish, and the contrast is low with numerous background impurities. The cells are densely distributed, with the number of cell instances in a single cell image far exceeding the number of instances in natural image segmentation.

[0076] Currently, medical institutions use manual microscopy to examine cells and other components in cell images. However, this method requires a significant amount of time from staff, easily leading to staff fatigue, which reduces both efficiency and accuracy. Furthermore, the results of manual microscopy rely excessively on the experience of the staff and are subject to considerable subjective factors, lacking objective testing standards.

[0077] With the development of machine learning technology, cell images can be preprocessed first, then image segmentation methods can be used to obtain candidate regions that may contain cells. Features such as color, texture, and shape can then be extracted from these candidate regions. Finally, classification methods such as support vector machines can be used to classify and identify the features extracted from the candidate regions. However, this approach requires meticulous algorithm optimization and additional human intervention, such as adjusting parameters to adapt to scenarios with large differences in cell morphology and high cell density, resulting in a low level of automation.

[0078] With the development of deep learning technology, convolutional neural networks (CNNs) have been widely applied to image analysis. In cell segmentation tasks, a two-stage segmentation architecture, represented by Mask R-CNN (an object detection and segmentation algorithm), is relied upon. This architecture includes a Region Proposal Network (RPN) and an R-CNN (Region-based Convolutional Neural Network) detection head. Generally, RPM is first used to generate high-quality regions of interest (ROIs) from the anchor boxes, then the ROI reddening operator is used to extract more precise features from the ROIs, and finally, the detection head is used to generate detection boxes and segmentation results. However, due to inherent limitations of the algorithm, the computational cost increases linearly with the number of candidate regions. When the cell image is densely packed with cells, it faces enormous computational pressure, resulting in not only time-consuming processing but also poor accuracy.

[0079] In view of this, embodiments of this application provide a cell segmentation method. This method includes acquiring a biological tissue image, extracting features from the biological tissue image to obtain tissue features, then determining corresponding convolution kernel information, pixel score information, and pixel detection boxes based on the tissue features, then determining a corresponding score mask based on the score information of each pixel, determining the corresponding convolution position based on the detection box of each pixel, and performing modulated deformable convolution processing on the tissue features based on the score mask and convolution position to obtain focused features. This method reuses the score information and detection boxes output by the detection head, thus eliminating the need for machine learning to obtain the score mask and convolution position, shortening processing time, and improving processing efficiency. Finally, based on the convolution kernel information and focused features, the segmentation result of cell segmentation in the biological tissue image is determined. Therefore, compared with traditional manual microscopic examination, this method not only improves processing efficiency but also improves the accuracy of the segmentation results.

[0080] In some embodiments, the cell segmentation method described above can be executed by a processing device, which can be a terminal or a server. Of course, in other embodiments, the method may be executed by other devices instead of a processing device. This application does not specifically limit the executing entity of the cell segmentation method described above; those skilled in the art can select the executing entity based on actual needs.

[0081] To make the technical solution of this application clearer and easier to understand, the cell segmentation method provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0082] like Figure 1 As shown, this figure is a flowchart of a cell segmentation method provided in an embodiment of this application. The method includes:

[0083] S101. Obtain images of biological tissues.

[0084] As mentioned above, biological tissue images can include cell images, tissue images, organ images, biological specimen images, etc. This embodiment uses cell images as an example of biological tissue images for illustration.

[0085] In some embodiments, cell images can be acquired using optical microscopes, electron microscopes, laser scanning confocal microscopes, or similar methods. In other embodiments, cell images can also be acquired by receiving cell images transmitted from other devices.

[0086] S102. Extract features from biological tissue images to obtain tissue features.

[0087] Tissue features refer to the features obtained after feature extraction from biological tissue images. In some embodiments, feature extraction networks can be used to extract features from biological tissue images to obtain tissue features.

[0088] like Figure 2 As shown in the figure, this is a schematic diagram of a network architecture provided in an embodiment of this application. The cell segmentation method will be described below with reference to this schematic diagram of the network architecture.

[0089] This network architecture can be based on the CondInst algorithm, a target detection algorithm based on instance segmentation. CondInst stands for Conditional Convolutions for InstanceSegmentation. The main idea of ​​CondInst is to introduce conditional convolution operations into convolutional neural networks to improve the accuracy and efficiency of instance segmentation.

[0090] It should be noted that the above is only an example of the CondInst algorithm. The embodiments of this application can also be based on other algorithms for cell segmentation.

[0091] In some examples, after obtaining an image of biological tissue, the image can be input into a feature extraction network, which then outputs the tissue features. A feature extraction network is a network structure in deep learning used to extract useful information from input data (such as images of biological tissue). It is typically composed of convolutional neural networks, which obtain feature representations through operations such as convolution and pooling.

[0092] In other examples, the resolution of the biological tissue image can be adjusted to a preset resolution before it is input into the feature extraction network. For example, the preset resolution could be 1024*1024, 2048*2048, or 512*512.

[0093] It should be noted that this application does not specifically limit the size of the preset resolution. Those skilled in the art can set the above-mentioned preset resolution based on the computing power of the processing device.

[0094] S103. Based on the organizational features, determine the convolution kernel information corresponding to the organizational features, the pixel score information in the organizational features, and the detection box of the pixels.

[0095] After the feature extraction network outputs tissue features, these tissue features can be input into the detection head. For example... Figure 3 As shown in the figure, this is a schematic diagram of a detection head provided in an embodiment of this application. The detection head includes three branches, each branch comprising two sets of convolutional layers (CNN), a normalization layer (BN), and an activation function. The activation function can be SiLU, but other activation functions can also be selected; this application does not specifically limit the choice. Each branch of the detection head outputs a different result. For example, the first branch can output the pixel score information in the tissue feature, the second branch can output the detection box of the pixel in the tissue feature, and the third branch can output the convolution kernel information corresponding to the tissue feature. Compared with traditional detection heads, this detection head has a simpler structure, reduces the computational power required during processing, and thus improves processing efficiency.

[0096] The pixel score is used to describe the confidence level that the pixel is a cell. Generally, the higher the confidence level, the greater the probability that the pixel is a cell. In some examples, a confidence threshold can be set; if the confidence level of a pixel is greater than the threshold, then the pixel is determined to be a cell. The detection box is used to cover the pixel and is generated from preset pixel positions in the network. It is generally a rectangle, but it can also be generated from anchor boxes in the convolutional neural network. It is generally a rectangle. This application does not specifically limit the generation method of the detection box.

[0097] S104. Based on the score information of each pixel, determine the corresponding score mask for each pixel, and based on the detection box of each pixel, determine the corresponding convolution position for each pixel.

[0098] After determining the score information and detection box of each pixel, the corresponding score mask can be determined based on the score information of each pixel, and the corresponding convolution position can be determined based on the detection box of each pixel.

[0099] In some embodiments, a corresponding score mask can be determined based on a preset first activation function and the score information of each pixel. The first activation function can be sigmoid() or other functions; this application does not specifically limit the first activation function.

[0100] Specifically, the score mask corresponding to a pixel can be determined using the following formula:

[0101] Δm p =sigmoid(score) p (1)

[0102] Where, Δm p Let be the score mask corresponding to the p-th pixel, sigmoid() be the first activation function, and score p This represents the score information for the p-th pixel.

[0103] In some embodiments, the corresponding convolution position can be determined based on a preset stride, the center position of the detection box of each pixel, the length of the detection box of each pixel, the width of the detection box of each pixel, the size of the convolution kernel corresponding to each pixel, and the position of the convolution kernel.

[0104] Specifically, the convolution position corresponding to a pixel can be determined using the following formula:

[0105]

[0106] in, Let S be the convolution position corresponding to the k-th position of the convolution kernel on the p-th pixel, and let S be the preset stride (x... ′ p ,y ′ p Let x be the center position of the detection box for the p-th pixel. ′ p Let y be the x-coordinate of the center position of the detection box at the p-th pixel. ′ p Let be the ordinate of the center position of the detection box at the p-th pixel, and size K w is the kernel size.p h is the width of the detection box at the p-th pixel. p Let p be the length of the detection box at the p-th pixel. k It represents the k-th position on the convolution kernel corresponding to the p-th pixel.

[0107] In this embodiment, the score information and detection box output by the detection head are reused, thus eliminating the need to obtain the score mask and convolution position through machine learning, shortening the processing time and improving the processing efficiency.

[0108] S105. Based on the score mask and convolution position, perform modulated deformable convolution processing on the tissue features to obtain focused features.

[0109] After obtaining the score mask and convolution position, the offset can be determined based on the convolution position. Then, based on the offset and the score mask, the tissue features are subjected to modulated deformable convolution processing to obtain the focused features.

[0110] Modulation-deformable convolution is a convolution operation in convolutional neural networks that allows adjustment of the convolution kernel's position, thereby improving the network's ability to perceive and adapt to targets. In modulation-deformable convolution, the convolution kernel is variable, adjusted by an offset. In traditional techniques, the offset needs to be learned by the network, for example, through an offset predictor network. In this application, it is not necessary to learn the offset through the network; instead, the convolution position is obtained by reprocessing the score information output by the detection head and the detection box, and the offset is determined based on the convolution position. This reduces computational load and improves processing efficiency.

[0111] In some embodiments, the tissue features can be modulated deformable convolution processing based on the score mask and convolution position to obtain sub-focused features, and then the sub-focused features are fused with the tissue features to obtain focused features. Figure 4 As shown in the figure, this figure is a schematic diagram of obtaining a focusing feature according to an embodiment of this application.

[0112] After performing modulated deformable convolution processing on tissue features based on score masks and convolution positions, sub-focused features are obtained. To avoid feature omission, the sub-focused features are further fused with the original tissue features to obtain focused features. This reduces information loss and deformation, improves the accuracy of focused features, and thus improves the accuracy of cell segmentation.

[0113] In some embodiments, the focusing features can be determined based on the following formula:

[0114]

[0115] Where PFM(p) is the feature vector corresponding to the p-th pixel, f is the feature vector of the p-th pixel, and Q is the feature vector of the p-th pixel. k The weights corresponding to the k-th position of the convolution kernel; Let be the feature vector at the convolution position, and D be the focused feature.

[0116] S106. Based on the convolution kernel information and focusing features, determine the segmentation result of the cells in the biological tissue image.

[0117] After obtaining the focusing features, the segmentation result of cell segmentation in biological tissue images can be determined based on the convolution kernel information and the focusing features.

[0118] In some embodiments, a non-maximum suppression (NMS) algorithm can be used to process the detection boxes of each pixel. For example, the detection boxes can be filtered based on score information, removing detection boxes with low confidence levels represented by the score information to obtain pixels with higher confidence levels.

[0119] Next, the focusing feature is combined with the positional encoding to obtain the target mask. The positional encoding refers to the encoding of each position in the focusing feature. The target mask is then fed into a conditional convolutional segmentation network to obtain an instance segmentation mask, which represents the segmentation result. The convolutional kernel of the segmentation network is adjusted to the kernel corresponding to the pixel obtained after filtering based on the pixel score information. In some embodiments, after obtaining the cell segmentation result, it can also be displayed on a display device.

[0120] Based on the above description, this application provides a cell segmentation method. This method includes acquiring a biological tissue image, extracting features from the biological tissue image to obtain tissue features, then determining corresponding convolution kernel information, pixel score information, and pixel detection boxes based on the tissue features, then determining a corresponding score mask based on the score information of each pixel, determining the corresponding convolution position based on the detection box of each pixel, and performing modulated deformable convolution processing on the tissue features based on the score mask and convolution position to obtain focused features. This method reuses the score information and detection boxes output by the detection head, thus eliminating the need for machine learning to obtain the score mask and convolution position, shortening processing time and improving processing efficiency. Finally, based on the convolution kernel information and focused features, the segmentation result of cell segmentation in the biological tissue image is determined. Therefore, compared with traditional manual microscopic examination, this method not only improves processing efficiency but also improves the accuracy of the segmentation results.

[0121] In the above embodiments, the principle of the cell segmentation method was introduced by taking one tissue feature as an example. The cell segmentation method provided in the embodiments of this application will be introduced below when there are multiple tissue features.

[0122] like Figure 5 As shown, this figure is a flowchart of another cell segmentation method provided in an embodiment of this application, the method including:

[0123] S501. Acquire images of biological tissues.

[0124] S501 is similar to S101. For details, please refer to S101. It will not be repeated here.

[0125] S502. Extract features from biological tissue images to obtain first tissue features, second tissue features, and third tissue features.

[0126] It should be noted that this embodiment does not specifically limit the number of tissue features. Tissue features can include two, three, or even more. For ease of understanding, the following description uses three tissue features as an example. For instance, the tissue features can include a first tissue feature, a second tissue feature, and a third tissue feature. When the number of tissue features is other values, the principle of the cell segmentation method provided in this application embodiment is similar. For example, when the number of tissue features is two, cell segmentation is also based on a similar principle, but the segmentation accuracy and processing efficiency will be affected. The more tissue features, the better the segmentation accuracy, but the worse the processing efficiency. Preferably, when the number of tissue features is three, both segmentation accuracy and processing efficiency can be effectively guaranteed.

[0127] The first, second, and third tissue features have different scales. For example, after scaling a biological tissue image to a preset resolution (e.g., 1024*1024), it is input into a feature extraction network. This feature extraction network can output three layers of multi-scale features with strides of 8, 16, and 32, such as the first, second, and third tissue features.

[0128] It should be noted that the step sizes of 8, 16, and 32 mentioned above are just examples, and other step sizes can also be used.

[0129] In some embodiments, different feature extraction networks can be used for different scenarios. For example, when the processing device has high computing power, CSPNeXt-large (a convolutional neural network model) with 256 output channels can be used; when the processing device has low computing power, CSPNeXt-tiny (a convolutional neural network model) with 96 output channels can be used. CSPNeXt-large is a larger model that uses more convolutional layers and channels to improve the model's accuracy and generalization ability. Compared to CSPNeXt-tiny, CSPNeXt-large has higher computational complexity and requires more computing resources and time for training and inference. CSPNeXt-tiny is a smaller model that uses fewer convolutional layers and channels to reduce the model's computational complexity, making it suitable for scenarios with limited computing resources. Compared to CSPNeXt-large, CSPNeXt-tiny has a smaller model size, lower computational complexity, and faster training and inference speeds.

[0130] It should be noted that the above two models are merely illustrative examples, and those skilled in the art can choose based on their actual needs.

[0131] As shown in Table 1 below, this table is a comparison table of the performance of the feature extraction network in the embodiments of this application.

[0132] Table 1:

[0133]

[0134] Compared with existing feature extraction networks (ResNet 50 and ResNeSt 200+DCN), the feature extraction networks (CSPNeXt-tiny and CSPNEXt-large) in this application not only improve the accuracy of the detection boxes (box) and the segmentation accuracy (segmentation), but also reduce the number of parameters and the number of computations (FLOPs).

[0135] like Figure 6 As shown, this figure is a schematic diagram of another network structure provided in an embodiment of this application. This network structure is similar to... Figure 2 Compared to the network structure shown, each organizational feature has its own corresponding detection head. For example, the first organizational feature corresponds to the first detection head, the second organizational feature corresponds to the second detection head, and the third organizational feature corresponds to the third detection head.

[0136] S503. Based on the first organizational feature, determine the convolution kernel information corresponding to the first organizational feature, the pixel score information in the first organizational feature, and the detection box of the pixel.

[0137] In some embodiments, the first tissue feature can be input into the first detection head, which can output the convolution kernel information corresponding to the first tissue feature, the pixel score information in the first tissue feature, and the detection box of the pixel. For details, please refer to the above embodiments, which will not be repeated here.

[0138] S504. Based on the second organizational feature, determine the convolution kernel information corresponding to the second organizational feature, the pixel score information in the second organizational feature, and the detection box of the pixel.

[0139] In some embodiments, the second tissue feature can be input into a second detection head, which can output convolution kernel information corresponding to the second tissue feature, pixel score information in the second tissue feature, and detection bounding box of the pixel. For details, please refer to the above embodiments, which will not be repeated here.

[0140] S505. Based on the third organizational feature, determine the convolution kernel information corresponding to the third organizational feature, the pixel score information in the third organizational feature, and the detection box of the pixel.

[0141] In some embodiments, the third tissue feature can be input into a third detection head, which can output the convolution kernel information corresponding to the third tissue feature, the pixel score information in the third tissue feature, and the detection box of the pixel. For details, please refer to the above embodiments, which will not be repeated here.

[0142] S506. Based on the score information of each pixel in the first organizational feature, determine the score mask corresponding to each pixel in the first organizational feature, and based on the detection box of each pixel in the first organizational feature, determine the convolution position corresponding to each pixel in the first organizational feature.

[0143] S507. Based on the score information of each pixel in the second organizational feature, determine the score mask corresponding to each pixel in the second organizational feature, and based on the detection box of each pixel in the second organizational feature, determine the convolution position corresponding to each pixel in the second organizational feature.

[0144] S508. Based on the score information of each pixel in the third organizational feature, determine the score mask corresponding to each pixel in the third organizational feature, and based on the detection box of each pixel in the third organizational feature, determine the convolution position corresponding to each pixel in the third organizational feature.

[0145] The specific process of steps S506-S508 can be found in the above embodiments and formulas (1) and (2), and will not be repeated here.

[0146] S509. Based on the pixel score mask and convolution position in the first organizational feature, perform modulated deformable convolution processing on the first organizational feature to obtain the first focused feature.

[0147] In some embodiments, the first organizational feature can be modulated and deformed convolutional processed based on the pixel score mask and convolution position in the first organizational feature to obtain a first sub-focusing feature. Then, this first sub-focusing feature is fused with the first organizational feature to obtain a first focusing feature. The specific process is similar to the above embodiments, and can be found in the above embodiments and... Figure 4 This will not be elaborated upon here.

[0148] S510. Based on the pixel score mask and convolution position in the second tissue feature, perform modulated deformable convolution processing on the second tissue feature to obtain the second focused feature.

[0149] In some embodiments, the second organizational feature can be modulated and deformed convolutional processed based on the pixel score mask and convolution position in the second organizational feature to obtain a second sub-focusing feature. Then, this second sub-focusing feature is fused with the second organizational feature to obtain a second focusing feature. The specific process is similar to the above embodiments, and can be found in the above embodiments and... Figure 4 This will not be elaborated upon here.

[0150] S511. Based on the pixel score mask and convolution position in the third organizational feature, perform modulated deformable convolution processing on the third organizational feature to obtain the third focused feature.

[0151] In some embodiments, the third organizational feature can be modulated and deformed convolutionally processed based on the pixel score mask and convolution position in the third organizational feature to obtain a third sub-focusing feature. Then, this third sub-focusing feature is fused with the third organizational feature to obtain the third focusing feature. The specific process is similar to the above embodiments, and can be found in the above embodiments and... Figure 4 This will not be elaborated upon here.

[0152] S512. The first focusing feature, the second focusing feature and the third focusing feature are fused together to obtain the focusing feature.

[0153] In some embodiments, when obtaining the first focusing feature, the second focusing feature, and the third focusing feature, the first focusing feature, the second focusing feature, and the third focusing feature can be melted to obtain the focusing feature.

[0154] In some embodiments, the first focusing feature, the second focusing feature, and the third focusing feature can be fused together based on channel attention and / or spatial attention to obtain the focusing feature. For example, the first focusing feature, the second focusing feature, and the third focusing feature can be fused together based on channel attention, spatial attention, or both channel attention and spatial attention.

[0155] Channel attention learns the importance of each channel and then weights the features of different channels; spatial attention weights different positions in the feature map, thereby increasing the model's attention to information at different positions. Therefore, in this embodiment, the feature fusion process is not a simple summation, but rather employs channel attention and / or spatial attention, which captures richer and more discriminative feature information, thus improving the representational power of the fused features.

[0156] like Figure 7 As shown, this figure is a schematic diagram of a feature fusion process provided in an embodiment of this application. In some examples, channel attention can be implemented through one-dimensional convolution with a kernel size of 5. Spatial attention is achieved by using max pooling and average pooling for dimensionality compression, generating weights through 7*7 convolutions, and then performing element-wise multiplication to achieve attention. The first, second, and third focusing features are unified to a stride of 8 and summed pixel by pixel. Then, a mask feature with a stride of 4 and 8 channels is output using four 3*3 convolution kernels and one deconvolution, which is the focusing feature obtained after the above fusion process.

[0157] S513. Based on the convolution kernel information and focusing features, determine the segmentation result of the cells in the biological tissue image.

[0158] After obtaining the focusing features, the segmentation result of cell segmentation in biological tissue images can be determined based on the focusing features and convolution kernel information.

[0159] In some embodiments, the focusing features can be combined with positional encoding to obtain a target mask with 10 channels, where the positional encoding has 2 channels, representing the horizontal and vertical coordinates respectively. Positional encoding refers to the encoding of each position in the fused focusing features, calculated based on the resolution of the input and the stride of the fused feature map. For example, if the input resolution is 1024*1024 and the stride of the fused feature map is 4, then the resolution of the fused feature map is 256*256. Based on this, a target mask of size 10*256*256 can be obtained. This target mask is fed into a conditional convolutional segmentation network to obtain an instance segmentation mask, which is used to characterize the segmentation result.

[0160] Based on the above description, this application provides a cell segmentation method. This method includes acquiring a biological tissue image, extracting features from the biological tissue image to obtain tissue features, then determining corresponding convolution kernel information, pixel score information, and pixel detection boxes based on these tissue features, then determining a corresponding score mask based on the score information of each pixel, determining the corresponding convolution position based on the detection box of each pixel, and performing modulated deformable convolution processing on the tissue features based on the score mask and convolution position to obtain focused features. This method reuses the score information and detection boxes output by the detection head, thus eliminating the need for machine learning to obtain the score mask and convolution position, shortening processing time and improving processing efficiency. Finally, based on the convolution kernel information and focused features, the segmentation result of cell segmentation in the biological tissue image is determined. Therefore, compared with traditional manual microscopic examination, this method not only improves processing efficiency but also improves the accuracy of the segmentation result. Furthermore, this method obtains multiple tissue features at different scales during feature extraction, further improving the accuracy and processing efficiency of cell image segmentation.

[0161] The following is based on Figure 6 Taking the network architecture shown as an example, the training process of this network architecture will be introduced.

[0162] like Figure 8 As shown, this figure is a flowchart of a training method provided in an embodiment of this application. The method includes:

[0163] S701. Acquire historical biological tissue images.

[0164] Historical biological tissue images are images in which cell locations have already been labeled.

[0165] S702. Perform data augmentation processing on historical biological tissue images to obtain enhanced images.

[0166] After obtaining historical biological tissue images, data augmentation processing can be performed on the historical biological tissue images using multi-slice fusion technology to obtain enhanced images.

[0167] In some examples, the number of cells in a single historical biological tissue image is too high, affecting training efficiency. Therefore, this application extracts slices from historical biological tissue images, with the number of cells in the slices being much smaller than the number of cells in the historical biological tissue image. Then, slices from multiple historical biological tissue images are fused to obtain an enhanced image.

[0168] like Figure 9 As shown in the figure, this is a schematic diagram of a multi-slice fusion technology provided in an embodiment of this application. Taking the fusion of slices from five historical biological tissue images as an example, firstly, slices are extracted from each historical biological tissue image. Then, four slices from different historical biological tissue images undergo mosaic enhancement processing and are stitched together. Next, they are randomly scaled within a preset ratio (e.g., 0.5 to 2), then random slicing and random flipping are performed. Finally, the slices are mixed with another slice to achieve data enhancement and obtain the enhanced image.

[0169] It should be noted that the above example only uses the blending of 5 slices. In other examples, more or fewer slices can be blended to obtain an enhanced image.

[0170] Data augmentation of historical biological tissue images can reduce the number of cells in the image, thus avoiding the impact of densely packed cells on training efficiency.

[0171] S703. Input the enhanced image into the feature extraction network to obtain the first historical organization feature, the second historical organization feature, and the third historical organization feature.

[0172] The specific process is similar to the aforementioned embodiments, and will not be repeated here.

[0173] S704. The first historical organization feature is fed into the first detection head to obtain the convolution kernel information corresponding to the first historical organization feature, as well as the pixel score information and detection box in the first historical organization feature.

[0174] S705. The second historical organization feature is fed into the second detection head to obtain the convolution kernel information corresponding to the second historical organization feature, as well as the pixel score information and detection box in the second historical organization feature.

[0175] S706. The third historical organization feature is fed into the third detection head to obtain the convolution kernel information corresponding to the third historical organization feature, as well as the pixel score information and detection box in the third historical organization feature.

[0176] In some embodiments, the above score information can be used to calculate the loss value using the Quality Focal Loss function, and then the model weights can be adjusted; the above detection box can be used to calculate the loss value using the GIoU Loss function, and then the model weights can be adjusted.

[0177] It should be noted that the above loss function is merely an example, and other functions can also be used to calculate the loss value in the embodiments of this application.

[0178] S707. Determine the first historical focusing feature based on the pixel score information and detection box in the first historical organization feature.

[0179] The specific process is similar to the aforementioned embodiments, and will not be repeated here.

[0180] S708. Determine the second historical focus feature based on the pixel score information and detection box in the second historical organization feature.

[0181] The specific process is similar to the aforementioned embodiments, and will not be repeated here.

[0182] S709. Based on the pixel score information and detection box in the third historical organization feature, determine the third historical focus feature.

[0183] The specific process is similar to the aforementioned embodiments, and will not be repeated here.

[0184] S710. The first historical focus feature, the second historical focus feature, and the third historical focus feature are fused together to obtain the historical focus feature.

[0185] After obtaining the historical focus features, an additional 3*3 convolution can be added to generate a semantic segmentation mask. Then, the Dice Loss function and semantic annotations are used to calculate the loss value, and the weights of the model are adjusted accordingly.

[0186] S711. Determine the instance segmentation mask based on the convolution kernel information and historical focusing features.

[0187] After obtaining the instance segmentation mask, the Dice Loss function can be used to calculate the loss value with the instance annotations, and then the weights of the model can be adjusted.

[0188] like Figure 10 As shown in the figure, this is a schematic diagram of a cell segmentation device provided in an embodiment of this application. The cell segmentation device includes:

[0189] Acquisition module 901 is used to acquire images of biological tissues;

[0190] Feature extraction module 902 is used to extract features from the biological tissue image to obtain tissue features;

[0191] The detection module 903 is used to determine the convolution kernel information corresponding to the tissue feature, the score information of the pixels in the tissue feature, and the detection box of the pixel based on the tissue feature.

[0192] The focusing module 904 is used to determine the corresponding score mask based on the score information of each pixel, and to determine the corresponding convolution position based on the detection box of each pixel; and to perform modulated deformable convolution processing on the tissue features based on the score mask and the convolution position to obtain the focusing features.

[0193] The segmentation module 905 is used to determine the segmentation result of the cells in the biological tissue image based on the convolution kernel information and the focusing features.

[0194] Optionally, the focusing module 904 is specifically used to perform modulated deformable convolution processing on the tissue features according to the score mask and the convolution position to obtain sub-focusing features; and to fuse the sub-focusing features with the tissue features to obtain focusing features.

[0195] Optionally, the focusing module 904 is specifically used to determine the corresponding score mask based on the preset first activation function and the score information of each pixel.

[0196] Optionally, the focusing module 904 is specifically used to determine the score mask using the following formula:

[0197] Δm p =sigmoid(score) p )

[0198] Where, Δm p Let be the score mask corresponding to the p-th pixel, sigmoid() be the first activation function, and score p This represents the score information for the p-th pixel.

[0199] Optionally, the focusing module 904 is specifically used to determine the corresponding convolution position based on the preset step size, the center position of the detection box of each pixel, the length of the detection box of each pixel, the width of the detection box of each pixel, the size of the convolution kernel corresponding to each pixel, and the position of the convolution kernel.

[0200] Optionally, the focusing module 904 is specifically used to determine the convolution position using the following formula:

[0201]

[0202] in, Let S be the convolution position corresponding to the k-th position of the convolution kernel on the p-th pixel, and let S be the preset stride (x...′ p ,y ′ p Let be the center position of the detection box at the p-th pixel, and size be... K w is the kernel size. p h is the width of the detection box at the p-th pixel. p Let p be the length of the detection box at the p-th pixel. k It represents the k-th position on the convolution kernel corresponding to the p-th pixel.

[0203] Optionally, the focusing module 904 is specifically used to obtain the focusing feature using the following formula:

[0204]

[0205] Where PFM(p) is the feature vector corresponding to the p-th pixel, f is the feature vector of the p-th pixel, and Q is the feature vector of the p-th pixel. k The weights corresponding to the k-th position of the convolution kernel; Let be the feature vector at the convolution position, and D be the focused feature.

[0206] Optionally, the organizational features include at least a first organizational feature and a second organizational feature, wherein the first organizational feature and the second organizational feature have different scales; the focusing module 904 is specifically used to perform modulated deformable convolution processing on the first organizational feature according to the score mask and convolution position corresponding to the first organizational feature to obtain a first focusing feature; perform modulated deformable convolution processing on the second organizational feature according to the score mask and convolution position corresponding to the second organizational feature to obtain a second focusing feature; and fuse the first focusing feature and the second focusing feature to obtain the focusing feature.

[0207] Optionally, the focusing module 904 is specifically used to fuse the first focusing feature with the second focusing feature based on channel attention and / or spatial attention.

[0208] In addition, this application also provides a computing device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any of the above embodiments.

[0209] In addition, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the method described in any of the above embodiments.

[0210] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0211] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0212] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0213] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of cell segmentation, the method comprising: The method comprises the following steps: acquiring a biological tissue image; extracting features of the biological tissue image to obtain tissue features; determining, according to the tissue features, convolution kernel information corresponding to the tissue features, score information of a pixel in the tissue features, and a detection frame of the pixel; determining a score mask corresponding to each pixel according to the score information of the pixel, and determining a convolution position corresponding to each pixel according to the detection frame of the pixel, wherein the score information of the pixel is used to describe a confidence degree of the pixel being a cell, and the score mask is determined by the score information of the pixel and a first activation function; performing modulation deformable convolution processing on the tissue features according to the score mask and the convolution position to obtain focused features; determining a segmentation result of segmenting cells in the biological tissue image according to the convolution kernel information and the focused features.

2. The method of claim 1, wherein, The modulation deformable convolution processing on the tissue features according to the score mask and the convolution position to obtain focused features comprises: performing modulation deformable convolution processing on the tissue features according to the score mask and the convolution position to obtain sub-focused features; fusing the sub-focused features and the tissue features to obtain the focused features.

3. The method of claim 1, wherein, The determination of the score mask corresponding to each pixel according to the score information of each pixel comprises: determining the score mask corresponding to each pixel according to a preset first activation function and the score information of each pixel.

4. The method of claim 3, wherein, The score mask is determined by the following formula: ； wherein, is a score mask corresponding to the pth pixel pair, is a first activation function, is score information of the pth pixel.

5. The method of claim 1, wherein, The determination of the convolution position corresponding to each pixel according to the detection frame of each pixel comprises: determining the convolution position corresponding to each pixel according to a preset step length, a center position of the detection frame of each pixel, a length of the detection frame of each pixel, a width of the detection frame of each pixel, a convolution kernel size corresponding to each pixel, and each position of the convolution kernel.

6. The method of claim 5, wherein, The convolution position is determined by the following formula: ； wherein, is the convolution position corresponding to the kth position of the convolution kernel on the pth pixel, S is a preset step size, is the center position of the detection frame of the pth pixel, is the size of the convolution kernel, is the width of the detection frame of the pth pixel, is the length of the detection frame of the pth pixel, is the kth position on the convolution kernel corresponding to the pth pixel.

7. The method of claim 2, wherein, The focused features are obtained by the following formula: ； wherein, is a feature vector corresponding to the pth pixel, f p is a feature vector of the pth pixel, is a weight corresponding to the kth position of the convolution kernel, is a convolution position corresponding to the kth position of the convolution kernel on the pth pixel, is a score mask corresponding to the pth pixel; is a feature vector on the convolution position, and D is the focused feature.

8. The method according to any one of claims 1 to 7, characterized in that, The tissue features at least comprise first tissue features and second tissue features, and the first tissue features and the second tissue features are different in scale; The modulation deformable convolution processing on the tissue features according to the score mask and the convolution position to obtain focused features comprises: performing modulation deformable convolution processing on the first tissue features according to the score mask and the convolution position corresponding to the first tissue features to obtain first focused features; performing modulation deformable convolution processing on the second tissue features according to the score mask and the convolution position corresponding to the second tissue features to obtain second focused features; fusing the first focused features and the second focused features to obtain the focused features.

9. The method of claim 8, wherein, The fusion of the first focused features and the second focused features comprises: fusing the first focused features and the second focused features based on channel attention and / or spatial attention.

10. A cell segmentation apparatus, characterized by, The method comprises the following steps: an acquisition module, configured to acquire a biological tissue image; a feature extraction module, configured to extract features of the biological tissue image to obtain tissue features; detecting a convolution kernel corresponding to the tissue feature, score information of a pixel in the tissue feature, and a detection frame of the pixel according to the tissue feature; focusing on a value mask corresponding to each pixel according to the score information of each pixel, and a convolution position corresponding to each pixel according to the detection frame of each pixel, wherein the score information of each pixel is used to describe a confidence degree of the pixel being a cell, and the value mask is determined by the score information of the pixel and a first activation function; and performing a modulated deformable convolution on the tissue feature according to the value mask and the convolution position to obtain a focused feature; segmenting a segmentation result of cells in the biological tissue image according to the convolution kernel and the focused feature.

11. An apparatus, comprising: comprise: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the computer program, the method in any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, and when the instructions run on the terminal equipment, the terminal equipment executes the method in any one of claims 1 to 9.

Images