Multi-immunofluorescence staining cell target detection method and device based on environmental information fusion

Abstract

The invention discloses a multi-immunofluorescence staining cell target detection method and device based on environmental information fusion. The method comprises the following steps: 1) constructing an image detection framework combined with environmental information fusion; 2) detecting network anchor point adjustment and positioning; 3) accurately positioning a feature map and generating cells in a classified manner; and 4) performing feature accurate positioning target detection based on environmental information guidance. According to the method, target category image target detection based on environment information conversion is realized through a feature vector accurate positioning form based on environment information of a DAPI image.

Description

Technical Field

[0001] The present invention belongs to the field of image segmentation, and in particular relates to a method and device for detecting multiple immunofluorescence-stained cell targets based on environmental information fusion. Background Art

[0002] Cell detection, a fundamental task in computational pathology, is crucial for automated disease diagnosis, cell counting, and tissue structure analysis. In particular, in the auxiliary diagnosis and treatment of major diseases like cancer, doctors often need to make judgments based on the number, type, distribution, and spatial structure of cells. Therefore, high-precision and efficient cell detection algorithms are of practical value in promoting intelligent pathology image analysis.

[0003] Multiplex immunohistochemistry fluorescence (mIHC) technology can simultaneously detect multiple markers on a single tissue section, and through a combination of different fluorescently labeled antibodies, it can achieve high-throughput analysis of complex biological scenarios such as tumor microenvironment and immune cell infiltration. Among these fluorescent markers, DAPI (4',6-diamidino-2-phenylindole) is an indispensable nuclear counterstain in mIHC because it combines with the abundant and highly concentrated DNA in the cell nucleus, clearly marking the cell nuclear morphology and forming a spatial distinction from the fluorescent signals of other targets. With the growing demand for improved technology, this technology has been introduced and applied in research and clinical fields, enabling people to comprehensively study cell composition, functional status and cell-to-cell interactions, thereby improving diagnostic efficiency.

[0004] With the continuous advancement of computer technology, artificial intelligence (AI) has made rapid progress in the medical field. In particular, machine learning (ML) and deep learning have demonstrated remarkable performance in representing complex medical images. However, the fuzzy boundaries, large color variations, complex textures, dense cell arrangement, and high overlap in pathological images can hinder the success of cell phenotyping methods. This challenge is particularly challenging in real-world mIHC data.

[0005] When performing cell identification, doctors need to use the cell information of the DAPI channel image to determine the cell type in the current image. However, some of the current cell instance segmentation methods only use the cell information of a single channel, which can easily misidentify stained or autofluorescent areas as cells, or use multi-channel images as unified input to determine the types of all cells at the same time. Such methods will transmit interference information from different staining channels to each other, affecting the final classification effect.

[0006] Anchor-based object detection methods select anchor points in the input image with a certain step size, and then extract the features of a square anchor box with the anchor point as the center to determine the step size side length to identify the type of this anchor box. However, the distribution of cells is random, and matching the anchor point to the center of the cell cannot accurately utilize the features of the corresponding position. Summary of the Invention

[0007] The present invention aims to solve the above-mentioned problems in the prior art and provides a method and device for detecting cell targets by multiple immunofluorescence staining based on environmental information fusion.

[0008] When doctors identify cell types, they need to observe cell morphology with the help of images that combine the DAPI channel with the current channel. However, existing multiple fluorescent immunohistochemical staining cell detection algorithms all perform detection based on single-channel grayscale images, which does not solve the problem that cell identification is easily interfered with and the fuzzy positioning of cell features in existing detection networks. The present invention designs a multiple immunofluorescence staining cell target detection method and device based on environmental information fusion. It utilizes the cell nucleus information of the existing DAPI channel in multiple fluorescence, converts the environmental information into cell identification in the current image in the form of auxiliary features, and accurately extracts the surrounding features of the anchor point during the cell identification process, realizing image target detection assisted by environmental information.

[0009] The first aspect of the present invention relates to a method for detecting cell targets by multiple immunofluorescence staining based on environmental information fusion, comprising the following steps:

[0010] 1) Preprocessing multiple fluorescent staining datasets to extract environmental information and calculate the average cell size;

[0011] 2) Combine environmental information fusion to build an image detection framework, extract and process target feature maps and environmental feature maps;

[0012] 3) Initialize the detection network anchor point and adjust the anchor point based on the target feature map and the environment feature map;

[0013] 4) Accurately locate the anchor classification feature map based on the adjusted anchor points and generate the anchor classification results.

[0014] Preferably, step 1) specifically includes: performing cell size statistical analysis on the existing fluorescent staining data set, inputting the marked cell points into an advanced segmentation model such as SAM for preliminary segmentation, manually reviewing and screening the segmentation results, eliminating erroneous segmentation masks, and accurately calculating the cell radius using OpenCV image processing methods; finally, combining the clinical experience of professional doctors to determine the average cell size C; and manually screening and excluding two types of unqualified samples: images with abnormal staining and images containing abnormal tissue areas.

[0015] Preferably, step 2) specifically includes:

[0016] Select ConvNeXt network as the subject detection network F, and CTransPath as the environment information feature extraction network F ′ , the input target detection picture is a two-channel picture pair (I,I ′ ), where I is the target channel image to be detected, I ′ It is the DAPI channel image of the auxiliary detection; the output of F is the multi-scale main feature map extracted from I by ConvNeXt Where N is determined by the size of I, F ′ The output of CTransPath is from I ′ The extracted lowest level feature map f 2 ;Then Input FPN network for feature fusion and output multi-scale feature maps f 2 The convolution layer with a stride of 1 is adjusted to the feature dimension The same auxiliary feature map Avoid the auxiliary feature dimension being too long, which may affect the main feature.

[0017] Preferably, step 3) specifically includes:

[0018] 4.1 Setting the original anchor points: To set the initialization to uniformly distributed anchor points as close to the cell centroid as possible, the average size of cells in the image is investigated in advance and the anchor point interval is set to the average cell size C to obtain the original anchor points.

[0019] 4.2 Preliminary deformation anchor point: During the deformation process of the anchor point offset to the cell center of mass, the high resolution contains the finest grained features that are critical for cell positioning. The features are initially regressed through the FFN regression head and then added to the original anchor coordinates to obtain the initial deformed anchor points. The FFN regression head is a multi-layer perceptron (MLP), which is composed of a stack of fully connected layers and nonlinear activation functions. It is a basic feedforward neural network structure.

[0020] 4.3 Generating the final anchor point: using grid sampling and After uniform feature scale changes, these feature maps are spliced ​​into the final feature map, and then fused through a convolutional layer with a step size of 1 to obtain the feature map affected by environmental information. Input the FFN regression head for the second regression and add it to the initial deformation anchor point to get the final anchor point coordinates

[0021] 4.4 Extracting Environmental Information: Auxiliary Feature Maps Input a general binary classification convolutional neural network to generate a DAPI channel mask to participate in model training to ensure further improvement of the effectiveness of environmental information.

[0022] Preferably, step 4) specifically includes:

[0023] 5.1 Mapping anchor point coordinates: In order to enhance the adaptive perception of coordinate offset in feature space, the predicted coordinates in the previous step With the original anchor point The large offset problem caused by the predicted coordinates Map to In the feature space, the feature space coordinates are obtained

[0024] 5.2 Alignment and reorganization of anchor coordinate geometry: Since the feature space scale is generally smaller than the original image size (usually one-eighth of the original image size), there will be precision errors in the mapping process of the predicted coordinates. At the same time, in order to ensure that the extracted features can cover the entire cell, the feature map size is enlarged and the horizontal / vertical offset (d x / d y )Select The feature vector of the multiple of the left / right / up / down position of the mapping coordinate corresponding to the feature map is used as the feature vector of the predicted coordinate, and finally the large-scale feature map after the shift is obtained. Will Input the FFN regression head to obtain the final cell classification result.

[0025] The auxiliary feature map based on environmental information can effectively assist the target feature map in locating cells. At the same time, accurate extraction and reorganization of the feature map according to cell positioning can make cell classification more accurate, thereby achieving more accurate target detection.

[0026] Furthermore, the feature map size magnification in step 5.2 is four times, The feature size of the feature vector corresponding to the mapping coordinate left / up and down of the feature map is The feature map has a 2 by 2 size.

[0027] The second aspect of the present invention relates to a multiple immunofluorescence staining cell target detection device based on environmental information fusion, which is characterized in that it includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the image target detection method based on environmental information of the present invention.

[0028] The method of the present invention is a multiple immunofluorescence staining cell target detection method based on environmental information fusion. It is used under the guidance of DAPI channel images, based on the environmental information of DAPI images, to achieve target category image target detection based on environmental information transformation through precise positioning of feature vectors.

[0029] The beneficial effects of the present invention are as follows: for multiple immunofluorescence-stained cell images, with the assistance of clear and easily identifiable DAPI environmental information, the problem of difficulty in identifying fluorescently stained cell images can be solved; in addition, the feature precision method proposed in the present invention can better solve the classification error problem in the cell detection process, and achieve image target detection that is closer to professional doctors' identification of cells. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the method of the present invention.

[0031] Figure 2 This is the framework diagram of the method of the present invention. The symbols in the figure have the following meanings: subject detection network F, environmental information feature extraction network F ′ , FFN regression classification network, I is the target channel image that needs to be detected, I ′ This is the DAPI channel image for auxiliary detection. is the original anchor point coordinate, is the final anchor point coordinate, is the cell classification result, is a multi-scale subject feature map, is the auxiliary feature map, It is a feature map that integrates environmental information. It is a large-scale feature map for cell classification. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0033] Example 1

[0034] This embodiment provides a method for detecting cell targets using multiple immunofluorescence staining based on environmental information fusion, comprising the following steps:

[0035] 1) Construction of image detection framework combined with environmental information fusion;

[0036] In the selection of ConvNeXt network as the main detection network F, CTransPath as the environment information feature extraction network F ′ , the input target detection picture is a two-channel picture pair (I,I ′ ), where I is the target channel image to be detected, I ′Is the DAPI channel image of the auxiliary detection. The output of F is the multi-scale main feature map extracted from I by ConvNeXt Where N is determined by the size of the image, and F ′ The output of CTransPath is from I ′ The extracted lowest level feature map f 2 ;Then Input FPN network for feature fusion and output multi-scale feature maps f 2 The convolution layer with a stride of 1 is adjusted to the feature dimension The same auxiliary feature map To avoid the main features being affected by the excessive length of the auxiliary feature dimension, the auxiliary features can use the existing DAPI annotations to participate in training and further improve the effectiveness of the environment information;

[0037] 2) Detect network anchor point adjustment and positioning;

[0038] In order to achieve the goal of setting the initialization to uniformly distributed anchor points as close to the cell centroid as possible, the average size of cells in the image can be investigated in advance and the anchor point interval can be set to the average cell size C, so that the original anchor points can be obtained. During the deformation process of the anchor point offset to the cell center of mass, the high-resolution P2 contains the finest-grained features that are critical for the localization of small objects. The features are initially regressed through the FFN regression classification head and then added to the original anchor coordinates to obtain the initial deformed anchor points. Then use the grid sampling and After uniform feature scale changes, these feature maps are spliced ​​into the final feature map, and then fused through a convolutional layer with a step size of 1 to obtain the feature map affected by environmental information. Input the FFN regression classification network for the second regression and add it to the initial deformation anchor point to get the final anchor point coordinates

[0039] 3) Accurate positioning of feature maps and generation of cell classification;

[0040] In order to enhance the perception of coordinate offset in feature space, the coordinates predicted in the previous step With the original anchor point The algorithm will predict the coordinates of the large offset problem Map to In the feature space, the feature space coordinates are obtained Since the feature space scale is generally smaller than the original image size, there will be precision errors in the mapping process of the predicted coordinates. In order to ensure that the extracted features can cover the entire cell, the horizontal / vertical offset (d x / d y ) Select the 2x2 feature vector on the left / top / bottom of the mapping coordinate as the feature vector of the predicted coordinate, and finally get the large-scale feature map after the offset Will Input FFN regression classification network to obtain the final cell classification results

[0041] 4) Accurately locate and detect targets based on features guided by environmental information;

[0042] In order to achieve target detection guided by environmental information, we can use the existing clear cell nuclear morphology information of DAPI staining to realize cell detection in the target image; for the labeled target category images, the target detection network is trained through supervised loss; for open source non-target category images, accurate anchor point coordinates and feature maps at the anchor point coordinates are obtained through steps 2) and 3); finally, image target detection guided by environmental information is achieved.

[0043] Example 2

[0044] This embodiment relates to a multiple immunofluorescence staining cell target detection device based on environmental information fusion, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the multiple immunofluorescence staining cell target detection method based on environmental information fusion of Example 1.

[0045] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A multiplex immunofluorescence staining cell target detection method based on environmental information fusion, comprising the following steps: 1) Preprocessing multiple fluorescent staining datasets to extract environmental information and calculate the average cell size; 2) Combine environmental information fusion to build an image detection framework, extract and process target feature maps and environmental feature maps; 3) Initialize the detection network anchor point and adjust the anchor point based on the target feature map and the environment feature map; 4) Accurately locate the anchor classification feature map based on the adjusted anchor points and generate the anchor classification results.

2. The method for detecting cell targets by multiple immunofluorescence staining based on environmental information fusion according to claim 1, wherein: Step 1) specifically involves inputting the labeled cell points into an advanced segmentation model such as SAM for preliminary segmentation. The segmentation results are manually reviewed and screened, and after removing erroneous segmentation masks, the cell radius is accurately calculated using OpenCV image processing methods. Finally, the average cell size C is determined based on the clinical experience of professional physicians. Manual screening excludes two types of unqualified samples: images with abnormal staining and images containing abnormal tissue areas.

3. The method for detecting cell targets by multiple immunofluorescence staining based on environmental information fusion according to claim 1, wherein: Step 2) specifically includes: The ConvNeXt network is selected as the subject detection network F, and CTransPath is used as the environment information feature extraction network F′. The input target detection image is a two-channel image pair (I, I′), where I is the target channel image to be detected and I′ is the DAPI channel image for auxiliary detection. The output of F is the multi-scale subject feature map extracted from I by ConvNeXt. Where N depends on the size of I, and the output of F′ is the bottom-level feature map f extracted by CTransPath from I′ 2 ;Then Input FPN network for feature fusion and output multi-scale feature maps f 2 The convolution layer with a stride of 1 is adjusted to the feature dimension The same auxiliary feature map Avoid the auxiliary feature dimension being too long, which may affect the main feature.

4. The method for detecting cell targets by multiple immunofluorescence staining based on environmental information fusion according to claim 1, wherein: Step 3) specifically includes: 4.1 Setting the original anchor points: To set the initialization to uniformly distributed anchor points as close to the cell centroid as possible, the average size of cells in the image is investigated in advance and the anchor point interval is set to the average cell size C to obtain the original anchor points. 4.2 Preliminary deformation anchor point: During the deformation process of the anchor point offset to the cell center of mass, the high resolution contains the finest grained features that are critical for cell positioning. The features are initially regressed through the FFN regression head and then added to the original anchor coordinates to obtain the initial deformed anchor points. The FFN regression head is a multi-layer perceptron (MLP), which is composed of a stack of fully connected layers and nonlinear activation functions. It is a basic feedforward neural network structure. 4.3 Generating the final anchor point: using grid sampling and After uniform feature scale changes, these feature maps are spliced ​​into the final feature map, and then fused through a convolutional layer with a step size of 1 to obtain the feature map affected by environmental information. Input the FFN regression classification head for the second regression and add it to the initial deformation anchor point to get the final anchor point coordinates 4.4 Extracting Environmental Information: Auxiliary Feature Maps Input a general binary classification convolutional neural network to generate a DAPI channel mask to participate in model training to ensure further improvement of the effectiveness of environmental information.

5. The method for detecting cell targets by multiple immunofluorescence staining based on environmental information fusion according to claim 1, wherein: Step 4) specifically includes: 5.1 Mapping anchor point coordinates: In order to enhance the adaptive perception of coordinate offset in feature space, the predicted coordinates in the previous step With the original anchor point The large offset problem caused by the predicted coordinates Map to In the feature space, the feature space coordinates are obtained 5.2 Alignment and reorganization of anchor coordinates: Enlarge the size of the feature map and adjust the horizontal / vertical offset (d x / d y )Select The feature vector of the multiple of the left / right / up / down position of the mapping coordinate corresponding to the feature map is used as the feature vector of the predicted coordinate, and finally the large-scale feature map after the shift is obtained. Will Input the FFN regression head to obtain the final cell classification result.

6. The method for detecting cell targets by multiple immunofluorescence staining based on environmental information fusion according to claim 5, characterized in that: The feature map size magnification in step 5.2 is four times, The feature size of the feature vector corresponding to the mapping coordinate left / up and down of the feature map is The feature map has a 2 by 2 size.

7. A multiplex immunofluorescence staining cell target detection device based on environmental information fusion, characterized in that: The method comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method is used to implement the multiple immunofluorescence staining cell target detection method based on environmental information fusion according to any one of claims 1 to 6.

Images