Multi-immunofluorescence staining cell target detection method and device based on environmental information fusion

Abstract

The application discloses a multi-immunofluorescence staining cell target detection method and device based on environmental information fusion, and the method comprises the following steps: 1) combining an environmental information fusion image detection frame to build; 2) adjusting and positioning a network anchor point; 3) accurately positioning a feature map and generating a cell classification; and 4) accurately positioning a feature based on environmental information to guide target detection. The application realizes target detection of a target category image based on environmental information transformation in the form of accurate positioning of a feature vector based on environmental information of a DAPI image.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation, specifically relating to a method and apparatus for multiplex immunofluorescence staining of cell targets based on environmental information fusion. Background Technology

[0002] Cell detection, a fundamental task in computational pathology, is crucial for automating disease diagnosis, cell counting, and tissue structure analysis. Especially in the auxiliary diagnosis and treatment of serious diseases such as cancer, doctors often need to make judgments based on the number, type, distribution, and spatial structure of cells. Therefore, high-precision, high-efficiency cell detection algorithms have practical value in promoting the intelligentization of pathological image analysis.

[0003] Multiplex immunohistochemical fluorescence (mIHC) technology enables the simultaneous detection of multiple markers on a single tissue section. Through combinations of antibodies with different fluorescent labels, high-throughput analysis of complex biological scenarios such as the tumor microenvironment and immune cell infiltration can be achieved. Among these fluorescent markers, DAPI (4',6-diamidinyl-2-phenylindole) is an indispensable nuclear counterstain in mIHC because it binds to the abundant and highly concentrated DNA in the cell nucleus, clearly marking the cell nucleus morphology and spatially distinguishing it from the fluorescence signals of other targets. With the increasing demand for improved technology, this technique has been introduced and applied in research and clinical fields, enabling comprehensive studies of cellular composition, functional state, and intercellular interactions, thereby improving diagnostic efficiency.

[0004] With the continuous development of computer technology, artificial intelligence has made rapid progress in the medical field, especially machine learning (ML) and deep learning, which have demonstrated outstanding performance in learning representations of complex medical images. However, problems such as blurred cell boundaries, large color variations, complex textures, dense arrangement, and high overlap in pathological images hinder the success of cell phenotypic analysis methods. This problem is particularly challenging in real mIHC data.

[0005] When performing cell identification, doctors need to use cell information from DAPI channel images to determine the cell types in the current image. However, some existing cell instance segmentation methods only use cell information from a single channel, which can easily lead to misidentification of stained or autofluorescent areas as cells. Alternatively, some methods use multi-channel images as input to determine the types of all cells at the same time. Such methods will transmit interference information from different staining channels to each other, affecting the final classification results.

[0006] Anchor-based object detection methods select anchor points in the input image with a certain stride, and then extract features of square anchor boxes with the anchor point as the center point to determine the stride side length in order to identify the type of the anchor box. However, the distribution of cells is random, and the anchor point matching the center of the cell cannot accurately utilize the features of the corresponding position. Summary of the Invention

[0007] The present invention aims to solve the above-mentioned problems of the prior art by providing a method and device for multiplex immunofluorescence staining of cell targets based on environmental information fusion.

[0008] In identifying cell types, doctors need to observe cell morphology using images combining the DAPI channel and the current channel. However, existing multiplex immunofluorescence staining cell detection algorithms all perform detection based on single-channel grayscale images, failing to address the issues of cell recognition being easily interfered with and the ambiguous cell feature localization in existing detection networks. This invention designs a multiplex immunofluorescence staining cell target detection method and device based on environmental information fusion. It utilizes the cell nucleus information from the existing DAPI channel in the multiplex fluorescence image, transforming environmental information into cell recognition in the form of auxiliary features. Furthermore, it accurately extracts features around anchor points during cell recognition, achieving environment-assisted image target detection.

[0009] The first aspect of this invention relates to a method for detecting cell targets using multiplex immunofluorescence staining based on environmental information fusion, comprising the following steps:

[0010] 1) Preprocess the multiplex fluorescence staining dataset to purify environmental information and statistically analyze the average cell size;

[0011] 2) Construct an image detection framework by combining environmental information fusion, and extract and process target feature maps and environmental feature maps;

[0012] 3) Initialize the detection network anchor points and adjust them in conjunction with the target feature map and the environment feature map;

[0013] 4) Accurately locate the anchor point classification feature map based on the adjusted anchor points and generate the anchor point classification results.

[0014] Preferably, step 1) specifically includes: performing statistical analysis of cell size on the existing fluorescent staining dataset; inputting the labeled cell points into advanced segmentation models such as SAM for preliminary segmentation; manually reviewing and screening the segmentation results; removing erroneous segmentation masks; and using OpenCV image processing methods to accurately calculate the cell radius; finally, combining the clinical experience of professional doctors to determine the average cell size C; and manually screening out two types of unqualified samples: images with abnormal staining and images containing abnormal tissue regions.

[0015] Preferably, step 2) specifically includes:

[0016] ConvNeXt network was selected as the main detection network F, and CTransPath was selected as the environmental information feature extraction network F. ′ The input target detection image is a two-channel image pair (I,I) ′ ), where I is the target channel image to be detected, I ′ It is the DAPI channel image for auxiliary detection; the output of F is the multi-scale main feature map extracted from I by ConvNeXt. Where N is determined by the size of I, F ′ The output of CTransPath is from I ′ Extracted lowest-level feature map f 2 ;Then Inputting features into an FPN network and performing feature fusion to output multi-scale feature maps. f 2 Adjusting the feature dimension using a convolutional layer with a stride of 1 Same auxiliary feature map Avoid having excessively long auxiliary feature dimensions, which could negatively impact the main features.

[0017] Preferably, step 3) specifically includes:

[0018] 4.1 Setting Initial Anchor Points: To initialize the anchor points as evenly distributed as possible, they should be as close as possible to the cell centroid. The average cell size in the image is investigated beforehand, and the anchor point interval is set to the average cell size C, thus obtaining the initial anchor points.

[0019] 4.2 Initial Deformation Anchor Point: During the deformation process where the anchor point is offset to the cell centroid, because high resolution contains the finest-grained features crucial for cell localization, it will... The features are subjected to an initial regression using the FFN regression head, and then added to the original anchor point coordinates to obtain the initial deformed anchor points. The FFN regression head, also known as the multilayer perceptron (MLP), is composed of fully connected layers and nonlinear activation functions stacked together, and is a basic feedforward neural network structure.

[0020] 4.3 Generate final anchor points: using mesh sampling pairs and After performing uniform feature scale transformation, these feature maps are concatenated into a final feature map, which is then fused through a convolutional layer with a stride of 1 to obtain a feature map influenced by environmental information. After inputting the FFN regression head for a second regression, the coordinates are added to the initial deformation anchor points to obtain the final anchor point coordinates.

[0021] 4.4 Extracting Environmental Information: Auxiliary Feature Map A general binary classification convolutional neural network is input to generate DAPI channel masks for model training to ensure further improvement in the effectiveness of environmental information.

[0022] Preferably, step 4) specifically includes:

[0023] 5.1 Mapping Anchor Point Coordinates: To enhance the adaptive perception of coordinate offsets in the feature space, the coordinates predicted in the previous step are mapped... With the original anchor point The resulting large offset problem will affect the predicted coordinates. Mapped to At the feature space, the feature space coordinates are obtained.

[0024] 5.2 Alignment and Reassembly of Anchor Point Coordinate Geometry: Since the feature space scale is generally smaller than the original image size (typically one-eighth the size of the original image), the mapping process of predicted coordinates will have accuracy errors. Simultaneously, to ensure that the extracted features encompass the entire cell, the feature map size is enlarged, based on the horizontal / vertical offset (d) of the predicted coordinates between candidate boxes. x / d y Choose in The feature vectors corresponding to the mapped coordinates (left / right / up / down) are used as the feature vectors for the predicted coordinates, and finally, the offset large-scale feature map is obtained. Will Input the FFN regression head to obtain the final cell classification results.

[0025] Environmental information-based auxiliary feature maps can effectively assist target feature maps in locating cells. At the same time, precise extraction and recombination of feature maps based on cell location can make cell classification more accurate, thereby achieving more accurate target detection.

[0026] Furthermore, the feature map size magnification factor described in step 5.2 is four times. The feature size of the feature vector corresponding to the mapping coordinates left / right / up / down of the feature map is The feature map is 2x2 in size.

[0027] A second aspect of the present invention relates to a multiplex immunofluorescence staining cell target detection device based on environmental information fusion, characterized in that it includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the image target detection method based on environmental information of the present invention.

[0028] The method of the present invention is a multiplex immunofluorescence staining cell target detection method based on environmental information fusion. Under the guidance of DAPI channel images, it realizes target category image target detection based on environmental information transformation by accurately locating the target through feature vectors based on the environmental information of the DAPI images.

[0029] The beneficial effects of this invention are: for multiplex immunofluorescence stained cell images, with the assistance of clear and easily identifiable DAPI environmental information, the problem of difficulty in identifying fluorescently stained cell images can be solved; in addition, the feature-accurate method proposed in this invention can better solve the problem of classification errors in the cell detection process, and achieve image target detection that is closer to the identification of cells by professional doctors. Attached Figure Description

[0030] Figure 1 This is a flowchart of the method of the present invention.

[0031] Figure 2 This is a framework diagram of the method of the present invention. The symbols in the diagram have the following meanings: Subject detection network F, Environmental information feature extraction network F ′ FFN regression classification network, where I is the target channel image to be detected, I ′ These are DAPI channel images used for auxiliary detection. These are the original anchor point coordinates. These are the final anchor point coordinates. It is the result of cell classification. It is a multi-scale main feature map. It is an auxiliary feature map. It is a feature map that integrates environmental information. It is a large-scale feature map of cell classification. Detailed Implementation

[0032] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0033] Example 1

[0034] This embodiment provides a method for detecting cell targets using multiplex immunofluorescence staining based on environmental information fusion, including the following steps:

[0035] 1) Construct an image detection framework that integrates environmental information fusion;

[0036] ConvNeXt network was selected as the main detection network F, and CTransPath was selected as the environmental information feature extraction network F. ′ The input target detection image is a two-channel image pair (I,I) ′ ), where I is the target channel image to be detected, I ′This is the DAPI channel image used for auxiliary detection. The output of F is the multi-scale main feature map extracted from I by ConvNeXt. Where N is determined based on the image size, F ′ The output of CTransPath is from I ′ Extracted lowest-level feature map f 2 ;Then Inputting features into an FPN network and performing feature fusion to output multi-scale feature maps. f 2 Adjusting the feature dimension using a convolutional layer with a stride of 1 Same auxiliary feature map To avoid the main features being affected by excessively long auxiliary feature dimensions, auxiliary features can utilize existing DAPI annotations to participate in training, further improving the effectiveness of environmental information.

[0037] 2) Detect network anchor point adjustment and positioning;

[0038] To ensure that the initial anchor points, which are uniformly distributed, are as close as possible to the cell centroid, the average cell size in the image can be investigated beforehand, and the anchor point interval can be set to the average cell size C. This will yield the original anchor points. During the deformation process where the anchor point is offset to the cell centroid, because the high-resolution P2 contains the finest-grained features crucial for the localization of small objects, it will... After the features are subjected to an initial regression using the FFN regression classification head, the initial deformed anchor points are obtained by adding them to the original anchor point coordinates. Then grid sampling pairs were used. and After performing uniform feature scale transformation, these feature maps are concatenated into a final feature map, which is then fused through a convolutional layer with a stride of 1 to obtain a feature map influenced by environmental information. After inputting the FFN regression classification network for a second regression, the coordinates are added to the initial deformed anchor points to obtain the final anchor point coordinates.

[0039] 3) Precise feature map localization and cell classification generation;

[0040] To enhance the ability to perceive coordinate offsets in the feature space, the predicted coordinates from the previous step are... With the original anchor point To address the issue of significant offsets, this algorithm will predict coordinates. Mapped to At the feature space, the feature space coordinates are obtained. Since the feature space scale is generally smaller than the original image size, the mapping process of predicted coordinates will have accuracy errors. Furthermore, to ensure that the extracted features encompass the entire cell, the horizontal / vertical offset (d) of the predicted coordinates between candidate boxes is used. x / d y Select the 2x2 feature vectors to the left / right or up / down of the mapped coordinates as the feature vectors for the predicted coordinates, and finally obtain the large-scale feature map after offset. Will Inputting the data into the FFN regression classification network yields the final cell classification results.

[0041] 4) Feature-based precise target detection guided by environmental information;

[0042] To achieve target detection guided by environmental information, we can utilize the clear cell nucleus morphology information obtained from existing DAPI staining to detect cells in target images. For labeled target category images, we can train the target detection network using supervised loss. For open-source non-target category images, we can obtain accurate anchor coordinates and feature maps at the anchor coordinates through steps 2) and 3). Finally, we achieve image target detection guided by environmental information.

[0043] Example 2

[0044] This embodiment relates to a multiplex immunofluorescence staining cell target detection device based on environmental information fusion, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the multiplex immunofluorescence staining cell target detection method based on environmental information fusion of Embodiment 1.

[0045] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms described in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A multiplex immunofluorescence staining method for cell target detection based on environmental information fusion, comprising the following steps: 1) Preprocess the multiplex fluorescence staining dataset to purify environmental information and statistically analyze the average cell size; 2) Construct an image detection framework by integrating environmental information, and extract and process target feature maps and environmental feature maps; specifically including: ConvNeXt network was selected as the main detection network. CTransPath as an environmental information feature extraction network The input target detection image is a two-channel image pair. ,in These are the target channel images that need to be inspected. These are DAPI channel images used for auxiliary detection; The output is ConvNeXt from Extracted from Extracted multi-scale main feature maps N is based on It depends on the size. The output of CTransPath is from Extracted lowest level feature map ;Then Inputting features into an FPN network and performing feature fusion to output multi-scale feature maps. , Adjusting the feature dimension using a convolutional layer with a stride of 1 Same auxiliary feature map To avoid the main features being affected by excessively long auxiliary feature dimensions; 3) Initialize the detection network anchor points and adjust them in conjunction with the target feature map and the environment feature map; specifically including: 4.1 Setting Initial Anchor Points: To initialize the anchor points as evenly distributed as possible, they should be as close as possible to the cell centroid. Beforehand, investigate the average cell size in the image and set the anchor point interval to the average cell size. Obtain the original anchor point ; 4.2 Initial Deformation Anchor Point: During the deformation process where the anchor point is offset to the cell centroid, because high resolution contains the finest-grained features crucial for cell localization, it will... The features are subjected to an initial regression using the FFN regression head, and then added to the original anchor point coordinates to obtain the initial deformed anchor points. The FFN regression head, also known as the multilayer perceptron (MLP), is composed of fully connected layers and nonlinear activation functions stacked together, and is a basic feedforward neural network structure. 4.3 Generate final anchor points: using mesh sampling pairs and After performing uniform feature scale transformation, these feature maps are concatenated into a final feature map, which is then fused through a convolutional layer with a stride of 1 to obtain a feature map influenced by environmental information. After inputting the FFN regression classification header for a second regression, the coordinates are added to the initial deformed anchor points to obtain the final anchor point coordinates. ; 4.4 Extracting Environmental Information: Auxiliary Feature Map A general binary classification convolutional neural network is input to generate DAPI channel masks to participate in model training, so as to further improve the effectiveness of environmental information. 4) Accurately locate the anchor point classification feature map based on the adjusted anchor points and generate the anchor point classification results.

2. The method for detecting cell targets based on multiplex immunofluorescence staining using environmental information fusion as described in claim 1, characterized in that, Step 1) specifically includes: inputting the labeled cell points into advanced segmentation models such as SAM for initial segmentation; manually reviewing and filtering the segmentation results to remove erroneous segmentation masks; then using OpenCV image processing methods to accurately calculate the cell radius; finally, combining the clinical experience of professional doctors to determine the average cell size. Two types of unqualified samples were manually screened out: images with abnormal staining and images containing abnormal tissue areas.

3. The method for detecting cell targets based on multiplex immunofluorescence staining using environmental information fusion as described in claim 1, characterized in that, Step 4) specifically includes: 5.1 Mapping Anchor Point Coordinates: To enhance the adaptive perception of coordinate offsets in the feature space, the coordinates predicted in the previous step are mapped... With the original anchor point The resulting large offset problem will affect the predicted coordinates. Mapped to At the feature space, the feature space coordinates are obtained. ; 5.2 Alignment and Reorganization of Anchor Point Coordinates: Enlarge the feature map size and adjust the horizontal / vertical offsets between candidate boxes based on the predicted coordinates. / Choose in The feature vectors corresponding to the mapped coordinates (left / right / up / down) are used as the feature vectors for the predicted coordinates, and finally, the offset large-scale feature map is obtained. ,Will Input the FFN regression head to obtain the final cell classification results.

4. The method for detecting cell targets based on multiplex immunofluorescence staining using environmental information fusion as described in claim 3, characterized in that, The feature map size magnification factor described in step 5.2 is four times. The feature size of the feature vector corresponding to the mapping coordinates left / right / up / down of the feature map is The feature map is 2x2 in size.

5. A multiplex immunofluorescence staining cell target detection device based on environmental information fusion, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the multiplex immunofluorescence staining cell target detection method based on environmental information fusion as described in any one of claims 1-4.

Images