Tumor microenvironment detection method and system based on a single slide

Abstract

The present invention discloses a single-slide-based tumor microenvironment detection method and system. The method acquires a single-slide fluorescence microscope image captured by a microscope; performs image feature extraction on the fluorescence microscope image to generate a semantically-joint shallow-layer feature map of tumor growth status; and, based on the semantically-joint shallow-layer feature map of tumor growth status, determines the tumor growth status. This allows simultaneous imaging of multiple fluorescently labeled tumor cells and surrounding cells directly on a single slide, thereby obtaining comprehensive information about the tumor microenvironment.

Description

Technical Field

[0001] The present invention relates to the field of intelligent detection technology, and in particular to a tumor microenvironment detection method and system based on a single slide. Background Art

[0002] The tumor microenvironment refers to the complex environment where tumor cells interact with surrounding normal cells, blood vessels, immune cells, etc. It has an important impact on the occurrence, development, metastasis and treatment of tumors.

[0003] Currently, commonly used methods for detecting the tumor microenvironment include tissue sections, flow cytometry, and mass spectrometry. However, these methods all have limitations, such as requiring complex instrumentation. In recent years, single-slide fluorescence microscopy has emerged as a new method for detecting the tumor microenvironment. It offers advantages such as ease of operation, low cost, and rich information. It can simultaneously image multiple fluorescently labeled tumor cells and surrounding cells directly on a single slide, thereby obtaining comprehensive information about the tumor microenvironment.

[0004] However, determining the growth status of tumors from these highly complex fluorescence microscopy images remains a challenging problem, and a solution is therefore highly sought. Summary of the Invention

[0005] Embodiments of the present invention provide a single-slide-based tumor microenvironment detection method and system. The method acquires a single-slide fluorescence microscopy image captured by a microscope; performs image feature extraction on the fluorescence microscopy image to generate a semantically-joint shallow feature map of tumor growth status; and, based on the semantically-joint shallow feature map of tumor growth status, determines the tumor growth status. This allows simultaneous imaging of multiple fluorescently labeled tumor cells and surrounding cells directly on a single slide, thereby obtaining comprehensive information about the tumor microenvironment.

[0006] The present invention also provides a method for detecting a tumor microenvironment based on a single slide, which comprises:

[0007] Acquiring single slide-based fluorescence microscopy images collected by a microscope;

[0008] Performing image feature extraction on the fluorescence microscope imaging image to obtain a semantically combined shallow feature map of tumor growth status; and

[0009] The growth status of the tumor is determined based on the semantic combined shallow feature map of the tumor growth status.

[0010] The present invention also provides a single slide-based tumor microenvironment detection system, which includes:

[0011] An image acquisition module, used for acquiring a fluorescence microscope imaging image based on a single glass slide collected by a microscope;

[0012] an image feature extraction module, configured to extract image features from the fluorescence microscope image to obtain a semantically combined shallow feature map of tumor growth status; and

[0013] The tumor growth status determination module is used to determine the tumor growth status based on the semantic combined with the tumor growth status shallow feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0015] Figure 1 The figure is a flow chart of a tumor microenvironment detection method based on a single slide provided in an embodiment of the present invention.

[0016] Figure 2 Schematic diagram of the system architecture of a single slide-based tumor microenvironment detection method provided in an embodiment of the present invention.

[0017] Figure 3 This is a flowchart of the sub-steps of step 120 in a tumor microenvironment detection method based on a single slide provided in an embodiment of the present invention.

[0018] Figure 4 This is a block diagram of a tumor microenvironment detection system based on a single slide provided in an embodiment of the present invention.

[0019] Figure 5 This is a diagram of an application scenario of a tumor microenvironment detection method based on a single slide provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0021] Unless otherwise specified, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art in the art of the present application. The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit the scope of this application.

[0022] In the description of the embodiments of this application, it should be noted that, unless otherwise specified and limited, the term "connection" should be understood in a broad sense. For example, it can be an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meaning of the above terms can be understood according to the specific circumstances.

[0023] It should be noted that the terms "first, second, and third" in the embodiments of the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the terms "first, second, and third" can be interchanged to represent a specific order or precedence where permitted. It should be understood that the objects distinguished by "first, second, and third" can be interchanged where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0024] It should be understood that the tumor microenvironment refers to the complex environment surrounding tumor cells, including the extracellular matrix, immune cells, blood vessels, and other cell types. The tumor microenvironment plays a key regulatory role in tumor growth, invasion, and metastasis.

[0025] The main components and functions of the tumor microenvironment include:

[0026] 1. Extracellular Matrix (ECM): The ECM is a support structure composed of proteins, polysaccharides, and other molecules that provides the support and signals required for cell growth and migration. Tumor cells can promote tumor invasion and metastasis by altering the composition and structure of the ECM.

[0027] 2. Immune cells: The tumor microenvironment contains multiple types of immune cells, including macrophages, lymphocytes, and dendritic cells. Immune cells play a dual role in tumor development. On the one hand, they can recognize and kill tumor cells, inhibiting tumor growth. On the other hand, tumor cells can evade immune attack by manipulating immune cell function or producing immunosuppressive molecules.

[0028] 3. Blood vessels: The vascular system within the tumor microenvironment is crucial for tumor growth and metastasis. Tumor cells induce the formation of new blood vessels (angiogenesis) to obtain oxygen and nutrients, enabling tumor growth. Blood vessels also provide pathways for tumor cells to metastasize to other sites.

[0029] 4. Intercellular interactions: Tumor cells interact with surrounding cells in complex ways. This includes interactions with immune cells, endothelial cells, and other cell types. These interactions can influence tumor growth, invasion, and metastasis.

[0030] The characteristics and composition of the tumor microenvironment vary depending on tumor type, stage and individual differences. Understanding the characteristics of the tumor microenvironment is important for a deeper understanding of the development mechanism of tumors and provides important guidance for developing therapeutic strategies targeting the tumor microenvironment.

[0031] Furthermore, commonly used methods for detecting the tumor microenvironment include: Immunohistochemistry (IHC): Immunohistochemistry is a commonly used method for detecting specific protein markers in the tumor microenvironment. This method uses specific antibodies to bind to the target protein and uses a staining reaction to visualize the presence and location of the marker. For example, IHC can be used to detect immune cell infiltration, angiogenesis, extracellular matrix components, etc.

[0032] Immunofluorescence staining: Immunofluorescence staining is a method that uses fluorescently labeled antibodies to detect specific proteins. Similar to IHC, immunofluorescence staining can provide information on the localization and expression levels of target proteins in cells or tissues. It can be used to detect cell types, immune cell subsets, cytokines, etc. in the tumor microenvironment.

[0033] Gene expression analysis: Gene expression levels in the tumor microenvironment can be detected through gene expression analysis methods such as real-time quantitative polymerase chain reaction (qPCR) and gene chip technology. These methods can provide information on the activation status of immune cells, cytokine production, extracellular matrix composition, etc.

[0034] Fluorescence in situ hybridization (FISH): Fluorescence in situ hybridization is a method for detecting chromosomal abnormalities and gene rearrangements. It can be used to evaluate genetic abnormalities such as gene mutations, chromosome deletions or gains in the tumor microenvironment.

[0035] Multispectral imaging: Multispectral imaging technology combines optical microscopy and spectral analysis to simultaneously obtain morphological and spectral information of tissue samples. This method can be used to detect cell types, angiogenesis, metabolic status, etc. in the tumor microenvironment.

[0036] In recent years, single-slide fluorescence microscopy imaging technology has emerged as a new method for detecting the tumor microenvironment. It has the advantages of simple operation, low cost, and rich information. It can simultaneously image multiple fluorescently labeled tumor cells and surrounding cells directly on a single slide, thereby obtaining comprehensive information about the tumor microenvironment.

[0037] In one embodiment of the present invention, Figure 1 The figure is a flow chart of a tumor microenvironment detection method based on a single slide provided in an embodiment of the present invention. Figure 2 Schematic diagram of the system architecture of a tumor microenvironment detection method based on a single slide provided in an embodiment of the present invention. Figure 1 and Figure 2As shown, a tumor microenvironment detection method 100 based on a single glass slide according to an embodiment of the present invention includes: 110, acquiring a fluorescence microscope imaging image based on a single glass slide collected by a microscope; 120, performing image feature extraction on the fluorescence microscope imaging image to obtain a semantically combined shallow feature map of tumor growth status; and, 130, determining the growth status of the tumor based on the semantically combined shallow feature map of tumor growth status.

[0038] In step 110, when acquiring fluorescence microscopy images, care is taken to select appropriate fluorescent dyes and filters to ensure the specificity and intensity of the target structure or marker. Furthermore, appropriate exposure time and light source intensity are used to obtain clear, well-contrast images. Fluorescence microscopy images can provide spatial distribution information of specific proteins or markers in cells or tissues, facilitating the observation and analysis of cell types, immune cell infiltration, angiogenesis, and other aspects of the tumor microenvironment.

[0039] In step 120, image feature extraction is the process of converting an image into numerical features. When performing feature extraction, appropriate feature extraction methods are selected, such as texture features, shape features, color features, etc., to capture information related to the tumor growth status in the image. In addition, feature standardization and normalization are considered to ensure comparability between different images. Among them, by performing feature extraction on the fluorescence microscope imaging image, a semantic joint tumor growth status shallow feature map can be obtained. This map can reflect characteristics such as cell distribution, cell morphology, and cell tissue structure in the tumor microenvironment, which helps to further analyze and identify the growth status of the tumor.

[0040] In step 130, based on the semantically combined shallow feature map of tumor growth status, a machine learning algorithm or image analysis method can be applied to determine the tumor growth status. When determining the tumor growth status, an appropriate classifier or algorithm is selected and trained and validated. At the same time, feature selection and model optimization are considered to improve the accuracy and reliability of the growth status. Determining the tumor growth status based on the semantically combined shallow feature map of tumor growth status allows for quantitative analysis and classification of the tumor. This helps to understand the tumor's growth rate, invasiveness, metastatic potential, etc., providing important reference information for tumor diagnosis and treatment.

[0041] The single-slide tumor microenvironment detection method uses fluorescence microscopy to capture spatial information about the tumor microenvironment. It then extracts image features to generate a semantically combined shallow feature map of tumor growth. Finally, an analysis method based on the feature map determines the tumor's growth status. This provides quantitative tumor microenvironment analysis and provides valuable information for tumor diagnosis and treatment.

[0042] In response to the above technical problems, the technical concept of this application is to extract shallow image features and deep image features from fluorescence microscope imaging images, and integrate the semantic information contained in the deep features into the shallow features, narrow the semantic differences, and enrich the feature expression; and then use classification processing to make intelligent judgments on the growth status of the tumor.

[0043] It should be understood that shallow image features refer to underlying visual features, such as edges, textures, colors, etc. These features can capture the local details and surface morphology of the image, which are of great significance for judging the growth status of the tumor. Deep image features are high-level abstract features learned from the image, which have richer semantic expression capabilities and can capture higher-level features and semantic information of the image. Since shallow features mainly capture low-level details and surface morphology of the image, there may be certain limitations in their judgment of the growth status of the tumor, and they cannot fully express the high-level semantic information in the image. Deep features, through the learning of deep learning models, can extract more abstract and more semantic features, which can better describe the semantic information in the image. Therefore, in the technical concept of this application, it is expected that the semantic information in the deep features will be propagated to the shallow features, so that the shallow features can have richer semantic expression capabilities and improve the accuracy of judging the growth status of the tumor.

[0044] Specifically, in step 110, a fluorescence microscope image based on a single slide is acquired by a microscope. In the technical solution of the present application, a fluorescence microscope image based on a single slide is first acquired by a microscope. Fluorescence microscope images can be used to count and locate cells in tumor tissue, providing information about tumor cell density, cell distribution, and aggregation. For example, high cell density and aggregation may indicate high tumor proliferation activity. Fluorescence microscope images can reveal the location and distribution of specific molecular markers. For example, by labeling cell proliferation markers (such as Ki-67) or apoptosis markers (such as Caspase-3), the levels of tumor cell proliferation and apoptosis can be assessed. The location and intensity of these markers can provide information about tumor cell activity and growth status.

[0045] Fluorescently labeled antibodies can be used to label immune cells, such as lymphocytes and macrophages. Fluorescence microscopy images can be used to assess the degree of immune cell infiltration and distribution in tumor tissue. The extent of immune cell infiltration is associated with the tumor's immune response and prognosis. Using fluorescently labeled antibodies or dyes, angiogenesis in tumor tissue can be observed and assessed. The extent of angiogenesis is related to the tumor's nutrient supply and growth potential. Fluorescence microscopy images can provide information on vascular density, vascular morphology, and vascular network structure.

[0046] By analyzing and extracting useful information from fluorescence microscopy images, quantitative and qualitative information about the tumor growth status can be obtained. This information can be used to assess the tumor's aggressiveness, grade, prognosis, etc., and provide guidance for personalized treatment.

[0047] It should be understood that fluorescence microscopy images captured by a microscope and based on a single slide play an important role in determining the growth status of a tumor. On the one hand, they can provide spatial distribution information. Fluorescence microscopy images can provide information on the spatial distribution of specific proteins or markers within the tumor microenvironment. These markers may include immune cells, blood vessels, and extracellular matrix components. By observing the distribution of markers in the images, information about the tumor microenvironment structure and tissue morphology can be obtained.

[0048] On the one hand, fluorescence microscopy can be used to observe cell morphology and type. Images from this technique can reveal the morphology and types of different cells within the tumor microenvironment. By observing cell morphological characteristics, such as size, shape, and nuclear chromatin distribution, a preliminary assessment of tumor cell activity and proliferation can be made. Furthermore, different cell types, such as tumor cells, immune cells, and endothelial cells, can be identified, thereby understanding their relative proportions and distribution within the tumor microenvironment.

[0049] On the other hand, fluorescence signal intensity can be measured. Fluorescence microscopy images provide information on fluorescence signal intensity. Fluorescence signal intensity can reflect the expression level of a specific protein or marker. By comparing fluorescence signal intensities across different regions or cells, differences in the expression levels of specific proteins within the tumor microenvironment can be assessed. This is beneficial for understanding the functional status of tumor cells and the degree of immune cell activation.

[0050] On the other hand, fluorescence microscopy images can provide a basis for quantitative analysis. By processing and analyzing these images, image features such as cell morphology and fluorescence signal intensity can be extracted for further calculation and modeling. These quantitative analysis results can be used to determine tumor growth status, such as growth rate, invasiveness, and metastatic potential.

[0051] That is, obtaining fluorescence microscopy images based on a single glass slide collected by a microscope plays an important role in finally determining the growth status of the tumor. It provides spatial information of the tumor microenvironment, cell morphology and type, fluorescence signal intensity, and the basis for quantitative analysis, which helps to fully understand the growth status and related characteristics of the tumor.

[0052] Specifically, in step 120, Figure 3 Flowchart of the sub-steps of step 120 in a tumor microenvironment detection method based on a single slide provided in an embodiment of the present invention, as shown in FIG. Figure 3 As shown, image feature extraction is performed on the fluorescence microscope imaging image to obtain a semantically combined shallow feature map of tumor growth status, including: 121, extracting shallow feature information and deep feature information of the fluorescence microscope imaging image to obtain a shallow feature map of tumor growth status and a deep feature map of tumor growth status; and, 122, propagating the semantic information of the deep feature map of tumor growth status to the shallow feature map of tumor growth status to obtain the semantically combined shallow feature map of tumor growth status.

[0053] First, in the field of image processing and computer vision, various methods can be used to extract feature information from fluorescence microscope imaging images. These features can be divided into shallow features and deep features.

[0054] Shallow features are extracted from low-level visual features of the image, such as color, texture, shape, etc. Among them, shallow feature extraction methods include gray-level co-occurrence matrix (GLCM), local binary pattern (LBP), histogram of oriented gradients (HOG), etc. These features can provide information about the local structure and texture in the image.

[0055] Deep features are high-level semantic information learned from images using deep learning models. Deep learning models such as convolutional neural networks (CNNs) can be used to extract deep features of images. By training on large-scale datasets, CNNs can automatically learn abstract features in images, such as edges, textures, and shapes. These deep features can provide higher-level semantic information.

[0056] Therefore, by extracting shallow features from fluorescence microscope imaging images and using a deep learning model to extract deep features, shallow feature maps and deep feature maps of tumor growth status can be obtained.

[0057] Then, once the deep feature map and shallow feature map of the tumor growth status are obtained, the semantically joint shallow feature map of the tumor growth status can be obtained by propagating the semantic information of the deep feature map into the shallow feature map.

[0058] Methods for propagating semantic information include feature fusion and attention mechanisms. For example, skip connections or attention mechanisms in convolutional neural networks can be used to fuse semantic information from deep feature maps with shallow feature maps. This allows shallow feature maps to acquire richer semantic information, improving understanding and discrimination of tumor growth status.

[0059] By fusing the semantic information of deep and shallow features, we can generate a semantically combined shallow feature map of tumor growth. This feature map combines the low-level visual information of shallow features with the high-level semantic information of deep features, helping to more accurately describe and understand the tumor growth status.

[0060] This semantically combined feature map can provide more beneficial feature representation for subsequent tumor growth status analysis, classification, and prediction tasks, improving the understanding and prediction capabilities of tumor growth status.

[0061] For the step 121, extracting the shallow feature information and deep feature information of the fluorescence microscope imaging image to obtain a shallow feature map of the tumor growth state and a deep feature map of the tumor growth state includes: passing the fluorescence microscope imaging image through a tumor growth state feature extractor based on a pyramid network to obtain the shallow feature map of the tumor growth state and the deep feature map of the tumor growth state.

[0062] Next, shallow feature information and deep feature information of the fluorescence microscope imaging image are extracted to obtain a shallow feature map of the tumor growth state and a deep feature map of the tumor growth state.

[0063] In a specific example of the present application, the encoding process of extracting shallow feature information and deep feature information of the fluorescence microscope imaging image to obtain a shallow feature map of the tumor growth state and a deep feature map of the tumor growth state includes: passing the fluorescence microscope imaging image through a pyramid network-based tumor growth state feature extractor to obtain a shallow feature map of the tumor growth state and a deep feature map of the tumor growth state.

[0064] A pyramid network is a deep learning network architecture used for image processing and computer vision tasks. It is designed to extract features from a pyramid of images at different scales to capture multi-scale information in images. A pyramid network typically consists of multiple parallel branches, each processing input images at a different scale. Each branch contains a feature extractor, which can be a convolutional neural network (CNN) or other feature extraction method, to extract features from the image at the corresponding scale.

[0065] In feature extraction of tumor growth status, pyramid networks can be used to extract features from fluorescence microscopy images at different scales. Since cells and structures in tumor tissue vary in size and scale, using pyramid networks can better capture this multi-scale information. Through pyramid networks, both shallow and deep feature maps of tumor growth status can be obtained. The shallow feature map can provide local, low-level features such as texture and color, while the deep feature map can provide higher-level semantic information, such as tumor cell morphology and distribution.

[0066] By combining the multi-scale features of pyramid networks, more comprehensive and rich information about tumor growth status can be obtained. These features can be used for further analysis and judgment, such as tumor grading and prognosis assessment. Pyramid networks are a network structure for multi-scale feature extraction and can be applied to tumor growth status analysis in fluorescence microscopy images, providing both shallow and deep feature maps for more comprehensive information.

[0067] In this application, a pyramid network can process input images at different scales and extract features at each scale. For fluorescence microscopy images, the details and structure of tumor tissue may vary at different scales. Using a pyramid network can capture this multi-scale information, helping to fully understand the state of tumor growth.

[0068] Each branch in a deep pyramid network can contain a deep learning model, such as a convolutional neural network (CNN). These models can learn high-level semantic features in images, such as cell morphology and tissue structure. By applying these deep learning models at different scales, rich semantic information can be obtained, helping to accurately describe the growth status of tumors.

[0069] The multiple branches in a pyramid network can integrate feature information at various scales. This multi-scale fusion improves feature accuracy and robustness, reducing misjudgments caused by the limitations of single-scale features. By fusing shallow and deep features, the pyramid network can provide a more comprehensive and accurate picture of tumor growth status.

[0070] For the step 122, the semantic information of the deep feature map of the tumor growth state is propagated to the shallow feature map of the tumor growth state to obtain the semantic joint shallow feature map of the tumor growth state, including: performing global mean pooling on the deep feature map of the tumor growth state to obtain a semantic feature vector of the tumor growth state; passing the semantic feature vector of the tumor growth state through a semantic weight learner based on a point convolution layer to obtain a semantic weight vector; and using the semantic weight vector as a weight vector to weight each feature matrix along the channel dimension of the shallow feature map of the tumor growth state to obtain the semantic joint shallow feature map of the tumor growth state.

[0071] Then, the semantic information of the deep feature map of the tumor growth state is propagated to the shallow feature map of the tumor growth state to obtain the semantically combined shallow feature map of the tumor growth state. In a specific example of the present application, the encoding process of propagating the semantic information of the deep feature map of the tumor growth state to the shallow feature map of the tumor growth state to obtain the semantically combined shallow feature map of the tumor growth state includes: first performing global mean pooling on the deep feature map of the tumor growth state to obtain a semantic feature vector of the tumor growth state; then, passing the semantic feature vector of the tumor growth state through a semantic weight learner based on a point convolution layer to obtain a semantic weight vector; and then using the semantic weight vector as a weight vector to weight each feature matrix of the shallow feature map of the tumor growth state along the channel dimension to obtain a semantically combined shallow feature map of the tumor growth state.

[0072] It should be understood that deep feature maps contain high-level semantic information, while shallow feature maps provide more local details. By propagating the semantic information of deep feature maps to shallow feature maps, we can achieve the fusion of semantic information and detail information, thereby obtaining a more comprehensive and rich feature representation.

[0073] By performing global mean pooling on the deep feature map, a global semantic feature vector is generated. This global semantic feature vector captures the semantic information of the entire image and represents the overall characteristics of the image. This global semantic feature vector can provide a comprehensive understanding of the entire tumor growth state, helping to more comprehensively describe the tumor growth status.

[0074] The semantic weight learner based on the point convolutional layer can learn a semantic weight vector for each channel. The semantic weight vector represents the contribution of each channel to the overall semantics. By learning the semantic weight vector, the weight of each channel can be adaptively adjusted to highlight important semantic information and suppress unimportant information.

[0075] Using the learned semantic weight vectors, we can weight the individual feature matrices of the shallow feature map of tumor growth status. This weighting operation prioritizes important semantic information while suppressing less important information. Through weighted feature fusion, we can generate a semantically unified shallow feature map of tumor growth status, better capturing the balance between semantics and details.

[0076] By propagating the semantic information of the deep feature map of tumor growth status to the shallow feature map, and utilizing the global semantic feature vector, semantic weight learning and weighted feature fusion methods, a semantically united shallow feature map of tumor growth status can be obtained, providing a more comprehensive and rich feature representation, which helps to deeply understand and analyze the growth status of the tumor.

[0077] Specifically, in step 130, the growth status of the tumor is determined based on the semantic joint tumor growth status shallow feature map, including: optimizing the semantic joint tumor growth status shallow feature map to obtain an optimized semantic joint tumor growth status shallow feature map; and passing the optimized semantic joint tumor growth status shallow feature map through a classifier to obtain a classification result, and the classification result is used to represent the growth status label of the tumor.

[0078] In the technical solution of the present application, when the semantic weight vector is used as the weight vector to weight the various feature matrices along the channel dimension of the shallow feature map of the tumor growth state to obtain the semantic joint shallow feature map of the tumor growth state, the shallow image semantic features of the fluorescence microscope imaging image are weighted based on the channel distribution of the deep image semantic features expressed by the various feature matrices of the deep feature map of the tumor growth state. This will also cause the image semantic feature distribution of the semantic joint shallow feature map of the tumor growth state to deviate from the shallow image semantic feature distribution of the shallow feature map of the tumor growth state itself, thereby causing the semantic joint shallow feature map of the tumor growth state to have an image semantic distribution knowledge offset relative to the semantic distribution of the source image, thereby causing uncertainty in its image semantic feature distribution relative to the class probability understanding of the classifier, reducing the speed of classification training and the accuracy of the classification results.

[0079] Therefore, in a preferred embodiment, the semantic joint tumor growth state shallow feature map is optimized to obtain an optimized semantic joint tumor growth state shallow feature map, which specifically includes the following steps: determining the class probability value obtained by the classifier of the semantic joint tumor growth state shallow feature map, and subtracting the class probability value from one to obtain a class difference probability value; calculating the power function of each eigenvalue of the semantic joint tumor growth state shallow feature map with the class probability value as an exponent to obtain the semantic joint tumor growth state shallow class feature map; calculating the power function of each eigenvalue of the semantic joint tumor growth state shallow feature map with the class difference probability value as an exponent to obtain the semantic joint tumor growth state shallow class feature map. Long state shallow class difference feature map; perform point multiplication of the class probability value and the semantic joint tumor growth state shallow class difference feature map to obtain a first semantic joint tumor growth state shallow intermediate feature map, and perform point multiplication of the class difference probability value and the semantic joint tumor growth state shallow class feature map to obtain a second semantic joint tumor growth state shallow intermediate feature map; perform point multiplication of the first semantic joint tumor growth state shallow intermediate feature map and the second semantic joint tumor growth state shallow intermediate feature map, and further perform point addition with the dot product result of the class probability value and the semantic joint tumor growth state shallow class difference feature map to obtain an optimized semantic joint tumor growth state shallow feature map.

[0080] That is, when the semantically united shallow feature map of tumor growth status is classified by a classifier, in order to realize unsupervised domain adaptation of the feature distribution domain of the semantically united shallow feature map of tumor growth status to the probability distribution domain of classification probability, the class probability value and the class difference probability value obtained by the classifier of the semantically united shallow feature map of tumor growth status are used as domain proxies, and the moving average of the probability distribution of the semantically united shallow feature map of tumor growth status is performed through the interaction of the class distribution of the power function as an exponent, and the knowledge transfer from unlabeled classification features to labeled probability distribution is realized by superimposing the feature domain probability distribution knowledge of the semantically united shallow feature map of tumor growth status itself, thereby promoting the classification operation of the semantically united shallow feature map of tumor growth status through the classifier, that is, improving the speed of classification training and the accuracy of classification results.

[0081] Furthermore, the optimized semantic combined with the shallow feature map of tumor growth status is passed through a classifier to obtain a classification result, and the classification result is used to represent the growth status label of the tumor.

[0082] In summary, the single-slide-based tumor microenvironment detection method 100 according to an embodiment of the present invention is explained, which extracts shallow image features and deep image features from fluorescence microscope imaging images, and integrates the semantic information contained in the deep features into the shallow features, narrowing the semantic differences and enriching the feature expression; and then intelligently judges the growth status of the tumor through classification processing.

[0083] Figure 4 FIG. 1 is a block diagram of a tumor microenvironment detection system based on a single slide provided in an embodiment of the present invention. Figure 4 As shown, the tumor microenvironment detection system based on a single glass slide includes: an image acquisition module 210, used to obtain a fluorescence microscope imaging image based on a single glass slide collected by a microscope; an image feature extraction module 220, used to extract image features from the fluorescence microscope imaging image to obtain a semantically combined shallow feature map of the tumor growth status; and a tumor growth status determination module 230, used to determine the growth status of the tumor based on the semantically combined shallow feature map of the tumor growth status.

[0084] Specifically, in the tumor microenvironment detection system based on a single slide, the image feature extraction module includes: a feature information extraction unit, used to extract shallow feature information and deep feature information of the fluorescence microscope imaging image to obtain a shallow feature map of the tumor growth status and a deep feature map of the tumor growth status; and a semantic information propagation unit, used to propagate the semantic information of the deep feature map of the tumor growth status to the shallow feature map of the tumor growth status to obtain the semantic combined shallow feature map of the tumor growth status.

[0085] Specifically, in the tumor microenvironment detection system based on a single slide, the feature information extraction unit is used to: pass the fluorescence microscope imaging image through a tumor growth state feature extractor based on a pyramid network to obtain the shallow feature map of the tumor growth state and the deep feature map of the tumor growth state.

[0086] Specifically, in the tumor microenvironment detection system based on a single slide, the semantic information propagation unit is used for: a global mean pooling subunit, used to perform global mean pooling on the deep feature map of the tumor growth status to obtain a semantic feature vector of the tumor growth status; a semantic weight learning subunit, used to pass the semantic feature vector of the tumor growth status through a semantic weight learner based on a point convolution layer to obtain a semantic weight vector; and a weighting subunit, used to use the semantic weight vector as a weight vector to weight each feature matrix along the channel dimension of the shallow feature map of the tumor growth status to obtain the semantic joint shallow feature map of the tumor growth status.

[0087] Those skilled in the art will appreciate that the specific operations of each step in the above-mentioned single slide-based tumor microenvironment detection system have been described in detail above. Figures 1 to 3 The single slide-based tumor microenvironment detection method has been described in detail in the description of the single slide-based tumor microenvironment detection method, and therefore, its repeated description will be omitted.

[0088] As described above, the single-slide-based tumor microenvironment detection system 100 according to an embodiment of the present invention can be implemented in various terminal devices, such as a server for single-slide-based tumor microenvironment detection. In one example, the single-slide-based tumor microenvironment detection system 100 according to an embodiment of the present invention can be integrated into a terminal device as a software module and / or a hardware module. For example, the single-slide-based tumor microenvironment detection system 100 can be a software module in the terminal device's operating system, or an application developed specifically for the terminal device. Of course, the single-slide-based tumor microenvironment detection system 100 can also be one of the terminal device's many hardware modules.

[0089] Alternatively, in another example, the single slide-based tumor microenvironment detection system 100 and the terminal device may also be separate devices, and the single slide-based tumor microenvironment detection system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0090] Figure 5 FIG2 is an application scenario diagram of a tumor microenvironment detection method based on a single slide provided in an embodiment of the present invention. Figure 5 As shown, in this application scenario, first, a microscope (e.g., Figure 5 Single slide-based fluorescence microscopy images (e.g., Figure 5 Then, the acquired fluorescence microscope imaging image based on a single glass slide is input to a server (e.g., Figure 5 In S) as shown in , the server is capable of processing the single-slide-based fluorescence microscope imaging image based on a single-slide tumor microenvironment detection algorithm to determine the growth status of the tumor.

[0091] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting tumor microenvironment based on a single slide, characterized in that: include: Acquiring single slide-based fluorescence microscopy images collected by a microscope; performing image feature extraction on the fluorescence microscope imaging image to obtain a semantically combined shallow feature map of tumor growth status; as well as Determining the growth status of the tumor based on the semantic combined shallow feature map of the tumor growth status; Determining the tumor growth status based on the semantic combined shallow feature map of the tumor growth status includes: Optimizing the semantically combined tumor growth state shallow feature map to obtain an optimized semantically combined tumor growth state shallow feature map; and Passing the optimized semantic combined tumor growth status shallow feature map through a classifier to obtain a classification result, wherein the classification result is used to represent a tumor growth status label; Among them, the semantic joint tumor growth state shallow feature map is optimized to obtain an optimized semantic joint tumor growth state shallow feature map, which specifically includes the following steps: determining the class probability value obtained by the semantic joint tumor growth state shallow feature map through the classifier, and subtracting the class probability value from one to obtain a class difference probability value; calculating the power function of each eigenvalue of the semantic joint tumor growth state shallow feature map with the class probability value as an exponent to obtain the semantic joint tumor growth state shallow class feature map; calculating the power function of each eigenvalue of the semantic joint tumor growth state shallow feature map with the class difference probability value as an exponent to obtain the semantic joint tumor growth state shallow class difference feature map; perform point multiplication of the class probability value and the semantic joint tumor growth state shallow class difference feature map to obtain a first semantic joint tumor growth state shallow intermediate feature map, and perform point multiplication of the class difference probability value and the semantic joint tumor growth state shallow class feature map to obtain a second semantic joint tumor growth state shallow intermediate feature map; perform point multiplication of the first semantic joint tumor growth state shallow intermediate feature map and the second semantic joint tumor growth state shallow intermediate feature map, and further perform point addition with the dot product result of the class probability value and the semantic joint tumor growth state shallow class difference feature map to obtain an optimized semantic joint tumor growth state shallow feature map.

2. The method for detecting tumor microenvironment based on a single slide according to claim 1, wherein: Performing image feature extraction on the fluorescence microscope imaging image to obtain a semantically combined shallow feature map of tumor growth status, including: Extracting shallow feature information and deep feature information of the fluorescence microscope imaging image to obtain a shallow feature map of tumor growth status and a deep feature map of tumor growth status; and The semantic information of the deep feature map of the tumor growth state is propagated to the shallow feature map of the tumor growth state to obtain the semantic combined shallow feature map of the tumor growth state.

3. The method for detecting tumor microenvironment based on a single slide according to claim 2, wherein: Extracting shallow feature information and deep feature information of the fluorescence microscope imaging image to obtain a shallow feature map of tumor growth status and a deep feature map of tumor growth status, including: The fluorescence microscope imaging image is passed through a tumor growth state feature extractor based on a pyramid network to obtain the tumor growth state shallow feature map and the tumor growth state deep feature map.

4. The method for detecting tumor microenvironment based on a single slide according to claim 3, wherein: Propagating the semantic information of the deep feature map of the tumor growth state to the shallow feature map of the tumor growth state to obtain the semantic combined shallow feature map of the tumor growth state, including: performing global mean pooling on the deep feature map of tumor growth status to obtain a semantic feature vector of tumor growth status; Passing the semantic feature vector of tumor growth status through a semantic weight learner based on a point convolution layer to obtain a semantic weight vector; and The semantic weight vector is used as a weight vector to weight each feature matrix along the channel dimension of the shallow feature map of the tumor growth status to obtain the semantic joint shallow feature map of the tumor growth status.

5. A tumor microenvironment detection system based on a single slide, characterized in that: include: An image acquisition module, used for acquiring a fluorescence microscope imaging image based on a single glass slide collected by a microscope; an image feature extraction module, configured to extract image features from the fluorescence microscope image to obtain a semantically combined shallow feature map of tumor growth status; and a tumor growth status determination module, configured to determine the tumor growth status based on the semantics combined with the shallow feature map of the tumor growth status; Determining the tumor growth status based on the semantic combined shallow feature map of the tumor growth status includes: Optimizing the semantically combined tumor growth state shallow feature map to obtain an optimized semantically combined tumor growth state shallow feature map; and Passing the optimized semantic combined tumor growth status shallow feature map through a classifier to obtain a classification result, wherein the classification result is used to represent a tumor growth status label; Among them, the semantic joint tumor growth state shallow feature map is optimized to obtain an optimized semantic joint tumor growth state shallow feature map, which specifically includes the following steps: determining the class probability value obtained by the semantic joint tumor growth state shallow feature map through the classifier, and subtracting the class probability value from one to obtain a class difference probability value; calculating the power function of each eigenvalue of the semantic joint tumor growth state shallow feature map with the class probability value as an exponent to obtain the semantic joint tumor growth state shallow class feature map; calculating the power function of each eigenvalue of the semantic joint tumor growth state shallow feature map with the class difference probability value as an exponent to obtain the semantic joint tumor growth state shallow class difference feature map; perform point multiplication of the class probability value and the semantic joint tumor growth state shallow class difference feature map to obtain a first semantic joint tumor growth state shallow intermediate feature map, and perform point multiplication of the class difference probability value and the semantic joint tumor growth state shallow class feature map to obtain a second semantic joint tumor growth state shallow intermediate feature map; perform point multiplication of the first semantic joint tumor growth state shallow intermediate feature map and the second semantic joint tumor growth state shallow intermediate feature map, and further perform point addition with the dot product result of the class probability value and the semantic joint tumor growth state shallow class difference feature map to obtain an optimized semantic joint tumor growth state shallow feature map.

6. The single slide-based tumor microenvironment detection system according to claim 5, characterized in that: The image feature extraction module includes: a feature information extraction unit, configured to extract shallow feature information and deep feature information of the fluorescence microscope imaging image to obtain a shallow feature map of tumor growth status and a deep feature map of tumor growth status; and A semantic information propagation unit is used to propagate the semantic information of the deep feature map of the tumor growth state to the shallow feature map of the tumor growth state to obtain the semantic combined shallow feature map of the tumor growth state.

7. The single slide-based tumor microenvironment detection system according to claim 6, characterized in that: The feature information extraction unit is used to: The fluorescence microscope imaging image is passed through a tumor growth state feature extractor based on a pyramid network to obtain the tumor growth state shallow feature map and the tumor growth state deep feature map.

8. The single slide-based tumor microenvironment detection system according to claim 7, characterized in that: The semantic information propagation unit is used to: a global mean pooling subunit, configured to perform global mean pooling on the deep feature map of the tumor growth state to obtain a semantic feature vector of the tumor growth state; a semantic weight learning subunit, configured to pass the semantic feature vector of the tumor growth state through a semantic weight learner based on a point convolution layer to obtain a semantic weight vector; and A weighting subunit is used to weight each feature matrix along the channel dimension of the shallow feature map of the tumor growth state using the semantic weight vector as a weight vector to obtain the semantic joint shallow feature map of the tumor growth state.

Images