Tumor microenvironment protein marker detection system and method based on multicolor immunofluorescence

Abstract

The embodiment of the invention provides a tumor microenvironment protein marker detection system and method based on multicolor immunofluorescence, and belongs to the technical field of image processing. According to the system, a first segmentation module performs image segmentation on a multi-color fluorescence image by using a superpixel segmentation algorithm to obtain a first segmentation result; the second segmentation module performs image segmentation on the multicolor fluorescence image by using a feature clustering algorithm to obtain a second segmentation result; the segmentation fusion module fuses the first segmentation result and the second segmentation result to obtain a target segmentation result; a type determination module performs chromaticity identification on sub-segmentation results in the target segmentation result to obtain protein marker types; the information characterization module performs color characterization on each sub-segmentation result in the target segmentation result to obtain a first characterization result, and determines a second characterization result of the sub-segmentation results according to the protein marker type; and the result determination module is used for calculating a target distance between the first characterization result and the second characterization result and determining a protein marker detection result according to the target distance.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a tumor microenvironment protein marker detection system and method based on multicolor immunofluorescence. Background Art

[0002] Most tumors originate from epithelial cells, including common gastrointestinal cancer, lung cancer, liver cancer, cervical cancer, etc. These epithelial tumors are usually marked and identified by epithelial cell-specific adhesion molecules. In addition to epithelial tumors, the role of immune cells in the tumor microenvironment cannot be ignored. The recognition of immune cells mainly depends on surface markers such as CD3, CD8, CD4, CD19, CD20, CD21, and CD68. In addition, vascular endothelial cells and tumor-associated fibroblasts inside the tumor are also important research objects. Although the staining methods for various targets are different, no matter which method is used, ensuring that the staining image has a clear field of view under the microscope is the basis and key step for subsequent molecular dynamics analysis. This is crucial because clear staining images can provide accurate data, laying a solid foundation for in-depth analysis at the molecular level.

[0003] However, the related technology has certain limitations in post-staining image processing. Specifically, although the staining process can effectively mark the target area, it is difficult for the related technology to accurately distinguish and identify the stained area during the image analysis stage. This inaccuracy may lead to deviations in subsequent analysis results, which in turn affects the in-depth understanding of the tumor microenvironment and its related biological processes.

[0004] Therefore, it is urgent to design a technical solution to solve at least one of the above technical problems. Summary of the invention

[0005] The main purpose of the embodiments of the present invention is to provide a tumor microenvironment protein marker detection system and method based on multicolor immunofluorescence, aiming to solve the problem in related technologies that it is difficult to accurately distinguish and identify stained areas, thereby reducing the accuracy of subsequent analysis results.

[0006] In a first aspect, an embodiment of the present invention provides a tumor microenvironment protein marker detection system based on multicolor immunofluorescence, comprising:

[0007] An image acquisition module is used to obtain a multicolor fluorescent image of the object to be stained under a target staining mode;

[0008] A first segmentation module, configured to perform image segmentation on the multi-color fluorescent image using a superpixel segmentation algorithm to obtain a first segmentation result corresponding to the multi-color fluorescent image;

[0009] A second segmentation module, configured to perform image segmentation on the multi-color fluorescence image by using a feature clustering algorithm to obtain a second segmentation result corresponding to the multi-color fluorescence image;

[0010] A segmentation fusion module, configured to fuse the first segmentation result and the second segmentation result to obtain a target segmentation result corresponding to the multi-color fluorescence image;

[0011] A type determination module, configured to perform chromaticity recognition on each sub-segmentation result in the target segmentation result to obtain a protein marker type corresponding to the sub-segmentation result;

[0012] An information characterization module, configured to perform color characterization on each sub-segmentation result in the target segmentation result to obtain a corresponding first characterization result, and determine a second characterization result corresponding to the sub-segmentation result according to the protein marker type;

[0013] A result determination module, configured to calculate a target distance between the first characterization result and the second characterization result, and determine a protein marker detection result corresponding to the sub-segmentation result according to the target distance.

[0014] In a second aspect, an embodiment of the present invention provides a method for detecting protein markers in a tumor microenvironment based on multi-color immunofluorescence, including:

[0015] Obtaining a multi-color fluorescence image obtained by staining a to-be-stained object in a target staining manner;

[0016] Performing image segmentation on the multi-color fluorescence image by using a superpixel segmentation algorithm to obtain a first segmentation result corresponding to the multi-color fluorescence image;

[0017] Performing image segmentation on the multi-color fluorescence image by using a feature clustering algorithm to obtain a second segmentation result corresponding to the multi-color fluorescence image;

[0018] Fusing the first segmentation result and the second segmentation result to obtain a target segmentation result corresponding to the multi-color fluorescence image;

[0019] Performing chromaticity recognition on each sub-segmentation result in the target segmentation result to obtain a protein marker type corresponding to the sub-segmentation result;

[0020] Performing color characterization on each sub-segmentation result in the target segmentation result to obtain a corresponding first characterization result, and determining a second characterization result corresponding to the sub-segmentation result according to the protein marker type;

[0021] Calculating a target distance between the first characterization result and the second characterization result, and determining a protein marker detection result corresponding to the sub-segmentation result according to the target distance.

[0022] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any one of the tumor microenvironment protein marker detection systems based on multicolor immunofluorescence provided in the specification of the present invention are realized.

[0023] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any step of the tumor microenvironment protein marker detection system based on multicolor immunofluorescence as provided in the specification of the present invention.

[0024] The embodiment of the present invention provides a tumor microenvironment protein marker detection system and method based on multicolor immunofluorescence, the system comprising: an image acquisition module obtains a multicolor fluorescent image corresponding to the object to be stained by a target staining method, the multicolor fluorescent image can reflect the expression of different proteins, and provide basic data for subsequent analysis, and then the first segmentation module uses a superpixel segmentation algorithm to perform image segmentation on the multicolor fluorescent image to obtain a first segmentation result corresponding to the multicolor fluorescent image, and the second segmentation module uses a feature clustering algorithm to segment the multicolor fluorescent image to form different segmentation areas to obtain a second segmentation result, and then the segmentation fusion module fuses the first segmentation result of the superpixel segmentation and the second segmentation result of the feature clustering segmentation to combine the advantages of the two methods, and produce a more accurate and robust target segmentation result, thereby reducing the error that may be caused by a single method and improving the overall segmentation quality. Then, according to the type determination module, the sub-segmentation result after the target segmentation result is subjected to chromatic recognition to determine the type of protein marker corresponding to each sub-segmentation result. This step is based on the fact that different proteins show different fluorescent colors after staining, and the type of protein can be inferred by color analysis, so that the information characterization module performs color characterization on the target segmentation result to generate a first characterization result, and determines the second characterization result according to the protein marker type. This step aims to quantify the expression characteristics of proteins and provide numerical representation for subsequent comparison and analysis. Finally, the result determination module can evaluate the degree of match between the actual observed protein expression and the expected type by calculating the target distance between the first characterization result and the second characterization result. According to the target distance, the protein marker detection result corresponding to each sub-segmentation result can be determined, thereby realizing the automatic recognition and classification of proteins. Then, the system can effectively improve the accuracy and reliability of protein marker detection in multi-color fluorescence images, which is of great significance for biomedical research and clinical diagnosis. It also solves the problem in related technologies that it is difficult to accurately distinguish and identify stained areas, thereby reducing the accuracy of subsequent analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A schematic diagram of the module structure of a tumor microenvironment protein marker detection system based on multicolor immunofluorescence provided in an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of a process of a tumor microenvironment protein marker detection system based on multicolor immunofluorescence provided in an embodiment of the present invention;

[0028] Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0031] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0032] The embodiment of the present invention provides a tumor microenvironment protein marker detection system and method based on multicolor immunofluorescence. Among them, the tumor microenvironment protein marker detection system based on multicolor immunofluorescence can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The terminal device can be a server or a server cluster.

[0033] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] Please refer to Figure 1 , Figure 1 A schematic diagram of the module structure of a tumor microenvironment protein marker detection system 100 based on multicolor immunofluorescence provided in an embodiment of the present invention.

[0035] like Figure 1 As shown, the tumor microenvironment protein marker detection system 100 based on multi-color immunofluorescence includes an image acquisition module 101, a first segmentation module 102, a second segmentation module 103, a segmentation fusion module 104, a type determination module 105, an information representation module 106, and a result determination module 107, wherein the image acquisition module 101 is used to obtain a multi-color fluorescence image obtained by staining the object to be stained under the target staining mode; the first segmentation module 102 is used to perform image segmentation on the multi-color fluorescence image using a superpixel segmentation algorithm to obtain a first segmentation result corresponding to the multi-color fluorescence image; the second segmentation module 103 is used to perform image segmentation on the multi-color fluorescence image using a feature clustering algorithm to obtain a second segmentation result corresponding to the multi-color fluorescence image ; A segmentation and fusion module 104 is used to fuse the first segmentation result and the second segmentation result to obtain a target segmentation result corresponding to the multi-color fluorescent image; a type determination module 105 is used to perform chromaticity recognition on each sub-segmentation result in the target segmentation result to obtain a protein marker type corresponding to the sub-segmentation result; an information characterization module 106 is used to perform color characterization on each sub-segmentation result in the target segmentation result to obtain a corresponding first characterization result, and determine a second characterization result corresponding to the sub-segmentation result according to the protein marker type; a result determination module 107 is used to calculate a target distance between the first characterization result and the second characterization result, and determine a protein marker detection result corresponding to the sub-segmentation result according to the target distance.

[0036] Exemplarily, the image acquisition module 101 stains the biological sample of the object to be stained according to the target staining method. The target staining method generally involves the use of specific antibodies and fluorescent markers, so that different proteins emit fluorescence of different colors under the action of excitation light, and then use a high-sensitivity, multi-channel fluorescence microscope to collect multi-color fluorescence images. In addition, since different proteins have different staining methods, the target staining method can be one or more, thereby obtaining a multi-color fluorescence image corresponding to the object to be stained under the target staining method.

[0037] Exemplarily, the first segmentation module 102 uses a superpixel segmentation algorithm (such as SLIC, Simple Linear Iterative Clustering) to segment the multi-color fluorescence image into a plurality of superpixel regions. These regions have similarities in color or texture, thereby optimizing the boundaries of the superpixels, ensuring the smoothness and consistency of the segmentation results, and then obtaining the first segmentation result.

[0038] Exemplarily, the second segmentation module 103 extracts features of the multi-color fluorescence image (such as color histogram, texture features, etc.), and then applies a clustering algorithm (such as K-means, Mean Shift) to segment the image into different regions based on the extracted features, thereby obtaining a second segmentation result.

[0039] Exemplarily, the segmentation fusion module 104 performs data fusion on the first segmentation result and the second segmentation result by using a weighted fusion method or a decision layer fusion method to generate a final target segmentation result.

[0040] Exemplarily, the type determination module 105 performs chromaticity analysis on each sub-segmentation result in the target segmentation result in the color space, for example, converts the sub-segmentation result into the HSV color space for hue analysis to calculate the average hue value of all pixels in the sub-segmentation result.

[0041] Exemplarily, the type determination module 105 establishes a reference library containing known protein marker types and their corresponding chromaticity features. Each protein marker type has a chromaticity feature reference range, and then searches the chromaticity feature reference range according to the average hue value to obtain the protein marker type corresponding to the sub-segmentation result.

[0042] Exemplarily, the information characterization module 106 uses statistics such as color histogram, mean and variance to quantify the color features of each sub-segmentation result to generate a first characterization result, and obtains a second characterization result corresponding to the protein marker type when the protein is normal from the database.

[0043] Exemplarily, the result determination module 107 uses a metric method such as Euclidean distance and cosine similarity to calculate the distance between the first characterization result and the second characterization result to obtain a target distance, where the target distance refers to the difference between the chromaticity features of the sub-segmentation result actually detected and the chromaticity features of the expected protein marker type. The smaller the target distance, the closer the actual detection result is to the expected type; conversely, the larger the target distance, the greater the difference. A threshold is set to distinguish between normal and abnormal protein marker detection results. The selection of the threshold should be based on experimental data and experience to ensure that normal and abnormal situations can be effectively distinguished in practical applications. For example, the threshold can be dynamically adjusted according to specific application scenarios and requirements. For example, in some high-sensitivity detection scenarios, a lower threshold can be set to identify more abnormal situations; in other scenarios, a higher threshold can be set to avoid false alarms, and the calculated target distance is compared with the set threshold. If the target distance is less than the threshold, it indicates that the actual detected chromaticity feature is very close to the chromaticity feature of the expected protein marker type, and the protein marker detection result can be judged to be normal. If the target distance is greater than the threshold, it indicates that the actual detected chromaticity feature is significantly different from the chromaticity feature of the expected protein marker type, and the protein marker detection result can be judged to be abnormal.

[0044] In some embodiments, the first segmentation module includes: a data analysis module, which is used to perform color distribution analysis on the multi-color fluorescent image to obtain target distribution information, and determine the target entropy information corresponding to the multi-color fluorescent image based on the target distribution information; a number determination module, which is used to determine the image length and image width corresponding to the multi-color fluorescent image, and determine the number of superpixels corresponding to the multi-color fluorescent image based on the image length, the image width and the target entropy information; a feature analysis module, which is used to determine the initial cluster center from the multi-color fluorescent image based on the number of superpixels, and obtain the first color feature and the first spatial feature corresponding to the initial cluster center; a feature acquisition module, which is used to obtain the second color feature and the second spatial feature corresponding to the remaining pixel points of the multi-color fluorescent image except the initial cluster center; a distance calculation module , used to determine the distance information between the remaining pixels and the initial cluster center according to the first color feature, the first spatial feature, the second color feature and the second spatial feature; a first cluster analysis module, used to perform cluster analysis on the multi-color fluorescent image according to the distance information to obtain a first cluster result corresponding to the initial cluster center; an abnormal filtering module, used to perform abnormal filtering on each sub-cluster result in the first cluster result to obtain a second cluster result; a cluster updating module, used to update the initial cluster center according to the second clustering result to obtain a target cluster center, and re-perform cluster analysis according to the target cluster center to obtain a target cluster result; a first segmentation processing module, used to perform image segmentation on the multi-color fluorescent image according to the target cluster result to obtain the first segmentation result corresponding to the multi-color fluorescent image.

[0045] Exemplarily, the data analysis module calculates the color histogram of the multi-color fluorescent image and counts the pixel distribution of each color channel. The histogram of each color channel, such as the red, green, and blue channels, can be calculated separately, and then a joint histogram of the multi-channels can be generated to capture the relationship between different color channels, and then the target distribution information corresponding to the multi-color fluorescent image can be extracted from the color histogram and the joint histogram. Thus, the target entropy information corresponding to the multi-color fluorescent image can be obtained using the entropy value calculation formula according to the target distribution information.

[0046] Exemplarily, the number determination module obtains the target length and target width of the multicolor fluorescent image, and then adds the target length and target width, multiplies the sum with the target entropy information, and divides the sum by a preset number to obtain the number of superpixels, wherein the preset number is the minimum number in the superpixel region set based on experience.

[0047] Exemplarily, the feature analysis module randomly selects a number of superpixels as initial cluster centers in the multi-color fluorescence image, thereby extracting a first color feature of each initial cluster center and extracting a spatial position feature of each initial cluster center to obtain a first spatial feature.

[0048] Exemplarily, the feature acquisition module extracts the second color features of all pixel points that are not the initial cluster center in the multi-color fluorescent image and extracts the spatial position features of all pixel points that are not the initial cluster center to obtain the second spatial features.

[0049] Exemplarily, the distance calculation module uses Euclidean distance or other similarity measurement methods to calculate the color distance between each pixel point of the non-initial cluster center and the initial cluster center according to the first color feature and the second color feature, and uses Euclidean distance or other similarity measurement methods to calculate the spatial distance between each pixel point of the non-initial cluster center and the initial cluster center according to the first spatial feature and the second spatial feature, thereby fusing the color distance and the spatial distance in the form of a weighted sum to form the distance information between the pixel point of each non-initial cluster center and the initial cluster center.

[0050] Exemplarily, the first cluster analysis module allocates each pixel point that is not at the initial cluster center to the initial cluster center that is closest to the pixel point according to the distance information to form a first clustering result.

[0051] Exemplarily, the abnormal filtering module calculates the gray level co-occurrence matrix for each sub-clustering result in the first clustering result, thereby obtaining the homogeneity and correlation between each pixel and other pixels in the sub-clustering result according to the gray level co-occurrence matrix, and then obtaining the similarity between each pixel and other pixels according to the homogeneity and correlation, and then removing pixels with low similarity from the sub-clustering result to achieve abnormal filtering to obtain the second clustering result. Therefore, the second clustering result obtained after abnormal filtering provides good support for the subsequent cluster center update, thereby ensuring the quality and accuracy of the subsequent cluster center update.

[0052] Exemplarily, the cluster update module calculates the centroid of each sub-cluster result in the second cluster result as a new cluster center to determine as the target cluster center, and then recalculates the comprehensive distance between each pixel point of the non-initial cluster center and the target cluster center, thereby redistributing the pixel points of each non-initial cluster center according to the new comprehensive distance to form the target cluster result. In addition, if the cluster center of the target cluster result does not change again, the target cluster result is determined as the final cluster result; if the cluster center of the target cluster result changes after the cluster center calculation, it is necessary to continue to execute the abnormal filtering module and the cluster update model until the target cluster center does not change.

[0053] Exemplarily, the first segmentation processing module segments the multi-color fluorescent image according to the target clustering result, and assigns each pixel point to a corresponding cluster center to form a first segmentation result.

[0054] In some embodiments, the distance calculation module includes: a first feature disassembly module, which is used to determine the first feature information corresponding to the initial cluster center according to the first color feature and the first spatial feature; a second feature disassembly module, which is used to determine the second feature information corresponding to the remaining pixels according to the second color feature and the second spatial feature; a target calculation module, which is used to perform distance calculation between the initial cluster center and the remaining pixels according to the first feature information and the second feature information to obtain the distance information; wherein the distance information is obtained according to the following formula:

[0055]

[0056] Wherein, k represents the number of features corresponding to the first feature information or the second feature information, ij represents the distance information between the i-th initial cluster center and the j-th remaining pixel point, v it represents the tth first feature information of the i-th initial cluster center, v jt Represents the t-th second feature information of the j-th remaining pixel.

[0057] Exemplarily, the first feature decomposition module decomposes the first color feature and the first spatial feature of the initial cluster center to extract feature information for calculating the distance. For example, multiple color features corresponding to the initial cluster center, such as RGB values, HSV values, etc., are decomposed from the first color feature, and horizontal spatial features and vertical spatial features corresponding to the initial cluster center are decomposed from the first spatial feature, so that the multiple color features and the multiple spatial features are combined together as the first feature information.

[0058] Exemplarily, the second feature decomposition module decomposes the second color feature and the second spatial feature of the remaining pixels to extract feature information for calculating the distance. For example, multiple color features corresponding to the remaining pixels, such as RGB values, HSV values, etc., are decomposed from the second color feature, and horizontal spatial features and vertical spatial features corresponding to the remaining pixels are decomposed from the second spatial feature, so that the multiple color features and the multiple spatial features are combined together as the second feature information.

[0059] Exemplarily, the target calculation module calculates the distance information between the initial cluster center and the remaining pixels using the first feature information and the second feature information according to the following formula:

[0060]

[0061] Where k represents the number of features corresponding to the first feature information or the second feature information, ijRepresents the distance information between the i-th initial cluster center and the j-th remaining pixel point, v it represents the tth first feature information of the i-th initial cluster center, v jt Represents the t-th second feature information of the j-th remaining pixel.

[0062] For example, the above formula uses a combination of color features and spatial features, so that the distance calculation is not only based on color similarity, but also takes into account the proximity of spatial positions, thereby better preserving the boundaries and shapes of objects. By combining color and spatial features, clustering errors caused by relying on a single feature alone can be reduced, especially in areas with similar colors but large spatial position differences, or areas with spatial proximity but large color differences. This helps to improve the accuracy and stability of clustering.

[0063] In some embodiments, the second segmentation module includes: an image processing module, used to perform superpixel calculation on the multi-color fluorescent image to obtain a target pixel image corresponding to the multi-color fluorescent image; a center determination module, used to perform feature extraction on the target pixel image to obtain target pixel features, and perform cluster center calculation based on the target pixel features to obtain a first cluster center; a second cluster analysis module, used to perform cluster analysis on the multi-color fluorescent image using the feature clustering algorithm based on the first cluster centers and the target pixel features to obtain an image clustering result; a second segmentation processing module, used to obtain the second segmentation result corresponding to the multi-color fluorescent image based on the image clustering result.

[0064] Exemplarily, the image processing module uses a superpixel algorithm (such as SLIC, Felzenszwalb, etc.) to decompose the multicolor fluorescence image into several superpixel regions. Superpixels are small areas in the image that have similarities in color, texture, and spatial position, thereby reducing the complexity of image processing while retaining important image features, thereby treating each superpixel region as a pixel point to generate a new target pixel image. The features of each target pixel point can be the average color value, center position, etc. of the superpixel region.

[0065] Exemplarily, the center determination module extracts features from each target pixel in the target pixel image. These features may include color information (such as RGB values, HSV values, etc.), spatial position information (such as (x, y) coordinates), etc., thereby determining multiple pixel points with large differences between target pixel features as the first clustering centers.

[0066] Exemplarily, the second clustering analysis module uses a clustering algorithm (such as K-means, DBSCAN, etc.) to perform clustering analysis on the target pixel features using the first clustering center, and iteratively updates the first clustering center until the clustering result converges. In each iteration, the distance from each target pixel point to each clustering center is calculated, and the pixel points are reallocated to the nearest clustering center, and finally the image clustering result is obtained.

[0067] Exemplarily, the second segmentation processing module maps the image clustering result back to the original multi-color fluorescence image. Each clustering center corresponds to a region, and the superpixels in the same clustering can be merged to form the final second segmentation result.

[0068] Specifically, the superpixel calculation, feature extraction, clustering analysis, and segmentation processing of the multi-color fluorescence image are realized. These steps not only improve the efficiency of image processing but also can obtain more accurate segmentation results.

[0069] In some embodiments, the center determination module includes: an adjacent calculation module for calculating the distance according to the target pixel features to obtain the associated pixels adjacent to the relevant pixels in the target pixel image; a feature extraction module for obtaining the first pixel feature corresponding to the relevant pixels and the second pixel feature corresponding to the associated pixels from the target pixel features; a mean calculation module for calculating the pixel distance between the first pixel feature and the second pixel feature, and performing mean calculation after squaring and summing the pixel distances to obtain the first target mean; a first density calculation module for determining the first density information corresponding to the relevant pixels using the Gaussian kernel function and the first target mean; a second density calculation module for obtaining the second target mean corresponding to the associated pixels and determining the second density information corresponding to the associated pixels using the Gaussian kernel function and the second target mean; a first comparison module for comparing the first density information and the second density information, and when the first density information is greater than the second density information, determining the minimum distance between the relevant pixels and the associated pixels as the pixel relative distance; a second comparison module for when the first density information is less than or equal to the second density information, determining the maximum distance between the relevant pixels and the associated pixels as the pixel relative distance; a pixel characterization module for determining the pixel characterization value corresponding to the relevant pixels according to the pixel relative distance and the first density information; a center screening module for calculating the clustering center according to the pixel characterization value to obtain the first clustering center.

[0070] Exemplarily, the adjacent calculation module calculates the distance between each relevant pixel in the target pixel image and other pixel points and determines the associated pixels adjacent to each relevant pixel according to the calculated distance.

[0071] Exemplarily, the feature extraction module extracts features from relevant pixels, and these features may include color information (such as RGB value, HSV value, etc.), spatial position information (such as (x, y) coordinates), etc. to obtain the first pixel feature. Similarly, features are extracted from associated pixels. The feature type is consistent with the first pixel feature, and the second pixel feature is obtained.

[0072] Exemplarily, the mean calculation module calculates the Euclidean distance or Manhattan distance between the first pixel feature and the second pixel feature, and then squares all pixel distances and sums them, and then divides the sum by the number of associated pixels to obtain a first target mean.

[0073] Exemplarily, the first density calculation module uses a Gaussian kernel function to perform density estimation on the first target mean. The parameters of the Gaussian kernel function can be adjusted according to actual conditions, so as to substitute the first target mean into the Gaussian kernel function to obtain the first density information of the relevant pixels.

[0074] Exemplarily, the second density calculation module obtains the second target mean corresponding to the associated pixels in the same manner as the first target mean is calculated for the related pixels, and determines the second density information corresponding to the associated pixels using the Gaussian kernel function and the second target mean.

[0075] Exemplarily, the first comparison module compares the first density information with the second density information, and when the first density information is greater than the second density information, the minimum distance between the relevant pixel and the associated pixel is determined as the pixel relative distance.

[0076] Exemplarily, when the first density information is less than or equal to the second density information, the second comparison module determines the maximum distance between the relevant pixel and the associated pixel as the pixel relative distance.

[0077] Exemplarily, the pixel characterization module multiplies the pixel relative distance and the first density information, thereby determining the multiplication result as the pixel characterization value corresponding to the relevant pixel.

[0078] Exemplarily, the center screening module sorts the pixel representation values ​​of all relevant pixels from large to small, and then selects relevant pixels corresponding to a plurality of pixel representation values ​​arranged at the top as the first clustering center.

[0079] In some embodiments, the type determination module includes: a channel information processing module, which is used to calculate the color feature mean of the sub-segmentation result to obtain the first channel color information, the second channel color information and the third channel color information corresponding to the sub-segmentation result; a preset information determination module, which is used to determine the preset protein marker and the first preset color information, the second preset color information and the third preset color information corresponding to the preset protein marker; a first channel comparison module, which is used to compare the first channel color information with the first preset color information to obtain a first comparison result; a second channel comparison module, which is used to compare the second channel color information with the second preset color information to obtain a second comparison result; a third channel comparison module, which is used to compare the third channel color information with the third preset color information to obtain a third comparison result; a comparison fusion module, which is used to fuse the first comparison result, the second comparison result and the third comparison result to obtain the protein marker type corresponding to the sub-segmentation result.

[0080] Exemplarily, the channel information processing module performs mean calculation on each channel (usually RGB or HSV channel) of the sub-segmentation result, and then calculates the mean of each channel as the first channel color information (such as R channel), the second channel color information (such as G channel) and the third channel color information (such as B channel).

[0081] Exemplarily, the preset information determination module determines the preset protein marker and the first preset color information, the second preset color information, and the third preset color information corresponding to the preset protein marker. For example, according to known biological knowledge and experimental data, several common protein markers are determined. Corresponding color information is set for each preset protein marker. These color information can be reference values ​​based on known fluorescent colors or experimentally determined.

[0082] Exemplarily, the first channel comparison module compares the first channel color information with the first preset color information to obtain a first comparison result, for example, comparing the first channel color information (such as the R channel mean) with the first preset color information (such as the preset R channel reference value). The comparison can be performed using methods such as Euclidean distance and cosine similarity, so as to determine the first comparison result as a corresponding Boolean value, for example, the Boolean value indicates whether the first channel color information is within the information range corresponding to the first preset color information.

[0083] Exemplarily, the second channel comparison module compares the second channel color information with the second preset color information to obtain a second comparison result, for example, comparing the second channel color information (such as the G channel mean) with the second preset color information (such as the preset G channel reference value). The comparison method is the same as that of the first channel comparison module. Thus, the second comparison result is determined as a corresponding Boolean value, for example, the Boolean value indicates whether the second channel color information is within the information range corresponding to the second preset color information.

[0084] Exemplarily, the third channel comparison module compares the third channel color information with the third preset color information to obtain a third comparison result. For example, the third channel color information (such as the B channel mean) is compared with the third preset color information (such as the preset B channel reference value). The comparison method is the same as that of the first and second channel comparison modules, and the third comparison result is then determined as a corresponding Boolean value, for example, the Boolean value indicates whether the third channel color information is in the information range corresponding to the third preset color information.

[0085] Exemplarily, the comparison fusion module fuses the first comparison result, the second comparison result and the third comparison result. When the Boolean values ​​in the first comparison result, the second comparison result and the third comparison result all represent being within the corresponding information range, the protein marker type corresponding to the sub-segmentation result is set to the type corresponding to the corresponding preset protein marker.

[0086] Exemplarily, color features are extracted and compared based on the sub-segmentation results to determine the type of protein marker corresponding to the sub-segmentation results. This method not only takes into account the mean information of the color channel, but also combines the known preset color information to improve the accuracy and reliability of protein marker identification. This method has broad application prospects in the fields of multicolor fluorescence image analysis, cell biology research, etc.

[0087] In some embodiments, the information characterization module includes: a spectral feature extraction module, which uses the first feature extraction layer of the color characterization model to perform spectral feature extraction on the sub-segmentation result to obtain first characterization data; a texture feature extraction module, which uses the second feature extraction layer of the color characterization model to perform texture feature extraction on the sub-segmentation result to obtain second characterization data; and a feature fusion module, which uses the self-attention layer of the color characterization model to perform feature fusion on the first characterization data and the second characterization data to obtain a first characterization result corresponding to the sub-segmentation result.

[0088] Exemplarily, the spectral feature extraction module uses the first feature extraction layer of the color characterization model to extract spectral features of the sub-segmentation result to obtain first characterization data. For example, the first feature extraction layer generally includes a convolution layer, an activation function, and a pooling layer, etc., which are specially designed to extract spectral features of the image, such as color distribution, hue, saturation, etc., thereby generating first characterization data through processing by the first feature extraction layer, which contains the spectral feature information of the sub-segmentation result.

[0089] Exemplarily, the texture feature extraction module uses the second feature extraction layer of the color representation model to extract texture features of the sub-segmentation result to obtain second representation data. For example, the second feature extraction layer generally includes a convolution layer, an activation function, and a pooling layer, etc., which are specially designed to extract texture features of the image, such as texture patterns, structures, edges, etc., thereby generating second representation data through processing by the second feature extraction layer, which contains texture feature information of the sub-segmentation result.

[0090] Exemplarily, the feature fusion module uses the self-attention layer of the color representation model to perform feature fusion on the first representation data and the second representation data to obtain a first representation result corresponding to the sub-segmentation result. For example, the first representation data and the second representation data are combined, and possible methods include splicing, element-by-element addition, etc., and then the combined features are processed using the self-attention layer. The self-attention mechanism can automatically weight features at different positions and highlight important features. The specific implementation includes calculating the similarity between features, assigning attention weights, and obtaining the fused features by weighted summation, so that after being processed by the self-attention layer, a first representation result is generated, which combines spectral features and texture features, and enhances the performance of important features through the self-attention mechanism.

[0091] Please refer to Figure 2 , Figure 2 A schematic flow chart of a method for detecting tumor microenvironment protein markers based on multicolor immunofluorescence provided in an embodiment of the present invention.

[0092] like Figure 2 As shown, the tumor microenvironment protein marker detection system based on multicolor immunofluorescence includes steps S201 to 207.

[0093] Step S201: obtaining a multicolor fluorescent image of an object to be stained in a target staining mode.

[0094] Step S202: Perform image segmentation on the multi-color fluorescence image using a superpixel segmentation algorithm to obtain a first segmentation result corresponding to the multi-color fluorescence image.

[0095] Step S203: performing image segmentation on the multi-color fluorescence image using a feature clustering algorithm to obtain a second segmentation result corresponding to the multi-color fluorescence image.

[0096] Step S204: fusing the first segmentation result and the second segmentation result to obtain a target segmentation result corresponding to the multi-color fluorescence image.

[0097] Step S205: Perform chromaticity recognition on each sub-segmentation result in the target segmentation result to obtain the protein marker type corresponding to the sub-segmentation result.

[0098] Step S206: Perform color characterization on each of the sub-segmentation results in the target segmentation result to obtain a corresponding first characterization result, and determine a second characterization result corresponding to the sub-segmentation result according to the protein marker type.

[0099] Step S207 : calculating a target distance between the first characterization result and the second characterization result, and determining a protein marker detection result corresponding to the sub-segmentation result according to the target distance.

[0100] In some embodiments, the step of performing image segmentation on the multi-color fluorescent image using a superpixel segmentation algorithm to obtain a first segmentation result corresponding to the multi-color fluorescent image includes:

[0101] Performing color distribution analysis on the multi-color fluorescent image to obtain target distribution information, and determining target entropy information corresponding to the multi-color fluorescent image according to the target distribution information;

[0102] Determining an image length and an image width corresponding to the multi-color fluorescent image, and determining the number of superpixels corresponding to the multi-color fluorescent image according to the image length, the image width and the target entropy information;

[0103] Determining an initial cluster center from the multi-color fluorescent image according to the number of superpixels, and obtaining a first color feature and a first spatial feature corresponding to the initial cluster center;

[0104] Obtaining second color features and second spatial features corresponding to remaining pixel points of the multicolor fluorescent image except the initial cluster center;

[0105] Determine distance information between remaining pixels and the initial cluster center according to the first color feature, the first spatial feature, the second color feature, and the second spatial feature;

[0106] Performing cluster analysis on the multi-color fluorescent image according to the distance information to obtain a first clustering result corresponding to the initial clustering center;

[0107] Performing abnormal filtering on each sub-clustering result in the first clustering result to obtain a second clustering result;

[0108] According to the second clustering result, the initial cluster center is updated to obtain a target cluster center, and cluster analysis is re-performed according to the target cluster center to obtain a target clustering result;

[0109] The multi-color fluorescence image is segmented according to the target clustering result to obtain the first segmentation result corresponding to the multi-color fluorescence image.

[0110] In some implementations, determining the distance information between the remaining pixels and the initial cluster center according to the first color feature, the first spatial feature, the second color feature, and the second spatial feature includes:

[0111] Determining first feature information corresponding to the initial cluster center according to the first color feature and the first spatial feature;

[0112] Determine second feature information corresponding to the remaining pixels according to the second color feature and the second spatial feature;

[0113] Calculating the distance between the initial cluster center and the remaining pixels according to the first feature information and the second feature information to obtain the distance information;

[0114] The distance information is obtained according to the following formula:

[0115]

[0116] Wherein, k represents the number of features corresponding to the first feature information or the second feature information, ij represents the distance information between the i-th initial cluster center and the j-th remaining pixel point, v it represents the tth first feature information of the i-th initial cluster center, v jt Represents the t-th second feature information of the j-th remaining pixel.

[0117] In some embodiments, the step of performing image segmentation on the multi-color fluorescent image using a feature clustering algorithm to obtain a second segmentation result corresponding to the multi-color fluorescent image includes:

[0118] Performing superpixel calculation on the multi-color fluorescent image to obtain a target pixel image corresponding to the multi-color fluorescent image;

[0119] Extracting features of the target pixel image to obtain target pixel features, and calculating cluster centers based on the target pixel features to obtain first cluster centers;

[0120] Performing cluster analysis on the multi-color fluorescence image using the feature clustering algorithm according to the first cluster center and the target pixel feature to obtain an image clustering result;

[0121] The second segmentation result corresponding to the multi-color fluorescence image is obtained according to the image clustering result.

[0122] In some implementations, extracting features from the target pixel image to obtain target pixel features, and calculating cluster centers based on the target pixel features to obtain first cluster centers, includes:

[0123] Performing distance calculation according to the target pixel feature to obtain associated pixels adjacent to the relevant pixels in the target pixel image;

[0124] Obtaining a first pixel feature corresponding to the relevant pixel and a second pixel feature corresponding to the associated pixel from the target pixel feature;

[0125] Calculating a pixel distance between the first pixel feature and the second pixel feature, and performing mean calculation on the squared sum of the pixel distances to obtain a first target mean;

[0126] Determine first density information corresponding to the relevant pixels using a Gaussian kernel function and the first target mean;

[0127] Obtaining a second target mean value corresponding to the associated pixels, and determining second density information corresponding to the associated pixels using the Gaussian kernel function and the second target mean value;

[0128] comparing the first density information with the second density information, and when the first density information is greater than the second density information, determining the minimum distance between the relevant pixel and the associated pixel as the pixel relative distance;

[0129] When the first density information is less than or equal to the second density information, the maximum distance between the relevant pixel and the associated pixel is determined as the pixel relative distance;

[0130] Determine a pixel representation value corresponding to the relevant pixel according to the pixel relative distance and the first density information;

[0131] The first cluster center is obtained by performing cluster center calculation according to the pixel characterization value.

[0132] In some embodiments, performing colorimetric recognition on each sub-segmentation result in the target segmentation result to obtain a protein marker type corresponding to the sub-segmentation result includes:

[0133] Performing color feature mean calculation on the sub-segmentation results to obtain first channel color information, second channel color information, and third channel color information corresponding to the sub-segmentation results;

[0134] Determining a preset protein marker and first preset color information, second preset color information, and third preset color information corresponding to the preset protein marker;

[0135] Comparing the first channel color information with the first preset color information to obtain a first comparison result;

[0136] Comparing the second channel color information with the second preset color information to obtain a second comparison result;

[0137] Comparing the third channel color information with the third preset color information to obtain a third comparison result;

[0138] The first comparison result, the second comparison result and the third comparison result are integrated to obtain the protein marker type corresponding to the sub-segmentation result.

[0139] In some implementations, performing color characterization on each of the sub-segmentation results in the target segmentation result to obtain a corresponding first characterization result includes:

[0140] Using a first feature extraction layer of a color characterization model to extract spectral features from the sub-segmentation results to obtain first characterization data;

[0141] Using the second feature extraction layer of the color representation model to extract texture features from the sub-segmentation results to obtain second representation data;

[0142] The self-attention layer of the color representation model is used to perform feature fusion on the first representation data and the second representation data to obtain a first representation result corresponding to the sub-segmentation result.

[0143] In some embodiments, the tumor microenvironment protein marker detection method based on multicolor immunofluorescence can be applied to a terminal device.

[0144] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the tumor microenvironment protein marker detection method based on multicolor immunofluorescence described above can refer to the corresponding process in the aforementioned embodiment of the tumor microenvironment protein marker detection system based on multicolor immunofluorescence, and will not be repeated here.

[0145] See also Figure 3 , Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0146] like Figure 3 As shown, the terminal device 300 includes a processor 301 and a memory 302, and the processor 301 and the memory 302 are connected via a bus 303, such as an I2C (Inter-integrated Circuit) bus.

[0147] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0148] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0149] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the terminal device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0150] The processor is used to run a computer program stored in the memory, and implement any one of the tumor microenvironment protein marker detection systems based on multicolor immunofluorescence provided by the embodiments of the present invention when executing the computer program.

[0151] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0152] An image acquisition module is used to obtain a multicolor fluorescent image of the object to be stained under a target staining mode;

[0153] A first segmentation module, configured to perform image segmentation on the multi-color fluorescent image using a superpixel segmentation algorithm to obtain a first segmentation result corresponding to the multi-color fluorescent image;

[0154] A second segmentation module, used to perform image segmentation on the multi-color fluorescent image using a feature clustering algorithm to obtain a second segmentation result corresponding to the multi-color fluorescent image;

[0155] A segmentation and fusion module, used for fusing the first segmentation result and the second segmentation result to obtain a target segmentation result corresponding to the multi-color fluorescent image;

[0156] A type determination module, used for performing chromatic identification on each sub-segmentation result in the target segmentation result to obtain the protein marker type corresponding to the sub-segmentation result;

[0157] An information characterization module, configured to perform color characterization on each of the sub-segmentation results in the target segmentation result to obtain a corresponding first characterization result, and determine a second characterization result corresponding to the sub-segmentation result according to the protein marker type;

[0158] The result determination module is used to calculate the target distance between the first characterization result and the second characterization result, and determine the protein marker detection result corresponding to the sub-segmentation result according to the target distance.

[0159] In some implementations, the processor 301 executes, during the first segmentation module process:

[0160] A data analysis module, configured to perform color distribution analysis on the multi-color fluorescent image to obtain target distribution information, and determine target entropy information corresponding to the multi-color fluorescent image according to the target distribution information;

[0161] A number determination module, used to determine an image length and an image width corresponding to the multi-color fluorescent image, and determine the number of superpixels corresponding to the multi-color fluorescent image according to the image length, the image width and the target entropy information;

[0162] A feature analysis module, used to determine an initial cluster center from the multi-color fluorescent image according to the number of superpixels, and obtain a first color feature and a first spatial feature corresponding to the initial cluster center;

[0163] A feature acquisition module, used to obtain second color features and second spatial features corresponding to the remaining pixel points of the multi-color fluorescent image except the initial cluster center;

[0164] A distance calculation module, used to determine the distance information between the remaining pixels and the initial cluster center according to the first color feature, the first spatial feature, the second color feature and the second spatial feature;

[0165] A first cluster analysis module, used for performing cluster analysis on the multi-color fluorescent image according to the distance information to obtain a first cluster result corresponding to the initial cluster center;

[0166] An abnormality filtering module, used for performing abnormality filtering on each sub-clustering result in the first clustering result to obtain a second clustering result;

[0167] A clustering update module, configured to update the initial cluster center according to the second clustering result to obtain a target cluster center, and re-perform cluster analysis according to the target cluster center to obtain a target clustering result;

[0168] The first segmentation processing module is used to perform image segmentation on the multi-color fluorescent image according to the target clustering result to obtain the first segmentation result corresponding to the multi-color fluorescent image.

[0169] In some implementations, the processor 301, in the distance calculation module process, executes:

[0170] A first feature decomposition module, used for determining first feature information corresponding to the initial cluster center according to the first color feature and the first spatial feature;

[0171] A second feature decomposition module, used to determine second feature information corresponding to the remaining pixels according to the second color feature and the second spatial feature;

[0172] A target calculation module, used for performing distance calculation between the initial cluster center and the remaining pixel points according to the first feature information and the second feature information to obtain the distance information;

[0173] The distance information is obtained according to the following formula:

[0174]

[0175] Wherein, k represents the number of features corresponding to the first feature information or the second feature information, ij represents the distance information between the i-th initial cluster center and the j-th remaining pixel point, v it represents the tth first feature information of the i-th initial cluster center, v jt Represents the t-th second feature information of the j-th remaining pixel.

[0176] In some implementations, the processor 301 executes, during the second segmentation module process:

[0177] An image processing module, used for performing super-pixel calculation on the multi-color fluorescent image to obtain a target pixel image corresponding to the multi-color fluorescent image;

[0178] A center determination module, used for performing feature extraction on the target pixel image to obtain target pixel features, and performing cluster center calculation based on the target pixel features to obtain a first cluster center;

[0179] A second cluster analysis module, configured to perform cluster analysis on the multicolor fluorescence image using the feature clustering algorithm according to the first cluster center and the target pixel feature to obtain an image clustering result;

[0180] The second segmentation processing module is used to obtain the second segmentation result corresponding to the multi-color fluorescent image according to the image clustering result.

[0181] In some implementations, the processor 301, in the center determination module process, executes:

[0182] An adjacent calculation module, used for performing distance calculation according to the target pixel feature to obtain associated pixels adjacent to the relevant pixels in the target pixel image;

[0183] A feature extraction module, used to obtain a first pixel feature corresponding to the relevant pixel and a second pixel feature corresponding to the associated pixel from the target pixel feature;

[0184] a mean calculation module, used for calculating the pixel distance between the first pixel feature and the second pixel feature, and performing mean calculation after square summing the pixel distances to obtain a first target mean;

[0185] A first density calculation module, used for determining first density information corresponding to the relevant pixels by using a Gaussian kernel function and the first target mean;

[0186] A second density calculation module, used for obtaining a second target mean value corresponding to the associated pixels, and determining second density information corresponding to the associated pixels by using the Gaussian kernel function and the second target mean value;

[0187] a first comparison module, configured to compare the first density information with the second density information, and when the first density information is greater than the second density information, determine the minimum distance between the relevant pixel and the associated pixel as the pixel relative distance;

[0188] a second comparison module, configured to determine the maximum distance between the relevant pixel and the associated pixel as the pixel relative distance when the first density information is less than or equal to the second density information;

[0189] A pixel characterization module, configured to determine a pixel characterization value corresponding to the relevant pixel according to the pixel relative distance and the first density information;

[0190] A center screening module is used to calculate the cluster center according to the pixel representation value to obtain the first cluster center.

[0191] In some implementations, the processor 301, in the type determination module process, executes:

[0192] A channel information processing module, used for performing color feature mean calculation on the sub-segmentation results to obtain first channel color information, second channel color information and third channel color information corresponding to the sub-segmentation results;

[0193] A preset information determination module, used to determine a preset protein marker and first preset color information, second preset color information, and third preset color information corresponding to the preset protein marker;

[0194] A first channel comparison module, used for comparing the first channel color information with the first preset color information to obtain a first comparison result;

[0195] A second channel comparison module, used for comparing the second channel color information with the second preset color information to obtain a second comparison result;

[0196] A third channel comparison module, used for comparing the third channel color information with the third preset color information to obtain a third comparison result;

[0197] A comparison and fusion module is used to fuse the first comparison result, the second comparison result and the third comparison result to obtain the protein marker type corresponding to the sub-segmentation result.

[0198] In some implementations, the processor 301, in the information representation module process, executes:

[0199] A spectral feature extraction module, used for performing spectral feature extraction on the sub-segmentation result by using a first feature extraction layer of a color representation model to obtain first representation data;

[0200] A texture feature extraction module, used for performing texture feature extraction on the sub-segmentation result by using the second feature extraction layer of the color representation model to obtain second representation data;

[0201] A feature fusion module is used to use the self-attention layer of the color representation model to perform feature fusion on the first representation data and the second representation data to obtain a first representation result corresponding to the sub-segmentation result.

[0202] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned multi-color immunofluorescence-based tumor microenvironment protein marker detection system embodiment, and will not be repeated here.

[0203] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any step of a tumor microenvironment protein marker detection system based on multicolor immunofluorescence as provided in the description of the embodiment of the present invention.

[0204] The storage medium may be an internal storage unit of the terminal device in the aforementioned embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device.

[0205] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0206] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0207] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A tumor microenvironment protein marker detection system based on multicolor immunofluorescence, characterized in that: The system comprises: An image acquisition module is used to obtain a multicolor fluorescent image of the object to be stained under a target staining mode; A first segmentation module, configured to perform image segmentation on the multi-color fluorescent image using a superpixel segmentation algorithm to obtain a first segmentation result corresponding to the multi-color fluorescent image; A second segmentation module, used to perform image segmentation on the multi-color fluorescent image using a feature clustering algorithm to obtain a second segmentation result corresponding to the multi-color fluorescent image; A segmentation and fusion module, used for fusing the first segmentation result and the second segmentation result to obtain a target segmentation result corresponding to the multi-color fluorescent image; A type determination module, used for performing chromatic identification on each sub-segmentation result in the target segmentation result to obtain the protein marker type corresponding to the sub-segmentation result; An information characterization module, configured to perform color characterization on each of the sub-segmentation results in the target segmentation result to obtain a corresponding first characterization result, and determine a second characterization result corresponding to the sub-segmentation result according to the protein marker type; The result determination module is used to calculate the target distance between the first characterization result and the second characterization result, and determine the protein marker detection result corresponding to the sub-segmentation result according to the target distance.

2. The system according to claim 1, characterized in that The first segmentation module comprises: A data analysis module, configured to perform color distribution analysis on the multi-color fluorescent image to obtain target distribution information, and determine target entropy information corresponding to the multi-color fluorescent image according to the target distribution information; A number determination module, used to determine an image length and an image width corresponding to the multi-color fluorescent image, and determine the number of superpixels corresponding to the multi-color fluorescent image according to the image length, the image width and the target entropy information; A feature analysis module, used to determine an initial cluster center from the multi-color fluorescent image according to the number of superpixels, and obtain a first color feature and a first spatial feature corresponding to the initial cluster center; A feature acquisition module, used to obtain second color features and second spatial features corresponding to the remaining pixel points of the multi-color fluorescent image except the initial cluster center; A distance calculation module, used to determine the distance information between the remaining pixels and the initial cluster center according to the first color feature, the first spatial feature, the second color feature and the second spatial feature; A first cluster analysis module, configured to perform cluster analysis on the multi-color fluorescent image according to the distance information to obtain a first cluster result corresponding to the initial cluster center; an abnormality filtering module, used for performing abnormality filtering on each sub-clustering result in the first clustering result to obtain a second clustering result; A clustering update module, configured to update the initial cluster center according to the second clustering result to obtain a target cluster center, and re-perform cluster analysis according to the target cluster center to obtain a target clustering result; The first segmentation processing module is used to perform image segmentation on the multi-color fluorescent image according to the target clustering result to obtain the first segmentation result corresponding to the multi-color fluorescent image.

3. The system according to claim 2, characterized in that The distance calculation module comprises: A first feature decomposition module, used for determining first feature information corresponding to the initial cluster center according to the first color feature and the first spatial feature; A second feature decomposition module, used to determine second feature information corresponding to the remaining pixels according to the second color feature and the second spatial feature; A target calculation module, used for performing distance calculation between the initial cluster center and the remaining pixel points according to the first feature information and the second feature information to obtain the distance information; The distance information is obtained according to the following formula: Wherein, k represents the number of features corresponding to the first feature information or the second feature information, ij represents the distance information between the i-th initial cluster center and the j-th remaining pixel point, v it represents the tth first feature information of the i-th initial cluster center, v jt Represents the t-th second feature information of the j-th remaining pixel.

4. The system according to claim 1, characterized in that The second segmentation module comprises: An image processing module, used for performing super-pixel calculation on the multi-color fluorescent image to obtain a target pixel image corresponding to the multi-color fluorescent image; A center determination module, used for performing feature extraction on the target pixel image to obtain target pixel features, and performing cluster center calculation based on the target pixel features to obtain a first cluster center; A second cluster analysis module, configured to perform cluster analysis on the multicolor fluorescence image using the feature clustering algorithm according to the first cluster center and the target pixel feature to obtain an image clustering result; The second segmentation processing module is used to obtain the second segmentation result corresponding to the multi-color fluorescent image according to the image clustering result.

5. The system according to claim 4, characterized in that The center determination module includes: An adjacent calculation module, used for performing distance calculation according to the target pixel feature to obtain associated pixels adjacent to the relevant pixels in the target pixel image; A feature extraction module, used to obtain a first pixel feature corresponding to the relevant pixel and a second pixel feature corresponding to the associated pixel from the target pixel feature; a mean calculation module, used for calculating the pixel distance between the first pixel feature and the second pixel feature, and performing mean calculation after square summing the pixel distances to obtain a first target mean; A first density calculation module, used for determining first density information corresponding to the relevant pixels by using a Gaussian kernel function and the first target mean; A second density calculation module, used for obtaining a second target mean value corresponding to the associated pixels, and determining second density information corresponding to the associated pixels by using the Gaussian kernel function and the second target mean value; a first comparison module, configured to compare the first density information with the second density information, and when the first density information is greater than the second density information, determine the minimum distance between the relevant pixel and the associated pixel as the pixel relative distance; a second comparison module, configured to determine the maximum distance between the relevant pixel and the associated pixel as the pixel relative distance when the first density information is less than or equal to the second density information; A pixel characterization module, configured to determine a pixel characterization value corresponding to the relevant pixel according to the pixel relative distance and the first density information; A center screening module is used to calculate the cluster center according to the pixel representation value to obtain the first cluster center.

6. The system according to claim 1, characterized in that The type determination module comprises: A channel information processing module, used for performing color feature mean calculation on the sub-segmentation results to obtain first channel color information, second channel color information and third channel color information corresponding to the sub-segmentation results; A preset information determination module, used to determine a preset protein marker and first preset color information, second preset color information, and third preset color information corresponding to the preset protein marker; A first channel comparison module, used for comparing the first channel color information with the first preset color information to obtain a first comparison result; A second channel comparison module, used for comparing the second channel color information with the second preset color information to obtain a second comparison result; A third channel comparison module, used for comparing the third channel color information with the third preset color information to obtain a third comparison result; A comparison and fusion module is used to fuse the first comparison result, the second comparison result and the third comparison result to obtain the protein marker type corresponding to the sub-segmentation result.

7. The system according to claim 1, characterized in that The information representation module includes: A spectral feature extraction module, used for performing spectral feature extraction on the sub-segmentation result by using a first feature extraction layer of a color representation model to obtain first representation data; A texture feature extraction module, used for performing texture feature extraction on the sub-segmentation result by using the second feature extraction layer of the color representation model to obtain second representation data; A feature fusion module is used to use the self-attention layer of the color representation model to perform feature fusion on the first representation data and the second representation data to obtain a first representation result corresponding to the sub-segmentation result.

8. A method for detecting tumor microenvironment protein markers based on multicolor immunofluorescence, characterized in that: The method comprises: Obtaining a multicolor fluorescent image of the object to be stained by staining in a target staining mode; Performing image segmentation on the multi-color fluorescent image using a superpixel segmentation algorithm to obtain a first segmentation result corresponding to the multi-color fluorescent image; Performing image segmentation on the multi-color fluorescent image using a feature clustering algorithm to obtain a second segmentation result corresponding to the multi-color fluorescent image; fusing the first segmentation result and the second segmentation result to obtain a target segmentation result corresponding to the multi-color fluorescent image; Performing chromaticity recognition on each sub-segmentation result in the target segmentation result to obtain the protein marker type corresponding to the sub-segmentation result; Performing color characterization on each of the sub-segmentation results in the target segmentation result to obtain a corresponding first characterization result, and determining a second characterization result corresponding to the sub-segmentation result according to the protein marker type; A target distance between the first characterization result and the second characterization result is calculated, and a protein marker detection result corresponding to the sub-segmentation result is determined according to the target distance.

9. A terminal device, characterized in that: The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program and implement the tumor microenvironment protein marker detection system based on multicolor immunofluorescence as described in any one of claims 1 to 7 when executing the computer program.

10. A computer storage medium for computer storage, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the tumor microenvironment protein marker detection system based on multicolor immunofluorescence as described in any one of claims 1 to 7.

Images