Multiple immunohistochemical fluorescence colorimetric detection method and system based on a single slide

Abstract

The present application discloses a multiple immunohistochemical fluorescence colorimetric detection method and system based on a single slide, which combines deep learning algorithms and image processing technology to perform image processing and feature extraction on the fluorescence colorimetric images of tissue sections to achieve intelligent judgment of the category of tumor markers.

Description

Technical Field

[0001] The present application relates to the field of intelligent detection, and more specifically, to a multiple immunohistochemical fluorescence colorimetric detection method and system based on a single slide. Background Art

[0002] Tumor markers are substances that can reflect the existence, development and metastasis of tumors. They are expressed in tumor cells or normal cells, or secreted by tumor cells into body fluids. The detection of tumor markers is of great significance for the diagnosis, staging, prognosis evaluation and treatment effect monitoring of tumors. At present, the commonly used tumor marker detection methods are mainly immunohistochemistry and immunofluorescence, which can show the location and distribution of tumor markers on tissue sections.

[0003] Multiplex immunohistochemical fluorescence colorimetric detection is a method for simultaneously detecting multiple target proteins in cells or tissue sections. It combines immunohistochemistry and fluorescence microscopy techniques, uses specific antibodies to bind to target proteins and label them with fluorescent dyes, and observes and analyzes the distribution and location of different target proteins in cells or tissues through fluorescence microscopy.

[0004] However, existing multiple immunohistochemical fluorescence colorimetric detection has some problems, such as the quality of tissue section fluorescence colorimetric images is affected by factors such as fluorescence signal attenuation, noise interference and non-uniform brightness, which increases the difficulty of image segmentation and classification. Therefore, a solution is expected. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides a multiple immunohistochemical fluorescence colorimetric detection method and system based on a single slide, which combines deep learning algorithms and image processing technology to perform image processing and feature extraction on the fluorescence colorimetric images of tissue sections to achieve intelligent judgment of the category of tumor markers.

[0006] According to one aspect of the present application, a multiple immunohistochemical fluorescence colorimetric detection method based on a single slide is provided, comprising:

[0007] Acquire fluorescent colorimetric images of tissue sections;

[0008] Performing image feature extraction on the tissue slice fluorescence colorimetric image to obtain a semantically fused tissue slice fluorescence colorimetric shallow layer feature map; and

[0009] Based on the semantically fused tissue slice fluorescence coloration shallow layer feature map, the category of the tumor marker is determined.

[0010] According to another aspect of the present application, a multiplex immunohistochemical fluorescence colorimetric detection system based on a single slide is provided, comprising:

[0011] An image acquisition module, used for acquiring fluorescent color development images of tissue sections;

[0012] An image feature extraction module is used to extract image features from the tissue slice fluorescence colorimetric image to obtain a semantically fused tissue slice fluorescence colorimetric shallow layer feature map; and

[0013] The category generation module is used to determine the category of the tumor marker based on the semantic fusion tissue slice fluorescence coloration shallow layer feature map.

[0014] Compared with the prior art, the present application provides a multiple immunohistochemical fluorescence colorimetric detection method and system based on a single glass slide, which combines deep learning algorithms and image processing technology to perform image processing and feature extraction on tissue section fluorescence colorimetric images to achieve intelligent judgment of the category of tumor markers. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a flow chart of a multiple immunohistochemical fluorescent colorimetric detection method based on a single slide according to an embodiment of the present application;

[0017] Figure 2 A system architecture diagram of a multiple immunohistochemical fluorescent colorimetric detection method based on a single slide according to an embodiment of the present application;

[0018] Figure 3 Flow chart of sub-step S2 of the multiple immunohistochemical fluorescent colorimetric detection method based on a single slide according to an embodiment of the present application;

[0019] Figure 4 is a flow chart of sub-step S22 of the multiple immunohistochemical fluorescent colorimetric detection method based on a single slide according to an embodiment of the present application;

[0020] Figure 5 4 is a block diagram of a multiplex immunohistochemical fluorescent colorimetric detection system based on a single slide according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0022] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0023] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0024] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0025] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0026] Multiplex immunohistochemical fluorescence colorimetric detection is a method for simultaneously detecting multiple target proteins in cells or tissue sections. It combines immunohistochemical technology and fluorescence microscopy technology, uses specific antibodies to bind to target proteins and label them with fluorescent dyes, and observes and analyzes the distribution and localization of different target proteins in cells or tissues through fluorescence microscopy. However, there are some problems with existing multiplex immunohistochemical fluorescence colorimetric detection. For example, the quality of tissue section fluorescence colorimetric images is affected by factors such as fluorescence signal attenuation, noise interference, and non-uniform brightness, which increases the difficulty of image segmentation and classification. Therefore, a solution is expected.

[0027] In the technical solution of the present application, a multiplex immunohistochemical fluorescence colorimetric detection method based on a single slide is proposed. Figure 1 The flowchart is a multiple immunohistochemical fluorescent colorimetric detection method based on a single slide according to an embodiment of the present application. Figure 21 is a system architecture diagram of a multiple immunohistochemical fluorescence colorimetric detection method based on a single slide according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to an embodiment of the present application, a multiple immunohistochemistry fluorescence colorimetric detection method based on a single glass slide includes the steps of: S1, acquiring a tissue section fluorescence colorimetric image; S2, performing image feature extraction on the tissue section fluorescence colorimetric image to obtain a semantically fused tissue section fluorescence colorimetric shallow feature map; and, S3, determining the category of the tumor marker based on the semantically fused tissue section fluorescence colorimetric shallow feature map.

[0028] In particular, in step S1, a tissue section fluorescence colorimetric image is obtained. The tissue section fluorescence colorimetric image refers to an image obtained by observing under a fluorescence microscope after the tissue sample is cut into thin slices and the target molecules or structures are labeled with fluorescent dyes. This technology is often used to study the structure, cell type, protein localization, etc. of tissues. It is worth mentioning that fluorescence colorimetric technology is often used in fields such as biology, medicine, and materials science to study and observe the structure, function, and interaction of samples.

[0029] Accordingly, in a possible implementation, a fluorescent colorimetric image of a tissue section can be obtained through the following steps, for example: tissue fixation: prepare a buffer solution containing an appropriate fixative (such as formalin); place the tissue sample in the buffer solution for fixation, and adjust the time and temperature according to experimental requirements; rinse the sample to remove the fixative, and store the sample in an appropriate buffer solution; tissue sectioning: prepare a tissue slicer and a slicer blade; place the fixed tissue sample in the slicer and slice it. The thickness of the slices is usually between a few microns and tens of microns, which can be adjusted as needed; collect the slices on slides or slice boxes and mark them for subsequent processing; fluorescence staining: prepare appropriate fluorescent dyes and staining buffer; soak the slices in staining buffer to allow the fluorescent dyes to interact with the target molecules or structures; perform appropriate staining time and temperature treatment according to the needs and requirements of staining; rinse the slices to remove excess dyes and store the slices in appropriate buffer; fluorescence microscopy observation: prepare a fluorescence microscope and appropriate filter combinations; place the stained slices on the fluorescence microscope stage and observe them using appropriate eyepieces; adjust the excitation light source and filters of the fluorescence microscope to adapt to the fluorescent dyes used; adjust the microscope parameters (such as focal length, exposure time, etc.) to obtain clear fluorescent color images; image acquisition and analysis: record fluorescent color images using a camera or image acquisition system; import the images into a computer and use image processing software to make necessary adjustments and enhancements; perform image analysis, such as fluorescence intensity measurement, colocalization analysis, etc.

[0030] In particular, in step S2, image feature extraction is performed on the tissue slice fluorescence color image to obtain a semantically fused tissue slice fluorescence color shallow layer feature map. In particular, in a specific example of the present application, Figure 3 As shown, the S2 includes: S21, performing image preprocessing on the tissue slice fluorescence chromogen image to obtain an enhanced tissue slice fluorescence chromogen image; and S22, extracting and fusing the shallow features and deep features of the enhanced tissue slice fluorescence chromogen image to obtain the semantically fused tissue slice fluorescence chromogen shallow feature map.

[0031] Specifically, the S21 performs image preprocessing on the tissue slice fluorescence chromogenic image to obtain an enhanced tissue slice fluorescence chromogenic image. That is, the noise and interference factors in the tissue slice fluorescence chromogenic image are removed by means of image preprocessing. In particular, in a specific example of the present application, the S21 includes: performing bilateral filtering on the tissue slice fluorescence chromogenic image to obtain the enhanced tissue slice fluorescence chromogenic image. Among them, bilateral filtering has the functions of denoising, smoothing and edge protection in the processing of tissue slice fluorescence chromogenic images. It can reduce the noise in the image, maintain the detail information and edge clarity of the image, and provide a better basis for subsequent image analysis and feature extraction.

[0032] It is worth noting that bilateral filtering is a nonlinear filter that is often used for smoothing and denoising operations in image processing. It performs weighted averaging on the image by considering the spatial distance between pixels and the similarity between grayscale values, thereby achieving smoothing while retaining edge information. In traditional linear filters, the average value of a pixel is obtained by simply weighted averaging the neighboring pixels. This averaging operation blurs the edges of the image and causes loss of details. The bilateral filter considers two aspects of similarity to adjust the weighting coefficient, namely spatial similarity and grayscale value similarity.

[0033] It is worth mentioning that in other specific examples of the present application, the tissue section fluorescence chromogenic image can also be preprocessed in other ways to obtain an enhanced tissue section fluorescence chromogenic image, for example: Denoising: Use an appropriate denoising algorithm, such as a median filter or a Gaussian filter, to reduce the noise in the image. This helps to improve the clarity and contrast of the image; select appropriate filter parameters, such as filter size or standard deviation, according to the characteristics of the image and the type of noise; adjust brightness and contrast: use methods such as histogram equalization or adaptive histogram equalization to enhance the contrast of the image. This helps to improve the visibility of details in the image; adjust the brightness and contrast of the image to make the overall brightness and details of the image more clearly visible; enhance details: use a sharpening filter or an edge enhancement algorithm to enhance the details in the image. This can make the structure and cell boundaries clearer; pay attention to avoid excessive enhancement when enhancing details to cause image noise; color correction: if there is color shift or unevenness in the image, color correction can be performed. This can make the color of the image more accurate and consistent; use color correction algorithms, such as the gray world algorithm or the white balance algorithm, to adjust the color balance of the image; image smoothing: if there is noise or granular interference in the image, you can use smoothing filters, such as mean filters or Gaussian filters, to reduce the noise and make the image smoother; edge protection: when performing image enhancement, you need to pay attention to protecting the edge information in the image to avoid over-smoothing or loss of details; you can use edge protection filters, such as bilateral filters or guided filters, to smooth the image while retaining edge information.

[0034] Specifically, the S22 extracts and fuses the shallow features and deep features of the enhanced tissue slice fluorescence colorimetric image to obtain the semantically fused tissue slice fluorescence colorimetric shallow feature map. It should be understood that shallow features generally refer to the extraction of low-level information of the image, such as edges, textures, etc. These features play an important role in distinguishing different regions and structures in cells or tissues. By extracting the shallow features of the enhanced fluorescence colorimetric image, the morphological and texture information of cells or tissues can be captured, helping to accurately locate and identify the distribution of target proteins. The deep features refer to high-level abstract representations learned from the image. Deep features can capture more semantic and abstract information and have stronger representation capabilities. Extracting deep features from the enhanced fluorescence colorimetric image can better capture the specific signals of the target protein. Therefore, in the technical solution of the present application, it is expected to extract and fuse the shallow features and deep features of the enhanced tissue slice fluorescence colorimetric image to obtain the semantically fused tissue slice fluorescence colorimetric shallow feature map. In particular, in a specific example of the present application, such as Figure 4As shown, the S22 includes: S221, passing the enhanced tissue slice fluorescence colorimetric image through an image multi-scale feature extractor based on a pyramid network to obtain a tissue slice fluorescence colorimetric shallow feature map and a tissue slice fluorescence colorimetric deep feature map; and, S222, using a semantic joint propagation module to fuse the tissue slice fluorescence colorimetric shallow feature map and the tissue slice fluorescence colorimetric deep feature map to obtain the semantically fused tissue slice fluorescence colorimetric shallow feature map.

[0035] More specifically, in S221, the enhanced tissue slice fluorescence color image is passed through an image multi-scale feature extractor based on a pyramid network to obtain a tissue slice fluorescence color shallow feature map and a tissue slice fluorescence color deep feature map. It should be understood that in the tissue slice fluorescence color image, different cells or tissue structures may have different scales, so the image content can be more comprehensively described by multi-scale feature extraction. The pyramid network can process inputs of different scales and extract features at different scales. This makes the network robust to scale changes and can better adapt to scale changes of cells or tissue structures in tissue slice fluorescence color images.

[0036] It is worth noting that the pyramid network is a deep learning network structure designed to handle features of different scales. It effectively captures multi-scale information in images by building feature pyramids of multiple scales. The basic idea of ​​the pyramid network is to introduce multiple parallel branches into the network, each of which is responsible for processing inputs of different scales. Each branch can contain different convolutional layers or pooling layers to meet the requirements of feature extraction at different scales. The output feature maps of each branch are integrated or fused to obtain the final feature representation. The advantage of the pyramid network is that it can process information of multiple scales at the same time, thereby improving the network's perception of objects or features of different scales. It can effectively solve the problems of scale change and scale invariance in some visual tasks. The pyramid network has a wide range of applications in the field of computer vision, especially in object detection and image segmentation tasks. By introducing feature pyramids of multiple scales, the network can better detect objects of different scales or segment objects of different scales.

[0037] Accordingly, in a possible implementation, the enhanced tissue slice fluorescence color image can be passed through a pyramid network-based image multi-scale feature extractor to obtain a tissue slice fluorescence color shallow feature map and a tissue slice fluorescence color deep feature map through the following steps, for example: obtaining an enhanced tissue slice fluorescence color image as input; designing and constructing a pyramid network-based image multi-scale feature extractor. A pyramid network is usually composed of multiple parallel branches, each branch processing inputs of different scales; performing feature extraction on the input image at different scales. Each branch can include operations such as convolutional layers and pooling layers to capture features of different scales. Shallower branches are usually used to extract coarse features, while deeper branches extract more abstract and semantically richer features; feature maps extracted at different scales are fused or integrated. This can be achieved by splicing feature maps, weighted summing, or using an attention mechanism. The fused feature map contains information of different scales and has richer semantic expression capabilities; a shallow feature map is extracted from the fused feature map. Shallow feature maps usually contain more local and detailed features, which are used to capture subtle changes and detail information in the image; deep feature maps are extracted from the fused feature maps. Deep feature maps have higher-level semantic information and can be used for tasks such as target classification and positioning; the shallow feature map of tissue section fluorescence colorimetry and the deep feature map of tissue section fluorescence colorimetry are obtained as the output of the pyramid network. These feature maps can be used for subsequent tasks such as target detection, image segmentation, feature analysis, etc.

[0038] More specifically, in S222, a semantic joint propagation module is used to fuse the tissue slice fluorescence colorization shallow feature map and the tissue slice fluorescence colorization deep feature map to obtain the semantically fused tissue slice fluorescence colorization shallow feature map. More specifically, in an embodiment of the present application, a semantic joint propagation module is used to fuse the tissue slice fluorescence colorization shallow feature map and the tissue slice fluorescence colorization deep feature map to obtain an encoding process of a semantically fused tissue slice fluorescence colorization shallow feature map, including: first, upsampling the tissue slice fluorescence colorization deep feature map to obtain a resolution reconstruction feature map; then, performing point convolution, batch normalization and ReLU-based inactivation function operations on the global mean feature vector obtained by global mean pooling of the resolution reconstruction feature map to obtain a global semantic vector; then, performing point convolution, batch normalization and ReLU-based inactivation function operations on the resolution reconstruction feature map to obtain a local semantic vector; then, performing point convolution, batch normalization and ReLU-based inactivation function operations on the resolution reconstruction feature map to obtain a local semantic vector; then, the global semantic vector and the local semantic vector are point-added to obtain a semantic weight vector; further, using the semantic weight vector as a weight vector, weighting the tissue slice fluorescence colorization shallow feature map to obtain a semantic joint feature map; then, fusing the tissue slice fluorescence colorization shallow feature map with the semantic joint feature map to obtain the semantically fused tissue slice fluorescence colorization shallow feature map.

[0039] It is worth mentioning that in other specific examples of the present application, the shallow features and deep features of the enhanced tissue slice fluorescence color image can also be extracted and fused in other ways to obtain the semantically fused tissue slice fluorescence color shallow feature map, for example: Image preprocessing: preprocessing the original tissue slice fluorescence color image, including denoising, adjusting brightness contrast, enhancing details, color correction and other steps. These preprocessing steps help to improve image quality and contrast, laying the foundation for subsequent feature extraction; shallow feature extraction: extract shallow features from the preprocessed image using computer vision technology, such as convolutional neural networks. These features are usually low-level visual features, such as edges, textures, colors, etc.; deep feature extraction: extract deep features from the preprocessed image using deep learning technology, such as pre-trained convolutional neural networks or custom deep neural networks. Deep features have higher-level semantic information and can capture more abstract and complex image features; feature fusion: fuse shallow features and deep features to obtain a semantically fused tissue slice fluorescence color shallow feature map. Fusion can be performed by simple weighted averaging, feature concatenation, or using a specific fusion network. The goal of fusion is to comprehensively utilize the advantages of shallow and deep features to enhance the semantic information expressed by features; Feature map analysis and application: Further analysis and application of the fused semantic fusion tissue section fluorescence color shallow feature map. This may involve tasks such as image segmentation, target detection, classification, and positioning. Select the corresponding algorithms and methods according to specific needs.

[0040] It is worth mentioning that in other specific examples of the present application, the image feature extraction of the tissue section fluorescence color image can also be performed in other ways to obtain a semantically fused tissue section fluorescence color shallow feature map, for example: Image preprocessing: preprocessing the fluorescence color image, such as denoising, smoothing and enhancement, to improve image quality and contrast; Feature extraction: extracting shallow features of the image using computer vision or deep learning methods. Common feature extraction methods include traditional image processing algorithms (such as edge detection, texture analysis, etc.) and convolutional neural network (CNN) models based on deep learning; for traditional image processing algorithms, edge detection algorithms (such as Canny edge detection), texture feature extraction algorithms (such as grayscale co-occurrence matrix), etc. can be applied to capture the surface features of the image; for CNN-based methods, pre-trained network models (such as VGG, ResNet, Inception, etc.) can be used for feature extraction. The fluorescence color image is input into the network model, and the feature representation of the middle layer of the network is extracted as a shallow feature; semantic fusion: the extracted shallow features are fused with the previously obtained semantic information to enhance the semantic meaning of the features. Semantic information can come from existing knowledge of tissue structure, cell type or molecular localization. Fusion methods such as feature-level fusion (such as feature concatenation, feature weighted averaging, etc.) or model-level fusion (such as multi-model fusion, attention mechanism, etc.) can be used to combine shallow features with semantic information. Feature map generation: The fused features are input into an appropriate image generation model, such as a generative adversarial network (GAN) or an autoencoder (AE), to generate a semantically fused shallow feature map of tissue section fluorescence colorimetry. These models can learn high-level feature representations of images and generate feature maps with better semantic information.

[0041] In particular, in step S3, the category of the tumor marker is determined based on the semantically fused tissue section fluorescence chromogenic shallow feature map. In particular, in a specific example of the present application, the S3 includes: passing the semantically fused tissue section fluorescence chromogenic shallow feature map through a classifier to obtain a classification result, and the classification result is used to represent the category label of the tumor marker. Specifically, the semantically fused tissue section fluorescence chromogenic shallow feature map is expanded into a classification feature vector based on a row vector or a column vector; the classification feature vector is fully connected encoded using multiple fully connected layers of the classifier to obtain an encoded classification feature vector; and the encoded classification feature vector is passed through the Softmax classification function of the classifier to obtain the classification result.

[0042] A classifier is a machine learning model or algorithm that is used to classify input data into different categories or labels. A classifier is part of supervised learning and performs classification tasks by learning the mapping relationship from input data to output categories.

[0043] A fully connected layer is a common type of layer in a neural network. In a fully connected layer, each neuron is connected to all neurons in the previous layer, and each connection has a weight. This means that each neuron in a fully connected layer receives inputs from all neurons in the previous layer, and performs a weighted sum of these inputs through weights, and then passes the result to the next layer.

[0044] The Softmax classification function is a commonly used activation function for multi-classification problems. It converts each element of the input vector into a probability value between 0 and 1, and the sum of these probability values ​​is equal to 1. The Softmax function is often used in the output layer of a neural network, especially for multi-classification problems, because it can map the network output to the probability distribution of each category. During the training process, the output of the Softmax function can be used to calculate the loss function and update the network parameters through the back propagation algorithm. It is worth noting that the output of the Softmax function does not change the relative size relationship between the elements, but only normalizes them. Therefore, the Softmax function does not change the characteristics of the input vector, but only converts it into a probability distribution form.

[0045] In the technical solution of the present application, the tissue slice fluorescence colorization shallow feature map and the tissue slice fluorescence colorization deep feature map respectively express the image semantic features of the tissue slice fluorescence colorization image at different scales and depths based on the pyramid network. When the semantic joint propagation module is used to fuse the tissue slice fluorescence colorization shallow feature map and the tissue slice fluorescence colorization deep feature map, the tissue slice fluorescence colorization shallow feature map is weightedly fused through the global semantic feature representation of the tissue slice fluorescence colorization deep feature map. At the same time, the overall feature distribution of the semantically fused tissue slice fluorescence colorization shallow feature map will deviate from the image source domain semantic feature distribution of the tissue slice fluorescence colorization image. Therefore, in the classification scenario, the probability of the semantically fused tissue slice fluorescence colorization shallow feature map will converge and diverge during the action of the classifier's weight matrix, thereby affecting the accuracy of the classification result.

[0046] Therefore, when the semantically fused tissue slice fluorescence coloration shallow feature map is classified by a classifier, the present application first optimizes the semantically fused tissue slice fluorescence coloration shallow feature map, including the following steps:

[0047] Determine the median eigenvalue, the maximum eigenvalue and the minimum eigenvalue in the feature set of the semantically fused tissue section fluorescence coloration shallow layer feature map, and divide the median eigenvalue by the difference between the maximum eigenvalue and the minimum eigenvalue to obtain the semantically fused tissue section fluorescence coloration shallow layer distribution probability value, that is, p = fmid / (f max -f min ), where f mid 、f max 、f min They respectively represent the median eigenvalue, the maximum eigenvalue and the minimum eigenvalue in the feature set of the semantically fused tissue section fluorescence coloration shallow layer feature map, and p represents the semantically fused tissue section fluorescence coloration shallow layer distribution probability value;

[0048] Each eigenvalue in the semantically fused tissue section fluorescence coloration shallow layer feature map is normalized to the maximum value to obtain the semantically fused tissue section fluorescence coloration shallow layer probability eigenvalue f i ;

[0049] The probability characteristic value f of the shallow layer of fluorescence coloration of each semantic fusion tissue slice is calculated by subtracting the shallow layer distribution probability value p of the semantic fusion tissue slice as an index. i The power function is used to obtain the first semantic fusion tissue section fluorescence color shallow layer probability convergence eigenvalue f 1i =f i 1-p , where p represents the probability value of the shallow layer distribution of fluorescence coloration in semantic fusion tissue slices, and f i represents the probability feature value of the shallow layer of fluorescence coloration in semantic fusion tissue slices, f 1i It represents the convergence eigenvalue of the probability of the shallow layer of the fluorescence coloration of the first semantic fusion tissue section;

[0050] The probability value p of the shallow layer of fluorescence coloration in the semantic fusion tissue slice is used as an index to calculate a probability characteristic value f of the shallow layer of fluorescence coloration in each semantic fusion tissue slice. i The power function of the difference of the second semantic fusion tissue section fluorescence color shallow layer probability convergence eigenvalue f 2i =(1-f i ) p , where f 2i It represents the second semantic fusion tissue section fluorescence color shallow layer probability convergence eigenvalue;

[0051] The probability feature value f of the shallow layer of each semantic fusion tissue slice fluorescence coloring i The difference between the probability value p of the shallow layer of fluorescence coloration of the semantic fusion tissue slice and one is multiplied to obtain the first semantic fusion tissue slice fluorescence coloration shallow layer probability limiting feature value f 3i =f i ×(1-p), where f 3i Indicates the probability-limited feature value of the shallow layer of the fluorescence coloration of the first semantic fusion tissue section;

[0052] The probability feature value f of the shallow layer of the fluorescent coloration of each semantic fusion tissue slice isi The difference between the values ​​of and is multiplied by the probability value p of the shallow layer of fluorescence coloration of the semantic fusion tissue slice to obtain the second semantic fusion tissue slice fluorescence coloration shallow layer probability limiting feature value f 4i =(1-f i )×p, where f 4i Indicates the second semantic fusion tissue section fluorescence color shallow layer probability limit feature value;

[0053] After multiplying the first semantic fusion tissue slice fluorescence coloration shallow layer probability convergence eigenvalue and the second semantic fusion tissue slice fluorescence coloration shallow layer probability convergence eigenvalue, further add the first semantic fusion tissue slice fluorescence coloration shallow layer probability limit eigenvalue and the second semantic fusion tissue slice fluorescence coloration shallow layer probability limit eigenvalue to obtain the eigenvalue f′ of the optimized semantic fusion tissue slice fluorescence coloration shallow layer feature map i =f 1i ×f 2i +f 3i +f 4i , where f′ y Represents the feature value of the shallow feature map of the optimized semantic fusion tissue section fluorescence color.

[0054] Finally, the optimized semantic fusion tissue slice fluorescence coloration shallow layer feature map is passed through a classifier to obtain a classification result, and the classification result is used to represent the category label of the tumor marker.

[0055] Based on this, by using the cross entropy form power series of the regression probability of the semantically fused tissue section fluorescence coloration shallow feature map as the probability distribution convergence limit, the regression probability convergence approximation is performed on the basis of the combination of the feature set distribution and the probability density distribution of the semantically fused tissue section fluorescence coloration shallow feature map, so as to avoid the fatal potential divergence of the semantically fused tissue section fluorescence coloration shallow feature map. On the premise of conservative convergence control, the probability distribution cross entropy is further used as the objective function to guide the limited recovery strategy, so as to achieve the common agility of the convergence of the probability convergent and divergent feature set of the semantically fused tissue section fluorescence coloration shallow feature map to the probability density distribution space, thereby improving the accuracy of the classification results.

[0056] It is worth mentioning that in other specific examples of the present application, the category of the tumor marker can also be determined based on the semantically fused tissue section fluorescence chromogenic shallow feature map in other ways, for example: first, the enhanced tissue section fluorescence chromogenic image is obtained through the steps described above. Then, the semantically fused tissue section fluorescence chromogenic shallow feature map is obtained using feature extraction and fusion methods; and data sets for training and testing are prepared. These data sets should contain sample images with tumor marker categories already labeled; the semantically fused tissue section fluorescence chromogenic shallow feature map is used as input, and the feature representation of the image is extracted by the feature extraction method. Convolutional neural network (CNN) or other feature extraction models can be used to extract features; using the labeled data set, the feature representation is used as input to train a tumor marker classification model. Common classification models include support vector machine (SVM), random forest (RandomForest), deep learning model, etc.; another independent test data set is used to evaluate the trained classification model. The performance of the model is evaluated by calculating indicators such as accuracy, recall, F1 score, etc.; the trained classification model is used to predict the new tissue section fluorescence chromogenic shallow feature map. Based on the output of the model, the category of tumor markers is determined.

[0057] In summary, the multiple immunohistochemical fluorescence colorimetric detection method based on a single glass slide according to the embodiment of the present application is explained, which combines deep learning algorithms and image processing technology to perform image processing and feature extraction on the fluorescence colorimetric images of tissue sections to achieve intelligent judgment of the category of tumor markers. Furthermore, a multiple immunohistochemical fluorescence colorimetric detection system based on a single glass slide is also provided.

[0058] Figure 5 FIG. 4 is a block diagram of a multiple immunohistochemical fluorescence colorimetric detection system based on a single slide according to an embodiment of the present application. Figure 5 As shown, according to an embodiment of the present application, a multiple immunohistochemistry fluorescence colorimetric detection system 300 based on a single glass slide includes: an image acquisition module 310, used to acquire a tissue section fluorescence colorimetric image; an image feature extraction module 320, used to extract image features from the tissue section fluorescence colorimetric image to obtain a semantically fused tissue section fluorescence colorimetric shallow feature map; and a category generation module 330, used to determine the category of the tumor marker based on the semantically fused tissue section fluorescence colorimetric shallow feature map.

[0059] As described above, the multiple immunohistochemical fluorescence colorimetric detection system 300 based on a single glass slide according to the embodiment of the present application can be implemented in various wireless terminals, such as a server having a multiple immunohistochemical fluorescence colorimetric detection algorithm based on a single glass slide. In a possible implementation, the multiple immunohistochemical fluorescence colorimetric detection system 300 based on a single glass slide according to the embodiment of the present application can be integrated into a wireless terminal as a software module and / or a hardware module. For example, the multiple immunohistochemical fluorescence colorimetric detection system 300 based on a single glass slide can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the multiple immunohistochemical fluorescence colorimetric detection system 300 based on a single glass slide can also be one of the many hardware modules of the wireless terminal.

[0060] Alternatively, in another example, the multiple immunohistochemistry fluorescence colorimetric detection system 300 based on a single slide and the wireless terminal may be separate devices, and the multiple immunohistochemistry fluorescence colorimetric detection system 300 based on a single slide may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0061] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A multiplex immunohistochemical fluorescence colorimetric detection method based on a single slide, characterized in that: include: Acquire fluorescent colorimetric images of tissue sections; Performing image feature extraction on the tissue slice fluorescence colorimetric image to obtain a semantically fused tissue slice fluorescence colorimetric shallow layer feature map; as well as Based on the semantically fused tissue slice fluorescence coloration shallow layer feature map, the category of the tumor marker is determined.

2. The multiple immunohistochemical fluorescence colorimetric detection method based on a single slide according to claim 1, characterized in that: Performing image feature extraction on the tissue slice fluorescence color image to obtain a semantically fused tissue slice fluorescence color shallow feature map, including: Performing image preprocessing on the tissue slice fluorescence colorimetric image to obtain an enhanced tissue slice fluorescence colorimetric image; and The shallow features and deep features of the enhanced tissue slice fluorescence colorimetric image are extracted and fused to obtain the semantically fused tissue slice fluorescence colorimetric shallow feature map.

3. The multiple immunohistochemical fluorescence colorimetric detection method based on a single slide according to claim 2, characterized in that: Performing image preprocessing on the tissue slice fluorescence colorimetric image to obtain an enhanced tissue slice fluorescence colorimetric image, comprising: The tissue slice fluorescence chromogenic image is subjected to bilateral filtering to obtain the enhanced tissue slice fluorescence chromogenic image.

4. The multiple immunohistochemical fluorescence colorimetric detection method based on a single slide according to claim 3, characterized in that: Extracting and fusing the shallow features and deep features of the enhanced tissue slice fluorescence color image to obtain the semantic fusion tissue slice fluorescence color shallow feature map, including: The enhanced tissue slice fluorescence color image is passed through an image multi-scale feature extractor based on a pyramid network to obtain a tissue slice fluorescence color shallow layer feature map and a tissue slice fluorescence color deep layer feature map; and A semantic joint propagation module is used to fuse the tissue slice fluorescence coloration shallow feature map and the tissue slice fluorescence coloration deep feature map to obtain the semantically fused tissue slice fluorescence coloration shallow feature map.

5. The multiple immunohistochemical fluorescence colorimetric detection method based on a single slide according to claim 4, characterized in that: Based on the semantically fused tissue section fluorescence coloration shallow layer feature map, the category of the tumor marker is determined, including: The semantically fused tissue slice fluorescence coloration shallow layer feature map is passed through a classifier to obtain a classification result, and the classification result is used to represent the category label of the tumor marker.

6. A multiplex immunohistochemical fluorescence colorimetric detection system based on a single slide, characterized in that: include: An image acquisition module, used for acquiring fluorescent color development images of tissue sections; An image feature extraction module is used to extract image features from the tissue slice fluorescence color image to obtain a semantically fused tissue slice fluorescence color shallow layer feature map; as well as The category generation module is used to determine the category of the tumor marker based on the semantic fusion tissue slice fluorescence coloration shallow layer feature map.

Images