A method for analyzing multiple immunohistochemical fluorescence staining images and related equipment

Abstract

The present invention provides an analysis method and related equipment for multiple immunohistochemical fluorescence staining images, which are applied to the field of data processing technology. The present application obtains original biological sample data; slices the original biological sample data to generate a number of initial biological slice information; processes the initial biological slice information to generate a fluorescent staining slice image; processes the fluorescent staining slice image to generate an aligned fluorescent staining slice image; performs fusion processing on the aligned fluorescent staining slice image to generate a target fluorescent staining image; processes the target fluorescent staining image to generate initial region information; processes the initial region information to generate multiple images of different fluorescent channels, wherein the images of multiple different fluorescent channels are single fluorescent channel images; processes the images of multiple different fluorescent channels to generate cell clustering information.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an analysis method for multiple immunohistochemical fluorescence staining images and related equipment. Background Art

[0002] The occurrence and development of tumors involve a continuous and dynamic interaction between tumor cells and the tumor immune microenvironment (TIME). TIME has a diverse composition and complex pathophysiological mechanisms, including all non-tumor host cells in the tumor, including fibroblasts, endothelial cells, neurons, adipocytes, adaptive and innate immune cells, and their non-cellular components, including the extracellular matrix and soluble products such as chemokines, cytokines, growth factors, and extracellular vesicles.

[0003] Immunohistochemistry (IHC) is currently the most widely used histopathological diagnostic technology. However, this technology only allows the labeling of a single marker for each tissue section, which is not conducive to obtaining information about diagnosis and prognosis from patient samples. However, the immune markers of each component in TIME are complex and large in number, and the several markers that can be detected by multiple fluorescent immunohistochemistry (mIHC) are increasingly unable to meet the needs of researchers. In addition, the prior art also has the slicing of IHC. IHC slice fusion usually uses a double multi-resolution pyramid model based on the SIFT feature matching algorithm for registration. First, the tissue level registration is quickly completed based on the high-level low-resolution image, and then the registration parameters are applied to the large image with high resolution at the bottom layer. Although the registration speed has been improved to a certain extent, relatively speaking, its registration accuracy at the cellular level is not as good as the method of fusion of mIHC slices based on a complex machine learning model proposed in this application, and it cannot meet the needs of accurate analysis of the subtle structures and relationships between cells.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present application is to provide an analysis method and related equipment for multiple immune tissue fluorescent staining images, which at least to some extent overcomes the problems existing in the prior art, performs multiple fluorescent immunohistochemical staining on multiple tissue sections, and each section is marked with specific cell components related to tumor immunity. Through a series of steps such as layer extraction, manual feature point selection, calculation and application of affine transformation matrix, the alignment and fusion of multiple section images are achieved, and more immune markers are indirectly detected, which can reflect the spatial distribution and interaction of more types of cells in the tumor immune microenvironment, and more comprehensively characterize the characteristics of the tumor immune microenvironment, which can help researchers systematically, deeply and accurately understand the information of the tumor immune microenvironment.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present invention.

[0007] According to one aspect of the present application, a method for analyzing multiple immune tissue fluorescent staining images is provided, comprising: acquiring original biological sample data; performing slicing processing on the original biological sample data to generate a number of initial biological slice information, wherein the initial biological slice information includes a target marker, and the target marker includes at least one of lymphocytes, myeloid cells, and interstitial cells; processing the initial biological slice information to generate a fluorescent staining slice image, wherein the fluorescent staining slice image is an image of multiple fluorescent channels, and the fluorescent staining slice image is generated based on multiple fluorescent immunohistochemistry technology; processing the fluorescent staining slice image to generate an aligned fluorescent staining slice image; performing fusion processing on the aligned fluorescent staining slice image to generate a target fluorescent staining image; processing the target fluorescent staining image to generate initial region information, wherein the initial region information is used to characterize a region where tumors and / or immune cells are enriched; processing the initial region information to generate multiple images of different fluorescent channels, wherein the images of the multiple different fluorescent channels are single fluorescent channel images; processing the images of the multiple different fluorescent channels to generate cell clustering information, wherein the cell clustering information is used to characterize the immune status of the tumor immune microenvironment.

[0008] Another aspect of the present application is a multiplex immunohistochemical fluorescence staining image analysis device, characterized in that it includes: an acquisition module for acquiring original biological sample data; a processing module for slicing the original biological sample data to generate a number of initial biological slice information, wherein the initial biological slice information includes a target marker, and the target marker includes at least one of lymphocytes, myeloid cells, and interstitial cells; processing the initial biological slice information to generate a fluorescent staining slice image, wherein the fluorescent staining slice image is an image of multiple fluorescent channels, and the fluorescent staining slice image is generated based on multiple fluorescent immunohistochemistry technology; The fluorescent staining slice image is processed to generate an aligned fluorescent staining slice image; the aligned fluorescent staining slice image is fused to generate a target fluorescent staining image; the target fluorescent staining image is processed to generate initial region information, wherein the initial region information is used to characterize the region where tumor and / or immune cells are enriched; the initial region information is processed to generate images of multiple different fluorescent channels, wherein the images of multiple different fluorescent channels are single fluorescent channel images; the images of multiple different fluorescent channels are processed to generate cell clustering information, wherein the cell clustering information is used to characterize the immune status of the tumor immune microenvironment.

[0009] According to another aspect of the present application, an electronic device is characterized in that it includes: a first processor; and a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the above-mentioned multiple immunotissue fluorescence staining image analysis method by executing the executable instructions.

[0010] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a second processor, the above-mentioned method for analyzing multiple immunohistochemical fluorescence staining images is implemented.

[0011] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein the computer program implements the above-mentioned multiple immunohistochemical fluorescence staining image analysis method when executed by a third processor.

[0012] The present application provides a method for analyzing multiple immunohistochemical images and related equipment. The server performs multiple fluorescent immunohistochemical staining on multiple tissue slices. Each slice is marked with specific cell components related to tumor immunity. Through layer extraction, manual feature point selection, affine transformation matrix calculation and application of a series of steps, multiple slice images are aligned and fused, and more immune markers are detected indirectly. It can reflect the spatial distribution and interaction of more types of cells in the tumor immune microenvironment, and more comprehensively characterize the characteristics of the tumor immune microenvironment, which can help researchers systematically, deeply and accurately understand the information of the tumor immune microenvironment. Compared with the mIHC of a single slice, the present invention constructs a fusion algorithm to fuse the mIHC fluorescence images marked by different markers from several slices into one image, integrates a variety of cell components and non-cellular component information contained in each slice, and can analyze their spatial distribution and interaction, which is conducive to researchers to conduct a more comprehensive study on the cell composition, functional state and cell-cell interaction of TIME. It can better help the clinic diagnose, treat and predict the prognosis of tumors, and also has the advantages of high sensitivity and low cost.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flowchart showing a method for analyzing multiple immunohistochemical fluorescence staining images provided by an embodiment of the present application is shown;

[0015] Figure 2 A schematic diagram of the structure of a device for analyzing multiple immunohistochemical fluorescence staining images provided in one embodiment of the present application is shown;

[0016] Figure 3 A schematic diagram of the structure of a machine learning model provided in one embodiment of the present application is shown;

[0017] Figure 4 A flow chart of a storage medium provided by an embodiment of the present application is shown;

[0018] Figure 5 A flow chart of fusing fluorescence images of different regions of the same sample and performing cell segmentation is shown in one embodiment of the present application;

[0019] Figure 6 A schematic diagram of cell clustering provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0021] Combine the following Figure 1 To describe the analysis method of multiple immunohistochemical fluorescence staining images according to an exemplary embodiment of the present application. It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this regard. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0022] In one embodiment, the present application also provides a method for analyzing multiple immunohistochemical fluorescence staining images and related equipment. Figure 1 As shown, the method is applied to a server, comprising:

[0023] S101, obtaining original biological sample data.

[0024] In one embodiment, clinical information, genetic test results, etc. of 90 patients with lung adenocarcinoma are retrospectively collected. Specifically, demographic characteristics such as age, gender, and body mass index (BMI) of the patients are obtained. This information helps to understand the basic composition of the patient population and may be helpful for analyzing the differences in the tumor immune microenvironment of patients with different characteristics. Smoking history: smoking is an important risk factor for lung adenocarcinoma, which may affect the occurrence and development of the tumor and the state of the immune microenvironment. Tumor staging information, including tumor size, whether there is lymph node metastasis, and whether there is distant metastasis, is critical for assessing the severity and prognosis of the tumor. Symptoms, such as the presence and severity of symptoms such as cough, hemoptysis, chest pain, and dyspnea, can reflect the impact of the tumor on the body and the progression of the disease. Treatment history, including whether the patient has received surgical treatment, chemotherapy, radiotherapy, etc., as well as the specific treatment plan and treatment effect, is of great significance for subsequent studies on the response of the tumor to treatment and changes in the immune microenvironment. In addition, epidermal growth factor receptor (EGFR) gene mutation is one of the focuses in lung adenocarcinoma research. The test results may show mutant or wild type. Different genotypes may have different sensitivities to targeted therapeutic drugs, affecting treatment decisions and prognosis. KRAS gene mutation. The KRAS gene also has a certain mutation frequency in lung adenocarcinoma, and its mutation status is related to the biological behavior and treatment response of the tumor. Other related gene tests may also include the detection of genes related to tumor immune regulation, such as the expression of programmed death receptor 1 (PD-1) and its ligand (PD-L1) genes. These genes play an important role in the mechanism of tumor immune escape, and their test results are of great value for studying the tumor immune microenvironment and the application of immunotherapy. Detect changes in gene expression related to tumor cell proliferation, apoptosis, metastasis, etc. The abnormal expression of these genes may affect the development process of the tumor. By analyzing its test results, we can gain a deep understanding of the biological characteristics of the tumor.

[0025] S102, slicing the original biological sample data to generate a number of initial biological slice information.

[0026] In one embodiment, the initial biological slice information includes target markers, and the target markers include at least one of lymphocytes, myeloid cells, and interstitial cells. According to the research needs, 6 markers of lymphocytes (L), myeloid cells (M), and interstitial cells (N) were selected, for a total of 18 markers. FFPE wax blocks with relatively enriched tumors were selected, and continuous sections were performed, and three 4um sections were cut, corresponding to the detection of L, M, and N related markers. OpalTM 7-color multiple fluorescent immunohistochemistry kit was used for multiple staining of the selected 6 different markers for the three sections of L, M, and N. For example, lymphocyte markers may include CD8a (Cytotoxic T), CD4 (Helper T), etc.; myeloid cell markers may include CD86 (Macrophage I), CD68 (Macrophages), etc.; interstitial cell markers may include Pan-cytokeratin (Keratinocytes), aSMA (Myofibroblasts, pericytes), etc. In this way, the corresponding target markers are marked on the three slices, and initial biological slice information containing the target markers is generated.

[0027] S103, processing the initial biological slice information to generate a fluorescent stained slice image.

[0028] In one embodiment, the fluorescently stained slice image is an image of multiple fluorescent channels. In a study of lung adenocarcinoma patient samples, after staining three slices L, M, and N, each QPTIFF image is in a multi-layer pyramid format, containing multiple layers of different resolution sizes, each layer containing 8 channels, which are the images of the fluorescent channels.

[0029] Get the preset matching feature points, use the matplotlib library to display and create a window containing three sub-images L, M, and N. Use the PointCollector class to process the mouse click event on the image, and then use the onclick function to capture the location where the user clicks the image, and record the coordinates of this location, which is the feature point manually selected by the user. Select 5 or 6 groups of feature points on each image. For example, when operating on the slice image of a lung adenocarcinoma sample, select characteristic points such as corners, holes, and gaps on the three images L, M, and N as feature points.

[0030] The fluorescent stained slice images are subjected to layer extraction processing to generate grayscale images, which are used to align fluorescent stained slice images of different categories. Specifically, the first channel (DAPI) of the same numbered images in categories L, M, and N are extracted as grayscale images for alignment.

[0031] The fluorescent stained slice image is processed based on the preset matching feature points and the grayscale image to generate an affine transformation matrix. The method includes obtaining a calculation formula for the affine transformation matrix, which is:

[0032]

[0033]

[0034] Where T is the affine transformation matrix; a and d are scaling factors (horizontal and vertical scaling); b and c are shearing factors (used to control the tilt of the image); is the translation amount, is the coordinate point in the fluorescent slice image; is the coordinate point in the grayscale image; is the parameter vector we want to solve for, is the transpose of the matrix M; yes The inverse matrix of .

[0035] Specifically, assume that the coordinates of the three points selected on the fluorescent slice image are (1, 2), (3, 4), and (5, 6), and the coordinates of the corresponding three points on the grayscale image are (2, 3), (4, 5), and (6, 7).

[0036] For the points (1, 2) and (2, 3), according to the formula It can be seen that .

[0037] For the points (3, 4) and (4, 5), we can derive: .

[0038] For the points (5, 6) and (6, 7), we can derive: .

[0039] Writing these equations in matrix form:

[0040]

[0041] Combined into one large matrix equation:

[0042] By solving this matrix equation, we can get The value of , thereby determining the affine transformation matrix .

[0043] The affine transformation matrix is ​​processed to generate the inverse mapping and target interpolation, including obtaining the calculation formula of the inverse mapping, which is: ;

[0044] Among them, x and y: represent the coordinate points in the original image (i.e. the image you want to get after alignment), which are the result coordinates obtained after inverse mapping calculation and are used to determine the corresponding positions of points in the target image in the original image; : represents the coordinate point in the target image, usually the coordinate of the point in the image that has not been aligned; These parameters are the coefficients associated with the affine transformation matrix, and are obtained by inverting the affine transformation matrix. To some extent, it can be understood as being related to the scaling relationship between the original image and the target image. For example, if A larger value of may mean that the target image is greatly stretched relative to the original image in the horizontal direction; A similar effect occurs in the vertical direction. Related to the tilt of the image. They determine the tilt of the target image in the horizontal and vertical directions when it is mapped to the original image. is the translation parameter. Controls the horizontal translation. Controls the vertical translation. If If it is a positive value, the points in the target image will be moved to the right by the corresponding unit in the horizontal direction when mapped to the original image; A positive value will move the element vertically upward by that amount.

[0045] Assume that the affine transformation matrix has been calculated , its inverse matrix .

[0046] For a point (3, 4) in the target image, according to the inverse mapping formula ;

[0047] , we can conclude that:

[0048] .

[0049] So the corresponding position of this point in the original image is (0.5, 3). After the transformation, some points in the target image may be mapped outside the boundaries of the original image. Here, a constant value (usually 0) is used to fill in the area beyond the boundary.

[0050] The method includes obtaining a calculation formula for the target interpolation, which is:

[0051] ;

[0052] in, Represents the horizontal and vertical distance respectively.

[0053] Assume that the target pixel position (0.5, 3) is between the four neighboring pixels (0.2)(1, 2)(0, 3)(1, 3), the pixel value in the original image .set up , .

[0054] According to the calculation formula of target interpolation ;

[0055] It can be concluded that .

[0056] By performing the above inverse mapping and target interpolation operations on each point in image B, an aligned fluorescent staining slice image can be finally obtained. That is, the cell structure in image B is aligned with the cell structure in image A in spatial position, which is convenient for subsequent analysis and processing, such as image fusion and other operations.

[0057] S104, processing the fluorescent staining slice image to generate an aligned fluorescent staining slice image.

[0058] In one embodiment, taking the study of a tumor sample as an example, a tissue sample is obtained from a patient and sliced. The slice is fluorescently stained to obtain a fluorescently stained slice image. Assume that we have three slice images, labeled as image A, image B, and image C, corresponding to different cell marker staining. Each fluorescently stained slice image is in a multi-layer pyramid format, containing multiple layers of different resolution sizes, and each layer contains 8 channels. The first channel (DAPI) of the same numbered image in image A, image B, and image C is extracted as a grayscale image for alignment. For example, the first channel of the first layer is extracted from the multi-layer structure of image A as its corresponding grayscale image, and the same operation is performed on image B and image C. A window containing three sub-images (corresponding to image A, image B, and image C, respectively) is created using a related image processing library (such as the matplotlib library). The mouse click event on the image is processed by a specific class (such as the PointCollector class), and 5-6 groups of feature points are selected on each image. These feature points can be easy-to-identify and match locations such as the corners of the cell structure and the center of a specific cell area. For example, several corner points of the cell boundary are selected as feature points in image A, and similar feature points are also selected near the corresponding positions in images B and C.

[0059] Using the selected points on image A as the reference, match the points of image B and image C with the points of image A. Assume that the coordinates of a feature point on image A are , the corresponding feature point coordinates on image B are , the corresponding feature point coordinates on image C are By the least squares method, according to the formula ; For example, for a set of points in image A and image B, we have By calculating multiple groups of points, we can get the parameters in the affine transformation matrix: , thereby determining the affine transformation matrix.

[0060] After calculating the affine transformation matrix T, find its inverse matrix For each point in the target image (such as image B and image C relative to image A), , according to the inverse mapping formula Find its corresponding position (x, y) in the original image (such as image A). Since (x, y) may not be an integer coordinate, bilinear interpolation is used. Assume that the pixel value in the original image is , target pixel position Between four adjacent pixels (x1, y1), (x2, y1), (x1, y2), (x2,y2), according to the target interpolation formula ,in Calculate the pixel value at that location.

[0061] Through inverse mapping and interpolation processing, the above operations are performed on each pixel in image B and image C, so that image B and image C are aligned with image A in spatial position, and finally an aligned fluorescent staining slice image is generated. For example, the positions of cells in image B and image C are inconsistent with the corresponding cell positions in image A. After processing, they accurately correspond in space, which is convenient for subsequent analysis and research. Through the above inverse mapping, the corresponding position of each point in the target image in the original image is found, and then the pixel value of the position is determined by target interpolation. Through such inverse mapping and interpolation operations, a preliminary aligned image is obtained, that is, an initial fluorescent staining slice image, which at least contains relevant markers for characterizing lymphocytes, myeloid cells, interstitial cells or extracellular matrix (such as the various cell markers mentioned above). The initial fluorescent staining slice image is processed based on the machine learning model to generate an aligned fluorescent staining slice image, which will be explained in detail later.

[0062] In another embodiment, the initial fluorescent staining slice image is processed based on the machine learning model to generate an aligned fluorescent staining slice image, as follows:

[0063] like Figure 3As shown, the initial fluorescent stained slice image is processed based on the machine learning model to generate feature maps of different levels, wherein the feature maps of the same level include feature information of different channel dimensions, and the initial fluorescent stained slice image includes a reference image and a target image. Assuming that we are studying the immune microenvironment of a tumor sample, we first obtain an initial fluorescent stained slice image by processing the sample, including a reference image (L) for characterizing lymphocytes and a target image (M, N) for characterizing myeloid cells and interstitial cells. These images are obtained through the steps of multiple fluorescent immunohistochemical staining, scanning, image alignment (preliminary alignment based on inverse mapping and interpolation) mentioned above. The initial fluorescent stained slice image is in a multi-layer pyramid format, containing multiple layers of different resolution sizes, each layer has 8 channels (taking the QPTIFF image in the embodiment as an example), and these initial fluorescent stained slice images (L, M, N) are input into the machine learning model. The convolutional network architecture in the model, i.e. (CNN), starts working as a feature extractor. The convolutional network architecture, i.e. (CNN), contains multiple convolutional blocks, for example, there are three convolutional blocks, and each convolutional block has a different number of convolutional layers (such as the first convolutional block has 3 convolutional layers, the second convolutional block has 4 convolutional layers, the third convolutional block has 5 convolutional layers, etc. The specific number can be designed according to the actual model).

[0064] When the input image passes through the first convolution block, the feature information of the image is initially extracted to generate the first layer feature map, whose dimension may be (Assume H1=512, W1=512, C1=64, where the number of channels C1 is the new channel dimension obtained after the convolution operation, representing different feature information.) As the image continues to pass through subsequent convolution blocks, feature maps of different levels are generated. For example, the dimension of the second-layer feature map may become (Assuming H2=256, W2=256, C2=128), the dimension of the third layer feature map is (Assume H3=128, W3=128, C3=256), etc. The feature map at each level contains rich image feature information. For example, the feature map of the baseline image (L) representing lymphocytes may contain feature information such as cell morphology and distribution density; the feature map of the target image (M, N) representing myeloid cells and interstitial cells also contains various feature information of the corresponding cells, and these feature information are reflected in different channel dimensions. For example, some channels may focus more on cell contours, while some channels may focus more on the internal structure of cells.

[0065] Based on the machine learning model, the feature maps of different levels are processed respectively to generate several feature maps with the same channel dimension. Taking the feature map of a certain level as an example (such as the second layer feature map), its dimension is ). The gated block module in the model, namely (MuGI Block) structure, comes into play. The gated block module, namely (MuGI Block), divides the feature map into two halves in the channel dimension, namely the first half C2 / 2=64 and the second half C2 / 2=64. The second half is activated by the sigmoid function as the weight of the gating mechanism, and then element-by-element multiplication is performed with the first half feature map of the other stream. Assuming that the pixel value of a certain area of ​​the first half feature map is [10, 20, 30], and the weight value of the corresponding area after the sigmoid activation function of the second half is [0.5, 0.6, 0.7], the result of element-by-element multiplication is [5, 12, 21]. After such a gated interaction, the information of the feature map is recombined and adjusted to generate a new feature map (the number is the same as the number of input feature maps, and the channel dimension may remain unchanged or change after subsequent processing. Here, it is assumed that the channel dimension is still 128). The same operation is performed on the feature maps of other levels in sequence, thereby generating several feature maps with the same channel dimension. These feature maps have been optimized at different levels to better extract and fuse the feature information in the image, preparing for the subsequent generation of the deformation field.

[0066] The feature maps of the same channel dimension are processed separately to generate local deformation fields and target loss values, where the target loss value includes the loss value and smoothing loss value used to measure the structural similarity of two fluorescent stained slice images. For the feature maps of the same channel dimension obtained by the previous processing (taking a certain level as an example), the model further processes them through the self-attention module, i.e., (Self-Attention) and the cross-attention module, i.e., (Cross-Attention) mechanism.

[0067] In the self-attention module, the query, key and value matrices are calculated from the feature map through linear transformation. Assume that on a certain feature map, through a specific linear projection matrix, (These matrices are the parameters learned by the model during training) and the feature map is converted into the Query matrix Q, the Key matrix K and the Value matrix V. The dimensions of each matrix are (assuming Then, by calculating the dot product of Query and Key, we can get the relationship between each pixel and other pixels. For example, after calculating the attention matrix, we can get the output feature map through weighted calculation (the dimension is still ). This output feature map is residually connected with the reference image feature (assuming the reference image feature is F, the result after residual connection is F+Output), and then normalized by LayerNorm to ensure the stability of the output value range. It is then sent to the MLP (Multi-layer Perceptron) for nonlinear transformation. The MLP contains two fully connected layers and activation functions (such as ReLU activation function) to further improve the feature expression ability, and finally the processed feature map is obtained for subsequent calculations.

[0068] The processing flow of the cross-attention module is similar to that of the self-attention module, but the query, key, and value come from the reference image features, the output of the high-level feature self-attention, and the target image features, respectively. For example, the query may come from the features of the reference image after being processed by the self-attention module, and the key and value come from the target image features and the output of the high-level feature self-attention, respectively. After the residual connection, LayerNorm, and MLP processing, the feature information used to generate the deformation field is obtained. Based on these processed feature information, a local deformation field is generated. The dimension of the deformation field is (Assume H=1024, W=1024), where the displacement vector of each pixel represents the displacement in the x and y directions. For example, at a certain pixel position (x, y), the value of the deformation field is (dx, dy), which means that the pixel needs to be moved units in the x direction and dy units in the y direction to achieve alignment with the reference image.

[0069] At the same time, the target loss value is calculated. The target loss value consists of two parts:

[0070] The loss value used to measure the structural similarity of two fluorescent stained slice images .calculate ,in is the image transformed by the deformation field (the aligned image, such as the image M or N that is aligned with the reference image L after the deformation field transformation), is the reference image (L image). SSIM is an indicator to measure the structural similarity of images, ranging from 0 to 1. The higher the value, the higher the structural similarity. , the model can try to make the aligned image more similar in structure to the reference image to optimize the image alignment effect.

[0071] Smoothing loss value ,in is the displacement vector of the deformation field at position p, denote the gradient of the deformation field in the x and y directions respectively. The smoothness loss is used to constrain the generated deformation field to make it smooth in space and avoid abrupt deformation. For example, if the gradient of the deformation field in a certain area in the x direction changes greatly, that is, the displacement of adjacent pixels changes dramatically, then Increasing the value will cause the model to adjust the deformation field to make the deformation smoother.

[0072] The local deformation field and target loss value are processed to generate the target deformation field. For each slice (such as M slice and N slice relative to the reference L slice), after calculating the local deformation field and target loss value, the model back-propagates and updates the network parameters according to the loss value. For example, if in is the weight parameter used to smooth the loss, assuming If the value is large, it means that the current deformation field effect is not ideal. The model adjusts the relevant parameters of the generated deformation field (such as the convolution kernel parameters in the convolutional network architecture (CNN), the projection matrix in the Attention mechanism, etc.) through back propagation to optimize the deformation field.

[0073] When processing overlapping areas, since the local deformation fields calculated by different slices may have slight differences, the weighted average method is used to fuse the deformation fields of the overlapping areas. For example, in the overlapping area, for a certain pixel position, the weight is calculated according to its distance to the center of the overlapping area (assuming that the closer the distance to the center is, the weight is 1, and the farther the distance is, the weight gradually decreases to 0.5, etc.), and then the deformation field values ​​of different slices at this position are weighted summed to obtain the fused deformation field value. This can ensure a smooth transition of the deformation field in the overlapping area and avoid abrupt displacement jumps.

[0074] After the local deformation fields of all slices are optimized and fused, the deformation fields of these slices are spliced ​​back to the original image position to form a global deformation field. For example, the deformation fields of the processed M slices and N slices are integrated with the information of the reference L slice to obtain a global deformation field covering the entire image. This global deformation field describes the deformation of each pixel point from the target image (unaligned image) to the reference image in the x and y directions, so that the target image (M, N) can be accurately aligned to the reference image (L).

[0075] The machine learning model can achieve accurate registration from the tissue level to the cell level. It improves the accuracy of image alignment, especially at the cell level, by selecting feature points, calculating affine transformation matrices, and generating and optimizing deformation fields. Using convolutional neural networks (convolutional network architecture, i.e. (CNN)) as feature extractors, the model can extract multi-level feature information from images of different resolutions, which helps to more comprehensively understand and analyze the structure and characteristics of cells. The model contains structures such as the gated block module, i.e. (MuGI Block), the self-attention module, i.e. (Self-Attention), the cross-attention module, i.e. (Cross-Attention), and the concatenation block module, i.e. (Concat Block), which can process and optimize feature maps and generate more accurate local deformation fields, thereby improving the quality of image alignment.

[0076] When processing overlapping areas of images, the model uses a weighted average method to fuse the deformation field, ensuring a smooth transition of the deformation field in the overlapping area and avoiding abrupt displacement jumps. The machine learning model can integrate information from multiple fluorescence channels and comprehensively analyze the various characteristics of cells, which is crucial for understanding the expression and functional status of different markers of cells. Through the Attention mechanism, the model can focus on key areas and features in the image, enhance attention to important cell structures and relationships, and improve the accuracy of registration and analysis. The model can evaluate the quality of the preset fluorescent staining image by calculating indicators such as correlation coefficient value, structural similarity index, sample set similarity value, mean square error value, and peak signal-to-noise ratio to ensure that the generated target fluorescent staining image meets the requirements.

[0077] Based on the cell subtype information, the model can use a machine learning clustering algorithm to perform cluster analysis on the sample data, thereby revealing the distribution of different cell subtypes in the tumor immune microenvironment, which helps to characterize the immune status of the tumor immune microenvironment. The machine learning model can help researchers more comprehensively characterize the characteristics of the tumor immune microenvironment, including cell composition, functional status, and cell-cell interactions, which is of great significance for the diagnosis, treatment, and prognosis of tumors. Compared with the mIHC of a single slice, the present invention constructs a fusion algorithm to fuse the mIHC fluorescence images marked by different markers from several slices into one image, integrating a variety of cell components and non-cellular component information contained in each slice, and can analyze their spatial distribution and interaction, which is conducive to researchers to conduct a more comprehensive study of the cell composition, functional status, and cell-cell interactions of TIME, while having the advantages of high sensitivity and low cost. In summary, the machine learning model provides an efficient, accurate, and comprehensive method in the analysis of multiple immune tissue fluorescence staining images, which can improve the quality and depth of image processing and provide a powerful tool for the study of tumor immune microenvironment.

[0078] The initial fluorescent stained slice image is processed based on the target deformation field to generate an aligned fluorescent stained slice image. The initial fluorescent stained slice image (M, N image) is processed based on the target deformation field, and the pixels in the target image are moved according to the displacement vector in the deformation field to achieve precise alignment. For example, for a pixel in the M image , whose displacement vector in the deformation field is , then move the pixel to a new position . After such processing, the M and N images are accurately aligned with the reference image L, and the aligned fluorescent staining slice images are obtained. These aligned images can more accurately reflect the spatial distribution and interaction relationship of different cell components (such as lymphocytes, myeloid cells, stromal cells, etc.) in the tumor immune microenvironment, and provide a more reliable data basis for subsequent cell segmentation, clustering analysis and other studies. In the embodiment, after such processing, the distribution of different cell subsets in the image can be more clearly observed, such as the distribution and mutual relationship of Cytotoxic T cells expressing CD8a, Macrophage II cells expressing CD68 and CD206, and Endothelial cells expressing CD31, which is helpful for in-depth study of the characteristics of the tumor immune microenvironment.

[0079] S105, performing fusion processing on the aligned fluorescent staining slice images to generate a target fluorescent staining image.

[0080] In one embodiment, the channel information of the aligned fluorescent staining slice images is obtained. In the study of lung adenocarcinoma samples, three aligned fluorescent staining slice images L, M, and N are obtained after the previous processing. Each image has 8 channel information. Figure 4 and Figure 5 As shown, for example, the 8 channels of the L image may correspond to different marker staining conditions, such as channel 1 is DAPI staining for displaying the cell nucleus, and channel 2 may be a staining of a certain cell surface marker, etc. Based on the channel information of the differently aligned fluorescent staining slice images, sequential stacking processing is performed to generate a preset fluorescent staining image. The three aligned images L, M, and N are sequentially stacked. Specifically, the 8 channels at each image position are sequentially stacked, first taking the 8 channels of the L image, then the 8 channels of the M image, and finally the 8 channels of the N image, so that the final number of channels is 24, and a preset fluorescent staining image is generated.

[0081] The preset fluorescent staining image is processed based on the preset verification rules to generate a target fluorescent staining image. The preset verification rules include correlation coefficient, Dice coefficient, mean square error, peak signal-to-noise ratio, structural similarity index, etc. Taking the correlation coefficient as an example, the pixel value similarity between the aligned image and the reference image is calculated. The value range of the correlation coefficient is between [-1,1]. The closer the value is to 1, the stronger the linear relationship between the images. By calculating these indicators, the quality of the preset fluorescent staining image is evaluated. If the indicators meet the requirements, the preset fluorescent staining image is the target fluorescent staining image. If some indicators do not meet the requirements, it may be necessary to adjust the previous processing steps, regenerate the preset fluorescent staining image and perform verification until a target fluorescent staining image that meets the requirements is obtained.

[0082] In the process of image registration of mIHC slices, the method based on inverse mapping and target interpolation and machine learning models (including convolutional network architecture, i.e. (CNN), gated block module, i.e. (MuGI Block), splicing block module, i.e. (ConcatBlock), Attention mechanism and other structures) can achieve accurate registration from tissue level to cell level. Through a series of operations such as feature point selection, affine transformation matrix calculation (based on least squares method), feature extraction, deformation field generation and optimization (through self-attention module, i.e. (Self-Attention) and cross-attention module, i.e. (Cross-Attention) mechanism), when processing overlapping areas, the weighted average method and other fine processing methods are used to make the image registration more accurate, especially at the cellular level, which can achieve accurate alignment and ensure the accuracy of subsequent analysis. mIHC (multiple immunohistochemistry) slice fusion can simultaneously label multiple different antibodies (or antigens) at the same site and compare and observe their expression. For example, when studying the tumor immune microenvironment, multiple markers related to lymphocytes (CD8a, CD4, etc.), myeloid cells (CD86, CD68, etc.), and stromal cells (Pan-cytokeratin, αSMA, etc.) can be simultaneously labeled to clearly show the distribution and relationship of different cell types and their markers in tissues, providing rich information for in-depth research on the cell composition and function of the tumor microenvironment. The expression intensity of multiple markers can be objectively judged, and the abundance and distribution changes of various cell subpopulations can be quantitatively analyzed through different fluorescent labels or staining intensities, which helps to more accurately study the interaction between cells and the association with disease progression.

[0083] In addition, the analysis of the relatively low registration accuracy of IHC slice fusion at the cellular level described in the background technology is as follows:

[0084] 1. Differences in registration algorithm principles:

[0085] Registration algorithm for mIHC slice fusion: mIHC slice fusion adopts a method based on inverse mapping, target interpolation and machine learning model. When calculating the affine transformation matrix, by manually selecting feature points (such as selecting characteristic points such as corners, holes, gaps, etc. on the image), the matrix parameters are calculated using the least squares method. This method can more accurately capture the characteristic relationship of the cell structure in the image, especially for the positioning of subtle features at the cellular level. For example, in the study of tumor immune microenvironment, the selection of characteristic points of cell distribution expressing different markers can more carefully reflect the spatial relationship between cells. The convolutional network architecture in the machine learning model, that is, (CNN) as a feature extractor, performs multi-level feature extraction on the image through multiple convolution blocks, which can obtain rich feature information of cells at different resolutions, and gradually refine the understanding of cell structure from macro to micro levels. The gated block module, i.e., the (MuGI Block) structure, optimizes the feature map through gated interaction, and the self-attention module, i.e., the (Self-Attention) and cross-attention modules, i.e., the (Cross-Attention) mechanism further processes the feature map, and can more accurately consider the relative position relationship between cells when generating the local deformation field. By calculating the Query, Key, and Value matrices, the relationship between each pixel and other pixels is obtained, so as to more accurately align the cell structure. For example, when processing densely populated areas, it can accurately distinguish the boundaries and relationships between different cell types, and improve the registration accuracy.

[0086] Registration algorithm for IHC slice fusion: IHC slice fusion mainly adopts a dual multi-resolution pyramid model based on the SIFT feature matching algorithm. Although preliminary registration parameters can be quickly obtained by first registering on a high-level low-resolution image, the high-level image relatively loses some detailed information at the cell level, which may result in the inability to fully and accurately correspond to the fine structure at the cell level when the registration parameters are applied to the large image with high resolution at the bottom level. For example, the tiny organelle structures of some cells or the weak connection structures between cells may not be clearly reflected in the high-level image, which in turn affects the registration accuracy. When processing cell images, the SIFT feature matching algorithm mainly matches based on the local feature points of the image, and has relatively limited understanding of the overall structure of the cell and the complex relationship between cells. Compared with the complex machine learning model in mIHC, it lacks the ability to analyze cell features at multiple levels and dimensions, and it is difficult to accurately capture the changes and relationships of the fine structures between cells. Registration deviations may occur in areas with dense cells or complex structures.

[0087] 2. Different capabilities in processing cell feature information:

[0088] Feature processing capabilities of mIHC slice fusion: During the processing of mIHC slice fusion, the machine learning model can make full use of the channel information in the image. For example, each layer of the fluorescent stained slice image contains multiple channels, and different channels can correspond to the staining of different cell markers. The model can integrate these channel information and comprehensively analyze the various characteristics of the cell. When generating the deformation field, the position and morphological changes of cells expressing different markers can be comprehensively considered, so that the registration process is more in line with the actual structure and functional state of the cell. For example, when analyzing the interaction between immune cells and tumor cells in the tumor immune microenvironment, more accurate registration can be achieved based on the channel information of multiple immune cell markers and tumor cell markers at the same time. The Attention mechanism in the model can focus on the key areas and features in the cell image and enhance the focus on important cell structures and relationships. For example, when distinguishing tumor cells from surrounding immune cells, key areas such as the contact area between immune cells and tumor cells can be highlighted, so that these important structures can be more accurately aligned during the registration process, improving the accuracy of cell-level registration.

[0089] Feature processing capabilities of IHC slice fusion: When using the SIFT feature matching algorithm, IHC slice fusion mainly focuses on the local feature points of the image, and its ability to comprehensively utilize the overall characteristics of the cell and multi-channel information is relatively weak. It may not be able to fully explore the complex cell states and relationships reflected by the staining of different cell markers. In the registration process, it lacks consideration of the comprehensive characteristics of the cell and is prone to ignoring the differences in subtle structures between cells, resulting in limited registration accuracy at the cellular level. Due to the lack of complex model structures like those in mIHC to integrate and optimize feature information, IHC slice fusion is insufficient in processing complex structural relationships between cells. For example, when faced with various types of interactions between cells (such as immune synapse formation and structural relationships related to intercellular signaling), it is impossible to accurately capture and utilize this information to optimize registration, making it difficult to meet the needs of accurate analysis of subtle structures and relationships between cells.

[0090] 3. Differences in the ability to cope with complex cell structures and overlapping areas:

[0091] Strategies for mIHC slice fusion: When dealing with complex cell structures and overlapping areas, mIHC slice fusion works together through multiple structures in the machine learning model. For example, after generating the local deformation field, the weighted average method is used to fuse the overlapping areas, and the weight is calculated based on the distance from the pixel to the center of the overlapping area. This method can handle the transition of cell structures in overlapping areas more smoothly and avoid distortion or dislocation of cell structures due to improper registration. When faced with areas with complex cell structures, such as tumor tissue and surrounding immune cell infiltration areas, the model can accurately distinguish the boundaries and relationships between different cell types through multi-level feature extraction and Attention mechanisms to ensure registration accuracy.

[0092] Strategies for IHC slice fusion: Although IHC slice fusion has a certain registration effect based on the dual multi-resolution pyramid model when processing overlapping areas, its processing method may not be fine enough. In complex cell structure areas, such as when there are multiple cell types that are highly mixed and the cell morphology is irregular, the SIFT feature matching algorithm may not be able to fully adapt to the diversity and complexity of the cell structure, resulting in deviations in the registration at the cell level, and unable to accurately restore the true structural relationship between cells, thus affecting the precise analysis of the subtle structure and relationship between cells.

[0093] In another embodiment, a preset fluorescent staining image is processed based on a preset verification rule to generate a target fluorescent staining image, including:

[0094] Obtain a preset reference image, where the preset reference image is used to characterize the image information of other fluorescent staining images. In the study of lung adenocarcinoma samples, a processed fluorescent staining slice image with good quality that can represent the overall characteristics may be selected as a preset reference image. For example, the L image that has been aligned and preliminarily verified is selected as the preset reference image, which can characterize the image information of other fluorescent staining images (such as M image and N image).

[0095] Processing the preset fluorescent staining image and the preset reference image to generate a correlation coefficient value and a structural similarity index, wherein the correlation coefficient value is used to characterize the linear relationship between the preset fluorescent staining and the preset reference image, and the structural similarity index is used to characterize the index of similarity between the preset fluorescent staining and the preset reference image in terms of structure, brightness, contrast, etc.;

[0096] The preset fluorescent staining image (assuming it is the image after the fusion of the three images L, M, and N) and the preset reference image (L image) are processed. The correlation coefficient is used to measure the linear relationship between two variables. For images, the correlation coefficient can be used to measure the similarity of pixel values ​​between the aligned image and the reference image. The range of the correlation coefficient is between [-1, 1]. The closer the value is to 1, the stronger the linear relationship between the images. Assume that the gray value of the fused image at a certain pixel position is , the gray value of the reference image at the corresponding position is , through the formula Calculate the correlation coefficient value.

[0097] Structural similarity index calculation: The structural similarity index is an indicator to measure the similarity of two images in terms of structure, brightness, contrast, etc. Assume that the average value of the fused image is , the variance is The average value of the reference image is , the variance is , the covariance is xy, through the formula Calculate the structural similarity index.

[0098] Sample set similarity calculation: The sample set similarity is used to characterize the similarity between the preset fluorescent staining image and the initial biological slice information. Assume that the gray value of the ideal image corresponding to the initial biological slice information at a certain pixel position is , the gray value of the fluorescent staining image at the corresponding position is preset to , the similarity value of the sample set is calculated by similarity measurement method (such as Dice coefficient method, which can be used for binary images, where the image needs to be properly processed and converted into binary image form).

[0099] Mean square error calculation: The mean square error is used to characterize the average square value of the pixel value difference between the preset fluorescent staining image and the initial biological slice information. Calculate the mean square content error difference.

[0100] Peak signal-to-noise ratio calculation: Peak signal-to-noise ratio is used to characterize the image quality of a preset stained fluorescent image. Assuming that the maximum pixel value of the image is L and the mean square error is MSE, the formula Calculate the peak signal-to-noise ratio.

[0101] The method includes obtaining a calculation formula for the correlation coefficient value, and the calculation formula is:

[0102]

[0103] in, Represents the preset fluorescent staining images and preset reference images The gray value at pixel i, is an image The average gray value of the molecular part, n is the total number of pixels in the image; Represents the covariance of the image grayscale values, and the denominator represents the product of the standard deviations of the two images;

[0104] The method includes obtaining a calculation formula for a structural similarity index, the calculation formula being:

[0105]

[0106] in, and are the average values ​​of the preset fluorescent staining image and the preset reference image, respectively, indicating the brightness information of the image; and Preset Fluorescence The variance between the light-stained image and the preset reference image represents the contrast information of the image; is the covariance between the preset fluorescent staining image and the preset reference image, which represents the structural information of the image; and Represents a constant used to stabilize the denominator and numerator.

[0107] Processing the preset fluorescent staining image to generate a sample set similarity value, a mean square error value, and a peak signal-to-noise ratio, wherein the sample set similarity value is used to characterize the similarity between the preset fluorescent staining image and the initial biological slice information, the mean square error value is used to characterize the average square value of the pixel value difference between the preset fluorescent staining image and the initial biological slice information, and the peak signal-to-noise ratio is used to characterize the image quality of the preset staining fluorescent image;

[0108] The correlation coefficient value, structural similarity index, sample set similarity value, mean square error value and peak signal-to-noise ratio are processed to generate image alignment index results;

[0109] The method includes obtaining a calculation formula for the similarity value of a sample set, and the calculation formula is:

[0110]

[0111] Among them, A and B are pixel sets of the preset fluorescent staining image and the preset reference image, are the total number of pixels with a value of 1 in the preset fluorescent staining image and the preset reference image (i.e., the number of “white” pixels in the two images), is the number of pixels in the intersection of the two images where the pixels are 1;

[0112] The method includes obtaining a calculation formula for a mean square error value, and the calculation formula is:

[0113]

[0114] Where n is the total number of pixels in the image, Represents the preset fluorescent staining images and preset reference images The gray value at pixel i, Represents the sum of square errors of all pixels;

[0115] The method includes obtaining a calculation formula for a peak signal-to-noise ratio, which is:

[0116]

[0117] Among them, L is the maximum pixel value of the image, MSE is the mean square error value, and L represents the maximum value of the signal. Represents the average error of the image; a target fluorescent staining image is generated based on the image alignment index results.

[0118] By combining multiple evaluation indicators, the quality of image alignment can be more comprehensively evaluated to ensure that the aligned images are highly consistent in terms of structure, brightness, contrast, etc., thereby improving the accuracy of subsequent analysis. Accurate image alignment indicator results can reduce misjudgments caused by alignment errors and improve the reliability of subsequent steps such as cell clustering and cell subtype analysis. Through the feedback of alignment indicator results, parameters in the image processing process can be adjusted and optimized, such as the selection of feature points and the calculation of affine transformation matrices, to obtain better alignment effects. Combining multiple evaluation indicators can comprehensively evaluate image quality, including linear relationships, structural similarity, pixel value differences, image quality, etc., to provide more comprehensive guidance for image processing. Significantly improve the accuracy, reliability and efficiency of multiple immunohistochemical fluorescence staining image analysis.

[0119] S106, processing the target fluorescent staining image to generate initial region information.

[0120] In one embodiment, the initial region information is used to characterize the region where the tumor and / or immune cells are enriched. Taking the target fluorescent staining image obtained from the study of lung adenocarcinoma samples as an example, the image has already contained the staining information of multiple cell markers after a series of previous processing steps (including section staining, image alignment, fusion and verification, etc.). The target fluorescent staining image is analyzed using relevant tools (such as Qupath (v0.4.4)). According to the actual situation of the sample and the requirements of the research design, a certain number of 1mm² regions (such as 5-10 regions of tumor enrichment or immune cell enrichment are often selected in oncology and immunology research) are manually selected from the appropriate position of the sample image as the initial region information. For example, in the image, the tumor enrichment region can be identified by observing the staining intensity and distribution of cells. Tumor cells may have specific staining reactions to certain markers and show strong fluorescence signals. Immune cell enrichment regions may surround tumor cells or be near certain specific tissue structures, and their staining of related immune cell markers is also more obvious. By carefully observing and manually selecting these regions, initial region information for characterizing tumor and / or immune cell enrichment can be generated.

[0121] S107, processing the initial region information to generate multiple images of different fluorescence channels.

[0122] In one embodiment, images of multiple different fluorescence channels are single fluorescence channel images. Take the tumor-enriched area selected from the target fluorescence staining image of a lung adenocarcinoma sample as the initial region information as an example. This region contains staining information of multiple cell markers and is a comprehensive region information. Since the image corresponding to the initial region information is a comprehensive image containing information of multiple fluorescence channels. For each fluorescence channel, we can separate it and generate a single fluorescence channel image. For example, if the image corresponding to the initial region information has 8 fluorescence channels (channel 1 may be DAPI staining for displaying the cell nucleus, channel 2 may be a certain cell surface marker staining, etc.), we can use image processing software or algorithms to extract the information of each channel separately to obtain 8 single fluorescence channel images.

[0123] Image processing software is a tool specifically used to perform various operations and analyses on images. It usually has a graphical user interface (GUI) that allows users to operate intuitively. For example, in the above process of extracting a single fluorescent channel image from a comprehensive image, the software may provide menu options or tool buttons to implement the channel separation function. Common software examples: Adobe Photoshop: This is a widely used image processing software. It has powerful image editing functions, including color adjustment, image synthesis, channel operations, etc. When extracting a single fluorescent channel image, you can use its channel panel to obtain a single channel image by selecting a specific channel and copying or saving it. ImageJ: This is a free and open source image processing software that is widely used in the field of biomedical image processing. It provides a large number of image processing tools and plug-ins. For extracting a single fluorescent channel image, you can use its built-in channel separation function, which can be achieved through simple operation steps. Image processing algorithms are a series of mathematical methods and calculation steps used to process image data. These algorithms can achieve image enhancement, segmentation, feature extraction, and other functions. In the above scenario, the algorithm is used to accurately separate the information of a single channel from an image containing multiple channel information.

[0124] Take the channel separation algorithm based on matrix operation as an example: image data is usually stored in the form of a matrix. For an image with multiple channels, the data of each channel can be regarded as a two-dimensional matrix. By performing specific operations on the image matrix, the matrix of each channel can be extracted to obtain a single fluorescence channel image. For example, if the image is an RGB image (with three channels: red, green, and blue), three separate two-dimensional matrices corresponding to the images of the three channels of red, green, and blue can be obtained by slicing the entire image matrix according to the channel dimension. Take the channel separation algorithm based on filtering as an example: this algorithm uses filtering technology to separate channels. For example, a filter can be designed that only allows the frequency components of a specific channel to pass through and suppresses the frequency components of other channels. By applying this filter to the image, a single fluorescence channel image can be obtained. This algorithm may have a better effect when processing fluorescence channel images with complex frequency characteristics. This application does not limit the specific image processing software or algorithm, and it can be selected according to actual needs. Each single fluorescence channel image only displays the staining of the marker corresponding to the channel, which is more convenient for analyzing the expression of a specific marker in the area.

[0125] S108, processing the images of multiple different fluorescence channels to generate cell clustering information.

[0126] In one embodiment, images of multiple different fluorescence channels are corrected to generate target region information, wherein the target region information is used to characterize the cell expression data corresponding to the initial region information. Taking 8 different fluorescence channel images obtained from lung adenocarcinoma samples as an example, each channel image corresponds to a different cell marker staining condition. For example, channel 1 may be DAPI stained to display the cell nucleus, channel 2 may be stained with a certain immune cell surface marker, etc. Images of these different fluorescence channels are corrected using tools such as R (v4.2). The data with the highest 1% and the lowest 1% of each marker expression are corrected, and the sample information of each initial region information (such as the selected tumor enrichment region) is spliced ​​with the cell segmentation information to obtain the target region information, which can characterize the cell expression data corresponding to the initial region information.

[0127] The target region information is processed to generate cell subtype information. CELESTA is used to predict the positive probability of each marker expression to obtain the positive probability value of each cell at different protein expression levels. These probability values ​​are used as the characteristics of each cell, and cells with similar characteristics are merged to divide the entire sample into different cell subpopulations to generate cell subtype information. For example, a Cytotoxic T cell subpopulation expressing CD8a, a Macrophage II subpopulation expressing CD68 and CD206, etc. may be divided. The following is an example of cell subpopulations: From the study of lung adenocarcinoma samples, the target region information (including cell expression data corresponding to multiple different fluorescent channel images after correction processing) is analyzed. CD8a - Cytotoxic T cell subpopulation division, CELESTA is used to process the cell expression data to predict the positive probability of each marker expression. For the CD8a marker, when the positive probability of CD8a expression of a cell reaches a certain threshold (for example, 0.5), the cell is divided into the Cytotoxic T cell subpopulation expressing CD8a. These cells may show specific distribution characteristics in the image, such as being relatively concentrated around tumor tissue, which may be related to the immune response of tumor cells. CD68 and CD206 Macrophage II subpopulation division: CELESTA is also used to predict the positive probability of the expression of CD68 and CD206 markers. When a cell satisfies the corresponding thresholds for the positive probabilities of CD68 and CD206 expression (for example, 0.4 and 0.4, respectively), the cell is divided into the Macrophage II subpopulation expressing CD68 and CD206. In the image, these cells may be distributed inside or around the tumor tissue, participate in the regulation of the tumor immune microenvironment, and may have the function of phagocytizing tumor cell fragments or secreting related immune regulatory factors.

[0128] By dividing these cell subsets, we can better understand the distribution and functional status of different cell types in the tumor immune microenvironment. For example, understanding the number and activity of the Cytotoxic T cell subset can evaluate the body's immune killing ability against tumor cells; understanding the characteristics of the Macrophage II subset can explore whether its role in tumor immunity is to promote tumor growth (such as by secreting certain factors that promote tumor cell proliferation) or inhibit tumor growth (such as by phagocytizing tumor cells).

[0129] The cell subtype information is processed to generate cell clustering information. Based on the cell subtype information, a clustering algorithm based on machine learning (such as a large-scale data clustering algorithm based on graph theory (PhenoGraph)) is used to perform cluster analysis on the sample data, and the entire sample is divided into different cell subtype combinations. These combination information is the cell clustering information. For example, cluster analysis may be used to obtain the distribution of different cell subtypes in the tumor immune microenvironment, thereby characterizing the immune status of the tumor immune microenvironment. For example, it is found that there are more immunosuppressive cell subpopulations clustered around the tumor, or the distribution characteristics of a certain immune activation cell subpopulation in a specific area.

[0130] Suppose we have obtained the following information about cell subtypes from a study of lung adenocarcinoma samples: Cytotoxic T cell subset expressing CD8a, with a cell number of , each cell has an associated feature vector (e.g., a positive probability value predicted based on the expression of different markers, etc.). Macrophage II subpopulation expressing CD68 and CD206, with a cell number of . Immune checkpoint cell subsets expressing PD-1, cell number wait.

[0131] For each cell subtype, calculate the similarity between cells. For example, for two Cytotoxic T cells, calculate the similarity measure (such as the inverse of the Euclidean distance, etc.) based on their feature vectors (such as marker expression probability, etc.). Consider cells of all cell subtypes as nodes of the graph, and use the similarity between cells as the weight of the edge to construct a similarity graph. Use the PhenoGraph algorithm to perform community discovery on the constructed similarity graph. The algorithm divides the nodes into different communities, that is, different cell clusters, based on the weight of the edge. For example, it may be found that some Cytotoxic T cells and some Macrophage II cells are divided into the same cluster because of their interactions (high similarity), although they belong to different cell subtypes.

[0132] like Figure 6As shown in the figure, after cluster analysis, different cell clusters are obtained. For example, cluster 1 may contain some Cytotoxic T cells, some Macrophage II cells, and a small number of immune checkpoint cells expressing PD-1; cluster 2 may be mainly composed of other Cytotoxic T cells and some other cell subtypes. These clustering information is the cell clustering information, which reflects the combination of different cell subtypes in the tumor immune microenvironment. The immune status of the tumor immune microenvironment is analyzed from the cell clustering information. If more clusters containing immunosuppressive cell subpopulations are found around the tumor (such as more cells expressing PD-1 and clustered with other immunosuppressive-related cells), it indicates that the tumor immune microenvironment may be in an immunosuppressive state, and tumor cells may be more likely to escape immune attacks. On the contrary, if a region is found to have more clusters containing immune-activated cell subpopulations (such as Cytotoxic T cells and related activated auxiliary cells clustered together), it means that the region may have strong immune activity and may have a strong ability to kill tumor cells.

[0133] This application performs multiple fluorescent immunohistochemistry staining on multiple tissue sections. Each section is marked with specific cellular components related to tumor immunity. Through a series of steps such as layer extraction, manual feature point selection, calculation and application of affine transformation matrix, the alignment and fusion of multiple section images are achieved, and more immune markers are indirectly detected. It can reflect the spatial distribution and interactions of more types of cells in the tumor immune microenvironment, and more comprehensively characterize the characteristics of the tumor immune microenvironment, which can help researchers systematically, deeply and accurately understand the information of the tumor immune microenvironment.

[0134] Based on the mature mIHC technology, several consecutive tissue slices are marked with different markers. After obtaining the corresponding several fluorescent images, a mIHC image fusion algorithm model is constructed using machine learning. Through layer extraction, manual feature point selection, calculation of affine transformation matrix and application of a series of steps, the alignment and fusion of multiple slice images are realized. After cell segmentation and clustering, the several markers marked by mIHC can be expanded several times to dozens of markers, thereby allowing a more comprehensive study of the cell composition, functional state and cell-cell interaction of TIME. Compared with the mIHC of a single slice, the present invention constructs a fusion algorithm to fuse the mIHC fluorescent images marked by different markers from several slices into one image, integrates a variety of cell components and non-cellular component information contained in each slice, and can analyze their spatial distribution and interaction, which is conducive to researchers to conduct a more comprehensive study of the cell composition, functional state and cell-cell interaction of TIME. It can better help the clinic diagnose, treat and predict the prognosis of tumors, and also has the advantages of high sensitivity and low cost.

[0135] Fluorescently stained slice images generated based on multiple fluorescent immunohistochemistry (mIHC) technology is a technology that can simultaneously label multiple different antibodies (or antigens) on the same tissue slice. This technology allows researchers to observe the expression and distribution of multiple proteins in the same tissue environment, thereby gaining a more comprehensive understanding of biological processes or pathological states. Suppose we have a biological sample from a patient with lung adenocarcinoma, and we want to study the spatial relationship between different immune cell subsets (such as CD8-positive cytotoxic T cells, CD68-positive macrophages) and tumor cells (whose proliferation state may be identified by Ki-67 labeling) in the tumor microenvironment.

[0136] First, several thin slices (e.g., 4-micron thick slices) are cut from the patient's tumor sample and stained with specific antibodies. For example, we might use the following combination of antibodies: CD8 (to identify cytotoxic T cells), CD68 (to identify macrophages), Ki-67 (to identify proliferating cells), DAPI (for nuclear staining to observe cell nuclei). The stained slices are scanned using a fluorescence microscope to obtain images containing multiple fluorescent channels. Each channel corresponds to a specific fluorescent marker that can reveal the distribution of a specific antibody.

[0137] Since slices may be spatially misaligned due to deformation during processing or inconsistent slice thickness, a machine learning model is used to align the images to ensure that images from different channels accurately correspond to the same tissue area. The aligned multiple channel images are fused to generate a comprehensive image that contains information on all markers. In this way, we can intuitively see the spatial distribution of different cell types and tumor cells on the same image. The machine learning algorithm is used to perform cell clustering analysis on the fused images, identify different cell subpopulations, and analyze their distribution and interactions in the tumor microenvironment.

[0138] Through this multiplex fluorescent staining and image analysis technology, we found that: CD8-positive cytotoxic T cells are mainly concentrated at the edge of the tumor and may be involved in immune surveillance of tumor cells. CD68-positive macrophages are distributed inside and around the tumor and may be involved in the inflammatory response of the tumor microenvironment. The high expression area of ​​Ki-67 indicates the active proliferation area of ​​tumor cells. This helps researchers to have a deeper understanding of the complexity of the tumor immune microenvironment.

[0139] In one embodiment, if Figure 2 As shown, the present application also provides an analysis device for multiple immunohistochemical fluorescence staining images, comprising:

[0140] An acquisition module 201 is used to acquire original biological sample data;

[0141] The processing module 202 is used to slice the original biological sample data to generate a number of initial biological slice information, wherein the initial biological slice information includes a target marker, and the target marker includes at least one of lymphocytes, myeloid cells, and interstitial cells; process the initial biological slice information to generate a fluorescent stained slice image, wherein the fluorescent stained slice image is an image of multiple fluorescent channels, and the fluorescent stained slice image is generated based on multiple fluorescent immunohistochemistry technology; process the fluorescent stained slice image to generate an aligned fluorescent stained slice image; perform fusion processing on the aligned fluorescent stained slice image to generate a target fluorescent stained image; process the target fluorescent stained image to generate initial region information, wherein the initial region information is used to characterize the region where tumors and / or immune cells are enriched; process the initial region information to generate multiple images of different fluorescent channels, wherein the images of multiple different fluorescent channels are single fluorescent channel images; process the images of multiple different fluorescent channels to generate cell clustering information, wherein the cell clustering information is used to characterize the immune status of the tumor immune microenvironment.

[0142] Each embodiment in the present application is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the analysis method, electronic device, electronic device, and readable storage medium embodiment for evaluating multiple immunohistofluorescence staining images, since they are basically similar to the analysis method embodiment of multiple immunohistofluorescence staining images described above, the description is relatively simple, and the relevant parts can be referred to the partial description of the analysis method embodiment of multiple immunohistofluorescence staining images described above.

Claims

1. A method for analyzing multiple immunohistochemical fluorescence staining images, characterized in that: include: Obtaining raw biological sample data; Slice the original biological sample data to generate a number of initial biological slice information, wherein the initial biological slice information includes a target marker, and the target marker includes at least one of a lymphocyte, a myeloid cell, and an interstitial cell; Processing the initial biological slice information to generate a fluorescently stained slice image, wherein the fluorescently stained slice image is an image of multiple fluorescent channels and is generated based on a multiplex fluorescent immunohistochemistry technique; Processing the fluorescent staining slice image to generate an aligned fluorescent staining slice image; Performing fusion processing on the aligned fluorescent staining slice images to generate a target fluorescent staining image; Processing the target fluorescent staining image to generate initial region information, wherein the initial region information is used to characterize regions where tumors and / or immune cells are enriched; Processing the initial region information to generate multiple images of different fluorescence channels, wherein the multiple images of different fluorescence channels are single fluorescence channel images; Processing images of multiple different fluorescence channels to generate cell clustering information, wherein the cell clustering information is used to characterize the immune status of the tumor immune microenvironment; Processing the fluorescently stained slice image to generate an aligned fluorescently stained slice image, including: obtaining preset matching feature points; performing layer extraction processing on the fluorescently stained slice image to generate a grayscale image, wherein the grayscale image is used to align fluorescently stained slice images of different categories; processing the fluorescently stained slice image based on the preset matching feature points and the grayscale image to generate an affine transformation matrix; processing the affine transformation matrix to generate an inverse mapping and a target interpolation; processing the fluorescently stained slice image based on the inverse mapping and the target interpolation to generate an initial fluorescently stained slice image, wherein the initial fluorescently stained slice image is a preliminary aligned fluorescently stained slice image, and the initial fluorescently stained slice includes at least any one marker for characterizing lymphocytes, myeloid cells, interstitial cells or extracellular matrix; processing the initial fluorescently stained slice image based on a machine learning model to generate an aligned fluorescently stained slice image; The method includes obtaining a calculation formula for an affine transformation matrix, wherein the calculation formula is: ; ; ; ; ; Where T is the affine transformation matrix; a and d are scaling coefficients for horizontal and vertical scaling; b and c are shear coefficients for controlling the tilt of the image; and is the translation amount; and is the coordinate point in the fluorescent slice image; and is the coordinate point in the grayscale image; is the parameter vector to be solved; is the transpose of the matrix M; yes The inverse matrix of The method includes obtaining a calculation formula for inverse mapping, and the calculation formula is: ; ; Where x and y represent the coordinate points in the original image; and Represents the coordinate point in the target image; These parameters are the coefficients associated with the affine transformation matrix; and is the translation parameter; The method includes obtaining a calculation formula for target interpolation, and the calculation formula is: ;in, ; Respectively indicate distance horizontal and vertical distances.

2. The method according to claim 1, characterized in that The initial fluorescent staining slice image is processed based on the machine learning model to generate an aligned fluorescent staining slice image, including: Processing the initial fluorescent stained slice image based on a machine learning model to generate feature maps of different levels, wherein the feature maps of the same level include feature information of different channel dimensions, and the initial fluorescent stained slice image includes a reference image and a target image; Based on the machine learning model, feature maps of different levels are processed respectively to generate several feature maps with the same channel dimension; Processing the feature maps of the same channel dimension respectively to generate a local deformation field and a target loss value, wherein the target loss value includes a loss value and a smoothing loss value for measuring the structural similarity of two fluorescent stained slice images; Processing the local deformation field and the target loss value to generate a target deformation field; Processing the initial fluorescent staining slice image based on the target deformation field to generate an aligned fluorescent staining slice image; The machine learning model includes a calculation formula for obtaining a target loss value, and the calculation formula is: ； ； in, is the displacement vector of the deformation field at position P, and Identify the gradients of the deformation field in the x and y directions, respectively, is the weight parameter used to smooth the loss, is the image transformed by the deformation field, is the reference image.

3. The method according to claim 1, characterized in that The aligned fluorescent staining slice images are fused to generate a target fluorescent staining image, including: Obtain channel information of aligned fluorescent staining slice images; Based on the channel information of the differently aligned fluorescent staining slice images, sequential stacking processing is performed to generate a preset fluorescent staining image; The preset fluorescent staining image is processed based on preset verification rules to generate a target fluorescent staining image.

4. The method according to claim 3, characterized in that The preset fluorescent staining image is processed based on a preset verification rule to generate a target fluorescent staining image, including: Acquire a preset reference image, wherein the preset reference image is used to represent image information of other fluorescent staining images; Processing the preset fluorescent staining image and the preset reference image to generate a correlation coefficient value and a structural similarity index, wherein the correlation coefficient value is used to characterize the linear relationship between the preset fluorescent staining and the preset reference image, and the structural similarity index is used to characterize an index of similarity between the preset fluorescent staining and the preset reference image in terms of structure, brightness, and contrast; Processing the preset fluorescent staining image to generate a sample set similarity value, a mean square error value, and a peak signal-to-noise ratio, wherein the sample set similarity value is used to characterize the similarity between the preset fluorescent staining image and the initial biological slice information, the mean square error value is used to characterize the average square value of the pixel value difference between the preset fluorescent staining image and the initial biological slice information, and the peak signal-to-noise ratio is used to characterize the image quality of the preset staining fluorescent image; Processing the correlation coefficient value, structural similarity index, sample set similarity value, mean square error value and peak signal-to-noise ratio to generate an image alignment index result; A target fluorescent staining image is generated based on the image alignment index result.

5. The method according to claim 4, characterized in that The preset fluorescent staining image is processed based on the preset verification rule to generate a target fluorescent staining image, and further includes: The method includes obtaining a calculation formula for a correlation coefficient value, and the calculation formula is: ； in, and Represents the preset fluorescent staining images and preset reference images Gray value at pixel i; and is an image and The average gray value of the molecular part represents the covariance of the image grayscale values, and the denominator represents the product of the standard deviations of the two images; n is the total number of pixels in the image; The method includes obtaining a calculation formula for a sample set similarity value, and the calculation formula is: ； Among them, A and B are pixel sets of the preset fluorescent staining image and the preset reference image, and are the total number of pixels with a value of 1 in the preset fluorescent staining image and the preset reference image, is the number of pixels in the intersection of the two images where the pixels are 1; The method includes obtaining a calculation formula for a mean square error value, and the calculation formula is: ； Where n is the total number of pixels in the image, and Represents the preset fluorescent staining images and preset reference images The gray value at pixel i, Represents the sum of square errors of all pixels; The method includes obtaining a calculation formula for the peak signal-to-noise ratio, which is: ； Among them, L is the maximum pixel value of the image, MSE is the mean square error value, and L represents the maximum value of the signal. Represents the average error of the image; The method includes obtaining a calculation formula for the structural similarity index, and the calculation formula is: ； in, and are the average values ​​of the preset fluorescent staining image and the preset reference image, respectively, indicating the brightness information of the image; and are the variances of the preset fluorescent staining image and the preset reference image, respectively, indicating the contrast information of the image; is the covariance between the preset fluorescent staining image and the preset reference image, which represents the structural information of the image; and Represents a constant used to stabilize the denominator and numerator.

6. The method according to claim 1, characterized in that Process images of multiple different fluorescence channels to generate cell clustering information, including: Performing correction processing on images of multiple different fluorescence channels to generate target region information, wherein the target region information is used to characterize cell expression data corresponding to the initial region information; Processing the target region information to generate cell subtype information; The cell subtype information is processed to generate cell clustering information.

7. An analysis device for multiple immunohistochemical fluorescence staining images, characterized in that: The device comprises: An acquisition module, used to acquire raw biological sample data; a processing module, configured to slice the original biological sample data to generate a number of initial biological slice information, wherein the initial biological slice information includes a target marker, and the target marker includes at least one of lymphocytes, myeloid cells, and interstitial cells; process the initial biological slice information to generate a fluorescently stained slice image, wherein the fluorescently stained slice image is an image of multiple fluorescent channels, and the fluorescently stained slice image is generated based on multiple fluorescent immunohistochemistry technology; process the fluorescently stained slice image to generate an aligned fluorescently stained slice image; perform fusion processing on the aligned fluorescently stained slice image to generate a target fluorescently stained image; process the target fluorescently stained image to generate initial region information, wherein the initial region information is used to characterize regions where tumors and / or immune cells are enriched; process the initial region information to generate images of multiple different fluorescent channels, wherein the images of multiple different fluorescent channels are single fluorescent channel images; process the images of multiple different fluorescent channels to generate cell clustering information, wherein the cell clustering information is used to characterize the immune status of the tumor immune microenvironment; Processing the fluorescently stained slice image to generate an aligned fluorescently stained slice image, including: obtaining preset matching feature points; performing layer extraction processing on the fluorescently stained slice image to generate a grayscale image, wherein the grayscale image is used to align fluorescently stained slice images of different categories; processing the fluorescently stained slice image based on the preset matching feature points and the grayscale image to generate an affine transformation matrix; processing the affine transformation matrix to generate an inverse mapping and a target interpolation; processing the fluorescently stained slice image based on the inverse mapping and the target interpolation to generate an initial fluorescently stained slice image, wherein the initial fluorescently stained slice image is a preliminary aligned fluorescently stained slice image, and the initial fluorescently stained slice includes at least any one marker for characterizing lymphocytes, myeloid cells, interstitial cells or extracellular matrix; processing the initial fluorescently stained slice image based on a machine learning model to generate an aligned fluorescently stained slice image; The device includes a calculation formula for obtaining an affine transformation matrix, and the calculation formula is: ； ； ; ; ; Where T is the affine transformation matrix; a and d are scaling coefficients for horizontal and vertical scaling; b and c are shear coefficients for controlling the tilt of the image; and is the translation amount; and is the coordinate point in the fluorescent slice image; and is the coordinate point in the grayscale image; is the parameter vector to be solved; is the transpose of the matrix M; yes The inverse matrix of The device includes a calculation formula for obtaining an inverse mapping, and the calculation formula is: ; ; Where x and y represent the coordinate points in the original image; and Represents the coordinate point in the target image; These parameters are the coefficients associated with the affine transformation matrix; and is the translation parameter; The device includes a calculation formula for obtaining a target interpolation value, and the calculation formula is: ;in, , Respectively indicate distance horizontal and vertical distances.

8. An electronic device, characterized in that: include: a first processor; and a memory for storing executable instructions of the first processor; Wherein, the first processor is configured to execute the method for analyzing multiple immunohistochemical fluorescence staining images according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the second processor, the method for analyzing multiple immunohistochemical fluorescence staining images according to any one of claims 1 to 6 is implemented.

Images