Immunofluorescent staining image processing method and related device

Abstract

The invention discloses an immunofluorescent staining image processing method and a related device, and relates to the technical field of medical image processing and analysis. Comprising the following steps: acquiring a plurality of dyeing image sets corresponding to target biological tissues; each dyeing image set comprises a reference image representing the spatial distribution of the same cell in the target biological tissue; taking one reference image as a fixed image, and taking each dyeing image set which does not contain the fixed image as a variable dyeing image set; constructing an affine matrix between the reference image and the fixed image in the variable dyeing image set by using the first prediction model; and transforming each image in the variable dyeing image set to a coordinate system where the fixed image is located by using the affine matrix to obtain a first target image. Therefore, the same tissue unit composed of a plurality of cells in the sample biological tissue can be close to the same tissue unit in different dyeing images as much as possible, namely, the space alignment of the images in different dyeing image sets is realized.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing and analysis, and in particular to a method and related device for processing immunofluorescence staining images. Background Art

[0002] Immunofluorescence staining is an intuitive and reliable biological tissue imaging technology that can mark and observe the expression of specific proteins in biological tissues, providing a basis for molecular diagnosis and personalized treatment in the medical field.

[0003] After obtaining the sample biological tissue, the sample biological tissue can be stained multiple times by immunofluorescence staining technology to obtain multiple staining images. However, in the process of decolorizing and re-staining the sample biological tissue, the sample biological tissue will have tissue displacement and deformation caused by elution and re-staining, resulting in different positions of the same biological tissue in the staining images obtained after different rounds of staining, which makes it difficult to analyze and compare the staining images.

[0004] How to make the position of the same biological tissue unit in the sample biological tissue as similar as possible in different staining images has become a technical problem that needs to be solved urgently in the field of medical image processing and analysis technology. Summary of the invention

[0005] Based on the above problems, the present application provides a method for processing immunofluorescence staining images, so as to make the positions of the same tissue unit composed of multiple cells in the sample biological tissue as close as possible in different staining images.

[0006] The first aspect of the present application provides a method for processing an immunofluorescence staining image, comprising:

[0007] Acquire a plurality of staining image sets corresponding to the target biological tissue; each image in each of the staining image sets represents the spatial distribution of different types of cells in the target biological tissue; each of the staining image sets includes a reference image representing the spatial distribution of the same type of cells;

[0008] any one of the reference images in the plurality of dyed image sets is used as a fixed image; the dyed image set not including the fixed image is a variable dyed image set;

[0009] For each of the variable staining image sets, a reference image in the variable staining image set and the fixed image are input into a first prediction model to obtain an affine matrix; the affine matrix indicates spatial displacement information between a tissue in the fixed image and a corresponding tissue in the reference image; the first prediction model is a pre-trained model for generating an affine matrix between two images; the tissue includes a plurality of cells;

[0010] For each of the variable dyed image sets, based on the affine matrix, each image in the variable dyed image set is transformed to the coordinate system where the fixed image is located, so as to obtain a first target image set.

[0011] In an optional implementation, the method for processing the immunofluorescence staining image further includes:

[0012] Acquire a target reference image in each of the first target image sets; the target reference image is an image obtained by performing an affine transformation on a reference image in the variable dye image set based on an affine matrix;

[0013] For each of the first target image sets, a target reference image and the fixed image in the first target image set are input into a second prediction model to obtain a target deformation field; a vector in the target deformation field indicates spatial deformation information of a cell in the target reference image and a corresponding cell in the fixed image; the second prediction model is a pre-trained model for generating a deformation field between two images;

[0014] For each of the first target image sets, the position of cells in each image in the first target image set is adjusted based on the target deformation field to obtain a second target image set.

[0015] In an optional implementation, the first prediction model includes N+2 convolution modules and N downsampling convolution modules, where N is an integer greater than or equal to 4; for each of the variable stained image sets, the reference image and the fixed image in the variable stained image set are input into the first prediction model to obtain an affine matrix, including:

[0016] Generate a first feature vector according to the reference image and the fixed image in the variable stained image set by using the first two convolution modules and the N downsampling convolution modules in the first prediction model;

[0017] Generate a second feature vector according to the reference image and the fixed image in the variable stained image set by using the first three convolution modules and the N downsampling convolution modules in the first prediction model;

[0018] Similarly, the N+2 convolution modules and the N downsampling convolution modules in the first prediction model are used to generate an N+1th feature vector according to the reference image and the fixed image in the variable stained image set;

[0019] The affine matrix is ​​generated through the first eigenvector, the second eigenvector, and up to the N+1th eigenvector.

[0020] In an optional implementation, for each of the first target image sets, inputting the target reference image and the fixed image in the first target image set into a second prediction model to obtain a target deformation field includes:

[0021] Generate, by using a plurality of convolution modules in the second prediction model, M feature vectors corresponding to the target reference image in the first target image set; and generate, by using the fixed image, M feature vectors corresponding to the fixed image; wherein M is an integer greater than or equal to 4;

[0022] splicing the Mth eigenvector corresponding to the target reference image and the Mth eigenvector corresponding to the fixed image to obtain a first splicing vector; and iteratively optimizing the first splicing vector to obtain a first deformation field;

[0023] Based on the first deformation field, spatially transform the M-1th eigenvector corresponding to the target reference image, splice the result of the spatial transformation with the M-1th eigenvector corresponding to the fixed image to obtain a second splicing vector, and iteratively optimize the second splicing vector to obtain a second deformation field increment;

[0024] generating the second deformation field based on the second deformation field increment and the first deformation field;

[0025] The target deformation field is generated based on the second deformation field, the first M-2 feature vectors corresponding to the target reference image, and the first M-2 feature vectors corresponding to the fixed image.

[0026] In an optional implementation manner, after obtaining a plurality of second target image sets, the method further includes:

[0027] Input each image in each of the second target image sets into the cell segmentation model to obtain a set of identification images corresponding to each of the second target image sets; an identification image in the identification image set corresponds to an image in the second target image set; the identification image is used to distinguish between objects identified by a dye marker and objects not identified by a dye marker.

[0028] In an optional implementation, the cell segmentation model includes a feature extraction module, which extracts features of each image in each of the second target image sets; the feature extraction module includes a residual convolution module, a window attention module, a fusion sampling module and the residual convolution module connected in sequence.

[0029] In an optional implementation, after acquiring the plurality of identification images, the method further includes:

[0030] Based on each of the identified images, determining statistical information of each cell on the identified image; the statistical information includes the center of mass, volume and fluorescence intensity of the cell;

[0031] For each cell on the identification image, if the distance between the centroid of the cell and the centroid of the target cell in other identification images is less than or equal to a preset distance threshold, then obtaining the information of the target cell as the co-expression information of the cell;

[0032] Based on the statistical information of each cell and the co-expression information of each cell, the spatial distribution result of each cell in the target biological tissue is obtained.

[0033] A second aspect of the present application provides a device for processing an immunofluorescence staining image, the device comprising:

[0034] A staining image set acquisition module, used to acquire multiple staining image sets corresponding to the target biological tissue; each image in each of the staining image sets represents the spatial distribution of different types of cells in the target biological tissue; each of the staining image sets includes a reference image representing the spatial distribution of the same type of cells;

[0035] A basic data determination module, configured to use any reference image in the plurality of dyeing image sets as a fixed image; a dyeing image set not including the fixed image is a variable dyeing image set;

[0036] An affine matrix generation module is used for inputting the reference image in the variable staining image set and the fixed image into the first prediction model for each variable staining image set to obtain an affine matrix; the affine matrix indicates the spatial displacement information of the tissue in the fixed image and the corresponding tissue in the reference image; the first prediction model is a model for generating an affine matrix between two images; the tissue includes a plurality of cells;

[0037] The first target image set generating module is used to transform each image in the variable dyeing image set to the coordinate system where the fixed image is located based on the affine to obtain the first target image set.

[0038] A third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method introduced in any implementation manner of the first aspect.

[0039] A fourth aspect of the present application provides an electronic device, including:

[0040] a memory having a computer program stored thereon;

[0041] A processor is used to execute the computer program in the memory to implement the steps of the method introduced in any implementation manner of the first aspect.

[0042] Compared with the prior art, this application has the following beneficial effects:

[0043] The present application discloses a method for processing immunofluorescence staining images, which obtains multiple staining image sets corresponding to target biological tissues; each staining image set includes a reference image that characterizes the spatial distribution of the same cell in the target biological tissue; a reference image is used as a fixed image, and each staining image set that does not include the fixed image is used as a variable staining image set; an affine matrix between the reference image and the fixed image in the variable staining image set is constructed using a first prediction model; and then the affine matrix is ​​used to transform each image in the variable staining image set corresponding to the affine matrix to the coordinate system where the fixed image is located, so as to obtain a first target image.

[0044] Since the fixed image is also a reference image, the reference images all represent the spatial distribution of the same cell in the target biological tissue. In theory, the same position in different reference images corresponds to the same tissue unit of the target biological tissue. The affine matrix indicates the spatial displacement information of the tissue in the fixed image and the corresponding tissue in the reference image in the variable staining image set. By using the affine matrix to transform the spatial coordinates of the image in the variable staining image set to the coordinate system of the fixed staining image, the spatial alignment of the staining images obtained after two rounds of staining can be achieved, so that the position of the same biological tissue unit in the sample biological tissue in different staining images is as close as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 A flowchart of a method for processing an immunofluorescence staining image provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of the structure of a first prediction model provided in an embodiment of the present application;

[0048] Figure 3 A flowchart of another method for processing an immunofluorescence staining image provided in an embodiment of the present application;

[0049] Figure 4A A schematic diagram of the structure of a second prediction model provided in an embodiment of the present application;

[0050] Figure 4B A schematic diagram of the structure of each module in a second prediction model provided in an embodiment of the present application;

[0051] Figure 5A A schematic diagram of the structure of a cell segmentation model provided in an embodiment of the present application;

[0052] Figure 5B A schematic diagram of the structure of each module in a cell segmentation model provided in an embodiment of the present application;

[0053] Figure 6 A flow chart for obtaining the spatial distribution results of each cell in a biological tissue provided in an embodiment of the application;

[0054] Figure 7 A schematic diagram of the structure of an immunofluorescence staining image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] Immunofluorescence staining is an intuitive and reliable biological tissue imaging technology that can mark and observe the expression of specific proteins in biological tissues, providing a basis for molecular diagnosis and personalized treatment in the medical field.

[0056] After obtaining the sample biological tissue, the sample biological tissue can be stained multiple times by immunofluorescence staining technology to obtain multiple staining images. However, in the process of decolorizing and re-staining the sample biological tissue, the sample biological tissue will have tissue displacement and deformation caused by elution and re-staining, resulting in different positions of the same biological tissue in the staining images obtained after different rounds of staining, which makes it difficult to analyze and compare the staining images.

[0057] How to make the position of the same biological tissue unit in the sample biological tissue as similar as possible in different staining images has become a technical problem that needs to be solved urgently in the field of medical image processing and analysis technology.

[0058] The present application discloses a method for processing an immunofluorescence staining image, which is used to make the position of the same biological tissue unit in a sample biological tissue in different staining images as similar as possible.

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] Figure 1 A flowchart of a method for processing an immunofluorescence staining image provided in an embodiment of the present application. Figure 1 As shown, the method for processing immunofluorescence staining images disclosed in the present application includes:

[0061] S101, obtaining a plurality of staining image sets corresponding to target biological tissues.

[0062] The target biological tissue in the present application is a biological tissue that needs to be subjected to multiple rounds of staining treatment using immunofluorescence staining technology.

[0063] Staining the target biological tissue by immunofluorescence technology means using the high specificity of the antigen-antibody reaction to mark the antibody or antigen with a fluorescent dye (staining marker) that does not affect the activity of the antigen and antibody, thereby achieving the labeling and detection of multiple different antigens on the target biological tissue.

[0064] In one round of staining, multiple different types of cells in the target biological tissue can be marked with multiple staining markers (objects such as blood vessels and amyloid proteins can also be marked). For example, immune T cells can be marked with one staining marker, channel vascular cells can be marked with another staining marker, tumor cells can be marked with another staining marker, and so on in one round of staining.

[0065] After a round of staining of the target biological tissue, the stained target biological tissue is placed under a microscopic imaging device, and multiple staining images corresponding to the target biological tissue after this round of staining can be observed. For example, if three staining markers are used to mark the channel vascular cells, tumor cells, and immune T cells respectively during this round of staining, then under the microscopic imaging device, a staining image representing the spatial distribution of the channel vascular cells can be seen, a staining image representing the spatial distribution of tumor cells can be seen, and a staining image representing the spatial distribution of immune T cells can be seen.

[0066] Exemplarily, the size of each image in the multiple staining image sets corresponding to the target biological tissue in the present application is 75×3500×2500; that is, each image has 75 pixel units in the Z-axis direction, 3500 pixel units in the X-axis direction, and 2500 pixel units in the Y-axis direction.

[0067] It is understandable that after a round of staining, the number of staining images that can be observed by the microscopic imaging device is closely related to how many staining markers are used to mark the cells in the target biological tissue during this round of staining. If only one marker is used to mark one cell of the target organism in this round of staining, the microscopic imaging device can only observe one staining image; if two markers are used to mark two cells in the target biological tissue, the microscopic imaging device can observe two staining images.

[0068] In this application, the target biological tissue needs to be stained multiple times, and the multiple staining processes may involve the use of different types of fluorescent dyes and antibodies to label different antigens in the target biological tissue. After each round of staining, the labeled antigen or antibody is rinsed by a specific method (such as heat repair), and then the antigen or antibody is stained again with a new marker.

[0069] After a round of staining of the target biological tissue, a plurality of staining images corresponding to the target biological tissue in the round of staining images can be obtained using a microscopic imaging device. These images are images in a set of staining images corresponding to the target biological tissue in the round of staining. Each image in the staining image set, that is, each staining image, represents the spatial distribution of different types of cells in the target biological tissue. After multiple rounds of staining of the target biological tissue, multiple sets of staining images corresponding to the target biological tissue can be obtained.

[0070] It is understandable that a round of staining of the target biological tissue will result in multiple images. Since the images in the set of stained images obtained by this round of staining are obtained by imaging at the same time, there is no positional offset and tissue deformation in the multiple images obtained by this round of staining. There is no need to align the multiple images obtained by this round of staining, and there is no need to align the images again on the basis of image alignment. However, after re-staining, the target biological tissue will have cell displacement and cell deformation caused by wash-off and re-staining. At this time, the results obtained will have positional mismatches such as misaligned contours and misaligned cells with the images obtained by the previous staining.

[0071] To solve the above problems, the same staining marker is used in each round of staining to mark the same type of cells in the target biological tissue. For example, in each round of staining, the target marker is used to mark immune T cells, so that each set of staining images includes a staining image that characterizes the spatial distribution of immune T cells, i.e., a reference image.

[0072] It is understandable that, although the staining rounds are different, the reference images in the staining image set corresponding to different staining rounds all represent the spatial distribution of the same type of cells in the target biological tissue. In theory, the same position in the reference image should be the same cell in the sample biological tissue. Since the spatial distribution of the same type of cells should be similar between different rounds, with only the difference of misaligned positions, two reference images can be used for image alignment and image registration.

[0073] It is understandable that after obtaining a plurality of staining image sets corresponding to the target biological tissue, the existing image processing technology can be used to preprocess the images in the staining image set to eliminate the background and irrelevant information in the image, so that the cells in the preprocessed image are more prominent. The present application does not limit the operation of preprocessing the image. Those skilled in the art can use operations such as image denoising, enhancement, geometric correction, grayscale, binarization, histogram equalization, etc. to preprocess the image to enhance the image quality.

[0074] S102, taking any reference image in the plurality of dyed image sets as a fixed image; a dyed image set not including the fixed image is a variable dyed image set.

[0075] In S101, multiple staining image sets are obtained, each of which includes a reference image representing the spatial distribution of the same type of cells. In the present application, any reference image in the multiple staining image sets is taken as a fixed image, the staining image set including the fixed image is taken as a fixed staining image set, and the staining image set not including the fixed image is taken as a variable staining image set.

[0076] Exemplarily, the reference image obtained by the first dyeing can be selected as the fixed image in the present application, the dyed image set obtained by the first dyeing can be selected as the fixed dyed image set, and the remaining dyed image sets can be selected as the variable dyed image set.

[0077] S103: For each of the variable stained image sets, input the reference image and the fixed image in the variable stained image set into a first prediction model to obtain an affine matrix.

[0078] After determining the fixed image, for each set of variable staining images, the reference image and the fixed image in the set of variable staining images are input into the first prediction model, and the reference image and the fixed image are processed by the first prediction model to generate an affine matrix. The affine matrix indicates the spatial displacement information of the tissue in the fixed image and the corresponding tissue in the reference image. The tissue in the present application includes a plurality of cells, that is, the tissue in the present application is a set composed of a plurality of adjacent cells of the same type in the target biological tissue.

[0079] Figure 2 A schematic diagram of the structure of a first prediction model provided in an embodiment of the present application. Figure 2 As shown, the process of inputting the reference image and the fixed image in the variable stained image set into the first prediction model to generate the affine matrix is ​​described in detail.

[0080] Combination Figure 2 As shown, the first prediction model in the present application includes N+2 convolution modules and N downsampling convolution modules, where N is an integer greater than or equal to 4. Figure 2 The solid black arrow in the figure indicates the feature vector generated after the input image is processed by the convolution module or the downsampling convolution module.

[0081] Among them, each of the N+2 convolution modules consists of three first operations; each of the N downsampling convolution modules consists of one first operation and two second operations. The first operation consists of a convolution operation with a step size of 1, a normalization operation, and an activation function in series; the second operation consists of a convolution operation with a step size of 2, a normalization operation, and an activation function in series. In this application, a rectified linear unit (ReLU) is used as the activation function in the first operation and the second operation. Since the convolution operation and the normalization operation are well known to those skilled in the art, this application will not elaborate on this content.

[0082] For each set of changing stained images, after the reference image and the fixed image in the set of changing stained images are input into the first prediction model, the first feature vector is obtained through the first two convolution modules (i.e., the first convolution module and the second convolution module) and N downsampling convolution modules (i.e., the first downsampling convolution module, the second downsampling convolution module, up to the N-1th downsampling convolution module and the Nth downsampling convolution module) of the first prediction model.

[0083] At the same time, for each set of changing stained images, after the reference image and the fixed image in the set of changing stained images are input into the first prediction model, the second feature vector is generated through the first three convolution modules (i.e., the first convolution module, the second convolution module, and the third convolution module) and N downsampling convolution modules of the first prediction model.

[0084] The process of generating the second eigenvector can be described in detail as follows: obtain the first eigenvector generated in the previous step, and obtain the intermediate eigenvector generated by the first convolution module and the first N-1 downsampling convolution modules in the first prediction model after inputting the reference image and the fixed image in the variable staining image set into the first prediction model; input the first eigenvector into the third convolution module, and input the intermediate vector into the third convolution module through a jump connection, and the third convolution module outputs the second eigenvector.

[0085] By analogy, for each set of changing stained images, after the reference image and the fixed image in the set of changing stained images are input into the first prediction model, the N+1th feature vector is generated through the N+2 convolution modules in the first prediction model (i.e., the first convolution module, the second convolution module, the third convolution module, up to the Nth convolution module, the N+1th convolution module and the N+2th convolution module) and N downsampling convolution modules.

[0086] The process of generating the N+1th feature vector can be described in detail as follows: after the reference image and the fixed image in the variable stained image set are input into the first prediction model, the Nth feature vector is generated through the first N+1 convolution modules and N downsampling convolution modules in the first prediction model; at the same time, the reference image and the fixed image in the variable stained image set are input into the first prediction model, and the feature vector generated by the first convolution module is obtained; the Nth feature vector is input into the N+2th convolution module, and the feature vector generated by the first convolution module is input into the N+2th convolution module through a jump connection, and the N+2th convolution module generates the N+1th feature vector

[0087] The first prediction model also includes N+1 alignment heads. Each alignment head is used to determine what affine parameters can enable the reference image and the fixed image in the variable staining image set to achieve a rough alignment at the tissue level including multiple cells. Each alignment head is composed of a convolution layer, an average pooling layer, and a flattening layer (also called a Flatten Layer) connected in sequence.

[0088] The first feature vector is input into the first alignment head, and the second feature vector is input into the second alignment head, until the N+1th feature vector is input into the N+1th alignment head, and then the vectors output by the above N+1 alignment heads are weighted summed to generate affine parameters consisting of rotation angle, scaling factor, spatial displacement and shear angle; and then an affine matrix is ​​generated according to the affine parameters. Since the concept of affine parameters and the generation of affine matrices according to affine parameters are numerical contents of technicians in this field, this application does not elaborate on this content.

[0089] It is understandable that the first prediction model is a model trained based on the training data. Figure 2The untrained model in is used as the first model to be trained, and two sample reference images corresponding to any two rounds of staining obtained after multiple rounds of staining of the sample biological tissue are used as a set of training data. An unsupervised training method is adopted, and the normalized cross-correlation loss (NCC LOSS) is used as the loss function to train the first model to be trained. When the training cutoff condition is met, the training is stopped, and the trained first model to be trained is used as the first prediction model. Since training the model to be trained based on the training data is a content well known to those skilled in the art, this content will not be further elaborated in this application.

[0090] S104, for each of the variable dyeing image sets, based on the affine matrix, transform each image in the variable dyeing image set to the coordinate system where the fixed image is located, to obtain a first target image set.

[0091] Affine transformation in a plane or space refers to a transformation that maps a graphic to another position through operations such as translation, rotation, scaling, and shearing. Affine matrix is ​​a matrix used to represent affine transformation. The affine matrix in two-dimensional space is a 3x3 matrix, and the affine matrix in three-dimensional space is a 4x4 matrix. In computer vision, affine matrices can be used to correct images and align images.

[0092] The principle of image alignment in this application is that the reference images obtained by two rounds of staining that characterize the spatial distribution of the same type of cells in the target biological tissue should be completely consistent in principle, but there is an offset in position. The affine matrix is ​​a matrix generated based on affine parameters consisting of rotation angle, zoom ratio, spatial displacement and shear angle, which represents the offset of the spatial position of objects between two images, such as corresponding tissues. The reference image is transformed using a reflection matrix, that is, the reference image is transformed to the coordinate system where the fixed image is located, so that the reference image and the fixed image can be aligned in position.

[0093] It can be understood that if the images in the stained image set obtained after staining the target biological tissue in the present application are three-dimensional images, then the affine matrix in the present application is a 4x4 matrix; if the images in the stained image set obtained after staining the target biological tissue in the present application are two-dimensional images, then the affine matrix in the present application is a 3x3 matrix.

[0094] It can be understood that the multiple staining images in a staining image set obtained by performing each round of staining on the target biological tissue have the same degree of positional offset relative to the coordinate system where the fixed image is located.

[0095] Based on this, after obtaining the affine matrix, for each variable staining image set, the affine matrix is ​​used to transform each image in the variable staining image set to the coordinate system where the fixed image is located, so as to obtain the first target image.

[0096] Since the coordinate transformation of each image in the variable dyeing image set based on the affine matrix is ​​well known to those skilled in the art, no further elaboration is made on this content in this application.

[0097] In summary, the present application discloses a method for processing immunofluorescence staining images, including: obtaining multiple staining image sets corresponding to target biological tissues; each staining image set includes a reference image that characterizes the spatial distribution of the same cell in the target biological tissue; taking a reference image as a fixed image, and each staining image set that does not include a fixed image as a variable staining image set; using a first prediction model to construct an affine matrix between the reference image and the fixed image in the variable staining image set; and then using the affine matrix to transform each image in the variable staining image set corresponding to the affine matrix to the coordinate system where the fixed image is located, to obtain a first target image.

[0098] Since the fixed image is also a reference image, the reference images all represent the spatial distribution of the same cell in the target biological tissue. In theory, the same position in different reference images corresponds to the same tissue unit of the target biological tissue. The affine matrix indicates the spatial displacement information of the tissue in the fixed image and the corresponding tissue in the reference image in the variable staining image set. By using the affine matrix to transform the images in the variable staining image set to the coordinate system where the fixed image is located, the spatial alignment of the staining images obtained after two rounds of staining can be achieved. This solves the problem that during the decolorization and re-staining process of the target biological tissue, the sample biological tissue will have tissue offset caused by elution and re-staining, that is, the position of the same biological tissue in the staining images obtained after different rounds of staining is not exactly the same.

[0099] The method disclosed in the aforementioned embodiment solves the problem of "tissue deviation" during multiple rounds of staining, but has not yet solved the problem of "tissue deformation" during multiple rounds of staining, that is, the same cell in the target biological tissue has different sizes and shapes in different rounds of staining images.

[0100] Based on this, the present application discloses a flowchart of another method for processing immunofluorescence staining images. Figure 3 A flowchart of another method for processing an immunofluorescence staining image provided in an embodiment of the present application. Figure 3 As shown, another method for processing immunofluorescence staining images disclosed in the present application includes:

[0101] S301, obtaining a plurality of staining image sets corresponding to target biological tissues.

[0102] S302 , taking any reference image in the plurality of dyed image sets as a fixed image; a dyed image set not including the fixed image is a variable dyed image set.

[0103] S303: For each of the variable stained image sets, input the reference image and the fixed image in the variable stained image set into a first prediction model to obtain an affine matrix.

[0104] S304 , for each of the variable dyed image sets, based on the affine, transform each image in the variable dyed image set to the coordinate system where the fixed image is located, to obtain a first target image set.

[0105] For the contents in S301-S304, refer to the introduction of S101-S104 in the aforementioned embodiment, which will not be repeated here.

[0106] S305: Acquire a target reference image in each of the first target image sets.

[0107] From the introduction of the aforementioned embodiments, it can be known that each variable staining image set has an affine matrix relative to the fixed image. The affine matrix represents the spatial displacement information of the tissue of the reference image in the variable staining image set relative to the corresponding tissue in the fixed image. After obtaining the affine matrix corresponding to each variable staining image set, the affine matrix can be used to perform spatial transformation on each image in each variable staining image set, that is, each image in the variable staining image set is transformed into the coordinate system where the fixed image is located to obtain the first target image set; that is, each image in the first target image set is an image obtained by transformation using the affine matrix.

[0108] The target reference in the present application is an image obtained by performing an affine matrix transformation on the reference image in the variable staining image set. The target reference image also represents the spatial distribution of the specific cells shown in the reference image mentioned in the above embodiments.

[0109] S306: For each of the first target image sets, input the target reference image and the fixed image in the first target image set into a second prediction model to obtain a deformation field.

[0110] After obtaining a plurality of first target image sets, for each first target image set, a target reference image and a fixed image in the first target image set are input into a second prediction model to generate a deformation field.

[0111] In the field of image processing, a deformation field is a two-dimensional or three-dimensional vector field, in which each pixel or voxel position corresponds to an offset vector. These offset vectors describe the position change of each pixel or voxel in the image during the deformation process. By applying the deformation field to the original image, the image can be distorted, deformed, bent, etc., which is crucial for tasks such as image correction, registration, and enhancement.

[0112] The second prediction model in the present application includes a feature extraction part and a multi-level deformation field fusion part; the feature extraction part includes a first branch for extracting a feature vector of a target reference image, and a second branch for extracting a feature vector of a fixed image; the first branch and the second branch of the feature extraction part are both composed of M convolution modules connected in series, where M is an integer greater than or equal to 4.

[0113] For each first target image set, after the target reference image in the first target image set is input into the second prediction model, M serially connected convolution modules in the first branch of the feature extraction part of the second prediction model process the target reference image, each convolution module outputs a feature vector of the target reference image, and finally outputs M feature vectors corresponding to the target reference image.

[0114] Similarly, after the fixed image is input into the second prediction model, the M convolution modules in series of the second branch of the feature extraction part of the second prediction model process the fixed image, each convolution module outputs a feature vector of the fixed image, and finally outputs M feature vectors corresponding to the fixed image. It can be understood that the target reference image and the fixed image are input into the second prediction model at the same time.

[0115] After obtaining the M feature vectors corresponding to the target reference image and the M feature vectors corresponding to the fixed image, the multi-level deformation field fusion part of the second prediction model is entered. If M is 4, four deformation field fusions are required to generate the target deformation field.

[0116] Specifically, in the multi-level deformation field fusion part of the second prediction model, the Mth eigenvector corresponding to the target reference image and the Mth eigenvector corresponding to the fixed image are spliced ​​to obtain a first splicing vector; the first splicing vector is iterated multiple times to obtain a first deformation field; the first deformation field is upsampled, and the result obtained after the upsampling operation is spatially transformed with the M-1th eigenvector corresponding to the target reference image, and the result after the spatial network transformation is spliced ​​with the M-1th eigenvector of the fixed image to obtain a second splicing vector; the second splicing vector is iterated multiple times to obtain a second deformation field increment; the second deformation field increment and the result obtained after the upsampling operation of the first deformation field are weightedly summed to obtain a second deformation field; and so on, based on the second deformation field, the first M-2 eigenvectors corresponding to the target reference image and the first M-2 eigenvectors corresponding to the fixed image, a target deformation field is generated.

[0117] Figure 4A A schematic diagram of the structure of a second prediction model provided in an embodiment of the present application; Figure 4B A schematic diagram of the structure of each module in a second prediction model provided in an embodiment of the present application. Figure 4A and Figure 4B As shown, the process of inputting the target reference image and the fixed image in the first target image set into the second prediction model to generate the target deformation field is described in detail. M in FIG4 is 4.

[0118] Combination Figure 4A As shown, the first branch of the feature extraction part in the second prediction model includes 4 convolution modules, and the second branch includes 4 convolution modules. Specifically, the 4 convolution modules included in the first branch are convolution module (1), convolution module (2), and two identical convolution modules (3) in FIG4 ; wherein the convolution module (3) is also called a window attention module. It should be noted that in FIG4 , (1) and a black solid arrow represent the convolution module (1); (2) and a black solid arrow represent the convolution module (2); and (3) and a black solid arrow represent the convolution module (3).

[0119] Combination Figure 4B As shown, the convolution module 1 consists of a convolution layer, instance normalization, an activation function, a convolution layer, instance normalization, and an activation function connected in series.

[0120] Exemplarily, a leaky linear rectified unit (Leaky ReLU) may be used as an activation function in the convolution module 1.

[0121] Exemplarily, after the feature vector a to be processed is input into the convolution module 1, it is processed by a series of convolution layers, instance normalization, activation functions, convolution layers, instance normalization and activation functions to obtain feature vector b; feature vector b is weightedly summed with feature vector a to obtain feature vector c.

[0122] Among them, the convolution module 2 is composed of two serially connected convolution modules 1. The content of the convolution module 2 can refer to the introduction of the relevant content of the convolution module 1, which will not be repeated here.

[0123] Among them, convolution module 3 is composed of two serially connected window attention modules. Each window attention module consists of linear normalization (Linear Norm), window based multi-head self-attention (W-MSA), multilayer perceptron (MLP) and shifted window multi-head self-attention (SW-MSA).

[0124] Exemplarily, after the feature vector 1 to be processed is output to the convolution module 3, feature vector 2 is generated in sequence through linear scaling and window-based multi-head attention; feature vector 2 is weightedly summed with feature vector 1 to obtain feature vector 3; feature vector 3 is subjected to linear scaling operation and multi-layer perceptron to obtain feature vector 4; feature vector 4 is weightedly summed with feature vector 3 to obtain feature vector 5; feature vector 5 is subjected to linear scaling and moving window multi-head attention mechanism to obtain feature vector 6; feature vector 6 is weightedly summed with feature vector 5 to obtain feature vector 7; feature vector 7 is subjected to linear scaling and multi-layer perceptron to obtain feature vector 8; feature vector 8 is weightedly summed with feature vector 7 to obtain output feature vector 9 corresponding to the feature vector 1 to be processed.

[0125] Combination Figure 4A As shown, when the target reference image and the fixed image are input into the second prediction model, the four convolution modules in the first branch of the feature extraction part of the second prediction model process the target reference image to obtain the first feature vector generated by the convolution module (1), the second feature vector generated by the convolution module (2), the third feature vector generated by the first convolution module (3), and the fourth feature vector generated by the second convolution module (3). At the same time, the four convolution modules in the second branch of the feature extraction part of the second prediction model process the fixed image to obtain the first feature vector generated by the convolution module (1), the second feature vector generated by the convolution module (2), the third feature vector generated by the first convolution module (3), and the fourth feature vector generated by the second convolution module (3).

[0126] In the multi-level deformation field fusion part, the fourth eigenvector of the target reference image and the fourth eigenvector of the fixed image are spliced ​​to obtain a first spliced ​​vector, and the first spliced ​​vector is subjected to the operation of the convolution module (4) and the iterative operation to generate a first deformation field D1; the first deformation field D1 is upsampled to obtain an upsampled deformation field D1 up ; The deformation field D1 obtained after upsampling up A spatial network transformation operation is performed on the third eigenvector of the target reference image, and the vector obtained after the spatial network transformation is spliced ​​with the third eigenvector of the target reference image to obtain a second spliced ​​vector. The second spliced ​​vector is subjected to the operation of the convolution module (4) and the iterative operation to generate a second deformation field increment, ΔD2; the second deformation field increment ΔD2 is combined with the deformation field D1 obtained after upsampling up Weighted summation is performed to obtain the second deformation field D2; the second deformation field is upsampled to obtain the upsampled deformation field D2 up ; Deformation field D2 after upsampling up A spatial network transformation operation is performed on the second eigenvector of the target reference image, and the vector obtained after the spatial network transformation is spliced ​​with the second eigenvector of the fixed image to obtain a third spliced ​​vector; the third spliced ​​vector is subjected to the operation of a convolution module (5) and an iterative operation to generate a third deformation field increment ΔD3; the third deformation field increment ΔD3 is combined with the deformation field D2 obtained after upsampling up Perform weighted summation to obtain the third deformation field; perform upsampling on the third deformation field to obtain the upsampled deformation field D3 up ; Deformation field D3 after upsampling up The spatial network transformation is performed on the first feature vector of the target reference image, and the vector generated by the spatial network transformation is concatenated with the first feature vector of the fixed image to obtain a fourth concatenated vector; the fourth concatenated vector is subjected to the operation of the convolution module (5) and the iterative operation to generate a fourth deformation field increment, that is, the final deformation field increment ΔD fin ; The fourth deformation field increment and the deformation field D3 obtained after upsampling up Perform weighted summation to obtain the target deformation field D fin .

[0127] The convolution module (5) is composed of two convolution modules (1) connected in series, and the convolution module (4) is composed of two convolution modules (5) connected in series.

[0128] It can be understood that the second prediction model is also a model obtained through pre-training. Figure 4AThe untrained model shown in the figure is used as the second model to be trained. After multiple rounds of staining using sample biological tissue, the two subsequent reference images of any two rounds of staining images are used as a set of training data. An unsupervised training method is adopted, and NCCLOSS is used as the loss function to train the second model to be trained. The trained model is used as the second prediction model.

[0129] S307 , for each of the first target image sets, adjusting the position of cells in each image in the first target image set based on the deformation field to obtain a second target image set.

[0130] It can be known from the introduction of the above embodiments that the vector in the deformation field is a vector indicating the spatial deformation information of the cells in the target reference image and the corresponding cells in the fixed image in the first target image set.

[0131] For each first target image set, after obtaining the deformation field of the target reference image in the first target image set relative to the fixed image, the position of the cells in each image in the first target image set is adjusted using the deformation field to obtain the second target image set. Since image processing based on deformation fields is a technology well known to those skilled in the art, this application does not elaborate on this content.

[0132] The process of image registration in this application is to first use the reference image to align the reference images in different stained image sets; then use the aligned images for image registration. The reason for performing image alignment first is that there will be performance loss if image registration is performed directly. Image registration pays more attention to the mapping relationship of local details. If there is a large position deviation between the two images as a whole, direct image registration will lead to very unsatisfactory registration results. Therefore, it will be better to align the images first and then perform image registration based on the aligned images.

[0133] To summarize, the present application obtains the affine matrix corresponding to each variable staining image set, and transforms each image in the variable staining image set to the coordinate system of the fixed image based on the affine matrix to obtain multiple first target image sets (realizing the contour alignment at the tissue level in the image). The present application further discloses a method of inputting the target reference image and the fixed image in the first target image set into a second prediction model to generate a target deformation field of the target reference image in the first target image set relative to the fixed image, and based on the target deformation field, spatially transforms the cells in each image in the first target image set to obtain a second target image set (realizing image alignment at the cell level).

[0134] In the field of medical image processing and analysis technology, after obtaining multiple staining image sets corresponding to target biological tissues and using the method disclosed in the aforementioned embodiments to achieve image alignment at the tissue (multiple cells) level with images in other staining image sets, as well as image registration at the cell level, the images in the multiple staining image sets after image alignment and image registration can be analyzed to achieve molecular diagnosis in the medical field and provide a basis for personalized treatment.

[0135] In an optional implementation, after obtaining multiple second target image sets, each image in the second target image set can be input into a cell segmentation model, and each image is processed by the cell segmentation model to obtain an identification image set with the same number as the second target image set; one identification image in the identification image set corresponds to one image in the second target image set.

[0136] Although the images in the second target image set show the spatial distribution of cells marked by the marker, the cells marked by the marker cannot be well highlighted and displayed in the images due to the presence of background noise. The identification images in the identification image set clearly show the objects marked by the dyed marker and the objects not marked by the dyed marker.

[0137] In this application, a model similar to 3D U-Net (3D U-Net Convolutional Neural Network) is used as the third model to be trained. The sample stained images of the sample biological tissue and the sample stained images with artificially labeled cells are used as training data. The third model to be trained is trained with cross entropy loss (CE Loss) and Dice loss (Dice Loss) as loss functions to obtain a cell segmentation model.

[0138] Among them, the difference between the third model to be trained and the traditional 3D U-Net model is that the convolutional layer of the traditional 3D U-Net model is replaced by a full convolutional module, the receptive field of a single convolutional layer is expanded, and multi-scale segmentation prediction is performed on the decoder to improve the segmentation accuracy.

[0139] Figure 5A A schematic diagram of the structure of a cell segmentation model provided in an embodiment of the present application; Figure 5B The schematic diagram of the structure of each module in a cell segmentation model provided in the embodiment of the present application is as follows. Figure 5A and Figure 5B , a process of using a cell segmentation model to process images in the second target image set and generate identification images is described in detail.

[0140] Combination Figure 5A As shown, the cell segmentation model includes an embedding layer, multiple residual convolution modules, multiple feature extraction modules and a segmentation head.

[0141] Figure 5A The embedding layer in is actually patch embedding, which is an embedding method used in deep learning. The core idea of ​​this method is to divide the input data (such as images or text) into several small patches or units, and then convert each small patch into a low-dimensional vector representation.

[0142] Exemplarily, when the size of each image in the plurality of stained image sets is 75×3500×2500, the size of the patches in the present application is 64×64×64, that is, the pixel size of the patches in the X-axis, Y-axis and Z-axis directions is 64.

[0143] Figure 5A The specific structure of the residual convolution module in Figure 5B Specifically, the residual convolution module is composed of a 3×3×3 convolution kernel, instance normalization, activation function, a 3×3×3 convolution kernel, instance normalization and activation function in series. Exemplarily, the feature vector a1 input to the residual convolution module is subjected to a 3×3×3 convolution kernel, instance normalization, activation function, a 3×3×3 convolution kernel, instance normalization and activation function to obtain a feature vector a2. The feature vector a2 is weightedly summed with the feature vector a1 to obtain a feature vector a3 output by the residual convolution module corresponding to the feature vector a1. Among them, the rectified linear unit is used as the Figure 5B Activation function in the residual convolution module.

[0144] Figure 5A The feature extraction module in is composed of a residual convolution module, two window attention modules, a fusion sampling module and a residual convolution module connected in sequence. Among them, the combination of the residual convolution module is as follows Figure 5B As shown, no further description is given here.

[0145] Figure 5A The window attention module (Swin transformer block) in the feature extraction module consists of linear normalization, window attention mechanism (W-MSA), multi-layer perceptron and moving window attention mechanism (SW-MSA).

[0146] Specifically, the feature vector b1 input into the window attention module generates feature vector b2 after linear normalization operation and window attention mechanism; feature vector b1 and feature vector b2 are weighted summed to obtain feature vector b3; feature vector b3 generates feature vector b4 after linear normalization and multi-layer perceptron; feature vector b3 and feature vector b4 are weighted summed to generate feature vector b5; feature vector b5 generates feature vector b6 after linear normalization and moving window attention; feature vector b6 and feature vector b5 are weighted summed to generate feature vector b7; feature vector b7 generates feature vector b8 after linear normalization and multi-layer perceptron; feature vector b8 and feature vector b7 are weighted summed to obtain feature vector b9 output by the window attention module corresponding to the input feature vector b1.

[0147] Figure 5A The main function of the segmentation head in the image processing model is to convert the feature map extracted by the previous network into a pixel-level classification prediction, that is, to assign a category label to each pixel in the image. The structure of the segmentation head is not limited in this application, and those skilled in the art can use a common segmentation head in the field of image processing as the segmentation head in the cell segmentation model in this application.

[0148] See also Figure 5A , when an image in the second target stained image set is input into the image segmentation model, it is processed by the embedding layer and the first residual convolution module to generate features Figure 1 ; After being processed by the embedding layer, the first residual convolution module and the first feature extraction module, the feature Figure 2 After being processed by the embedding layer, the first residual convolution module, the first feature extraction module and the second feature extraction module, the feature Figure 3 ; After being processed by the embedding layer, the first residual convolution module, the first feature extraction module, the second feature extraction module and the third feature extraction module, feature map 4 is generated; after being processed by the embedding layer, the first residual convolution module, the first feature extraction module, the second feature extraction module, the third feature extraction module and the fourth feature extraction module, feature map 5 is generated; after being processed by the second residual convolution module, feature Figure 6 .

[0149] Then, feature map 5 and feature map 4 are concatenated, and the concatenated result is processed by two series-connected residual convolution modules ( Figure 5A The residual convolution module in ×2) generates features Figure 7 ;feature Figure 7 With features Figure 3 The spliced ​​result is then processed by two series-connected residual convolution modules to generate feature map 8; feature map 8 and feature Figure 2The spliced ​​result is then processed by two series-connected residual convolution modules to generate feature map 9; feature map 9 and feature Figure 1 The spliced ​​result is then processed by two series-connected residual convolution modules to generate feature map 10; feature map 10 and feature Figure 6 The splicing is performed, and the spliced ​​result is processed by two residual convolution modules connected in series to generate a feature map 11; the feature map 11 is processed by the segmentation head to generate a logo image.

[0150] It can be understood that after inputting each image in the second target staining image set into the cell segmentation model, the benefit of obtaining an identification image corresponding to each input image is that the identification image clearly displays the cells represented by the identified object, so that the spatial distribution of the identified cells in each image can be seen more clearly.

[0151] In the present application, each image in the plurality of second target image sets is input into the cell segmentation model to generate a plurality of identification image sets. Since the second target image set is a set that realizes image alignment and image registration, the images in the generated plurality of identification image sets are identification images that realize image alignment and image registration.

[0152] It is understandable that the images in the set of multiple stained images corresponding to the target biological tissue can be directly input into the cell segmentation model, and the results obtained by the cell segmentation model are then image aligned according to the affine matrix, and the results of the image alignment are then image registered according to the target deformation field. Those skilled in the art can use any method to obtain the identification image corresponding to the aligned or registered image.

[0153] In an optional implementation, after obtaining a plurality of identification image sets, that is, obtaining a plurality of identification images, it is possible to use Figure 6 The method in is used to obtain the spatial distribution results of each cell in the target biological tissue. Figure 6 A flowchart for obtaining the spatial distribution results of each cell in a biological tissue is provided in the embodiment of the application. Figure 6 As shown, in this application, obtaining the spatial distribution results of each cell in the biological tissue includes the following steps:

[0154] S601, based on each of the identification images, determine statistical information of each cell on the identification image.

[0155] For each identification image, each cell is taken as an object and the statistical information of the cell is calculated. The statistical information in this application includes but is not limited to the information such as the centroid of the cell (the pixel coordinates of the cell in the identification image), the volume size and the fluorescence intensity. It should be noted that this process is also referred to as connected domain calculation. Since connected domain calculation is well known to those skilled in the art, this content will not be repeated in this application.

[0156] S602: For each cell on the identification image, if the distance between the centroid of the cell and the centroid of the target cell in other identification images is less than or equal to a preset distance threshold, then obtaining information of the target cell as the co-expression information of the cell.

[0157] It is understandable that, due to the existence of calculation errors, each cell in the sample biological tissue cannot be absolutely registered and aligned in different identification images even after image alignment and image registration in the aforementioned embodiments.

[0158] Based on this, after obtaining the statistical information of each cell on the identification image, it can be further determined whether the distance between the centroid of the cell and the centroid of any cell in other identification images (i.e., the target cell) is less than or equal to a preset distance threshold. Exemplarily, a 5-pixel interval can be used as a preset distance threshold. If the distance between the centroid of the cell and the centroid of the target cell in other identification images is less than or equal to the preset distance threshold, it is considered that the cell and the target cell are the same cell, except that the cell is identified by different markers in two different rounds of staining. At this time, the information of the target cell is obtained as the co-expression information of the cell.

[0159] S603, obtaining the spatial distribution result of each cell in the target biological tissue based on the statistical information of each cell and the co-expression information of each cell.

[0160] After obtaining the statistical information and co-expression information of each cell in the target biological tissue, the spatial distribution results of each cell in the target biological tissue can be obtained based on this information; and then various pathological analyses can be performed based on the spatial distribution results.

[0161] It should be noted that constructing the spatial distribution results of each cell in the target biological tissue based on the statistical information and co-expression information of each cell is well known to those skilled in the art and will not be further elaborated in this application.

[0162] Furthermore, the present application also discloses a device for processing immunofluorescence staining images. Figure 7 A schematic diagram of the structure of an immunofluorescence staining image processing device provided in an embodiment of the present application.

[0163] Combination Figure 7 As shown, the immunofluorescence staining image processing device 700 disclosed in the present application includes:

[0164] The staining image set acquisition module 701 is used to acquire multiple staining image sets corresponding to the target biological tissue; each image in each of the staining image sets represents the spatial distribution of different types of cells in the target biological tissue; each of the staining image sets includes a reference image representing the spatial distribution of the same type of cells;

[0165] A basic data determination module 702 is used to use any reference image in the plurality of dyed image sets as a fixed image; a dyed image set that does not include the fixed image is a variable dyed image set;

[0166] An affine matrix generation module 703 is used for inputting the reference image and the fixed image in the variable staining image set into a first prediction model for each variable staining image set to obtain an affine matrix; the affine matrix indicates the spatial displacement information between the tissue in the fixed image and the corresponding tissue in the reference image; the first prediction model is a model for generating an affine matrix between two images; the tissue includes a plurality of cells;

[0167] The first target image set generating module 704 is used to transform each image in the variable color image set to the coordinate system where the fixed image is located based on the affine to obtain the first target image set.

[0168] In an optional implementation, the immunofluorescence staining image processing device 700 further includes:

[0169] A target reference image determination module, used to obtain a target reference image in each of the first target image sets; the target reference image is an image obtained by performing an affine transformation on a reference image in the variable staining image set based on an affine matrix;

[0170] A target deformation field generation module is used for inputting a target reference image and the fixed image in the first target image set into a second prediction model for each of the first target image sets to obtain a target deformation field; the vector in the target deformation field indicates spatial deformation information of a cell in the target reference image and a corresponding cell in the fixed image; the second prediction model is a pre-trained model for generating a deformation field between two images;

[0171] The second target image set generating module is used to adjust the position of cells in each image in the first target image set based on the target deformation field for each of the first target image sets to obtain a second target image set.

[0172] In an optional implementation, the first prediction model includes N+2 convolution modules and N downsampling convolution modules, where N is an integer greater than or equal to 4, and the affine matrix generation module 703 includes:

[0173] A first feature vector generating unit, configured to generate a first feature vector according to a reference image and the fixed image in the variable stained image set by using the first two convolution modules and the N downsampling convolution modules in the first prediction model;

[0174] A second feature vector generating unit, configured to generate a second feature vector according to the reference image and the fixed image in the variable stained image set by using the first three convolution modules and the N downsampling convolution modules in the first prediction model;

[0175] an N+1th feature vector generating unit, configured to generate an N+1th feature vector by analogy, through the N+2 convolution modules and the N downsampling convolution modules in the first prediction model, according to the reference image and the fixed image in the variable stained image set;

[0176] An affine matrix generating unit is used to generate the affine matrix through the first eigenvector, the second eigenvector, and up to the N+1th eigenvector.

[0177] In an optional implementation, the target deformation field generation module includes:

[0178] A basic feature vector generating unit, configured to generate, according to the target reference image in the first target image set, M feature vectors corresponding to the target reference image through a plurality of convolution modules in the second prediction model; and to generate, according to the fixed image, M feature vectors corresponding to the fixed image; wherein M is an integer greater than or equal to 4;

[0179] A first deformation field generating unit is used to splice the Mth eigenvector corresponding to the target reference image and the Mth eigenvector corresponding to the fixed image to obtain a first splicing vector; and iteratively optimize the first splicing vector to obtain a first deformation field;

[0180] a second deformation field increment generating unit, configured to perform spatial transformation on the M-1th eigenvector corresponding to the target reference image based on the first deformation field, splice the result of the spatial transformation with the M-1th eigenvector corresponding to the fixed image to obtain a second splicing vector, and iteratively optimize the second splicing vector to obtain a second deformation field increment;

[0181] a second deformation field generating unit, configured to generate the second deformation field based on the second deformation field increment and the first deformation field;

[0182] The target deformation field generating unit is used to generate the target deformation field based on the second deformation field, the first M-2 feature vectors corresponding to the target reference image and the first M-2 feature vectors corresponding to the fixed image.

[0183] In an optional implementation, the immunofluorescence staining image processing device 700 further includes:

[0184] The identification image set generation module is used to input each image in each second target image set into the cell segmentation model to obtain the identification image set corresponding to each second target image set; one identification image in the identification image set corresponds to one image in the second target image set; the identification image is used to distinguish between objects marked by dye markers and objects not marked by dye markers. Wherein, the cell segmentation model includes a feature extraction module, which extracts the features of each image in each second target image set; the feature extraction module includes a residual convolution module, a window attention module, a fusion sampling module and the residual convolution module connected in sequence.

[0185] In an optional implementation, the immunofluorescence staining image processing device 700 further includes:

[0186] A statistical information acquisition unit, used to determine the statistical information of each cell on the identification image based on each identification image; the statistical information includes the center of mass, volume and fluorescence intensity of the cell;

[0187] a co-expression information acquisition unit, configured to, for each cell on the identification image, acquire information of the target cell as the co-expression information of the cell if the distance between the centroid of the cell and the centroid of the target cell in other identification images is less than or equal to a preset distance threshold;

[0188] The spatial distribution result generating unit is used to obtain the spatial distribution result of each cell in the target biological tissue based on the statistical information of each cell and the co-expression information of each cell.

[0189] Based on the immunofluorescence staining image processing method and device provided in the aforementioned embodiments, the present application also provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements some or all of the steps in the immunofluorescence staining image processing method mentioned above.

[0190] Based on the immunofluorescence staining image processing method and device provided in the above embodiments, the present application further provides an electronic device, including:

[0191] a memory having a computer program stored thereon;

[0192] A processor is used to execute the computer program in the memory to implement part or all of the steps in the method for processing immunofluorescence staining images provided in the above-mentioned embodiment.

[0193] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0194] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing an immunofluorescence staining image, characterized in that: The method comprises: Acquire a plurality of staining image sets corresponding to the target biological tissue; each image in each of the staining image sets represents the spatial distribution of different types of cells in the target biological tissue; each of the staining image sets includes a reference image representing the spatial distribution of the same type of cells; any one of the reference images in the plurality of dyed image sets is used as a fixed image; the dyed image set not including the fixed image is a variable dyed image set; For each of the variable staining image sets, a reference image in the variable staining image set and the fixed image are input into a first prediction model to obtain an affine matrix; the affine matrix indicates spatial displacement information between a tissue in the fixed image and a corresponding tissue in the reference image; the first prediction model is a pre-trained model for generating an affine matrix between two images; the tissue includes a plurality of cells; For each of the variable dyed image sets, based on the affine matrix, each image in the variable dyed image set is transformed to the coordinate system where the fixed image is located, so as to obtain a first target image set.

2. The method according to claim 1, characterized in that: The method further comprises: Acquire a target reference image in each of the first target image sets; the target reference image is an image obtained by performing an affine transformation on a reference image in the variable dye image set based on an affine matrix; For each of the first target image sets, a target reference image and the fixed image in the first target image set are input into a second prediction model to obtain a target deformation field; a vector in the target deformation field indicates spatial deformation information of a cell in the target reference image and a corresponding cell in the fixed image; the second prediction model is a pre-trained model for generating a deformation field between two images; For each of the first target image sets, the position of cells in each image in the first target image set is adjusted based on the target deformation field to obtain a second target image set.

3. The method according to claim 1, characterized in that The first prediction model includes N+2 convolution modules and N downsampling convolution modules, where N is an integer greater than or equal to 4; for each of the variable stained image sets, the reference image and the fixed image in the variable stained image set are input into the first prediction model to obtain an affine matrix, including: Generate a first feature vector according to the reference image and the fixed image in the variable stained image set by using the first two convolution modules and the N downsampling convolution modules in the first prediction model; Generate a second feature vector according to the reference image and the fixed image in the variable stained image set by using the first three convolution modules and the N downsampling convolution modules in the first prediction model; Similarly, the N+2 convolution modules and the N downsampling convolution modules in the first prediction model are used to generate an N+1th feature vector according to the reference image and the fixed image in the variable stained image set; The affine matrix is ​​generated through the first eigenvector, the second eigenvector, and up to the N+1th eigenvector.

4. The method according to claim 2, characterized in that: For each of the first target image sets, inputting the target reference image and the fixed image in the first target image set into a second prediction model to obtain a target deformation field, including: Generate, by using a plurality of convolution modules in the second prediction model, M feature vectors corresponding to the target reference image in the first target image set; and generate, by using the fixed image, M feature vectors corresponding to the fixed image; wherein M is an integer greater than or equal to 4; splicing the Mth eigenvector corresponding to the target reference image and the Mth eigenvector corresponding to the fixed image to obtain a first splicing vector; and iteratively optimizing the first splicing vector to obtain a first deformation field; Based on the first deformation field, spatially transform the M-1th eigenvector corresponding to the target reference image, splice the result of the spatial transformation with the M-1th eigenvector corresponding to the fixed image to obtain a second splicing vector, and iteratively optimize the second splicing vector to obtain a second deformation field increment; generating the second deformation field based on the second deformation field increment and the first deformation field; The target deformation field is generated based on the second deformation field, the first M-2 feature vectors corresponding to the target reference image, and the first M-2 feature vectors corresponding to the fixed image.

5. The method according to claim 2, characterized in that: After obtaining a plurality of sets of the second target images, the method further includes: Input each image in each of the second target image sets into the cell segmentation model to obtain a set of identification images corresponding to each of the second target image sets; an identification image in the identification image set corresponds to an image in the second target image set; the identification image is used to distinguish between objects identified by a dye marker and objects not identified by a dye marker.

6. The method according to claim 5, characterized in that The cell segmentation model includes a feature extraction module, which extracts the features of each image in each of the second target image sets; the feature extraction module includes a residual convolution module, a window attention module, a fusion sampling module and the residual convolution module connected in sequence.

7. The method according to claim 5, characterized in that After acquiring the plurality of identification images, the method further includes: Based on each of the identified images, determining statistical information of each cell on the identified image; the statistical information includes the center of mass, volume and fluorescence intensity of the cell; For each cell on the identification image, if the distance between the centroid of the cell and the centroid of the target cell in other identification images is less than or equal to a preset distance threshold, then obtaining the information of the target cell as the co-expression information of the cell; Based on the statistical information of each cell and the co-expression information of each cell, the spatial distribution result of each cell in the target biological tissue is obtained.

8. A device for processing immunofluorescence staining images, characterized in that: The device comprises: A staining image set acquisition module, used to acquire multiple staining image sets corresponding to the target biological tissue; each image in each of the staining image sets represents the spatial distribution of different types of cells in the target biological tissue; each of the staining image sets includes a reference image representing the spatial distribution of the same type of cells; A basic data determination module, configured to use any reference image in the plurality of dyeing image sets as a fixed image; a dyeing image set not including the fixed image is a variable dyeing image set; An affine matrix generation module is used for inputting the reference image in the variable staining image set and the fixed image into the first prediction model for each variable staining image set to obtain an affine matrix; the affine matrix indicates the spatial displacement information of the tissue in the fixed image and the corresponding tissue in the reference image; the first prediction model is a model for generating an affine matrix between two images; the tissue includes a plurality of cells; The first target image set generating module is used to transform each image in the variable dyeing image set to the coordinate system where the fixed image is located based on the affine to obtain the first target image set.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 9.

Images