Hierarchical Transform attention feature fusion-based iPSCs set pluripotency evaluation method

Abstract

The invention relates to the technical field of iPSCs colony pluripotency evaluation, in particular to an iPSCs colony pluripotency evaluation method based on hierarchical Transform attention feature fusion, and the method comprises the following steps: collecting images in an iPSCs culture process, and establishing a mapping relation between stem cell pluripotency and cell colony morphology; constructing an iPSCs image data set, enabling each pixel in the image to correspond to a pluripotent state category, and marking a pixel level; building an evaluation model based on Transform attention feature fusion, wherein the evaluation model comprises a Gabor edge feature enhancement module, a layered Transform trunk feature extraction module and an attention feature fusion module; the obtained feature layers are fused after being subjected to up-sampling and normalization, and then a prediction image, namely an evaluation result, is obtained after decoding; training the model by using the iPSCs image data set; and carrying out pluripotency evaluation on the image in the iPSCs culture process by using the model obtained by training. According to the method, human factor interference can be prevented, and the precision of iPSCs set pluripotency evaluation is high.

Description

Technical Field

[0001] The present invention relates to the technical field of iPSCs colony pluripotency assessment, and in particular to an iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion. Background Art

[0002] Induced pluripotent stem cells (iPSCs) are derived from adult cells through reprogramming technology and have the ability to continuously self-renew and differentiate into a variety of cell types. They have broad application prospects in the treatment of chronic and degenerative diseases, including Parkinson's disease, diabetes, heart disease, and spinal cord injury. However, the large number of cells required in these applications depends on the large-scale expansion of iPSCs. During expansion, genetic and epigenetic variations in iPSC colonies affect their differentiation ability and quality, which in turn affects their directed differentiation efficiency in disease modeling or therapeutic applications. In addition, minor differences in conventional culture can also cause variations in iPSC colonies. Therefore, strict quality control and evaluation of iPSC colonies during the culture stage are crucial to ensure their safety and effectiveness in clinical applications. Different from the quality assessment standards of general cells, the pluripotency of stem cells, that is, the potential to differentiate into multiple cells, is one of the important criteria for evaluating their quality. However, unlike the evaluation of phenotypic characteristics such as morphology, cell activity, and colony contamination, the evaluation of stem cell pluripotency is not easy to observe directly.

[0003] At present, the mapping relationship between stem cell pluripotency and morphology has not been clearly established, and most of them are classified a priori based on the experience of professionals. Therefore, the evaluation results will inevitably be interfered by human factors, resulting in large differences in the evaluation results of the same method in different laboratory strains, which limits its standardized and quantitative application as a quality assessment. In addition, when performing live cell imaging during iPSCs culture, identifying and monitoring iPSCs colonies of different pluripotencies faces multiple complex challenges. High-quality colonies usually have round edges, tight cell arrangement, and no voids, while low-quality colonies may show blurred edges, sparse cells, and internal voids. These morphological differences increase the complexity of image recognition, especially during the dynamic changes of colonies. In addition, colonies grow from small scale to mature colonies, and their morphology and size change continuously, requiring the recognition algorithm to be able to monitor continuously. The presence of floating cells also increases the difficulty of recognition, which may be attached to the edge of the colony or cover the top of the colony, affecting the accurate recognition of the colony boundary and the clarity of the internal structure.

[0004] Traditional segmentation algorithms, such as threshold- and edge-detection-based methods, rely on obvious intensity changes in the image to determine boundaries. However, the edges of iPSCs colonies are often not completely clear or continuous, especially when the colonies are differentiated or cell death occurs, the edges will be more blurred and irregular. This algorithm often fails in the absence of obvious boundaries and cannot accurately segment the colonies. The existing CNN framework for iPSCs pluripotency assessment is limited by the local receptive field and has difficulty capturing long-range spatial dependencies. The Transformer-based model uses its self-attention mechanism to directly calculate the correlation between any two elements in the input sequence, which can effectively integrate global and local information and improve the model's ability to handle complex features and long-range dependencies in images. However, this method is prone to losing details when analyzing complex samples, such as multiple phenotypes, cell overlap, and blurred edges.

[0005] In view of this, it is necessary to provide an accurate method for evaluating the pluripotency of iPSCs colonies. Summary of the invention

[0006] The purpose of the present invention is to propose an iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion, establish a mapping relationship between stem cell pluripotency and cell colony morphology, prevent interference from human factors, and use Gabor edge feature enhancement module and hierarchical Transformer trunk feature extraction module to improve the recognition accuracy of cell fuzzy edges and details, so as to achieve the effect of high accuracy of PSCs colony pluripotency assessment.

[0007] To achieve this object, the present invention adopts the following technical solutions:

[0008] A method for evaluating the pluripotency of iPSCs colonies based on hierarchical Transformer attention feature fusion, comprising the following steps:

[0009] (1) Collect images during the iPSCs culture process and establish a mapping relationship between stem cell pluripotency and cell colony morphology;

[0010] (2) construct an iPSCs image dataset, correspond each pixel in the image to a pluripotency state category, and label it at the pixel level;

[0011] (3) Building an evaluation model based on Transformer attention feature fusion:

[0012] The Gabor edge feature enhancement module is used to detect the image edges in the iPSCs image and enhance the edge features, and output the feature layer F g ;

[0013] Hierarchical Transformer backbone feature extraction module, n Transformer backbone feature extraction layers extract features from iPSCs images in turn, and output feature layers F in turn 1 、F 2 …F k …and F n ;

[0014] Attention feature fusion module is used to g 、F 1 F 2 …F k …and F n Multiple attention features are fused separately, and multiple fusion feature layers are output;

[0015] Fusion feature layer and feature layer F k …and F n After upsampling and normalization, they are fused and then decoded to obtain the predicted image, i.e., the evaluation result;

[0016] (4) Train the evaluation model based on Transformer attention feature fusion using the iPSCs image dataset;

[0017] (5) The trained Transformer attention feature fusion-based evaluation model is used to evaluate the pluripotency of images during the iPSCs culture process.

[0018] Furthermore, in the step (1), live cell bright field images are collected during the iPSCs culture process, and the image areas are classified into background, single cell clusters, colonies, and impurities;

[0019] The different colony morphologies during iPSCs culture were obtained by immunofluorescence staining, the pluripotency status of iPSCs colonies was classified according to the colony morphology, and the mapping relationship between stem cell pluripotency and cell colony morphology was established.

[0020] Furthermore, the pluripotent expression proteins OCT-3 / 4 and SOX2 produced during the iPSCs culture process were fluorescently stained by immunofluorescence staining; the average immunofluorescence intensity of the iPSCs colony state was calculated using ImageJ, and the average fluorescence intensity of the colony = the total fluorescence intensity of the colony area / the area of ​​the colony area.

[0021] Furthermore, in step (2), the pixels in the image are divided into six categories: iPSCs single cell clusters, high-quality iPSCs colonies, medium-quality iPSCs colonies, low-quality iPSCs, impurities and background, and the pixels of each category are converted into one-hot encoding, that is, labeled at the pixel level.

[0022] Furthermore, in step (3), the process of processing the iPSCs image by the Gabor edge feature enhancement module is as follows:

[0023] The input image is convolved with a multi-directional Gabor filter, and then passes through a 1×1 convolution layer, a 3×3 convolution layer, a batch normalization layer, a ReLU activation function, and a residual connection to output the feature layer F. g .

[0024] Furthermore, in step (3), the input image is processed in the Transformer backbone feature extraction layer by the overlapping block fusion layer and then enters the Transformer module for processing, and the output feature layer F 1 ; Feature layer F 1 Input to the next Transformer backbone feature extraction layer for processing to obtain F k ;

[0025] The feature layer processing process of the Transformer module is as follows:

[0026] Input feature layer x i Normalized by the layerNorm layer, and then passed through the ESA layer to obtain the feature layer x i ′;

[0027] Through the residual structure, the feature layer x i ′ and feature layer x i Add together to get feature layer s i ;

[0028] Feature Layers i Normalized by LayerNorm layer, and then passed through Mix-FFN layer to get feature layer s i ′;

[0029] Feature Layers i ′ and feature layer s i Information fusion is achieved through the residual structure, and the feature layer x is obtained by addition. i+1 , feature layer x i+1 , which is the output feature layer of the Transformer backbone feature extraction layer.

[0030] Furthermore, in step (3), the attention feature fusion module performs the feature layer F h and feature layer F L The steps for fusion are:

[0031] Feature layer F h and feature layer F L The feature layer F is obtained by ReLU activation function, convolution and upsampling respectively. h ′ and feature layer FL ′;

[0032] Feature layer F h ′ and feature layer F L ′ are fused after convolution and then activated by Sigmoid function to obtain the similarity feature layer F map ;

[0033] The feature layer F h ′ and feature layer F L ′; respectively with the similarity feature layer F map Perform weighted fusion and output the fused feature layer.

[0034] Furthermore, in step (3), the hierarchical Transformer backbone feature extraction module has four Transformer backbone feature extraction layers, which output feature layers F 1 、F 2 、F 3 and F 4 ;

[0035] Feature layer F g After downsampling and feature layer F 1 Merge into feature layer F C , feature layer F C The feature layer F is obtained by one-dimensional convolution C1 , feature layer F C1 With feature layer F 3 Through the attention feature fusion module, the output feature layer F f1 ; Among them, the feature layer F 3 and feature layer F 4 is a low-resolution feature layer, and the feature layer F C1 and feature layer F 2 is the high-resolution feature layer;

[0036] Feature layer F 2 and feature layer F 4 The feature layer F is output through the attention feature fusion module. f2 ;

[0037] Feature layer F f1 , feature layer F f2 , feature layer F 3 and feature layer F 4 After upsampling, the feature layer is added after passing through the convolution layer and MLP layer to obtain the feature layer F a ;

[0038] Feature layer F a Decoding is performed through convolutional layers and upsampling to output a predicted image.

[0039] Furthermore, in step (4), the training step is:

[0040] Input the images of the iPSCs image dataset into the evaluation model based on Transformer attention feature fusion and output the predicted image;

[0041] Compare the prediction results with the pre-labeled categories at the pixel level and calculate the Focal Loss formula as follows:

[0042]

[0043] in, is the predicted output of the model, y is the true label, C is the number of categories, and p i is the probability of the i-th class predicted by the model, α i is the category weight, γ is the adjustment factor of Focal Loss;

[0044] Use Adamw optimizer to perform error back propagation, calculate gradients and update model parameters θ. The update formula is: Where η is the learning rate, is the loss function L f The gradient of the parameter θ, λ is the weight decay coefficient.

[0045] A computer-readable storage medium storing computer-executable instructions for executing the above-mentioned iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion.

[0046] The technical solution provided by the present invention may include the following beneficial effects:

[0047] 1. In the present invention, by establishing a mapping relationship between stem cell pluripotency and cell colony morphology, interference from human factors is prevented; by corresponding pixels in the image to pluripotency status categories and marking them, accurate evaluation of the model is facilitated;

[0048] 2. The Gabor edge feature enhancement module and the hierarchical Transformer backbone feature extraction module form a powerful feature extraction network that can accurately locate and identify the fine structures of colonies and cells in complex cell images; and by applying Gabor filters in different directions, the model can more comprehensively perceive the texture and edge structure information of colonies of different sizes and qualities in the image.

[0049] 3. At the same time, based on the hierarchical Transformer backbone feature extraction module, the attention feature fusion module is introduced to enable the fusion of high-level semantic features in low-level features, thereby achieving better model segmentation performance. The model in the present invention can effectively identify various scale targets in iPSCs living cell bright field images, from large colonies to tiny single cells, and achieve accurate detection of cell colonies of different scales and morphologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flowchart of an iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion according to an embodiment of the present invention;

[0051] Figure 2 is the image region classification map ( Figure 2 a) and the expression levels of OCT-3 / 4 and SOX2 in fluorescence intensity analysis ( Figure 2 b);

[0052] Figure 3 It is a schematic diagram of the Gabor feature enhancement module structure;

[0053] Figure 4 It is a schematic diagram of the Transformer module structure;

[0054] Figure 5 This is a schematic diagram of the attention feature fusion module structure;

[0055] Figure 6 This is an evaluation model diagram based on Transformer attention feature fusion;

[0056] Figure 7 It is the ablation study vs. prediction map;

[0057] Figure 8 It is a performance bar chart of the model of the present invention for six categories under different evaluation indicators. DETAILED DESCRIPTION

[0058] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0059] Combine the following Figures 1 to 8 , describing the iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion according to an embodiment of the present invention.

[0060] The iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion in this embodiment includes the following steps:

[0061] (1) Collect images during the iPSCs culture process and establish a mapping relationship between stem cell pluripotency and cell colony morphology;

[0062] (2) construct an iPSCs image dataset, correspond each pixel in the image to a pluripotency state category, and label it at the pixel level;

[0063] (3) Building an evaluation model based on Transformer attention feature fusion:

[0064] The Gabor edge feature enhancement module is used to detect the image edges in the iPSCs image and enhance the edge features, and output the feature layer F g ;

[0065] Hierarchical Transformer backbone feature extraction module, n Transformer backbone feature extraction layers extract features from iPSCs images in turn, and output feature layers F in turn 1 、F 2 …F k …and F n ;

[0066] Attention feature fusion module is used to g 、F 1 F 2 …F k …and F n Multiple attention features are fused separately, and multiple fusion feature layers are output;

[0067] Fusion feature layer and feature layer F k …and F n After upsampling and normalization, they are fused and then decoded to obtain the predicted image, i.e., the evaluation result;

[0068] (4) Train the evaluation model based on Transformer attention feature fusion using the iPSCs image dataset;

[0069] (5) The trained Transformer attention feature fusion-based evaluation model is used to evaluate the pluripotency of images during the iPSCs culture process.

[0070] In the present invention, by establishing a mapping relationship between stem cell pluripotency and cell colony morphology, interference from human factors is prevented; by corresponding the pluripotency state category of pixels in the image and marking them, it is conducive to accurate evaluation of the model; a powerful feature extraction network is formed with the Gabor edge feature enhancement module and the hierarchical Transformer trunk feature extraction module, which can accurately locate and identify the fine structure of colonies and cells in complex cell images; at the same time, based on the hierarchical Transformer trunk feature extraction module, the attention feature fusion module is introduced to enable the fusion of high-level semantic features in low-level features, so as to achieve better model segmentation performance. The model in the present invention can effectively identify various scale targets in iPSCs bright field images, from large colonies to tiny single cells, and realizes accurate detection of cell colonies of different scales and morphologies.

[0071] In one embodiment of the present invention, in the step (1), a bright field image of living cells during the iPSCs culture process is collected, and the image area is classified into background, single cell clusters, colonies, and impurities;

[0072] The different colony morphologies during iPSCs culture were obtained by immunofluorescence staining, the pluripotency status of iPSCs colonies was classified according to the colony morphology, and the mapping relationship between stem cell pluripotency and cell colony morphology was established.

[0073] Specifically, the global live cell bright field image is used as the domain, and the image area is classified into background, single cell clusters, colonies, and impurities, including cell debris and dead cells. iPSCs are continuously passaged until the phenomenon of spontaneous differentiation of colonies is observed, thereby collecting a large number of images with pluripotency ranging from high to low. Further specifically, the culture cycle of iPSCs is generally 7 days, and within two days after cell inoculation, iPSCs appear as single cells or clusters. Under normal culture conditions, iPSCs on the 5th day after inoculation gradually present a colony state with densely arranged cells and clear edges. In order to better evaluate the colony state, we collected live cell bright field images on days 5-7. Exemplarily, an incubator microscope (Olympus IX83) was used to image iPSCs for live cells. The image channel is set to bright field, and the objective magnification is 10x. The resolution of the acquired bright field image is 1920x1200, and the format is TIFF. In the embodiment of the present invention, the pluripotency of iPSCs colonies is divided into three categories of "high", "medium" and "low" quality, and a mapping relationship between stem cell pluripotency and cell colony morphology is established.

[0074] iPSCs are characterized by their ability to self-renew indefinitely and produce all cell types of an organism. OCT-3 / 4 and SOX2 are two essential transcription factors that co-regulate target genes to maintain the pluripotency of iPSCs. Studies have shown that as iPSCs differentiation continues, the expression levels of OCT-3 / 4 and SOX2 decrease. The phenotypic differences of iPSCs colonies during culture are related to the expression of pluripotency markers. In one embodiment of the present invention, the pluripotent expression proteins OCT-3 / 4 and SOX2 produced during the iPSCs culture process are fluorescently stained by immunofluorescence staining, and the fluorescence intensity can accurately characterize the pluripotency state; ImageJ is used to calculate the average immunofluorescence intensity of the iPSCs colony state, and the iPSCs colony quality is divided into "high", "medium" and "low" to achieve a mapping relationship between the pluripotency and morphology of the iPSCs colony. Among them, the average fluorescence intensity of the colony = the sum of the fluorescence intensity of the colony area / the area of ​​the colony area.

[0075] Reference Figure 2 , exemplary, high-quality iPSCs colonies show uniform and strong OCT-3 / 4 and SOX2 fluorescence signals, with high fluorescence intensity, the average fluorescence intensity of OCT-3 / 4 is 25-35, and the average fluorescence intensity of SOX2 is 40-50, indicating that the cells in the colony maintain a good pluripotency state. The morphological characteristics of high-quality iPSCs colonies are: completely undifferentiated inside, morphologically manifested as: clear colony edges, uniform and tight arrangement of cells inside the colony, and cells with a high nuclear-cytoplasmic ratio.

[0076] In medium-quality iPSCs colonies, the fluorescence signals of OCT3 / 4 and SOX2 may vary to a certain extent, and some areas may show weaker fluorescence intensity. The average fluorescence intensity of OCT-3 / 4 is 15-25, and the average fluorescence intensity of SOX2 is 20-30. This may indicate that the pluripotency state of cells in these colonies is not completely uniform, and some cells may begin to lose pluripotency. The morphological characteristics of medium-quality iPSCs colonies are: some cells in the colony are differentiated, the edges of the colony are blurred, the cells are loosely stacked, and there is a bright bright field halo between cells.

[0077] Low-quality colonies may show lower and uneven OCT-3 / 4 and SOX2 fluorescence intensities, indicating that the pluripotency of the cells has been significantly affected. The average fluorescence intensity of OCT-3 / 4 of low-quality iPSCs colonies is between 5 and 15, and the average fluorescence intensity of SOX2 is between 10 and 20. The cells in these colonies may have begun to differentiate, and the state of iPSCs colonies is unstable. The morphological characteristics of medium-quality iPSCs colonies are: the cells inside the colonies are almost completely differentiated, cavities appear, and they show a fibroblastic cell morphology.

[0078] Furthermore, in step (2), the pixels in the image are divided into six categories: iPSCs single cell clusters (S), high-quality iPSCs colonies, medium-quality iPSCs colonies, low-quality iPSCs, impurities and background, and the pixels of each category are converted into one-hot encoding, that is, the pixel level is labeled. Specifically, the iPSCs single cell cluster is a cell in the early state, and each pixel in the prepared label is converted into a one-hot encoding, that is, the background pixel value is set to 0, and the RGB color mapping is (255, 255, 255). The pixel value of the iPSCs single cell cluster is set to 1, and the RGB color mapping is (6, 6, 6). The pixel value of the low-quality iPSCs colony is set to 2, and the RGB color mapping is (191, 191, 191). The pixel value of the medium-quality iPSCs colony is set to 3, and the RGB color mapping is (118, 113, 113). The pixel value of the high-quality iPSCs colony is set to 4, and the RGB color mapping is (59, 56, 56). The pixel value of other categories is set to 5, and the RGB color map is (218, 218, 218).

[0079] Reference Figure 3 The Gabor edge feature enhancement module in one embodiment of the present invention is composed of a Gabor filter, a 1×1 convolution layer, a 3×3 convolution layer, a batch normalization layer, a ReLU activation function and a residual connection. In the step (3), the Gabor edge feature enhancement module processes the iPSCs image as follows: the input image is convolved with a multi-directional Gabor filter, and then passes through a 1×1 convolution layer, a 3×3 convolution layer, a batch normalization layer, a ReLU activation function and a residual connection, and the output feature layer F g .

[0080] Gabor's 2D filter is used to extract edge texture features of an image by convolving the image with the Gabor kernel defined in the following formula:

[0081]

[0082] Where x′=xcosθ+ysinθ, y′=-xcosθ+ysinθ. x and y represent the pixel position in the spatial domain, θ is the direction of the Gabor filter, is the phase shift, σ is the standard deviation of the Gaussian function, λ represents the wavelength, and γ is the spatial aspect ratio.

[0083] The multi-directional Gabor filter in this technical solution helps the model capture features in different directions in the image, thereby improving the performance of the image segmentation task. By applying Gabor filters in different directions, the model can more comprehensively perceive the texture and edge structure information of colonies of different sizes and qualities in the image.

[0084] For example, the input image size is 512×512×3 (ie, height 512, width 512, number of channels 3), and the processing steps of the Gabor edge feature enhancement module are:

[0085] S310, the passing direction θ is 0, The output feature layer is 512×512×12. The large convolution kernel size of the filter is set to 15, and the parameters are σ, λ, and γ are 0, 4, 5, and 0.5, respectively;

[0086] S311, passing the output 512×512×12 feature layer through a 1x1 convolution layer, converting the number of output channels of the Gabor filter from the input channel number (12) to the specified output channel number (64), and the output feature layer size is 512×512×64;

[0087] S312, the feature layer with a size of 64 is further extracted through a 3x3 convolution layer, and then through a batch normalization layer and a ReLU activation function, the output feature layer size is 512×512×64;

[0088] The feature layers output by S313 and S311 are added to the feature layers output by S312 through residual connection to output the feature layer Fg.

[0089] In one embodiment of the present invention, in step (3), the input image is processed in the Transformer backbone feature extraction layer by the overlapping block fusion layer and then enters the Transformer module for processing, and the output feature layer F 1 ; Feature layer F 1 Input to the next Transformer backbone feature extraction layer for processing to obtain F k . The hierarchical Transformer backbone feature extraction module effectively captures contextual information of different scales and ranges in the image through its hierarchical architecture, and uses the self-attention mechanism to strengthen the association between local features and global features, thereby improving the semantic understanding of each region in the image. This structure not only enhances the model's ability to capture long-distance dependent features, but also speeds up the calculation speed through parallel processing, so that the model can more accurately identify and segment different semantic regions when processing images with complex structures and rich details, thereby improving the performance and accuracy of semantic segmentation. The processing steps of the overlapping block fusion layer are used to extract and synthesize local and global information of the image.

[0090] Reference Figure 4 , the feature layer processing process of the Transformer module is:

[0091] S3201, input feature layer x i After normalization through the layerNorm layer, the feature layer x is obtained through the ESA layer (Efficient self-attention layer) i ′;

[0092] S3202, through the residual structure, the feature layer x i ′ and feature layer x i Add together to get feature layer s i ;

[0093] S3203, feature layer s i Normalized by LayerNorm layer, and then passed through Mix-FFN layer to get feature layer s i ′;

[0094] S3204, feature layer s i ′ and feature layer s i Information fusion is achieved through the residual structure, and the feature layer x is obtained by addition. i+1 , feature layer x i+1 , which is the output feature layer of the Transformer backbone feature extraction layer.

[0095] The detailed calculation process of the input features through the Transformer module can be expressed as follows:

[0096] s i =ESA(LN(x i ))+x i

[0097] x i+1 =MFFN(LN(s i ))+s i

[0098] Among them, LN represents layer normalization (LayerNorm).

[0099] Exemplarily, the attention mechanism obtains weight coefficients by calculating the correlation between the query and each key, and uses these coefficients to perform weighted summation on the corresponding values ​​to obtain the attention value. The self-attention mechanism refers to the attention and utilization process of the query, key, and value at the same position. Its main function is to simulate the detailed interaction between pixels, and can capture the interaction between any positions on the image feature layer without distance restrictions, so that the Transformer model can model long-range dependencies in visual tasks very flexibly. The formula for self-attention is as follows:

[0100]

[0101] where the matrices Q, K, and V correspond to the query, key, and value, respectively, σ is the softmax function, T represents the transpose, and d k is the dimension of K. In order to minimize the number of parameters while maintaining the basic feature extraction capability of the transformer, the ESA layer introduces a reduction ratio R, which will be shown as follows:

[0102]

[0103] Among them, K represents the input sequence, which consists of Q, K and V, N is the length of the sequence, and C is the channel dimension. First, convert the shape of K from (N×C) to Then, the shape of K is changed from Convert to This method reduces the computational complexity from O(N 2 ) to

[0104] The Mix-FFN layer contains a linear layer, a residual module, and a nonlinear activation function GELU. The position information is "leaked" by directly using 3×3 convolution in FFN. This design allows the model to dynamically express the position relationship between patches instead of relying on fixed position encodings learned during training. This design not only reduces the model's dependence on position encoding, but also improves the model's adaptability to inputs of different resolutions. The calculation process of the Mix-FFN layer is as follows:

[0105] MFFN(X)=Conv 1 (GELU(Conv 3 (Conv 1 (X))))+X

[0106] Where X is the feature from the ESA layer, Conv 1 represents a point-by-point convolution with a stride of 1, a kernel size of 1, and padding of 0, while Conv 3 represents a depthwise convolution with stride 2, kernel size 3, and padding 1.

[0107] For example, refer to Figure 6 , the hierarchical Transformer backbone feature extraction module has four Transformer backbone feature extraction layers, and the processing steps are:

[0108] S320, first pass the 512×512×3 input image through the overlapping block fusion layer 1. The specific operation of this step is to divide the image into blocks and embed it into the specified dimension through 2D convolution operation. Its convolution kernel size is 7×7, the step size is 4, the padding size is 3, and the number of convolution kernels is 64. The input feature layer is straightened to a size of 16384×64 through the overlapping block fusion layer 1.

[0109] S321, then the 16384×64 vector is processed through Transformer Block 1 to process the feature sequence, and the output size is 128×128×64 feature layer F 1 Transformer Block 1 is the Transformer module. Set the scaling factor R of the self-attention mechanism to 8, the number of heads N of the self-attention mechanism to 1, the expansion rate E of the Mix-FFN layer to 8, and the number of encoders (ESA+MixFFN) L in each Transformer Block to 3.

[0110] S322, feature layer F 1 First, the image is passed through the overlapping block fusion layer 2 to reduce the resolution of the image, which plays a role of downsampling. Its convolution kernel size is 3×3, the step size is 2, the padding size is 1, and the number of convolution kernels is 128. Then, after TransformerBlock 2, the output size of the feature layer F is 64×64×128. 2 . R, N, E, and L are set to 4, 2, 8, and 3 respectively.

[0111] S323, feature layer F 2 First, the image is passed through the overlapping block fusion layer 3 to reduce the resolution of the image, which plays a role of downsampling. Its convolution kernel size is 3×3, the step size is 2, the padding size is 1, and the number of convolution kernels is 320. Then, after TransformerBlock 3, the output size of the feature layer F is 32×32×320. 3 . R, N, E, and L are set to 2, 5, 4, and 18 respectively.

[0112] S324, feature layer F 3 First, the image is passed through the overlapping block fusion layer 4 to reduce the resolution of the image, which plays a role of downsampling. Its convolution kernel size is 3×3, the step size is 2, the padding size is 1, and the number of convolution kernels is 512. Then, after TransformerBlock 4, the output size of the feature layer F is 16×16×512. 4 . R, N, E, and L are set to 1, 8, 4, and 3 respectively.

[0113] Further, see Figure 5 In step (3), the attention feature fusion module performs the feature layer F h and feature layer F L The steps for fusion are:

[0114] S330, feature layer F h and feature layer F L The feature layer F is obtained by ReLU activation function, convolution and upsampling respectively. h ′ and feature layer FL ′;

[0115] S331, feature layer F h ′ and feature layer F L ′ are fused after convolution and then activated by Sigmoid function to obtain the similarity feature layer F map ;

[0116] S332, feature layer F h ′ and feature layer F L ′; respectively with the similarity feature layer F map Perform weighted fusion and output the fused feature layer.

[0117] Furthermore, in step (3), the hierarchical Transformer backbone feature extraction module has four Transformer backbone feature extraction layers, which output feature layers F 1 、F 2 、F 3 and F 4 ;

[0118] Feature layer F g After downsampling and feature layer F 1 Merge into feature layer F C , feature layer F C The feature layer F is obtained by one-dimensional convolution C1 , feature layer F C1 With feature layer F 3 Through the attention feature fusion module, the output feature layer F f1 ; Among them, the feature layer F 3 and feature layer F 4 is a low-resolution feature layer, and the feature layer F C1 and feature layer F 2 is the high-resolution feature layer;

[0119] Feature layer F 2 and feature layer F 4 The feature layer F is output through the attention feature fusion module. f2 ;

[0120] Feature layer F f1 , feature layer F f2 , feature layer F 3 and feature layer F 4 After upsampling, the feature layer is added after passing through the convolution layer and MLP layer to obtain the feature layer F a ;

[0121] Feature layer F a Decoding is performed through convolutional layers and upsampling to output a predicted image.

[0122] In the multi-scale backbone feature extraction module, the low-resolution feature layer F 3 and F 4 The rich and accurate semantic information provided by the high-resolution feature layer F c1 and F 2 The detail parsing and boundary detection of F are crucial, and the low-resolution feature layers contain relatively few layers and channels. 3 and F 4 Considered as high-resolution feature F c1 and F 2 The high-resolution feature layer can selectively learn useful semantic features from the low-resolution feature layer. The vectors of the corresponding pixels in the feature layers of the high-resolution feature branch and the low-resolution feature branch are defined as and Then the output of the Sigmoid function can be expressed as:

[0123]

[0124] Where σ represents the probability that the two pixels belong to the same object, f h and f l represents a 1x1 convolution operation. If σ is high, we trust Because low-resolution features are semantically rich and accurate, and vice versa. Therefore, the output of the feature fusion module can be written as:

[0125]

[0126] F c1 and F 3 Take the fusion process as an example, refer to Figure 6 :

[0127] S330, feature layer F c1 The size is 128×128×64, the feature layer F 3 The size is 32×32×320. Feature layer F c1 The feature layer F with a size of 128×128×64 is obtained by ReLU activation function c1 ′. Feature layer F 3 Through ReLU activation function, 1×1 convolution, and upsampling, we get a feature layer F with a size of 128×128×64. 3 ′.

[0128] S331, feature layer F c1 ′ and F 3'Through 1×1 convolution, the number of channels is mapped to the number of intermediate channels 32, and the feature layer F with a size of 128×128×32 is obtained. c1 " and F 3 ";

[0129] The mapped feature layer F c1 " and F 3 "The elements corresponding to each channel in the matrix are multiplied to obtain a feature size of 128×128×32. Then, the 128×128×32 feature layer is summed in the channel dimension and passed through the Sigmoid activation function to obtain a feature layer F with a size of 128×128×1. map . F map Represents the similarity feature layer. This process can be regarded as the dot product of the vectors of corresponding pixels in the two feature layers (the corresponding elements of each pixel position in the two feature layers are multiplied and summed), and then σ is obtained through the Sigmoid function. σ represents the probability that the two pixels belong to the same target. Similarity feature layer F map It can also be viewed as a probability matrix consisting of σ obtained at each pixel position.

[0130] S332, based on the calculated similarity feature layer F map , the two input feature layers F c1 ′ and F 3 ′ performs weighted fusion to obtain the output feature F of the attention feature fusion module f1 , whose size is 128×128×64. σ represents the probability that these two pixels belong to the same object. Similarity feature layer F map It can be seen as a probability matrix composed of σ obtained at each pixel position. When the probability σ is high, we trust it more. Because low-resolution features are semantically rich and accurate, and vice versa.

[0131] Feature F 2 and F 4 The detailed fusion process of is the same as S1-S4. The output feature F of the attention feature fusion module is f2 The dimensions are 64×64×128.

[0132] Furthermore, in step (4), the training step is:

[0133] S401. Input the image of the iPSCs image dataset into the evaluation model based on Transformer attention feature fusion, and output the predicted image.

[0134] S402, compare the prediction result with the pre-labeled category at pixel level, and calculate the formula of Focal Loss as follows:

[0135]

[0136] in, is the predicted output of the model, y is the true label, C is the number of categories, and p i is the probability of the i-th class predicted by the model, α i is the category weight, and γ is the adjustment factor of Focal Loss; γ is used to reduce the weight of easy-to-classify samples, thereby paying more attention to difficult-to-classify samples.

[0137] S403, use Adamw optimizer to perform error back propagation, calculate gradient and update model parameter θ, the update formula is Where η is the learning rate, is the loss function L f The gradient of the parameter θ, λ is the weight decay coefficient. The AdamW optimizer also considers weight decay when updating parameters to prevent overfitting. The weight decay is set to 1e-2. This process is iterated until the model converges and finally a trained segmentation network is obtained.

[0138] Figure 7 For the ablation study and comparison of prediction graphs, the images Image1-4 are respectively input into the model of the present invention (Ours), the Segformer baseline model (Baseline), the model after adding the Gabor edge feature enhancement module to the Segformer baseline model (+Gabo), and the model after adding the attention feature fusion module to the Segformer baseline model (+Pag). The output prediction results are as follows: Figure 7 As shown, Figure 7 GT is the true result. It can be seen that the prediction result obtained by the model of the present invention has the highest similarity with the true result. The evaluation results of the ablation study are shown in the following table.

[0139]

[0140] According to the above table, when only the Gabor feature enhancement module is removed, the model performance has a significant decline, and the indicators mIoU, mPrecision, mRecall and F1-score have dropped by 2.81%, 1.06%, 2.74%, and 1.91% respectively. When only the attention feature fusion module (Pag) is removed, the model has a decline of 1.78.%, 0.29%, and 1.58% in mIoU, mPrecision and mRecall respectively. When the Gabor feature enhancement module and the attention feature fusion module (Pag) are removed, that is, when the Baseline is used, the model has the largest performance decline. This proves that in the constructed model, each module is indispensable. The attention feature fusion module (Pag) and the Gabor feature enhancement module have improved the segmentation performance of the model to a certain extent, and the two work together to greatly improve the detection accuracy of the model.

[0141] like Figure 8 The performance of the method of the present invention for each category of iPSCs colony pluripotency under different evaluation indicators was demonstrated. The prediction accuracy of high, medium and low quality iPSC colonies was averaged, and the results showed that the prediction accuracy of the model of the present invention for different iPSC states reached 98.6%.

[0142] Correspondingly, the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to execute the above-mentioned iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion.

[0143] Other structures and operations of the iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion according to an embodiment of the present invention are known to ordinary technicians in the field and will not be described in detail here.

[0144] In the description of this specification, the description with reference to the terms "embodiment", "example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0145] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for evaluating the pluripotency of iPSCs colonies based on hierarchical Transformer attention feature fusion, characterized in that: The following steps are involved: (1) Collect images during the iPSCs culture process and establish a mapping relationship between stem cell pluripotency and cell colony morphology; (2) construct an iPSCs image dataset, correspond each pixel in the image to a pluripotency state category, and label it at the pixel level; (3) Building an evaluation model based on Transformer attention feature fusion: The Gabor edge feature enhancement module is used to detect the image edges in the iPSCs image and enhance the edge features, and output the feature layer F g ; Hierarchical Transformer backbone feature extraction module, n Transformer backbone feature extraction layers extract features from iPSCs images in turn, and output feature layers F1, F2…F in turn k …and F n ; Attention feature fusion module is used to g 、F1 F2…F k …and F n Multiple attention features are fused separately, and multiple fusion feature layers are output; Fusion feature layer and feature layer F k …and F n After upsampling and normalization, they are fused and then decoded to obtain the predicted image, i.e., the evaluation result; (4) Train the evaluation model based on Transformer attention feature fusion using the iPSCs image dataset; (5) The trained Transformer attention feature fusion-based evaluation model is used to evaluate the pluripotency of images during the iPSCs culture process.

2. The method according to claim 1, characterized in that In the step (1), a bright field image of living cells during the iPSCs culture process is collected, and the image area is classified into background, single cell clusters, colonies, and impurities; The different colony morphologies during iPSCs culture were obtained by immunofluorescence staining, the pluripotency status of iPSCs colonies was classified according to the colony morphology, and the mapping relationship between stem cell pluripotency and cell colony morphology was established.

3. The method according to claim 2, characterized in that The pluripotent expression proteins OCT-3 / 4 and SOX2 produced during the iPSCs culture process were fluorescently stained by immunofluorescence staining; the average immunofluorescence intensity of the iPSCs colony state was calculated using ImageJ, and the average fluorescence intensity of the colony = the total fluorescence intensity of the colony area / the area of ​​the colony area.

4. The method according to claim 2, characterized in that: In the step (2), the pixels in the image are divided into six categories: iPSCs single cell clusters, high-quality iPSCs colonies, medium-quality iPSCs colonies, low-quality iPSCs, impurities and background, and the pixels of each category are converted into one-hot encoding, that is, labeled pixel level.

5. The method according to claim 1, characterized in that In the step (3), the process of processing the iPSCs image by the Gabor edge feature enhancement module is as follows: The input image is convolved with a multi-directional Gabor filter, and then passes through a 1×1 convolution layer, a 3×3 convolution layer, a batch normalization layer, a ReLU activation function and a residual connection to output the feature layer Fg.

6. The method according to claim 1, characterized in that In the step (3), the input image is processed by the overlapping block fusion layer in the Transformer backbone feature extraction layer and then enters the Transformer module for processing, outputting the feature layer F1; the feature layer F1 is input into the next Transformer backbone feature extraction layer for processing to obtain F k ; The feature layer processing process of the Transformer module is as follows: Input feature layer x i Normalized by the layerNorm layer, and then passed through the ESA layer to obtain the feature layer x i ′; Through the residual structure, the feature layer x i ′ and feature layer x i Add together to get feature layer s i ; Feature Layers i Normalized by LayerNorm layer, and then passed through Mix-FFN layer to get feature layer s i ′; Feature Layers i ′ and feature layer s i Information fusion is achieved through the residual structure, and the feature layer x is obtained by addition. i+1 , feature layer x i+1 , which is the output feature layer of the Transformer backbone feature extraction layer.

7. The method according to claim 1, characterized in that In step (3), the attention feature fusion module performs h and feature layer F L The steps for fusion are: Feature layer F h and feature layer F L The feature layer F is obtained by ReLU activation function, convolution and upsampling respectively. h ′ and feature layer F L ′; Feature layer F h ′ and feature layer F L ′ are fused after convolution and then activated by Sigmoid function to obtain the similarity feature layer F map ; The feature layer F h ′ and feature layer F L ′; respectively with the similarity feature layer F map Perform weighted fusion and output the fused feature layer.

8. The method according to claim 1, characterized in that In the step (3), the hierarchical Transformer backbone feature extraction module has four Transformer backbone feature extraction layers, and outputs feature layers F1, F2, F3 and F4 in sequence; Feature layer F g After downsampling, it is merged with feature layer F1 to form feature layer F C , feature layer F C The feature layer F is obtained by one-dimensional convolution C1 , feature layer F C1 The feature layer F3 is fused with the attention feature fusion module to output the feature layer F f1 ; Among them, feature layer F3 and feature layer F4 are low-resolution feature layers, and feature layer F C1 and feature layer F2 is a high-resolution feature layer; The feature layer F2 and the feature layer F4 are fused through the attention feature fusion module to output the feature layer F f2 ; Feature layer F f1 , feature layer F f2 , feature layer F3 and feature layer F4 are first upsampled, and then passed through the convolution layer and MLP layer, and the feature layers are added to obtain feature layer F a ; Feature layer F a Decoding is performed through convolutional layers and upsampling to output a predicted image.

9. The method according to claim 1, characterized in that: In step (4), the training step is: Input the images of the iPSCs image dataset into the evaluation model based on Transformer attention feature fusion and output the predicted image; Compare the prediction results with the pre-labeled categories at the pixel level and calculate the Focal Loss formula as follows: in, is the predicted output of the model, y is the true label, C is the number of categories, and p i is the probability of the i-th class predicted by the model, α i is the category weight, γ is the adjustment factor of Focal Loss; Use Adamw optimizer to perform error back propagation, calculate gradients and update model parameters θ. The update formula is: Where η is the learning rate, is the loss function L f The gradient of the parameter θ, λ is the weight decay coefficient.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion described in any one of claims 1-8.

Images