Method for evaluating pluripotency of ipscs colony based on hierarchical transformer attention feature fusion

Abstract

The present application relates to the technical field of iPSCs colony pluripotency evaluation, and particularly relates to an iPSCs colony pluripotency evaluation method based on hierarchical Transformer attention feature fusion, comprising the following steps: collecting images in the iPSCs culture process, and establishing a mapping relationship between stem cell pluripotency and cell colony morphology; constructing an iPSCs image dataset, corresponding pluripotency state categories to each pixel in the image, and marking the pixel level; building an evaluation model based on Transformer attention feature fusion, including a Gabor edge feature enhancement module, a hierarchical Transformer backbone feature extraction module and an attention feature fusion module; the obtained feature layers are fused after being up-sampled and normalized, and then the prediction image, i.e. the evaluation result, is obtained after decoding; the model is trained with the iPSCs image dataset; and the pluripotency of the images in the iPSCs culture process is evaluated with the trained model. The method of the present application can prevent human factor interference, and has high iPSCs colony pluripotency evaluation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of iPSCs colony pluripotency assessment, and in particular to an iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion. Background Art

[0002] Induced pluripotent stem cells (iPSCs) are derived from adult cells through reprogramming techniques and possess the ability to continuously self-renew and differentiate into a variety of cell types. They hold great promise for treating chronic and degenerative diseases, including Parkinson's disease, diabetes, heart disease, and spinal cord injury. However, the large numbers of cells required for these applications rely on large-scale expansion of iPSCs. During expansion, genetic and epigenetic variations in iPSC colonies can affect their differentiation capacity and quality, thereby impacting the efficiency of directed differentiation for disease modeling or therapeutic applications. Furthermore, subtle variations in routine culture can also lead to variability in iPSC colonies. Therefore, rigorous quality control and assessment of iPSC colonies during the culture phase are crucial to ensure their safety and efficacy in clinical applications. Unlike general cell quality assessment criteria, stem cell pluripotency—the potential to differentiate into multiple cell types—is a key quality criterion. However, unlike phenotypic assessments such as morphology, cell viability, and colony contamination, direct observation of stem cell pluripotency is difficult to perform.

[0003] Currently, the mapping between stem cell pluripotency and morphology has not been clearly established, and classification often relies on a priori classification based on the experience of professionals. Consequently, assessment results are inevitably subject to artifacts, leading to significant variability in assessment results for strains derived from the same method across different laboratories, limiting its standardized and quantitative application for quality assessment. Furthermore, live-cell imaging during iPSC culture presents multiple complex challenges in identifying and monitoring iPSC colonies of varying pluripotency. High-quality colonies typically exhibit rounded edges, densely packed cells, and the absence of cavities, while low-quality colonies may exhibit blurred edges, sparse cells, and internal cavities. These morphological differences complicate image recognition, especially during the dynamic evolution of colonies. Furthermore, as colonies grow from small to mature, their morphology and size continually change, requiring continuous monitoring by recognition algorithms. The presence of floating cells also complicates identification, potentially adhering to the edges of colonies or overlying them, compromising the accurate identification of colony boundaries and the clarity of internal structures.

[0004] Traditional segmentation algorithms, such as threshold-based and edge detection-based methods, rely on obvious intensity changes in the image to determine boundaries. However, the edges of iPSCs colonies are often not completely clear or continuous, especially when the colonies appear to be differentiating or cells are dying, the edges will be more blurred and irregular. Such algorithms often fail in the absence of obvious boundaries, and cannot accurately segment the colonies. Existing CNN frameworks for iPSCs pluripotency evaluation are limited by local receptive fields, making it difficult to capture spatial long-range dependencies. Transformer-based models use their self-attention mechanism to directly calculate the correlation between any two elements in the input sequence, effectively integrating global and local information, and improving the model's ability to handle complex features and long-range dependencies in images. However, this method can easily lose details when analyzing complex samples, such as multiple phenotypes, cell overlap, and blurred edges.

[0005] Therefore, it is necessary to provide an accurate iPSCs colony pluripotency evaluation method. SUMMARY

[0006] The purpose of the present application is to provide an iPSCs colony pluripotency evaluation method based on hierarchical Transformer attention feature fusion, to establish a mapping relationship between stem cell pluripotency and cell colony morphology, prevent human factor interference, and improve the recognition accuracy of cell blurred edges and details using Gabor edge feature enhancement module and hierarchical Transformer backbone feature extraction module, to achieve the effect of high PSCs colony pluripotency evaluation accuracy.

[0007] To achieve this purpose, the present application adopts the following technical solutions:

[0008] An iPSCs colony pluripotency evaluation method based on hierarchical Transformer attention feature fusion, comprising the following steps:

[0009] (1) Collecting images during the cultivation of iPSCs, establishing a mapping relationship between stem cell pluripotency and cell colony morphology;

[0010] (2) Constructing an iPSCs image dataset, corresponding the pluripotency state category to each pixel in the image, and marking the pixel level;

[0011] (3) Building an evaluation model based on Transformer attention feature fusion:

[0012] A Gabor edge feature enhancement module is used to detect and enhance the edge features of the image edges in the iPSCs image, and output the feature layer F g ;

[0013] Hierarchical Transformer backbone feature extraction module, n Transformer backbone feature extraction layers extract features from iPSCs images in sequence, and output feature layers F1, F2…F in sequence k ...and F n ;

[0014] Attention feature fusion module is used to g 、F1 F2…F k ...and F n Multiple attention features are fused separately, and multiple fusion feature layers are output;

[0015] Fusion feature layer and feature layer F k ...and F n After upsampling and normalization, they are fused and then decoded to obtain the predicted image, which is the evaluation result;

[0016] (4) Training the evaluation model based on Transformer attention feature fusion using the iPSCs image dataset;

[0017] (5) The trained evaluation model based on Transformer attention feature fusion is used to evaluate the pluripotency of images during the iPSCs culture process.

[0018] Furthermore, in step (1), bright field images of living cells during the iPSCs culture process are collected, and the image areas are classified into background, single cell clusters, colonies, and impurities;

[0019] Immunofluorescence staining was used to obtain the different colony morphologies during iPSCs culture, and the pluripotency status of iPSCs colonies was classified according to the colony morphology, and a mapping relationship between stem cell pluripotency and cell colony morphology was established.

[0020] Furthermore, the pluripotent expression proteins OCT-3 / 4 and SOX2 produced during the iPSCs culture process were fluorescently stained by immunofluorescence staining; the average immunofluorescence intensity of the iPSCs colony status was calculated using ImageJ, and the average fluorescence intensity of the colony = the total fluorescence intensity of the colony area / the area of ​​the colony area.

[0021] Furthermore, in step (2), the pixels in the image are divided into six categories: iPSCs single cell clusters, high-quality iPSCs colonies, medium-quality iPSCs colonies, low-quality iPSCs, impurities and background, and the pixels of each category are converted into one-hot encoding, that is, labeled at the pixel level.

[0022] Furthermore, in step (3), the Gabor edge feature enhancement module processes the iPSCs image as follows:

[0023] The input image is convolved with a multi-directional Gabor filter, and then passes through a 1×1 convolution layer, a 3×3 convolution layer, a batch normalization layer, a ReLU activation function, and a residual connection to output the feature layer F. g .

[0024] Furthermore, in step (3), the input image is processed in the Transformer backbone feature extraction layer through the overlapping block fusion layer and then enters the Transformer module for processing, outputting the feature layer F1; the feature layer F1 is input into the next Transformer backbone feature extraction layer for processing to obtain F k ;

[0025] The feature layer processing process of the Transformer module is as follows:

[0026] Input feature layer x i Normalized by the layerNorm layer, and then passed through the ESA layer to obtain the feature layer x i ';

[0027] Through the residual structure, the feature layer x i ′ and feature layer x i Add together to get the feature layer s i ;

[0028] Feature layers i Normalized by the LayerNorm layer and then passed through the Mix-FFN layer to obtain the feature layer s i ';

[0029] Feature layers i ′ and feature layer s i Information fusion is achieved through the residual structure, and the feature layer x is obtained by addition. i+1 , feature layer x i+1 , which is the output feature layer of the Transformer backbone feature extraction layer.

[0030] Furthermore, in step (3), the attention feature fusion module performs h and feature layer F L The steps for fusion are:

[0031] Feature layer F h and feature layer F L The feature layer F is obtained by ReLU activation function, convolution and upsampling respectively. h ′ and feature layer F L ';

[0032] Feature layer F h ′ and feature layer F L′ are fused after convolution and the similarity feature layer F is obtained after Sigmoid activation function map ;

[0033] The feature layer F h ′ and feature layer F L '; respectively with the similarity feature layer F map Perform weighted fusion and output the fused feature layer.

[0034] Furthermore, in step (3), the hierarchical Transformer backbone feature extraction module has four Transformer backbone feature extraction layers, which sequentially output feature layers F1, F2, F3, and F4;

[0035] Feature layer F g After downsampling, it is merged with feature layer F1 to form feature layer F C , feature layer F C The feature layer F is obtained by one-dimensional convolution C1 , feature layer F C1 The feature layer F3 is fused with the attention feature fusion module to output the feature layer F f1 Among them, feature layer F3 and feature layer F4 are low-resolution feature layers, feature layer F C1 and feature layer F2 is a high-resolution feature layer;

[0036] The feature layer F2 and the feature layer F4 are fused through the attention feature fusion module to output the feature layer F f2 ;

[0037] Feature layer F f1 , feature layer F f2 , feature layer F3 and feature layer F4 are first up-sampled, and then passed through the convolution layer and MLP layer to add the feature layers to obtain the feature layer F a ;

[0038] The feature layer Fa is decoded through the convolution layer and upsampling to output the predicted image.

[0039] Furthermore, in step (4), the training step is:

[0040] Input the images of the iPSCs image dataset into the evaluation model based on Transformer attention feature fusion and output the predicted image;

[0041] Compare the predicted results with the pre-labeled categories at the pixel level and calculate the Focal Loss formula as follows:

[0042]

[0043] in, is the predicted output of the model, y is the actual label, C is the number of categories, and p i is the probability of the i-th category predicted by the model, α i is the category weight, γ is the adjustment factor of Focal Loss;

[0044] Use Adamw optimizer to perform error back propagation, calculate the gradient and update the model parameters θ. The update formula is Where η is the learning rate, is the loss function L f The gradient of the parameter θ, λ is the weight decay coefficient.

[0045] A computer-readable storage medium stores computer-executable instructions for executing the above-mentioned iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion.

[0046] The technical solution provided by the present invention can have the following beneficial effects:

[0047] 1. In the present invention, by establishing a mapping relationship between stem cell pluripotency and cell colony morphology, human interference is prevented; by corresponding pixels in the image to pluripotency status categories and marking them, accurate evaluation of the model is facilitated;

[0048] 2. The Gabor edge feature enhancement module and the hierarchical Transformer backbone feature extraction module form a powerful feature extraction network that can accurately locate and identify the subtle structures of colonies and cells in complex cell images. Moreover, by applying Gabor filters in different directions, the model can more comprehensively perceive the texture and edge structure information of colonies of different sizes and qualities in the image.

[0049] 3. Based on the hierarchical Transformer backbone feature extraction module, an attention feature fusion module is introduced to integrate high-level semantic features with low-level features, achieving better model segmentation performance. The model in this paper can effectively identify targets of various scales in bright-field images of living iPSCs, from large colonies to tiny single cells, and accurately detect cell colonies of different scales and morphologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 1 is a flow chart of a method for assessing iPSC colony pluripotency based on hierarchical Transformer attention feature fusion according to an embodiment of the present invention;

[0051] Figure 2 is the image region classification map ( Figure 2a) and the expression levels of OCT-3 / 4 and SOX2 in fluorescence intensity analysis Figure 2 b);

[0052] Figure 3 is a schematic diagram of a Gabor feature enhancement module structure;

[0053] Figure 4 is a schematic diagram of a Transformer module structure;

[0054] Figure 5 is a schematic diagram of an attention feature fusion module structure;

[0055] Figure 6 is a Transformer attention feature fusion-based evaluation model diagram;

[0056] Figure 7 is an ablation study comparison prediction diagram;

[0057] Figure 8 is a performance column chart of the model of the present application on six categories under different evaluation indicators. DETAILED DESCRIPTION

[0058] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are only used to explain the present application and cannot be understood as a limitation of the present application.

[0059] The iPSCs colony pluripotency evaluation method based on hierarchical Transformer attention feature fusion of the embodiments of the present application is described below in combination with Figures 1 to 8 The iPSCs colony pluripotency evaluation method based on hierarchical Transformer attention feature fusion of the embodiments of the present application is described below in combination with

[0060] The iPSCs colony pluripotency evaluation method based on hierarchical Transformer attention feature fusion in the present embodiment includes the following steps:

[0061] (1) Collect images during the culture of iPSCs, and establish a mapping relationship between stem cell pluripotency and cell colony morphology;

[0062] (2) Construct an iPSCs image dataset, correspond the pluripotency state category to each pixel in the image, and mark the pixel level;

[0063] (3) Build an evaluation model based on Transformer attention feature fusion:

[0064] The Gabor edge feature enhancement module is used for detecting and enhancing the edge features of the image edges in the iPSCs image, and outputs the feature layer F g;

[0065] Hierarchical Transformer backbone feature extraction module, n Transformer backbone feature extraction layers extract features from iPSCs images in sequence, and output feature layers F1, F2…F in sequence k ...and F n ;

[0066] Attention feature fusion module is used to g 、F1 F2…F k ...and F n Multiple attention features are fused separately, and multiple fusion feature layers are output;

[0067] Fusion feature layer and feature layer F k ...and F n After upsampling and normalization, they are fused and then decoded to obtain the predicted image, which is the evaluation result;

[0068] (4) Training the evaluation model based on Transformer attention feature fusion using the iPSCs image dataset;

[0069] (5) The trained evaluation model based on Transformer attention feature fusion is used to evaluate the pluripotency of images during the iPSCs culture process.

[0070] In the present invention, by establishing a mapping relationship between stem cell pluripotency and cell colony morphology, interference from human factors is prevented; by corresponding pixels in the image to the pluripotency state category and marking them, it is conducive to accurate evaluation of the model; a powerful feature extraction network is formed with a Gabor edge feature enhancement module and a hierarchical Transformer backbone feature extraction module, which can accurately locate and identify the subtle structure of colonies and cells in complex cell images; at the same time, based on the hierarchical Transformer backbone feature extraction module, an attention feature fusion module is introduced to enable the fusion of high-level semantic features with low-level features, achieving better model segmentation performance. The model in the present invention can effectively identify various scale targets in iPSCs bright field images, from large colonies to tiny single cells, and achieves accurate detection of cell colonies of different scales and morphologies.

[0071] In one embodiment of the present invention, in step (1), bright field images of living cells during iPSCs culture are collected, and the image areas are classified into background, single cell clusters, colonies, and impurities;

[0072] Immunofluorescence staining was used to obtain the different colony morphologies during iPSCs culture, and the pluripotency status of iPSCs colonies was classified according to the colony morphology, and a mapping relationship between stem cell pluripotency and cell colony morphology was established.

[0073] Specifically, the global bright field image of living cells is used as the domain, and the image area is classified into background, single cell clusters, colonies and impurities, including cell debris and dead cells. iPSCs are continuously passaged until spontaneous differentiation of colonies is observed, thereby collecting a large number of images with pluripotency ranging from high to low. Further specifically, the culture cycle of iPSCs is generally 7 days. Within two days after cell inoculation, iPSCs appear in the state of single cells or clusters. Under normal culture conditions, iPSCs on the 5th day after inoculation gradually appear in a colony state with densely arranged cells and clear edges. In order to better evaluate the colony state, we collected bright field images of living cells on days 5-7. Exemplarily, an incubator microscope (Olympus IX83) was used to perform live cell imaging of iPSCs. The image channel was set to bright field, and the objective lens magnification was 10x. The resolution of the acquired bright field image was 1920x1200, and the format was TIFF. In the embodiment of the present invention, the pluripotency of iPSCs colonies was divided into three categories of "high", "medium" and "low" quality, and a mapping relationship between stem cell pluripotency and cell colony morphology was established.

[0074] iPSCs are characterized by their ability to self-renew indefinitely and generate all cell types of an organism. OCT-3 / 4 and SOX2 are two essential transcription factors that co-regulate target genes to maintain the pluripotency of iPSCs. Studies have shown that as iPSCs differentiate, the expression levels of OCT-3 / 4 and SOX2 decrease. Phenotypic differences in iPSC colonies during culture are associated with the expression of pluripotency markers. In one embodiment of the present invention, immunofluorescence staining of the pluripotency-expressing proteins OCT-3 / 4 and SOX2, produced during iPSC culture, is performed. Fluorescence intensity accurately characterizes pluripotency. ImageJ is used to calculate the average immunofluorescence intensity of iPSC colonies, and the quality of iPSC colonies is categorized as "high," "medium," or "low," enabling a mapping between the pluripotency and morphology of iPSC colonies. The average fluorescence intensity of a colony is calculated as the sum of the fluorescence intensity of the colony region divided by the area of ​​the colony region.

[0075] Reference Figure 2 Exemplary, high-quality iPSC colonies exhibit uniform and strong OCT-3 / 4 and SOX2 fluorescence signals, with high fluorescence intensities, ranging from 25 to 35 for OCT-3 / 4 and 40 to 50 for SOX2, indicating that the cells within the colony maintain a good pluripotent state. Morphological characteristics of high-quality iPSC colonies are: completely undifferentiated within, manifested by clear colony edges, uniform and dense cell arrangement within the colony, and a high nuclear-to-cytoplasmic ratio.

[0076] In medium-quality iPSC colonies, the fluorescence signals of OCT3 / 4 and SOX2 may vary somewhat, with some areas exhibiting weaker fluorescence intensities. The average fluorescence intensity of OCT-3 / 4 ranges from 15 to 25, and the average fluorescence intensity of SOX2 ranges from 20 to 30. This may indicate that the pluripotency of cells in these colonies is not completely uniform, and some cells may be beginning to lose pluripotency. Morphological characteristics of medium-quality iPSC colonies include: some differentiation of cells within the colony, blurred colony edges, loosely packed cells, and bright brightfield halos between cells.

[0077] Low-quality colonies may show lower and uneven OCT-3 / 4 and SOX2 fluorescence intensities, indicating that the pluripotency of the cells has been significantly affected. The average fluorescence intensity of OCT-3 / 4 in low-quality iPSC colonies is between 5 and 15, and the average fluorescence intensity of SOX2 is between 10 and 20. The cells in these colonies may have begun to differentiate, and the iPSC colony state is unstable. The morphological characteristics of medium-quality iPSC colonies are: the cells inside the colony are almost completely differentiated, with cavities and a fibroblast-like cell morphology.

[0078] Furthermore, in step (2), the pixels in the image are divided into six categories: iPSCs single cell clusters (S), high-quality iPSCs colonies, medium-quality iPSCs colonies, low-quality iPSCs, impurities, and background, and the pixels of each category are converted into one-hot encoding, that is, labeled at the pixel level. Specifically, the iPSCs single cell cluster is a cell in the early state. Each pixel in the prepared label is converted into a one-hot encoding, that is, the background pixel value is set to 0, and the RGB color mapping is (255, 255, 255). The pixel value of the iPSCs single cell cluster is set to 1, and the RGB color mapping is (6, 6, 6). The pixel value of the low-quality iPSCs colony is set to 2, and the RGB color mapping is (191, 191, 191). The pixel value of the medium-quality iPSCs colony is set to 3, and the RGB color mapping is (118, 113, 113). The pixel value of the high-quality iPSCs colony is set to 4, and the RGB color mapping is (59, 56, 56). The pixel value of other categories is set to 5, and the RGB color map is (218, 218, 218).

[0079] Reference Figure 3In one embodiment of the present invention, the Gabor edge feature enhancement module is composed of a Gabor filter, a 1×1 convolution layer, a 3×3 convolution layer, a batch normalization layer, a ReLU activation function and a residual connection. In step (3), the Gabor edge feature enhancement module processes the iPSCs image as follows: the input image is convolved with a multi-directional Gabor filter, and then passes through a 1×1 convolution layer, a 3×3 convolution layer, a batch normalization layer, a ReLU activation function and a residual connection, and the output feature layer F is obtained. g .

[0080] Gabor's 2D filter is used to extract edge texture features of an image by convolving the image with the Gabor kernel defined in the following formula:

[0081]

[0082] Where x′=xcosθ+ysinθ,y′=-xcosθ+ysinθ. x and y represent the pixel position in the spatial domain, θ is the direction of the Gabor filter, is the phase shift, σ is the standard deviation of the Gaussian function, λ represents the wavelength, and γ is the spatial aspect ratio.

[0083] The multi-directional Gabor filter in this technical solution helps the model capture features from different directions in the image, thereby improving the performance of image segmentation tasks. By applying Gabor filters in different directions, the model can more comprehensively perceive the texture and edge structure information of clusters of different sizes and qualities in the image.

[0084] For example, the input image size is 512×512×3 (i.e., height 512, width 512, and number of channels 3), and the processing steps of the Gabor edge feature enhancement module are as follows:

[0085] S310, the passing direction θ is respectively The convolution operation is performed on each RGB channel using four Gabor filters in different directions. The output feature layer is 512×512×12; the large convolution kernel size of the filter is set to 15, and the parameters are σ, λ, and γ are 0, 4, 5, and 0.5, respectively;

[0086] S311, the output 512×512×12 feature layer is passed through a 1x1 convolution layer, the output channel number of the Gabor filter is converted from the input channel number (12) to the specified output channel number (64), and the output feature layer size is 512×512×64;

[0087] S312, the feature layer of size 64 is further extracted through a 3x3 convolution layer, and then through a batch normalization layer and a ReLU activation function, the output feature layer size is 512×512×64;

[0088] The feature layers output by S313 and S311 are added to the feature layers output by S312 through residual connections, and the output feature layer F g .

[0089] In one embodiment of the present invention, in step (3), the input image is processed in the Transformer backbone feature extraction layer through the overlapping block fusion layer and then enters the Transformer module for processing, outputting a feature layer F1; the feature layer F1 is input into the next Transformer backbone feature extraction layer for processing to obtain F k . The hierarchical Transformer backbone feature extraction module effectively captures contextual information of different scales and ranges in the image through its hierarchical architecture, and uses the self-attention mechanism to strengthen the association between local features and global features, thereby improving the semantic understanding of various regions in the image. This structure not only enhances the model's ability to capture long-distance dependent features, but also speeds up the calculation speed through parallel processing, allowing the model to more accurately identify and segment different semantic regions when processing images with complex structures and rich details, thereby improving the performance and accuracy of semantic segmentation. The processing steps of the overlapping block fusion layer are used to extract and synthesize local and global information of the image.

[0090] Reference Figure 4 , the feature layer processing process of the Transformer module is:

[0091] S3201, input feature layer x i Normalized by the layerNorm layer, and then passed through the ESA layer (Efficient self-attention layer) to obtain the feature layer x i ';

[0092] S3202, through the residual structure, the feature layer x i ′ and feature layer x i Add together to get the feature layer s i ;

[0093] S3203, feature layer s i Normalized by the LayerNorm layer and then passed through the Mix-FFN layer to obtain the feature layer s i ';

[0094] S3204, feature layer s i ′ and feature layer s iInformation fusion is achieved through the residual structure, and the feature layer x is obtained by addition. i+1 , feature layer x i+1 , which is the output feature layer of the Transformer backbone feature extraction layer.

[0095] The detailed calculation process of the input features through the Transformer module can be expressed as follows:

[0096] s i =ESA(LN(x i ))+x i

[0097] x i+1 =MFFN(LN(s i ))+s i

[0098] Among them, LN represents layer normalization (LayerNorm).

[0099] For example, the attention mechanism calculates the correlation between the query and each key to obtain weight coefficients, and uses these coefficients to perform a weighted summation on the corresponding values ​​to obtain the attention value. The self-attention mechanism refers to the process of paying attention to and utilizing the query, key, and value at the same position. Its main function is to simulate the detailed interaction between pixels and capture the interaction between any positions on the image feature layer without distance restrictions, making the Transformer model very flexible in modeling long-range dependencies in visual tasks. The formula for self-attention is as follows:

[0100]

[0101] where matrices Q, K, and V correspond to queries, keys, and values, respectively, σ is the softmax function, T represents the transpose, and d k is the dimension of K. In order to minimize the number of parameters while maintaining the basic feature extraction capability of the transformer, the ESA layer introduces a reduction ratio R, which will be as follows:

[0102]

[0103] Among them, K represents the input sequence, which consists of Q, K and V, N is the length of the sequence, and C is the channel dimension. First, the shape of K is converted from (N×C) to Then, change the shape of K from Convert to This method reduces the computational complexity from O(N 2 ) to

[0104] The Mix-FFN layer consists of a linear layer, a residual module, and a nonlinear activation function called GELU. Position information is "leaked" by directly using 3×3 convolutions within the FFN. This design allows the model to dynamically express the positional relationships between patches, rather than relying on fixed, trained positional encodings. This design not only reduces the model's reliance on positional encodings but also improves its adaptability to inputs of varying resolutions. The computational process of the Mix-FFN layer is as follows:

[0105] MFFN(X)=Conv1(GELU(Conv3(Conv1(X))))+X

[0106] Where X is the feature from the ESA layer, Conv1 represents a point-wise convolution with stride 1, kernel size 1, and padding 0, and Conv3 represents a depth-wise convolution with stride 2, kernel size 3, and padding 1.

[0107] For example, refer to Figure 6 , the hierarchical Transformer backbone feature extraction module has four Transformer backbone feature extraction layers, and the processing steps are:

[0108] S320: First, pass the 512×512×3 input image through the Overlapping Block Fusion Layer 1. This step divides the image into blocks and embeds them into the specified dimensions through 2D convolution. The convolution kernel size is 7×7, the stride is 4, the padding size is 3, and the number of convolution kernels is 64. The Overlapping Block Fusion Layer 1 straightens the input feature layer to a size of 16384×64.

[0109] S321. Then, the 16384×64 vector is processed through Transformer Block 1 to process the feature sequence, outputting a feature layer F1 of size 128×128×64. Transformer Block 1 is the Transformer module. Set the scaling factor R of the self-attention mechanism to 8, the number of self-attention heads N to 1, the expansion rate E of the Mix-FFN layer to 8, and the number of encoders (ESA + MixFFN) L in each Transformer Block to 3.

[0110] S322, feature layer F1 first passes through overlapping block fusion layer 2 to reduce the image resolution, thus achieving downsampling. Its convolution kernel size is 3×3, stride is 2, padding size is 1, and the number of convolution kernels is 128. It then passes through TransformerBlock 2 to output feature layer F2 with a size of 64×64×128. R, N, E, and L are set to 4, 2, 8, and 3, respectively.

[0111] S323 and feature layer F2 first pass through overlapping block fusion layer 3 to reduce the image resolution, thus achieving downsampling. Its convolution kernel size is 3×3, with a stride of 2, a padding size of 1, and 320 kernels. It then passes through TransformerBlock 3 to output feature layer F3 with a size of 32×32×320. R, N, E, and L are set to 2, 5, 4, and 18, respectively.

[0112] S324 and feature layer F3 first pass through overlapping block fusion layer 4 to reduce the image resolution and achieve downsampling. Its convolution kernel size is 3×3, stride is 2, padding size is 1, and the number of convolution kernels is 512. It then passes through TransformerBlock 4 to output feature layer F4 with a size of 16×16×512. R, N, E, and L are set to 1, 8, 4, and 3, respectively.

[0113] Further, refer to Figure 5 In step (3), the attention feature fusion module performs the feature layer F h and feature layer F L The steps for fusion are:

[0114] S330, feature layer F h and feature layer F L The feature layer F is obtained by ReLU activation function, convolution and upsampling respectively. h ′ and feature layer F L ';

[0115] S331, feature layer F h ′ and feature layer F L ′ are fused after convolution and the similarity feature layer F is obtained after Sigmoid activation function map ;

[0116] S332, feature layer F h ′ and feature layer F L '; respectively with the similarity feature layer F map Perform weighted fusion and output the fused feature layer.

[0117] Furthermore, in step (3), the hierarchical Transformer backbone feature extraction module has four Transformer backbone feature extraction layers, which sequentially output feature layers F1, F2, F3, and F4;

[0118] Feature layer F g After downsampling, it is merged with feature layer F1 to form feature layer F C , feature layer F C The feature layer F is obtained by one-dimensional convolution C1 , feature layer F C1The feature layer F3 is fused with the attention feature fusion module to output the feature layer F f1 Among them, feature layer F3 and feature layer F4 are low-resolution feature layers, feature layer F C1 and feature layer F2 is a high-resolution feature layer;

[0119] The feature layer F2 and the feature layer F4 are fused through the attention feature fusion module to output the feature layer F f2 ;

[0120] Feature layer F f1 , feature layer F f2 , feature layer F3 and feature layer F4 are first up-sampled, and then passed through the convolution layer and MLP layer to add the feature layers to obtain the feature layer F a ;

[0121] Feature layer F a Decoding is performed through convolutional layers and upsampling to output the predicted image.

[0122] In the multi-scale backbone feature extraction module, the rich and accurate semantic information provided by the low-resolution feature layers F3 and F4 is very important for the high-resolution feature layer F c1 The detail parsing and boundary detection of F2 and F3 are crucial, and the low-resolution feature layers contain relatively few layers and channels. Therefore, we regard the low-resolution features F3 and F4 as high-resolution features F c1 and F2, and enable them to provide them with the required information. Compared with the simple addition or concatenation of features, we introduce an attention feature fusion module for the high-resolution feature layer to selectively learn useful semantic features from the low-resolution feature layer. The vectors of the corresponding pixels in the feature layers of the high-resolution feature branch and the low-resolution feature branch are defined as and Then the output of the Sigmoid function can be expressed as:

[0123]

[0124] Where σ represents the probability that the two pixels belong to the same object, f h and f l represents a 1x1 convolution operation. If σ is high, we trust Because low-resolution features are semantically rich and accurate, and vice versa. Therefore, the output of the feature fusion module can be written as:

[0125]

[0126] F c1 For example, the process of fusion with F3 is shown in Figure 2. Figure 6 :

[0127] S330, feature layer F c1 The size is 128×128×64, and the size of feature layer F3 is 32×32×320. Feature layer F c1 The feature layer F with a size of 128×128×64 is obtained by ReLU activation function c1 ′. The feature layer F3 is activated by ReLU, 1×1 convolution, and upsampling to obtain the feature layer F3′ with a size of 128×128×64.

[0128] S331, feature layer F c1 ′ and F3′ map the number of channels to the number of intermediate channels 32 through 1×1 convolution, and obtain the feature layer F with a size of 128×128×32 c1 "and F3";

[0129] The mapped feature layer F c1 The elements corresponding to each channel in "and F3" are multiplied to obtain a feature size of 128×128×32. Then, the 128×128×32 feature layer is summed in the channel dimension and passed through the Sigmoid activation function to obtain a feature layer F with a size of 128×128×1. map . F map Represents the similarity feature layer. This process can be seen as the dot product of the vectors of corresponding pixels in the two feature layers (the corresponding elements of each pixel position in the two feature layers are multiplied and summed), and then σ is obtained through the Sigmoid function. σ represents the probability that the two pixels belong to the same target. Similarity feature layer F map It can also be viewed as a probability matrix consisting of σ obtained at each pixel position.

[0130] S332, based on the calculated similarity feature layer F map , the two input feature layers F c1 ′ and F3′ are weighted fused to obtain the output feature F of the attention feature fusion module f1 , whose size is 128×128×64. σ represents the probability that these two pixels belong to the same object. Similarity feature layer F map It can be seen as a probability matrix composed of σ obtained at each pixel position. When the probability σ is high, we trust it more. Because low-resolution features are semantically rich and accurate, and vice versa.

[0131] The detailed process of fusion of features F2 and F4 is the same as S1-S4. The output feature F of the attention feature fusion module is f2 The dimensions are 64×64×128.

[0132] Furthermore, in step (4), the training step is:

[0133] S401. Input the image of the iPSCs image dataset into the evaluation model based on Transformer attention feature fusion and output the predicted image.

[0134] S402: Compare the predicted result with the pre-labeled category at pixel level, and calculate the Focal Loss formula as follows:

[0135]

[0136] in, is the predicted output of the model, y is the true label, C is the number of categories, pi is the probability of the i-th category predicted by the model, α i is the category weight, and γ is the adjustment factor of Focal Loss; γ is used to reduce the weight of easy-to-classify samples, thereby paying more attention to difficult-to-classify samples.

[0137] S403, use Adamw optimizer to perform error back propagation, calculate the gradient and update the model parameter θ, the update formula is Where η is the learning rate, is the loss function L f The gradient of the parameter θ is calculated, and λ is the weight decay coefficient. The AdamW optimizer also takes weight decay into account when updating parameters to prevent overfitting. Weight decay is set to 1e-2. This process is iterated until the model converges, ultimately obtaining a trained segmentation network.

[0138] Figure 7 For the ablation study and comparison of prediction graphs, the images Image1-4 are input into the model of the present invention (Ours), the Segformer baseline model (Baseline), the model with the Gabor edge feature enhancement module added to the Segformer baseline model (+Gabo), and the model with the attention feature fusion module added to the Segformer baseline model (+Pag). The output prediction results are as follows: Figure 7 As shown, Figure 7 GT is the true result. It can be seen that the prediction results obtained by the model of the present invention have the highest similarity with the true result. The evaluation results of the ablation study are shown in the following table.

[0139]

[0140] As shown in the table above, when only the Gabor feature enhancement module is removed, model performance significantly degrades, with mIoU, mPrecision, mRecall, and F1-score decreasing by 2.81%, 1.06%, 2.74%, and 1.91%, respectively. When only the attention feature fusion module (Pag) is removed, mIoU, mPrecision, and mRecall decrease by 1.78%, 0.29%, and 1.58%, respectively. The largest performance drop occurs when both the Gabor feature enhancement module and the attention feature fusion module (Pag) are removed, i.e., when the baseline is used. This demonstrates that each module in the proposed model is essential. Both the attention feature fusion module (Pag) and the Gabor feature enhancement module improve segmentation performance to some extent, and their synergy significantly improves the model's detection accuracy.

[0141] like Figure 8 The results demonstrate the performance of the method for each category of iPSC colony pluripotency under different evaluation indicators. The prediction accuracy of high-, medium-, and low-quality iPSC colonies was averaged, and the results showed that the model of the present invention achieved an accuracy of 98.6% for predicting different iPSC states.

[0142] Accordingly, the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to execute the above-mentioned iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion.

[0143] Other structures and operations of the iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion according to an embodiment of the present invention are well known to ordinary technicians in the field and will not be described in detail here.

[0144] Throughout this specification, reference to terms such as "embodiment" or "example" indicates that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0145] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for assessing iPSC colony pluripotency based on hierarchical Transformer attention feature fusion, characterized in that: The following steps are involved: (1) Collect images of iPSCs during culture and establish a mapping relationship between stem cell pluripotency and cell colony morphology; (2) Construct an iPSCs image dataset, assign each pixel in the image to a pluripotency state category, and label the pixel level; (3) Building an evaluation model based on Transformer attention feature fusion: Gabor edge feature enhancement module is used to detect image edges in iPSCs images and enhance edge features, outputting feature layer F g ; The hierarchical Transformer backbone feature extraction module has four Transformer backbone feature extraction layers, which output feature layers F1, F2, F3 and F4 in sequence; Feature layer F g After downsampling, it is merged with feature layer F1 to form feature layer F C , feature layer F C The feature layer F is obtained by one-dimensional convolution C1 , feature layer F C1 Fused with feature layer F3 through the attention feature fusion module, the output feature layer Among them, feature layer F3 and feature layer F4 are low-resolution feature layers, feature layer F C1 and feature layer F2 is a high-resolution feature layer; The feature layer F2 and the feature layer F4 are fused through the attention feature fusion module to output the feature layer ; Feature layer , feature layer , feature layer F3 and feature layer F4 are first up-sampled, and then passed through the convolution layer and MLP layer to add the feature layers to obtain the feature layer F a ; Feature layer F a Decode through convolutional layers and upsampling, and output the predicted image, which is the evaluation result; (4) Training the evaluation model based on Transformer attention feature fusion using the iPSCs image dataset; (5) The trained evaluation model based on Transformer attention feature fusion was used to evaluate the pluripotency of images during iPSCs culture.

2. The method according to claim 1, characterized in that In the step (1), a bright field image of living cells during the iPSCs culture process is collected, and the image area is classified into background, single cell clusters, colonies, and impurities; Immunofluorescence staining was used to obtain the different colony morphologies during iPSCs culture, and the pluripotency status of iPSCs colonies was classified according to the colony morphology, and a mapping relationship between stem cell pluripotency and cell colony morphology was established.

3. The method according to claim 2, characterized in that The pluripotent expression proteins OCT-3 / 4 and SOX2 produced during the iPSCs culture process were fluorescently stained by immunofluorescence staining; the average immunofluorescence intensity of the iPSCs colony status was calculated using ImageJ, and the average fluorescence intensity of the colony = the total fluorescence intensity of the colony area / the area of ​​the colony area.

4. The method according to claim 2, characterized in that In step (2), the pixels in the image are divided into six categories: iPSCs single cell clusters, high-quality iPSCs colonies, medium-quality iPSCs colonies, low-quality iPSCs, impurities and background, and the pixels of each category are converted into one-hot encoding, that is, labeled at the pixel level.

5. The method according to claim 1, wherein In step (3), the process of processing the iPSCs image by the Gabor edge feature enhancement module is as follows: The input image is convolved with a multi-directional Gabor filter, and then passes through a 1×1 convolution layer, a 3×3 convolution layer, a batch normalization layer, a ReLU activation function, and a residual connection to output the feature layer F. g .

6. The method according to claim 1, characterized in that In the step (3), the input image is processed by the overlapping block fusion layer in the Transformer backbone feature extraction layer and then enters the Transformer module for processing, outputting the feature layer F1; the feature layer F1 is input into the next Transformer backbone feature extraction layer for processing to obtain F k ; The feature layer processing process of the Transformer module is as follows: Input feature layer Normalized by the layerNorm layer, and then the feature layer is obtained by the ESA layer ; Through the residual structure, the feature layer With feature layer Add to get the feature layer ; Feature layer Normalized by the LayerNorm layer, and then the feature layer is obtained by the Mix-FFN layer ; Feature layer With feature layer Information fusion is achieved through residual structure and the feature layer is obtained by addition. , feature layer , which is the output feature layer of the Transformer backbone feature extraction layer.

7. The method according to claim 1, characterized in that In step (3), the attention feature fusion module performs C1 F3 and F2 are merged with F4. Let F3 be the feature layer. C1 and feature layer F2 as feature layer F h , feature layer F3 and feature layer F4 are feature layer F L , the steps for fusion are: Feature layer F h and feature layer F L The feature layer F is obtained by ReLU activation function, convolution and upsampling respectively. h ´ and feature layer F L ´; Feature layer F h ´ and feature layer F L After convolution, they are fused and activated by Sigmoid function to obtain the similarity feature layer F. map ; The feature layer F h ´ and feature layer F L ' respectively with the similarity feature layer F map Perform weighted fusion and output the fused feature layer.

8. The method according to claim 1, characterized in that In step (4), the training steps are: Input the images of the iPSCs image dataset into the evaluation model based on Transformer attention feature fusion and output the predicted image; Compare the predicted results with the pre-labeled categories at the pixel level and calculate the Focal Loss formula as follows: ； in, is the predicted output of the model, is the true label, C is the number of categories, The model predicts i The probability of the class, is the category weight, γ is the adjustment factor of Focal Loss; Use Adamw optimizer to perform error backpropagation, calculate gradients and update model parameters θ , the update formula is ,in η is the learning rate, is the loss function Parameters θ The gradient, λ is the weight decay coefficient.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the iPSCs colony pluripotency assessment method based on hierarchical Transformer attention feature fusion according to any one of claims 1 to 7.

Images