Cell microtubule array uncertainty quantitative segmentation method fused with large vision model

Abstract

The invention belongs to the technical field of image processing, and relates to a cell microtubule array uncertainty quantitative segmentation method fused with a large visual model, and the method comprises the steps: processing a cell microtubule microscopic image through a microtubule segmentation large visual model, and obtaining a microtubule likelihood graph of a pixel level; performing topological analysis on the microtube likelihood graph to obtain a plurality of manifold structures, and forming a microtube topological skeleton by a union set of the manifold structures; calculating an uncertainty interval of each manifold structure in the topology skeleton through a manifold uncertainty interval quantization network; and prior knowledge is introduced, a'pixel-topology-prior 'three-evidence-source fusion framework based on the D-S evidence theory is constructed, explicit modeling is performed on internal uncertainty of evidence sources, and thus an accurate microtube segmentation result is obtained. According to the method, the accuracy and the reliability of a final microtube segmentation result are improved by performing uncertainty modeling and fusion analysis on microtube form extraction results obtained from a pixel level, a topological structure level and a priori knowledge level respectively.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a cell microtubule array uncertainty quantification segmentation method fusing a large visual model. BACKGROUND

[0002] Microtubules, as important filamentous structures in cytoskeleton, are highly dynamic and complex supramolecular systems in eukaryotes, which play an irreplaceable role in maintaining cell morphology, mediating intracellular material transport, and regulating cell division. The biological function of microtubules is closely related to the array structure established by their dynamic behaviors such as growth, contraction, and fragmentation. Since different compounds can produce selective toxicity to diseased cells by inhibiting the topological evolution of microtubule arrays, it is of great significance to develop targeted therapeutic drugs to deeply understand the morphology of microtubule arrays and their change rules. Existing microscopy imaging technology can capture microtubule array images with good spatiotemporal resolution, making it possible to observe microtubule structures. However, how to accurately segment microtubule structures and extract fine microtubule morphological features from microscopic images is still a challenge. Traditional thresholding methods, active contour methods, and spline curve fitting methods are only effective in sparse microtubules, and for dense microtubule arrays with inherent geometric and morphological overlaps and topological complexity, the above methods all show great limitations.

[0003] In recent years, deep learning has become the preferred solution for processing complex microtubule microscopic images due to its high robustness to high noise, low contrast, and dense microtubule microscopic images, and the convenient end-to-end learning paradigm. However, deep learning methods have the disadvantage of relying heavily on the size and quality of training data. The cross-sectional outer diameter of microtubules is only about 25 nanometers, and according to the Abbe diffraction law, the resolution limit of a microscope based on optical imaging principles is about 200 nanometers. This observation limit is much higher than the size of microtubules, so traditional optical microscopes cannot directly observe clear microtubule structures. Super-resolution microscopes and electron microscopes can obtain clear structures, but they are limited by high cost. Therefore, there is a lack of data for research on morphological extraction and segmentation of cell microtubule microscopic images. In addition, the pixel-level classification strategy used by deep learning methods has inherent limitations. It processes each pixel independently and only relies on the prediction probability of a single pixel to determine whether it belongs to the microtubule region, ignoring the spatial correlation between pixels, which makes the segmentation result prone to microtubule branch breakage. SUMMARY

[0004] To address the above issues, this paper proposes a method for uncertainty quantification of cell microtubule array segmentation that integrates a large visual model. First, a large visual model for microtubule segmentation with few-shot learning capabilities is constructed, alleviating the data size dependency of deep learning methods. Secondly, the topological skeleton of microtubules is constructed using discrete Morse theory, and a manifold uncertainty interval quantification network is designed to analyze the uncertainty intervals of each manifold structure in the topological skeleton. A set of prior knowledge is then designed and constructed as a metric function. Finally, a three-source evidence fusion framework (pixel, topology, and prior) is constructed based on the Dempster-Shafer evidence theory.

[0005] The present invention is achieved through the following technical solutions: A method for quantifying uncertainty segmentation of cell microtubule arrays integrated with a large visual model, comprising the following steps: Step 1: Collect microscopic images of cell microtubules and annotate microtubule morphology information to construct a microtubule dataset; Step 2: Combine the deep trainable hint layer with the low-rank adaptive method to perform structural optimization and domain fine-tuning on the base large visual model to construct a large visual model for microtubule segmentation; Step 3: Use the large visual model for microtubule segmentation to process the cell microtubule microscopic image to obtain the microtubule likelihood map at the pixel level; Step 4: Perform topological analysis on the microtubule likelihood graph to obtain several manifold structures. The union of the manifold structures constitutes the microtubule topological skeleton. Step 5: Calculate the uncertainty interval of each manifold structure in the topological skeleton through the manifold uncertainty interval quantification network; Step 6: Develop a set of prior knowledge based on microtubule biological characteristics and heuristic rules, and design the prior knowledge into a computable metric function; Step 7: Construct a "pixel-topology-prior" three-source evidence fusion framework based on DS evidence theory. After explicitly modeling the internal uncertainty of the evidence source, the complementary or conflicting information of each evidence source is effectively integrated to obtain accurate microtubule segmentation results.

[0006] Further preferably, the large visual model for microtubule segmentation is composed of an image embedding module, an image encoder, and a mask decoder. The image embedding module is used to segment the input cell microtubule microscopic image into blocks to obtain image blocks, and then the image blocks are embedded into vectors respectively. The position codes of the image blocks are then incorporated into the embedded vectors to obtain the initial features of the cell microtubule microscopic image. Subsequently, the initial features of the cell microtubule microscopic image are processed by the image encoder and the mask decoder to finally obtain a microtubule likelihood map.

[0007] Further preferably, the image encoder, on the basis of the image encoder of the base large vision model, removes the prompt encoding module, and inserts a depth trainable prompt layer before each level of the image encoder, which mainly consists of convolution, regularization and activation operations.

[0008] Further preferably, a low-rank adaptive module is inserted into each level of the image encoder of the base large vision model using a low-rank adaptive method.

[0009] Further preferably, in each level of the image encoder of the microtubule segmentation large vision model, the depth trainable prompt layer first processes the vector input into this level, and the obtained feature vector is copied once. The two feature vectors are processed by the frozen base large vision model layer and the low-rank adaptive module, respectively. The feature vector is converted into a low-rank matrix through a lower projection matrix, and the low-rank matrix is converted into a feature vector through an upper projection matrix. The feature vectors output by the frozen base large vision model layer and the low-rank adaptive module are fused according to a certain weight, and the obtained fused vector is processed by another part of the frozen base large vision model layer. The obtained feature vector is transmitted into the next level and the neck module, respectively.

[0010] Further preferably, in step four, random Gaussian noise is first added to the microtubule likelihood map obtained by the microtubule segmentation large vision model to perturb the microtubule likelihood map, and then the perturbed microtubule likelihood map is analyzed using the discrete Morse theory to obtain a number of manifold structures, and the union of the manifold structures constitutes the microtubule topological skeleton.

[0011] Further preferably, the manifold uncertainty interval quantification network takes the manifold structure as input and has three branches. Branch one regards the manifold structure as a node to construct a graph structure. If there is an overlapping part between two manifold structures, it is considered that there is an edge between the two nodes. Then a graph convolution network is used to process the graph structure to obtain a spatial feature map that captures the adjacency relationship between the manifold structures. Branch two embeds all the manifold structures into vectors to construct a manifold feature. After the manifold feature is converted to the frequency domain through fast Fourier transform, it is processed by multiple Fourier operators in the frequency domain, and then it is restored from the frequency domain through inverse Fourier transform to obtain a frequency domain feature map. Branch three counts four kinds of manifold scalar values of the manifold structure, which are the average manifold size, the average manifold likelihood value, the persistent homology persistence of the manifold structure calculated by the persistent homology method, and the base noise. The spatial feature map, the frequency domain feature map and the manifold scalar value obtained by the last three branches are fused to calculate the uncertainty value and the logarithmic variance of each manifold structure.

[0012] Further preferably, in step five, Monte Carlo sampling is used and a certain proportion of manifold uncertainty interval quantization network internal parameters are randomly discarded to perturb the manifold uncertainty interval quantization network.

[0013] Further preferably, in step six, a set of priori knowledge includes two points: one is spatial continuity priori, and the other is edge false positive priori.

[0014] The application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to realize the cell microtubule array uncertainty quantization segmentation method.

[0015] The application combines the large visual model after field fine-tuning, the topological skeleton construction and the uncertainty analysis method, the microtubule morphology priori knowledge and the D-S evidence theory for processing multi-source uncertainty information fusion, and has the following advantages: 1. The large visual model is applied to the microtubule segmentation task, the parameter efficient fine-tuning and the few-sample learning strategy are used, the dependence of the traditional deep learning method on large-scale labeled data is reduced, and a feasible solution is provided for microtubule morphology analysis in a sample scarce scene.

[0016] 2. The microtubule segmentation large visual model is constructed, the low-rank adaptive method is used to fine-tune the base large visual model, so that the base large visual model can adapt to the microtubule segmentation task while ensuring general representation capability. In addition, a deep prompt layer is designed and the internal structure of the base large visual model is optimized, so that the segmentation paradigm is changed from artificial interaction to automatic recognition.

[0017] 3. The skeleton structure in the topological level of the microtubule is constructed, and the manifold uncertainty interval quantization network is designed to realize accurate calculation of the uncertainty of the topological branch. The accidental uncertainty and the cognitive uncertainty are comprehensively considered, and the topological uncertainty interval estimation method is proposed, so that the uncertainty evaluation is converted from a single scalar value to a probability interval, and the reliability of the topological analysis result is improved.

[0018] 4. The “pixel-topology-priori” three evidence source fusion framework based on the D-S evidence theory is proposed. After the internal uncertainty of the three kinds of evidence sources is explicitly modeled, the complementary or conflicting information of the evidence sources is effectively integrated by the basic probability distribution and the Dempster combination rule, so that a more accurate microtubule segmentation result is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 It is a cell microtubule array uncertainty quantization segmentation method process diagram provided by the application, which fuses a large visual model; Figure 2 It is a basic framework diagram of the microtubule segmentation large visual model; Figure 3 is a schematic diagram of a topological skeleton and its uncertainty interval construction method; Figure 4 is a microtubule microscopic image for testing; Figure 5 is a segmentation effect diagram of the scheme of the application in the comparative experiment; Figure 6 is a segmentation effect diagram of the Unet model in the comparative experiment; Figure 7 is a segmentation effect diagram of the Mask RCNN model in the comparative experiment; Figure 8 is a segmentation effect diagram of the pre-trained DPT model with a size of 224 in the comparative experiment; Figure 9 is a segmentation effect diagram of the pre-trained DPT model with a size of 384 in the comparative experiment; DETAILED DESCRIPTION

[0020] The application will be further described in detail below with reference to the accompanying drawings and examples.

[0021] Reference Figure 1 A cell microtubule array uncertainty quantification segmentation method fusing a large vision model, the steps are as follows: Step one: collect cell microtubule microscopic images and label microtubule morphology information, thereby constructing a microtubule dataset; Step two: combine a deep trainable prompt layer with a low-rank adaptive method to optimize the structure and fine-tune the field of the base large vision model, thereby constructing a microtubule segmentation large vision model; Step three: use the microtubule segmentation large vision model to process cell microtubule microscopic images to obtain a microtubule likelihood map at the pixel level; Step four: topological analysis is performed on the microtubule likelihood map to obtain a plurality of manifold structures, and the union set of the manifold structures constitutes a microtubule topological skeleton; Step five: calculate the uncertainty interval of each manifold structure in the topological skeleton through a manifold uncertainty interval quantification network; Step six: a set of prior knowledge is formulated based on microtubule biological characteristics and heuristic rules, and the prior knowledge is designed as a computable metric function; Step seven: construct a "pixel-topology-prior" three evidence source fusion framework based on D-S evidence theory, after explicitly modeling the uncertainty within the evidence source, effectively integrate the complementary or conflicting information of each evidence source, thereby obtaining a precise microtubule segmentation result.

[0022] Figure 2The basic framework of the microtubule segmentation large visual model constructed by the application is shown. The framework is mainly composed of an image embedding module, an image encoder, and a mask decoder. The image embedding module is used to divide the input cell microtubule microscopic image into image blocks, and then the image blocks are embedded into vectors, and the position encoding of the image blocks is fused into the embedded vectors to obtain the initial features of the cell microtubule microscopic image. Then the initial features of the cell microtubule microscopic image are processed by the image encoder and the mask decoder, and finally the microtubule likelihood map is obtained.

[0023] The existing base large visual model needs the user to specify external prompt words. Although this design ensures the zero-shot prediction ability of the model in different scenes, it also causes the problem that the model cannot automatically confirm the foreground object and needs to rely on artificial prompts, resulting in excessive human consumption and low interaction efficiency. For the microtubule segmentation task, although the dye color and imaging clarity are different, all microtubules in the cell microtubule microscopic image are presented in the form of fluorescent highlights, which form a clear contrast with the black background. And there is only one microtubule structure in the cell microtubule microscopic image, which belongs to a single target segmentation task, without the need to handle complex multi-segmentation object problems. Based on the above two reasons, the structure of the base large visual model is optimized, the prompt word encoding module is removed, and the four forms of external prompt words, including prompt points, prompt boxes, prompt texts, and prompt masks, are canceled, without the need for prompt word features.

[0024] In order to automatically extract prompt information from the cell microtubule microscopic image, a depth trainable prompt layer is inserted before each level of the image encoder of the base large visual model (the original layer of the base large visual model), so as to convert the original external specified hard prompt into adaptive soft prompt. The depth trainable prompt layer learns the implicit representation of the microtubule foreground through end-to-end fine-tuning, thereby realizing the conversion from the explicit instruction given by the user to the implicit feature encoding, and achieving fully automatic prediction without additional manual processing. The depth trainable prompt layer is mainly composed of convolution, regularization, and activation operations. In order to alleviate the problem of gradient vanishing or explosion, residual connection is adopted, and its data processing process can be represented by formula (1): (1); Wherein, num represents the level number of the image encoder of the base large visual model, x num is the input data of the depth trainable prompt layer, y num is the output data of the depth trainable prompt layer, Conv represents the convolution operation, Norm represents the regularization operation, is the activation function, and β is the residual coefficient.

[0025] In order to solve the problem that the base large vision model is more inclined to output block regions and cannot identify filamentous structures in the segmentation task, the low-rank adaptive method is used to insert a low-rank adaptive module into each level of the image encoder of the base large vision model, and then a small amount of cell microtubule microscopic images are used for field fine-tuning. This process can be represented by formula (2): (2); Wherein, represents the input vector of the num layer, represents the output vector of the num layer, is the pre-training weight of the base large vision model at the num layer, is the upper projection matrix, is the lower projection matrix, and both represent the data domain, d num represents the dimension size of the input and output feature vectors, r is the rank of the low-rank adaptive module, and alpha represents the fusion weight.

[0026] In each level of the image encoder of the microtubule segmentation large vision model, the depth trainable prompt layer first processes the vector input into the level, and the obtained feature vector is copied. The two feature vectors are processed by the frozen base large vision model layer and the low-rank adaptive module, respectively. The feature vector is converted into a low-rank matrix through the lower projection matrix, and the low-rank matrix is converted into a feature vector through the upper projection matrix. The feature vectors output by the frozen base large vision model layer and the low-rank adaptive module are fused according to a certain weight, and the obtained fusion vector is processed by another part of the frozen base large vision model layer. The obtained feature vector is transmitted into the next level and the neck module. The image encoder is stacked with N levels.

[0027] The microtubule segmentation large vision model takes pixels as the basic unit, and there is no explicit microtubule skeleton connectivity constraint between discrete pixels, so it inevitably has the problem of filamentous structure fracture. In order to alleviate this problem, the microtubule skeleton is analyzed and constructed from the topological level. As shown in Figure 3 , the present application first adds random Gaussian noise to the microtubule likelihood map obtained by the microtubule segmentation large vision model, thereby perturbing the microtubule likelihood map, and then analyzing the perturbed microtubule likelihood map using the discrete Morse theory to obtain a plurality of manifold structures (for example, N). The union of the manifold structures constitutes the microtubule topological skeleton.

[0028] To further calculate the uncertainty of each microtubule branch in the topological skeleton, the manifold uncertainty interval quantization network is constructed. The manifold uncertainty interval quantization network takes the manifold structure as input and has three branches. Branch one takes the manifold structure as a node to construct a graph structure. If there is an overlapping part between two manifold structures, it is considered that there is an edge between the two nodes. Then the graph convolution network is used to process the graph structure to obtain the spatial feature map which captures the adjacent relationship between the manifold structures. Branch two embeds all the manifold structures into vectors to construct the manifold features. After the fast Fourier transform, the manifold features are converted to the frequency domain and processed by multiple Fourier operators in the frequency domain. Then, the inverse Fourier transform is used to restore the frequency domain from the frequency domain, so as to obtain the frequency domain feature map which captures a large number of global features. Branch three counts four kinds of manifold scalar values of the manifold structure, which are the average manifold size, the average manifold likelihood value, the persistent homology persistence of the manifold structure calculated by the persistent homology method, and the basic noise. The spatial feature map, the frequency domain feature map and the manifold scalar value obtained by the last three branches are fused to calculate the uncertainty value and the logarithmic variance of each manifold structure.

[0029] Branch two uses fast Fourier transform, which can convert convolution in the spatial domain to element-by-element multiplication operation in the frequency domain, greatly reducing the computational cost and avoiding the problem that the operation cost and time consumption are unbearable due to the large number of manifold structures in the microtubule likelihood map. The data processing process of branch two can be represented by the following formulas: formula (3) represents the calculation process of the manifold feature from the spatial domain to the frequency domain; formula (4) is the element-by-element multiplication operation in the frequency domain; and formula (5) is the inverse Fourier transform, which restores the processing result in the frequency domain.

[0030] (3); (4); (5); Where S is the spatial feature size, n and m are the spatial row and column coordinates respectively, x(n,m) represents the spatial signal, k1 and k2 are the horizontal and vertical components in the frequency domain respectively, X(k1,k2) is the frequency domain signal, z is a complex number satisfying that the average of z is equal to -1, exp(·) is a complex exponential function, W(k1,k2) is a frequency domain convolution kernel, is a Hadamard product, and Y(k1,k2) is a frequency domain convolution result.

[0031] The uncertainty generally includes accidental uncertainty and cognitive uncertainty, the former depends on the inherent uncertainty existing in the sample, and the accidental uncertainty evaluation is possible by introducing random Gaussian noise into the microtubule likelihood map, and the value can be quantified directly from the output of the manifold uncertainty interval quantization network. However, the cognitive uncertainty reflects the degree of doubt of the manifold uncertainty interval quantization network to its own output, which is determined by the internal parameters of the manifold uncertainty interval quantization network, and cannot be obtained in the training process of the manifold uncertainty interval quantization network. Therefore, the present application uses Monte Carlo sampling and randomly discards a certain proportion of internal parameters to disturb the network, so as to judge the cognitive uncertainty of the network according to the change amplitude of the network output result.

[0032] After H times of independent sampling, the manifold uncertainty interval quantization network outputs a set of uncertainty prediction values of the manifold structure , i is the sampling number, and the sample variance of the set is the cognitive uncertainty of the manifold uncertainty interval quantization network ; the manifold uncertainty interval quantization network also outputs a set of log variances , the arithmetic mean of the set is obtained and is exponentiated to obtain the accidental uncertainty value . The sum of the two kinds of uncertainty is the total uncertainty , and the square root is the standard deviation, which together with the mean value of the uncertainty prediction value set constitutes the uncertainty interval of the manifold structure .

[0033] The present application uses D-S evidence theory to combine the pixel-level segmentation result obtained by the microtubule segmentation large visual model and the topological layer segmentation result obtained by the manifold uncertainty interval quantization network, and formulates a set of prior knowledge to assist in optimizing the segmentation performance. The present application first determines the evidence source as three parts: the microtubule likelihood map, the microtubule topological skeleton composed of the manifold set and the uncertainty interval of each manifold, and a set of prior knowledge. The set of prior knowledge specifically includes two points: one is the spatial continuity prior, since the microtubule has a continuous filamentous morphology, the randomly distributed isolated points may be background noise; the second is the edge false positive prior, according to the heuristic rule, the image edge is more likely to produce false positive results.

[0034] Next, the present application designs a set of prior knowledge into computable measurement functions. For the spatial continuity prior, the present application centers on the target pixel, creates a circular local window with a radius of R pixels, and computes the proportion of pixels within the window that belong to the microtubule region. The higher the proportion, the more likely the target pixel is located on a continuous microtubule structure rather than an isolated noise point. Therefore, the design of the spatial continuity prior as a measurement function for whether the target pixel belongs to the microtubule is shown in equation (6). For the edge false positive prior, the present application measures the Euclidean distance from each pixel to the nearest edge of the image, and maps the distance into a confidence degree, so the edge false positive prior measurement function can be expressed by equation (7).

[0035] (6); (7); where cont represents the degree of certainty of the spatial continuity prior that the target pixel belongs to the microtubule, represents the horizontal and vertical coordinates of the target pixel, represents the horizontal and vertical coordinates of each pixel within the local window, F represents the number of pixels within the local window that are identified as microtubules, M represents a binary mask that comes from the segmentation results at the pixel and topological levels, edge represents the degree of certainty of the edge false positive prior that the target pixel belongs to the microtubule, exp(·) is the exponential function, represents the Euclidean distance from the target pixel to the nearest edge, is a weight parameter.

[0036] A binary decision framework is then constructed, whose proposition space corresponds to the set C equal to where A represents the microtubule region, and ¬A represents the non-microtubule region. The power set of set C (the power set is all possible subsets of set C, including the empty set and set C itself, and the new set constructed is denoted as 2 C ), which can be defined as {∅, {A}, {¬A}, θ}, where ∅ belongs to the empty set, representing an impossible event; θ is set C itself, satisfying θ = C = {A, ¬A}, since θ exists both A representing the microtubule region and ¬A representing the non-microtubule region, so θ represents a completely uncertain state (since C belongs to the proposition space of the binary decision framework, and θ represents a completely uncertain state, even if the two sets are equal, there is a difference in meaning, so two different characters are used to represent them). The domain is the power set of set C (2 C ) on the basic probability function (BPA) is denoted by f, and the basic probability function f is a mapping that satisfies and To measure the degree of contradiction between the three different sources of evidence, the present invention sets a conflict factor K. If the value of K is close to 1, it indicates that there is a high degree of contradiction between the evidence; if its value is close to 0, it indicates that there is a high degree of consistency between the evidence. It can be calculated using formula (8).

[0037] (8) Where o and p are the numbers of the evidence sources. is the conflict factor between type o evidence and type p evidence, is the basic probability function of the oth type of evidence, is the basic probability function for the pth type of evidence. f1(A) represents the basic probability distribution function of the microtubule likelihood map, a source of evidence. It measures the support of the microtubule likelihood map for proposition A, while f1(¬A) represents the support of the microtubule likelihood map for proposition ¬A. f2(A) represents the basic probability distribution function of the uncertainty interval of the microtubule topological skeleton and each manifold, a source of evidence. It measures the support of this source for proposition A, while f2(¬A) represents the support of this source for proposition ¬A. f3(A) represents the basic probability distribution function of a set of prior knowledge, a source of evidence. It measures the support of a set of prior knowledge for proposition A, while f3(¬A) represents the support of a set of prior knowledge for proposition ¬A.

[0038] Next, the present invention designs the Dempster combination rule to handle conflicts and integrate evidence from different sources. This rule can be expressed as formula (9), which quantifies the degree of comprehensive support for proposition A after considering two sources of evidence o and p.

[0039] (9); in, It represents the comprehensive support degree of proposition A after the fusion analysis of evidence source o and evidence source p. E and F belong to the power set of set C (2 C ), E and F will form a set of proposition pairs. During the calculation process, the power set 2 C Then pick out only the pairs of propositions that satisfy the union A. is the initial support degree of the evidence source o for the proposition E, is the initial support degree of the evidence source p for proposition F.

[0040] The present application adopts a hierarchical method to step-by-step fuse the "pixel-topology-prior" three-layer evidence source. Since the topology evidence source only marks the microtubule skeleton, its width is very small, therefore, first, the morphological dilation operation is used to expand the skeleton to match the microtubule width of the likelihood map, to solve the scale mismatch problem between the abstractness of the topology representation and the concreteness of the visual feature; then the Dempster combination rule is used to preliminarily fuse the likelihood map and the topology manifold uncertainty map, and the consistency of the visual feature and the topology structure is dynamically evaluated through the real-time calculation of the conflict factor; subsequently, two kinds of constraints including spatial continuity prior and edge false positive prior are introduced for directional optimization, and the preliminary fusion result obtained by data-driven is calibrated in a knowledge-driven manner.

[0041] In order to better measure the few-shot learning ability of the scheme of the present application, 10 samples are randomly extracted from the microtubule data set as training data in the embodiment of the present application, and 3 classes of Out-of-Distribution (OOD) data are selected as test data, which are different from the training data in microtubule density, staining method and imaging device clarity.

[0042] In the embodiment, the pre-trained SAM2 is used as the base large visual model. The segmentation effect is measured by 7 indexes, which are accuracy (Acc), precision (Precious), recall (Recall), specificity (Specificity), intersection over union (IoU), dice (Dice), bookmaker informedness (BM) and Matthews correlation coefficient (MCC). The value range of information degree and Matthews correlation coefficient is [-1, 1], BM will be negative when the prediction performance of the model is worse than random guessing, and MCC will be negative when the prediction result of the model presents a negative correlation with the true label.

[0043] To prove that the scheme of the present application has an advantage in few-shot learning, a comparative experiment is designed. The method of the present application and the other four segmentation models are compared comprehensively on 3-class out-of-distribution data by using eight evaluation indexes. Among them, the Unet model and the MaskRCNN model belong to the model trained from scratch. They are first initialized as random weights, and then learn the features of microtubule microscopic images on 10 training images. MaskRCNN belongs to a two-stage model. The first stage is to obtain the candidate box of the object to be segmented, and the second stage is to perform target detection and instance segmentation. In order to make the MaskRCNN model more suitable for the microtubule segmentation task, the first stage processing of the MaskRCNN model is removed. Since microtubules generally occupy the entire microscopic image area, the complete image is used as the candidate box. In the second stage, the target detection related module is removed, and only the mask output module is retained. The DPT model is a model based on the visual Transformer (ViT) architecture for dense task prediction. Compared with the first two models, it has a larger parameter quantity. Therefore, the present application uses the pre-trained weight of the DPT model on the ImageNet dataset, and then freezes the image encoder of the pre-trained DPT model, and makes it perform parameter-efficient fine-tuning on 10 microtubule microscopic images. Considering that the input image size required by the pre-trained weight may affect the effect of parameter-efficient fine-tuning, the present application specially selects two sizes of pre-trained DPT model weights, namely 224 and 384, and scales the microtubule image to the corresponding size by linear interpolation before parameter-efficient fine-tuning. The few-shot segmentation performance of each model is shown in Table 1.

[0044] Table 1 Comparison of segmentation performance of each model in few-shot learning

[0045] As can be seen from the data in Table 1, the method of the present application performs well on 3-class out-of-distribution data different from the training samples, and is much higher than the other four segmentation models in indicators other than recall rate. In order to show the specific segmentation results of each model, the present application selects a test sample as shown in Figure 4 As can be seen from Figure 4 , the microtubule image is relatively dense and has some background noise. The segmentation result of the present application scheme for this sample is shown in Figure 5 , from which it can be found that the method of the present application can better handle the microtubule array microscopic image with certain complexity.

[0046] The values of the other four models on the seven segmentation indicators are all low, and only the recall rate is very high, basically above 0.95, and even up to 1. This shows that these segmentation models do not have the ability to identify the filament structure in the microtubule array when segmenting microtubules. They can only label the fuzzy area of the microtubule array, which is obviously invalid segmentation. Figure 4The segmentation results of the microtubule images are shown in Figure 6 The segmentation results of the Mask RCNN are shown in Figure 7 The segmentation results of the pre-trained DPT model with a size of 224 are shown in Figure 8 The segmentation results of the pre-trained DPT model with a size of 384 are shown in Figure 9 As can be seen from Figure 9 Under the condition that the scale of the training sample is limited, none of the four segmentation models can accurately segment the microtubule array, and they all tend to predict large block regions rather than filamentous regions.

[0047] The embodiment provides a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to realize the cell microtubule array uncertainty quantification segmentation method of the fused large visual model.

[0048] The above description of the application only expresses the implementation of the embodiments of the application, and cannot be understood as a limitation on the scope of the patent application, nor as a limitation on the structure of the embodiments of the application in any form. It should be noted that, for those skilled in the art, without departing from the concept of the embodiments of the application, a number of changes and improvements can be made, which are within the scope of protection of the embodiments of the application.

Claims

1. A cell microtubule array uncertainty quantification segmentation method integrating a large visual model, characterized by: Here are the steps: Step 1: Collect microscopic images of cell microtubules and annotate microtubule morphology information to construct a microtubule dataset; Step 2: Combine the deep trainable hint layer with the low-rank adaptive method to perform structural optimization and domain fine-tuning on the base large visual model to construct a large visual model for microtubule segmentation; Step 3: Use the large visual model for microtubule segmentation to process the cell microtubule microscopic image to obtain the microtubule likelihood map at the pixel level; Step 4: Perform topological analysis on the microtubule likelihood graph to obtain several manifold structures. The union of the manifold structures constitutes the microtubule topological skeleton. Step 5: Calculate the uncertainty interval of each manifold structure in the topological skeleton through the manifold uncertainty interval quantification network; Step 6: Develop a set of prior knowledge based on microtubule biological characteristics and heuristic rules, and design the prior knowledge into a computable metric function; Step 7: Construct a "pixel-topology-prior" three-source evidence fusion framework based on DS evidence theory. After explicitly modeling the internal uncertainty of the evidence sources, the complementary or conflicting information of each evidence source is effectively integrated to obtain accurate microtubule segmentation results.

2. The cell microtubule array uncertainty quantification segmentation method according to claim 1, characterized in that: The microtubule segmentation large visual model consists of an image embedding module, an image encoder, and a mask decoder. The image embedding module divides the input cell microtubule microscopic image into blocks to obtain image blocks. The image blocks are then embedded into vectors. The position codes of the image blocks are then incorporated into the embedded vectors to obtain the initial features of the cell microtubule microscopic image. The initial features of the cell microtubule microscopic image are then processed by the image encoder and mask decoder to finally obtain the microtubule likelihood map.

3. The cell microtubule array uncertainty quantification segmentation method according to claim 2, characterized in that: The image encoder is based on the image encoder of the base large visual model, and the prompt word encoding module is removed. A deep trainable prompt layer is inserted before each level of the image encoder, and the deep trainable prompt layer is mainly composed of convolution, regularization and activation operations.

4. The cell microtubule array uncertainty quantification segmentation method according to claim 2, characterized in that: A low-rank adaptation method is used to insert a low-rank adaptation module into each layer of the image encoder of the base large vision model.

5. The cell microtubule array uncertainty quantification segmentation method according to claim 4, characterized in that: In each layer of the image encoder of the microtubule segmentation large visual model, the deep trainable hint layer first processes the vector input to this layer. The feature vector obtained after processing will be copied, and the two feature vectors are processed by the frozen base large visual model layer and the low-rank adaptive module respectively. The feature vector is converted into a low-rank matrix through the down-projection matrix, and the low-rank matrix is ​​then converted into a feature vector through the up-projection matrix. The feature vectors output by the frozen base large visual model layer and the low-rank adaptive module are fused according to certain weights, and the resulting fused vector will be processed by another part of the frozen base large visual model layer. The feature vectors obtained after processing will be passed to the next layer and the neck module respectively.

6. The cell microtubule array uncertainty quantification segmentation method according to claim 1, characterized in that: In step 4, random Gaussian noise is first added to the microtubule likelihood map obtained by the large visual model of microtubule segmentation to perturb the microtubule likelihood map. Then, the perturbed microtubule likelihood map is analyzed using discrete Morse theory to obtain several manifold structures. The union of the manifold structures constitutes the microtubule topological skeleton.

7. The cell microtubule array uncertainty quantification segmentation method according to claim 1, characterized in that: The manifold uncertainty interval quantization network takes the manifold structure as input and has three branches; Branch 1 considers manifold structures as nodes to construct a graph structure. If there is overlap between two manifold structures, it is considered that there is an edge between the two nodes. Then, a graph convolutional network is used to process the graph structure to obtain a spatial feature map that captures the adjacent relationship between the manifold structures. The second branch embeds all manifold structures into vectors to construct manifold features. The manifold features are converted to the frequency domain after fast Fourier transform, processed by multiple Fourier operators in the frequency domain, and then restored from the frequency domain through inverse Fourier transform to obtain the frequency domain feature map. Four manifold scalar values ​​of the branch three statistical manifold structure are the average manifold size, the average manifold likelihood, the persistence of the manifold structure calculated by the persistence homology method, and the basic noise; The spatial domain feature maps, frequency domain feature maps and manifold scalar values ​​obtained from the last three branches will be fused to jointly calculate the uncertainty values ​​and logarithmic variances of each manifold structure.

8. The cell microtubule array uncertainty quantification segmentation method according to claim 7, characterized in that: In step 5, Monte Carlo sampling is used to randomly discard a certain proportion of the internal parameters of the manifold uncertainty interval quantization network to perturb the manifold uncertainty interval quantization network.

9. The cell microtubule array uncertainty quantification segmentation method according to claim 1, characterized in that: In step six, a set of prior knowledge includes two points: one is the spatial continuity prior, and the other is the edge false positive prior.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the cell microtubule array uncertainty quantification segmentation method according to any one of claims 1 to 9 is implemented.

Images