A small sample dense particle image segmentation method based on a visual large model fusion

Abstract

The application discloses a small sample dense particle image segmentation method based on a fusion visual large model. The method comprises the following steps: 1. image cropping and screening based on the image representative principle; 2. inputting a target image into a visual large model and a particle segmentation network; 3. training the overall model, and freezing the pre-training parameters of the image encoder of the visual large model during the training; 4. inputting a particle image to be segmented into the trained model to obtain edge information and particle internal distance transformation prediction of the target image, and combining a watershed algorithm to output a corresponding prediction mask as a particle segmentation result. The application utilizes the feature extraction capability of the visual large model, combines the particle segmentation network, fine-tunes the large model in the training process, and makes the large model more suitable for the dense particle dataset; a sample selection method based on clustering is adopted to ensure the effectiveness of the samples while reducing the number of samples, effectively reduces the number of training data, and enables the model structure to efficiently learn image features.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and artificial intelligence, specifically to a method for small-sample dense particle image segmentation that integrates a large visual model. Background Technology

[0002] Particle segmentation holds significant importance in fields such as biomedicine and materials science. In biomedicine, it can be used to separate and identify microstructures such as cells, aiding in treatment evaluation. In materials science, it can assist in analyzing particle morphology and microstructure, helping researchers understand material properties, interactions, and performance, and optimizing material preparation and applications. Therefore, particle segmentation provides quantitative and qualitative analysis capabilities for microstructures and particles in complex scenes, offering a foundation and support for scientific research, medical diagnosis, image processing, and engineering applications. With the continuous development of deep learning, instance segmentation methods have been widely applied in these fields. However, training these methods typically relies on sufficient densely labeled samples, which presents a significant challenge in particle segmentation tasks. Particles in real-world applications are often highly densely packed; due to their large number and variable size, pixel-level annotation is more time-consuming and labor-intensive than image-level annotation, indirectly leading to difficulties in obtaining dataset samples. Limited data often fails to leverage the powerful advantages of deep learning, further complicating the automatic and accurate segmentation of particle images and increasing the difficulty of particle segmentation.

[0003] OpenAI's Segment Anything Model (SAM), proposed last year, ushered in a new era of segmentation and validated its effectiveness in natural image scenes. Its powerful feature extraction capabilities have led to its widespread application in other fields. However, due to the lack of images of different particle types in the datasets used by the model, it still faces significant limitations in dense particle segmentation. Furthermore, applying SAM to particle segmentation presents a crucial challenge: SAM requires both point-based and bounding box-based cues to guide target segmentation and decoding. Although SAM provides an automatic mode that generates bounding box-based cues by filtering a sliding window of the input image, even simple manual annotation struggles to guarantee satisfactory segmentation predictions.

[0004] Therefore, how to fully utilize the feature extraction capabilities of large visual models and combine them with deep learning network models to ensure the accuracy of particle segmentation while reducing the number of training samples has become an important issue. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention proposes a small-sample dense particle image segmentation method that integrates a large visual model. This method is suitable for scenarios with a large number of densely distributed particles and can effectively overcome the challenge of ensuring the accuracy of particle segmentation when the dataset is small.

[0006] The technical solution adopted in this invention includes the following steps:

[0007] I. A Small-Sample Dense Particle Image Segmentation Method Integrating Large Visual Models

[0008] Includes the following steps:

[0009] 1) Cropping is performed on small sample dense particle images, and the cropped image patches are selected based on the principle of image representativeness;

[0010] 2) Input the filtered image patches into the image segmentation model for feature extraction. The image segmentation model consists of an image encoder of a large visual model and a particle segmentation network.

[0011] 3) Design a loss function to train the image segmentation model; during training, freeze the pre-training parameters of the large visual model image encoder;

[0012] 4) Input the particle image to be segmented into the trained image segmentation model to obtain the edge information of the target image and the predicted internal distance transformation of the particles. Combine the watershed algorithm to output the corresponding prediction mask as the particle segmentation result.

[0013] Step 1) specifically refers to:

[0014] A uniform sliding window with a fixed step size is used to randomly select an image from the training set for cropping, resulting in several image blocks of fixed size. The image blocks are then divided into several sub-regions.

[0015] The K-Means algorithm is used to classify image patches and their sub-regions: image patches and their sub-regions are input into a ResNet50 pre-trained ImageNet dataset to calculate feature vectors, and clustering is performed using the Euclidean distance of the feature vectors; the specific clustering method is as follows:

[0016] The image patch is divided into K1 clusters, and for each cluster C k The image block of size h×w is cut into 4 parts. The sub-regions are then divided into K2 clusters, resulting in a total of K1×K2 clusters.

[0017] Further evaluate the information representativeness of image patches and image sub-regions. The information representativeness of image patches and image sub-regions is calculated using the following formula:

[0018]

[0019] Where x represents an image patch of size h×w, x k For C k The selected image patch; f(·) is the feature extractor; x p Indicates that x is of size The subregion of C; K(x) represents C k The center vector in a cluster is the average of all feature vectors in that category; k(x) represents the center vector of the cluster with the most sub-regions.

[0020] In the formula, the two terms represent the representativeness of image patch information (the feature difference between each image patch and the cluster center in K1 categories) and the representativeness of image sub-region information (the sum of the feature differences between each sub-region and the cluster center with the most sub-region in K2 categories); the lower the value, the higher the representativeness of the image patch.

[0021] The above formula is used to calculate the cropped image blocks, and the calculation results are sorted according to the numerical value. The more representative image blocks are then selected based on the required quantity.

[0022] The large visual model in step 2) is the Segment Anything Model. The large visual model is pre-trained using the SA-1B dataset to obtain the pre-trained model parameter file. The image encoder is a ViT unit, which is used to extract features from the image to obtain the image feature vector.

[0023] The particle segmentation network in step 2) is designed based on the UNet network structure, including an encoder, a decoder, and two branches: edge prediction and distance transform regression. The encoder contains four downsampling operations, each preceded by two convolutional operations with a kernel size of 3*3. After each convolutional operation, a ReLU function is used as the activation layer. Downsampling is implemented using max pooling layers. The decoder contains four upsampling operations, each preceded by two convolutional operations with a kernel size of 3*3. After each convolutional operation, a ReLU function is used as the activation layer. Upsampling is implemented using nearest neighbor interpolation. The edge prediction branch includes two convolutional layers and an activation layer connected in sequence. The last layer is activated using the Sigmoid function to predict whether each pixel belongs to an edge point. The distance transform branch includes two convolutional layers to predict the value of each pixel and calculate the distance between the pixel and the maximum value point within its particle region.

[0024] Step 2) specifically involves: inputting the image into the image encoder of the large visual model and the encoder of the particle segmentation network to obtain the corresponding feature vectors; concatenating the image embedding vector obtained by the image encoder with the output of the encoder of the particle segmentation network and inputting it into the decoder of the particle segmentation network; inputting the output of the decoder into the edge prediction branch and the distance transformation branch respectively; and post-processing the outputs of the two branches using the watershed algorithm to obtain the final segmentation result.

[0025] The loss function expression in step 3) is as follows:

[0026]

[0027] L DIST =(x i -y i ) 2

[0028] L = L EDGE +L DIST

[0029] Where L is the total loss function, L EDGE Using cross-entropy loss, L DIST Using mean squared error loss, p i To predict the confidence level of the i-th pixel in the mask, g i y represents the confidence level of the i-th pixel in the edge ground truth mask, where N is the total number of pixels in the image; i Let x be the pixel value of the i-th pixel in the true distance transform. i Let be the pixel value of the i-th pixel in the prediction.

[0030] The particle image to be segmented is input into the trained image segmentation model to obtain edge contours and distance regression predictions, and then the particle segmentation result is obtained after watershed transformation.

[0031] II. A type of terminal

[0032] Including memory and processor;

[0033] The memory is used to store computer programs; the processor is used to implement the above-described small-sample dense particle image segmentation method that integrates large visual models when the computer programs are executed.

[0034] III. A computer-readable storage medium

[0035] The storage medium stores a computer program, which, when executed by a processor, implements the above-described method for small-sample dense particle image segmentation based on a fused visual large model.

[0036] The beneficial effects of this invention are:

[0037] This invention proposes a fine-tuning strategy that integrates a large visual model. During training, the large model's image encoder is fine-tuned simultaneously. By leveraging the feature extraction capabilities of the large visual model and combining it with a particle segmentation network, the large model becomes more adaptable to dense particle datasets, effectively reducing the amount of training data and ensuring the segmentation accuracy of the particle segmentation network.

[0038] In order to enable the model structure to learn image features efficiently, this invention adopts a clustering-based sample selection method, which ensures the effectiveness of the samples while keeping the number of samples small, and provides a new method for small-sample dense particle image segmentation. Attached Figure Description

[0039] Figure 1 This is a flowchart of the image block filtering process based on image representativeness according to an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the visual large model-particle segmentation model structure according to an embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram of the feature fusion module structure in the image segmentation model of this invention. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Figure 1 A flowchart of image-representative sample selection based on an embodiment of the present invention is disclosed. Figure 1 In the illustrated embodiment, the present invention proposes a sample selection method based on image representativeness.

[0044] This invention provides a small-sample dense particle image segmentation method that integrates a large visual model, comprising the following steps:

[0045] Step 1) Select an image from the dataset with a resolution of 768*1024; for example... Figure 1 As shown, several 512*512 image patches are cropped from the image using a uniform sliding window with a fixed step size of 15. Each image patch is uniformly cropped into four sub-regions of the same size. The K-Means algorithm is then used to group the image patches to further evaluate their representativeness. The image patches and their sub-regions are input into a ResNet50 pre-trained ImageNet dataset to calculate feature vectors, and clustering is performed using the Euclidean distance of these feature vectors.

[0046] First, the image patch is divided into K1 groups, and for each group C... kThe image patch, initially 512*512 pixels, is uniformly cropped into four 256*256 sub-regions. Then, each sub-region is divided into K2 groups, resulting in a total of K1×K2 clusters. Each category C... k Each image contains several 512*512 image blocks. The feature differences between all image blocks and their internal sub-regions are calculated using the following formula:

[0047]

[0048] Where x represents an image patch of size 512*512, x k For C k The selected image patch; f(·) is the feature extractor, x p K(x) represents a subregion of size 256*256 in x; K(x) refers to C. k The center vector in a cluster is the average of all feature vectors in that category; the cluster with the most sub-regions is designated as k. * , where k(x) represents the center vector of the classification. The two terms in the formula represent the coarse representativeness of the image information (the feature difference between each image patch and the cluster center in K1 categories) and the fine representativeness of the image information (the sum of feature differences between each sub-region and the cluster center with the most data in each K2 subgroup). Lower values ​​indicate higher representativeness and internal consistency of the image patch.

[0049] Based on this algorithm, the value of each image patch is calculated, and the minimum value corresponding to each category in K1 is output. In the specific experiment, K1=16 and K2=4 were selected after testing.

[0050] Step 2) During the training phase, a fine-tuning architecture is built for the large visual model, such as... Figure 2 As shown, the feature fusion module ensures that the image encoder output and the encoder output of the particle segmentation network are fused together before being input into the decoder of the particle segmentation network. Figure 3 As shown.

[0051] In the initial training phase, the image encoder of the large-scale visual model freezes its pre-trained parameters. In this embodiment, the image encoder is a Visual Transformer (ViT) pre-trained on the SA-1B dataset, consisting of 12 ViT units. Each ViT unit has a structure of LayerNorm, multi-head attention, LayerNorm, and multilayer perceptron. During training, the original image encoder parameters are frozen, and only the matrix values ​​are adjusted. The multilayer perceptron has a structure of linear layers, a GELU activation function, and another linear layer. The image encoder vector dimension is 756, the image encoder depth is 12, and the number of attention heads is 12. The training process uses step 1) to replace the random cropping operation to obtain image patches. The images are preprocessed (including image flipping, rotation, translation, etc.) to train the large-scale visual model-particle segmentation network model to adapt to the small-sample dense particle segmentation task.

[0052] In this implementation, the particle segmentation network is designed based on the UNet network structure, including an encoder, a decoder, and two branches: edge prediction and distance transform regression. The encoder includes four downsampling operations, preceded by two convolutional operations. Each convolutional kernel is set to a size of 3*3. After each convolutional operation, a ReLU function is used as the activation layer. Downsampling is achieved through max pooling layers. The decoder includes four upsampling operations, preceded by two convolutional operations. Each convolutional kernel is set to a size of 3*3. After each convolutional operation, a ReLU function is used as the activation layer. Upsampling is achieved using nearest neighbor interpolation. The last layer of the edge prediction branch uses the Sigmoid function for activation. The distance transform branch outputs the model's predicted values.

[0053] The image is input into the encoder of the basic vision model and the particle segmentation network to obtain the corresponding feature vectors. After feature fusion, the vectors are input into the decoder of the particle segmentation network to obtain the outputs of edge prediction and distance transformation.

[0054] The edge prediction is to predict whether a pixel in the target image belongs to an edge point, while the distance transform prediction is to predict the value of the pixel, which is used to calculate its distance to the maximum value point in the region.

[0055] This implementation method calculates the loss L using the following loss function:

[0056]

[0057] L DIST =(x i –y i ) 2

[0058] L = L EDGE +L DIST

[0059] Where L is the total loss function, L EDGE Using cross-entropy loss, L DIST Using mean squared error loss, p i To predict the confidence level of the i-th pixel in the mask, g i y represents the confidence level of the i-th pixel in the edge ground truth mask, where N is the total number of pixels in the image; i Let x be the pixel value of the i-th pixel in the true distance transform. i Let be the pixel value of the i-th pixel in the prediction. The optimizer uses Adam with an initial learning rate of 1e-4, decaying by 0.1 weights. After a certain number of training rounds, the learning rate is adjusted to 1e-5 to jointly fine-tune the parameters of the last ViT unit of the large model image encoder.

[0060] After the network structure described in step 3) is trained, the target image can be directly input into the network to obtain edge contours and distance regression predictions. After watershed transformation, particle segmentation results are obtained.

[0061] Experiments show that, when using only a single 768*1024 resolution particle image:

[0062] 1. The model trained on unfiltered image patches can achieve an Aggregated Jaccard Index of 0.6145; while the model trained on filtered images can achieve a reference performance of Aggregated Jaccard Index of 0.6217, demonstrating the effectiveness of image filtering.

[0063] 2. After training, the particle segmentation network with a large model image encoder achieves an AJI score of 0.6217, while the particle segmentation network without a large model image encoder achieves an AJI score of only 0.5957, and the performance of the untrained large model image is only 0.6056, demonstrating the effectiveness of the visual large model-particle segmentation network.

Claims

1. A method for small-sample dense particle image segmentation that integrates a large visual model, characterized in that, Includes the following steps: 1) Cropping is performed on small sample dense particle images, and the cropped image patches are selected based on the principle of image representativeness; Step 1) specifically refers to: A uniform sliding window with a fixed step size is used to randomly select an image from the training set for cropping, resulting in several image blocks of fixed size. The image blocks are then divided into several sub-regions. The K-Means algorithm is used to classify image patches and their sub-regions: the image patches and their sub-regions are input into the ImageNet pre-trained ResNet50 to calculate feature vectors, and clustering is performed using the Euclidean distance of the feature vectors; Further evaluate the information representativeness of image patches and image sub-regions; The clustering method is specifically as follows: Divide the image patch into There are clusters, for each category , will the size be The image block is cut into 4 The sub-regions are then divided into... Clusters were obtained, yielding a total of [number] clusters. One cluster; The information representativeness of an image patch and the information representativeness of an image sub-region are calculated using the following formula: in, Indicates size is Image blocks, From The selected image patch; For feature extractors; express medium size subregion; express The center vector in a cluster is the average of all feature vectors in that category; This represents the center vector of the cluster with the most sub-regions; In the formula, the two terms represent the representativeness of image patch information and the representativeness of image sub-region information, respectively; The above formula is used to calculate the image blocks after cropping, and the calculation results are sorted according to the numerical value. The lower the value, the more representative the image block is. The more representative image blocks are obtained by filtering according to the required number. 2) Input the cropped image patch into the image segmentation model for feature extraction. The image segmentation model consists of an image encoder of a large visual model and a particle segmentation network. Step 2) specifically involves: inputting the image into the image encoder of the large visual model and the encoder of the particle segmentation network to obtain the corresponding feature vectors; concatenating the image embedding vector obtained by the image encoder with the output of the encoder of the particle segmentation network and inputting it into the decoder of the particle segmentation network; inputting the output of the decoder into the edge prediction branch and the distance transformation branch respectively; and post-processing the outputs of the two branches using the watershed algorithm to obtain the final segmentation result. 3) Design a loss function to train the image segmentation model; during training, freeze the pre-training parameters of the large visual model image encoder; 4) Input the particle image to be segmented into the trained image segmentation model to obtain the edge information of the target image and the predicted internal distance transformation of the particles. Combine the watershed algorithm to output the corresponding prediction mask as the particle segmentation result.

2. The small-sample dense particle image segmentation method according to claim 1, characterized in that: The large visual model in step 2) is a segmentation model. The large visual model is pre-trained using the SA-1B dataset to obtain the pre-trained model parameter file. The image encoder is a ViT unit used to extract features from the image and obtain the image feature vector. The particle segmentation network in step 2) is designed based on the UNet network structure, including an encoder, a decoder, and two branches: edge prediction and distance transform regression. The encoder contains four downsampling operations, each preceded by two convolutional operations with a kernel size of 3*3. After each convolutional operation, a ReLU function is used as the activation layer. Downsampling is implemented using a max pooling layer. The decoder contains four upsampling operations, each preceded by two convolutional operations with a kernel size of 3*3. After each convolutional operation, a ReLU function is used as the activation layer. Upsampling is implemented using a nearest neighbor interpolation algorithm. The edge prediction branch includes two convolutional layers and an activation layer connected in sequence. The last layer is activated using a Sigmoid function to predict whether each pixel belongs to an edge point. The distance transform branch includes two convolutional layers to predict the value of each pixel and calculate the distance between the pixel and the maximum value point within its particle region.

3. The small-sample dense particle image segmentation method according to claim 1, characterized in that: The loss function expression in step 3) is as follows: in, For the total loss function, Using cross-entropy loss, Mean squared error loss is used. To predict the first in the mask Confidence level of each pixel For the edge truth mask, the first Confidence level of each pixel This represents the total number of pixels in the image. The first value in the distance transformation truth value The pixel value of each pixel. For the prediction of the first The pixel value of each pixel.

4. The small-sample dense particle image segmentation method according to claim 1, characterized in that: The particle image to be segmented is input into the trained image segmentation model to obtain edge contours and distance regression predictions, and then the particle segmentation result is obtained after watershed transformation.

5. A terminal, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the small-sample dense particle image segmentation method according to any one of claims 1 to 4 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the small-sample dense particle image segmentation method of any one of claims 1 to 4, which integrates large visual models.

Images