Image segmentation model training method, image segmentation method and system
By introducing uncertainty fusion and Gaussian distribution context information generation strategies into the SAM model, combined with contrast learning, the problem of poor sample segmentation effect of SAM model in medical images and complex scenarios is solved, and the accuracy of image segmentation is improved and the accuracy of image segmentation is improved. Great.
Patent Information
- Application Number
- CN202411132090.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-19
AI Technical Summary
The SAM model has poor segmentation effect in medical images and complex scenes, and its robustness and reliability are insufficient. The existing improvement methods have failed to effectively improve the segmentation performance of difficult pixels.
A difficult region learning strategy for uncertainty fusion is introduced, local difficult region images are generated through Gaussian distribution context information, nonlinear projection mapping is used to reduce dimensionality, and compute comparison losses in combination with contrast learning, to enhance the learning ability of SAM models for difficult regions.
The learning ability and segmentation accuracy of SAM model for difficult areas is improved, and the generalization and robustness of image segmentation are enhanced.
Smart Images

Figure CN118657944B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a training method for an image segmentation model, an image segmentation method and a system. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] Image segmentation is an important research direction in the field of computer vision. It aims to divide digital images into several regions with semantic information in order to understand and analyze the image content. Image segmentation has broad application prospects in medical image analysis, autonomous driving systems, security monitoring and other fields.
[0004] Recently, a general segmentation model called SAM (Segment Anything Model) has emerged in the field of visual segmentation. The model consists of three parts: image encoder, prompt encoder and mask decoder. It is obtained by training the model with the training set obtained from the data engine. It has zero-shot capabilities and can perform image segmentation in different downstream tasks. Although the SAM model can perform image segmentation in different downstream tasks and has good versatility, it also faces a series of challenges and difficulties. Specifically, although SAM has generalization, since most of SAM's training data sets are natural images, there is relatively little data training in other fields, and the segmentation effect is relatively poor on specific downstream tasks, such as: in the segmentation of lesions in medical images; SAM has poor results when segmenting difficult sample images in complex scenes, and its robustness and reliability are not good enough.
[0005] The above shortcomings and limitations limit SAM from becoming a stable and reliable general large model in the field of visual segmentation, which is undoubtedly a severe challenge for the development of large visual models. The inventors found that the existing deep learning methods for improving the SAM model mainly fine-tune the SAM for specific downstream tasks, transfer the knowledge of the SAM model to other models, or fuse the SAM model with other models to combine the characteristics of different models, but these methods do not perceive the difficult-to-segment areas, which limits the improvement of the model's segmentation performance for difficult pixels. Summary of the invention
[0006] In order to solve the above problems, the present invention provides a training method for an image segmentation model, an image segmentation method and a system. The scheme introduces a difficult area learning strategy of uncertainty fusion in the training of the SAM model, which effectively improves the SAM model's learning ability for uncertainty knowledge; by introducing a local difficult area generation strategy based on Gaussian distribution context information, the SAM's learning ability for difficult areas is effectively enhanced; at the same time, for the generated difficult areas, nonlinear projection mapping is used to reduce the dimension of the features of the local difficult area map, so that the SAM model can focus on the learning of the discriminative features of the difficult areas, and local difficult area images with the same difficult areas but different context information are used as positive and negative sample pairs, and the contrast loss based on the difficult areas is calculated by contrastive learning, which effectively enhances the distinguishability between the difficult areas and the background areas, and further improves the accuracy, generalization and robustness of image segmentation.
[0007] According to a first aspect of an embodiment of the present invention, a method for training an image segmentation model is provided, comprising:
[0008] Based on the training sample images, using the image segmentation model to be trained, an initial prediction result of the segmentation is obtained;
[0009] Based on the initial prediction results, an uncertainty estimation map is obtained by calculating the pixel-by-pixel difference between the segmentation mask and its label; and an entropy map is obtained by calculating the entropy value of the segmentation mask pixels;
[0010] Based on the obtained uncertainty estimation map and entropy map, a difficulty probability map is obtained by fusion;
[0011] Based on the obtained difficulty probability map, a local difficulty image based on the Gaussian distribution context is generated;
[0012] Based on the obtained difficulty probability map, a perceptual difficulty loss is constructed; based on the local difficulty image including the same difficult area, a difficult area contrast loss is constructed;
[0013] Based on the loss function of fusing the perceptual difficulty loss and the difficult area contrast loss, the image segmentation model is trained to obtain a trained image segmentation model.
[0014] Furthermore, the image segmentation model to be trained is used to obtain an initial prediction result of the segmentation, specifically: based on the sample image, feature extraction is performed through the image encoder of the image segmentation model; based on the extracted features and the output of the prompt encoder in the image segmentation model, the initial prediction result of the segmentation is obtained through the mask decoder of the image segmentation model.
[0015] Furthermore, the difficulty probability map is obtained by fusing the obtained uncertainty estimation map and entropy map, specifically: the uncertainty estimation map and the entropy map are normalized respectively, and the difficulty probability map is obtained by weighted summation based on the normalized uncertainty estimation map and entropy map.
[0016] Furthermore, based on the obtained difficulty probability map, a local difficulty image based on the Gaussian distribution context is generated, specifically: based on the probability value of each pixel point in the difficulty probability map, the center point of the difficulty area is calculated by weighted average; based on the center point of the difficulty area, another center point of the difficulty area is obtained in combination with the random offset generated by the Gaussian distribution; and two local difficulty images of the same size are generated according to preset length and width using the two center points as the starting points of the rectangular frame.
[0017] Furthermore, the perceived difficulty loss is specifically expressed as:
[0018]
[0019]
[0020] in, represents the size of the smallest batch of labeled images. represents the cross entropy loss, is the loss function based on the weights of the difficulty probability map and the labeled samples, w i is the importance weight matrix, y i is the initial prediction result of SAM for the i-th image, is the i-th sample label, is the pixel entropy value of the i-th image segmentation mask, is the probability value of the difficulty probability map of the i-th image.
[0021] Furthermore, the difficult area contrast loss is specifically expressed as:
[0022]
[0023] in, represents a positive sample pair, represents a negative sample pair, represents an exponential function that calculates the cosine similarity between two features. is the temperature parameter that controls the clarity of the contrast loss function, , N represents the number of pixels in the overlapping area, represents the sum of the difficulty probability values of the positive sample pairs, S represents the number of pixels in the non-overlapping area, Represents the sum of the difficulty probability values of the negative sample pairs. For two local difficult images A and B generated based on the same difficult area, is the feature of the overlapping area in image A after nonlinear projection mapping, is the feature of the overlapping area in the B image after nonlinear projection mapping, is the feature of the non-overlapping area in the B image after nonlinear projection mapping, and e is a natural constant.
[0024] Furthermore, the image segmentation model adopts a SAM network model, and the SAM network model includes an image encoder, a hint encoder and a mask decoder.
[0025] Furthermore, the image encoder adopts a Vision Transformer structure based on an attention mechanism.
[0026] According to a second aspect of an embodiment of the present invention, there is provided an image segmentation method, comprising:
[0027] Obtain an image to be segmented;
[0028] Based on the image, an image segmentation result is obtained using a pre-trained image segmentation model; wherein the image segmentation model adopts the above-mentioned training method of an image segmentation model.
[0029] According to a third aspect of an embodiment of the present invention, there is provided an image segmentation system, comprising:
[0030] An image acquisition unit, used for acquiring an image to be segmented;
[0031] An image segmentation unit is used to obtain an image segmentation result based on the image using a pre-trained image segmentation model; wherein the image segmentation model adopts the above-mentioned training method of an image segmentation model.
[0032] One or more of the above technical solutions have the following beneficial effects:
[0033] The present invention provides an image segmentation method, system, medium and device based on difficulty perception fusion. In the training of the SAM model, the scheme introduces a difficult area learning strategy of uncertainty fusion, fuses the uncertainty probability map and the entropy map to obtain a difficult probability map, and then assigns preset weights to the pixels in the difficult probability map, and applies the weights to the sample loss to enhance the learning ability of the SAM model for uncertainty knowledge. In order to further enhance the distinguishability of difficult area features from background areas, the scheme introduces a local difficult area generation strategy based on Gaussian distribution context information, based on the obtained difficult areas, obtains multiple circumscribed rectangular areas containing difficult areas by random cropping on the basis of the original image, and generates local difficult areas by randomly introducing different context information to these areas on the basis of the difficult areas, which can effectively enhance the learning ability of SAM for difficult areas. The solution of the present invention uses nonlinear projection mapping to reduce the dimension of the features of the local difficult area map for the generated difficult area, so that the SAM model can focus on the learning of the discriminative features of the difficult area, and uses the local difficult area images with the same difficult area but different context information as positive and negative sample pairs, and uses contrast learning to calculate the contrast loss based on the difficult area, which can effectively enhance the distinguishability between the difficult area and the background area, and further improve the accuracy, generalization and robustness of image segmentation. The advantages of the additional aspects of the present invention will be partially given in the following description, and part will become obvious from the following description, or will be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0035] Figure 1 It is a schematic diagram of the overall process of the image segmentation method based on difficulty perception fusion described in an embodiment of the present invention;
[0036] Figure 2 Schematic diagram of the SAM model structure described in an embodiment of the present invention;
[0037] Figure 3 Schematic diagram of the structure of an image segmentation system based on difficulty perception fusion described in an embodiment of the present invention. DETAILED DESCRIPTION
[0038] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0039] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.
[0040] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0041] Terminology explanation:
[0042] Difficult-to-segment areas include the following situations: the boundaries of the area are not clear, that is, the edges of the object are blurred or there are many small edge parts; the area has complex textures or structures, which makes it difficult for the segmentation algorithm to distinguish different objects or segment a single object; there are noisy or interfering areas in the image.
[0043] The difficult area refers to a difficult probability map obtained by fusing the uncertainty estimation map and the entropy map, and is an area in the difficult probability map where the probability value is greater than a preset threshold. In the scheme described in the present invention, the difficult segmentation area in the image is found through the difficult probability map.
[0044] Embodiment 1:
[0045] The purpose of this embodiment is to provide a training method for an image segmentation model.
[0046] like Figure 1 As shown, a training method for an image segmentation model specifically includes the following processing steps:
[0047] Step 1: Based on the training sample image, use the image segmentation model to be trained to obtain the initial prediction result of the segmentation;
[0048] In a specific implementation, the image segmentation model to be trained is used to obtain the initial prediction result of the segmentation, specifically: based on the sample image, feature extraction is performed through the image encoder of the image segmentation model; based on the extracted features and the output of the prompt encoder in the image segmentation model, the initial prediction result of the segmentation is obtained through the mask decoder of the image segmentation model.
[0049] Specifically, the following processing needs to be performed:
[0050] First, it is necessary to construct a training data set. During the data collection process, this embodiment targets various types of data sets, and the original data samples may have inconsistent image sizes. In order to enable the deep network model to learn more effectively, the image sizes of the original data sample set are uniformly sized. Secondly, in order to solve the problem of lack of labeled data sets, random data enhancement operations are performed on the data sets to expand the data sets. The original data sets are operated by random cropping, scaling, translation, horizontal or vertical flipping, etc., and the labels of the expanded labeled data sets remain unchanged. Finally, the data sets are divided into training sets and test sets in a ratio of 3:1.
[0051] Secondly, input the data of the dataset into the SAM network, firstly transform the image The image features are extracted by sending it to the image encoder, which is composed of a Vision Transformer based on the attention mechanism. Then, the SAM model is manually given a box prompt. We perform a prompt augmentation process to perturb the initial prompt by random shifting to generate an augmented bounding box prompt. , where b i refers to the i-th box prompt given, b in Refers to the nth augmented box hint after augmentation in the i-th box hint, where n is the predefined number of augmentations. The hint is fed into the hint encoder as a hint vector, and finally the features and hints are fed into the mask decoder for fusion. The predicted segmentation result is obtained by using the cross attention operation and the two-layer Transformer decoder decoding operation. ,in, is the augmented bounding box hint of the input, It is the function representation of SAM model.
[0052] In a specific implementation, the manual box prompt for the SAM model is specifically as follows: the operation of manually giving a box prompt outside the SAM model is to use the mouse to draw a rectangular box. For the inside of the SAM model, it is the following process: the representation of the box is constructed by two embedding pairs, which contain the position information of the two corners of the box and the learnable embedding vector representing the direction of the two corners. An embedding pair contains the position information of a corner of the box and the learnable embedding vector of the direction of this corner. The two corners are the upper left corner and the lower right corner of the rectangular box. The position information is represented by the position code (the spatial coordinates are multiplied by the vector of the Gaussian distribution to describe the position). The position code and the directional learnable embedding vector dimensions are mapped to 256 dimensions, and the position code and the directional learnable embedding vector are added to represent the input of the box prompt to the mask decoder.
[0053] Step 2: Based on the initial prediction results, obtain an uncertainty estimation map by calculating the pixel-by-pixel difference between the segmentation mask and its label; and obtain an entropy map by calculating the segmentation mask pixel entropy value;
[0054] Specifically, firstly, the uncertainty estimation map (Uncertainty Estimation Map) and the entropy map (Entropy Map) are obtained according to the initial prediction results.
[0055] The uncertainty estimate map is obtained by calculating the pixel-by-pixel difference between the segmentation mask and the label , the entropy map is obtained by calculating the entropy value of the segmentation mask pixels .
[0056] in, No. i Samples passed n After the prompt is expanded, n The average mask of masks, For the i The samples passed through j The mask obtained after the hint augmentation box hint, n To indicate the number of augmented boxes, For the i Sample labels, For the input i The image is passed through j The probability that the mask obtained after the box is enlarged is the true mask.
[0057] Entropy maps and uncertainty estimation maps are usually used to represent the amount of information or degree of uncertainty for each pixel in an image. Fusion of these two images can provide more comprehensive and accurate information for finding difficult areas from multiple perspectives.
[0058] Step 3: Based on the obtained uncertainty estimation map and entropy map, the difficulty probability map is obtained by fusion;
[0059] Specifically, first, the entropy map and uncertainty estimation map are normalized to map their value ranges to the same [0,1] interval, and then weights are assigned to the uncertainty estimation map and entropy map. Finally, the normalized entropy map and uncertainty estimation map are weighted and summed according to the assigned weights. The fused difficulty probability map. In order to better learn this part of knowledge, the pixels in the difficulty probability map are given a larger weight, and the learning of this type of pixel knowledge is focused.
[0060] The pixels in the difficulty probability map are given a larger weight, specifically by constructing an importance weight matrix , used to focus on learning the difficult areas; the construction of the importance weight matrix is specifically: the importance weight matrix is obtained by multiplying the entropy map of the pixel with the difficulty probability map, and the information amount of the data itself is used as the weight of the difficulty probability map.
[0061] Step 4: Based on the obtained difficulty probability map, generate a local difficulty image based on the Gaussian distribution context;
[0062] The method generates a local difficult image based on the Gaussian distribution context based on the obtained difficulty probability map, specifically: based on the probability value of each pixel point in the difficulty probability map, a weighted average method is used to calculate the center point of the difficult area; based on the center point of the difficult area, another center point of the difficult area is obtained in combination with a random offset generated by a Gaussian distribution; and two local difficult images of the same size are generated according to preset lengths and widths using the two center points as starting points of rectangular frames.
[0063] Specifically, the main purpose of this step is to enhance the learning of difficult area knowledge, which can be learned in different contextual information, while increasing the number of difficult areas in the fine-tuning training data.
[0064] This embodiment proposes a method for generating a local difficulty image based on context. First, the probability value of each pixel is obtained from the difficulty probability map. These probability values describe the difficulty level at each position in the image. An image processing library can be used to read and parse the image, such as the OpenCV library in Python. The probability values in the difficulty probability map are used to calculate the center point of the difficulty area. , the weighted average method is used for calculation, in which the pixel with higher difficulty probability has a greater impact on the center point.
[0065] in, H is the size of the input image, that is a OK b List, For the i The first a Line b Difficulty probability value for the pixels on the column.
[0066] Based on the center point obtained from the difficulty probability map, a rectangular box is generated using Gaussian distribution. For the center point of the difficult area, a random offset is generated using Gaussian distribution. , They are Gaussian distributions in Direction and The standard deviation of the direction, All from the standard normal distribution A random number is generated in . Another center point is obtained based on the offset , the generated offset is sampled according to the probability density function of the Gaussian distribution to ensure that the generated new center point is more likely to be located near the center of the difficult area. The two center points in the difficult area are used as the starting points of the rectangular box. The size of the box is specified by setting the length and width of the matrix greater than the offset. The center points are used to calculate the upper left corner coordinates and the lower right corner coordinates of the matrix to generate a matrix box of the same size. In this way, rectangular areas with different context information containing the same difficult area can be generated. These areas are used as local difficult areas generated by the difficult area.
[0067] Step 5: construct a perceptual difficulty loss based on the obtained difficulty probability map; construct a difficult region contrast loss based on the local difficult image including the same difficult region; wherein:
[0068] (1) Difficulty perception contrast learning:
[0069] For two local difficult images A and B generated based on the same difficult area, although these two images have different context information, they both have the same difficult area. First, they are sent to the image encoder of SAM Get the feature map In order to better focus on the discriminative features of the difficult region R, this method uses nonlinear projection mapping to reduce the dimension of the features learned by the encoder, and obtains two new feature maps after projection ,in, () is a nonlinear projection mapping operation. At the same time, the difficulty-aware ratio learning method is introduced to learn the distinction between background and foreground areas. The feature represents the key information of the data, while the uncertainty represents the confidence of the model in its prediction. Combining the two helps the model better understand the data and make more accurate predictions. The difficult area is integrated into the contrastive learning of the feature. By calculating the mean difficulty of the sample pair , N represents the number of pixels in the overlapping area, represents the sum of the difficulty probability values of the positive sample pairs, S represents the number of pixels in the non-overlapping area, Represents the sum of the difficulty probability values of the negative sample pairs; is the difficulty probability value of the area where the matrix box in image A overlaps with that in image B, is the difficulty probability value of the area in image B that overlaps with the matrix box in image A, is the difficulty probability value representing the non-overlapping area in the B image, is the mean difficulty of the positive sample pairs, is the mean difficulty of negative sample pairs;
[0070] The mean difficulty value is added as a weight to the contrast loss of the local image pair. The contrast loss of the difficult area will be given a larger weight, so that the model can focus on learning the discriminative features of the difficult area, thereby enhancing the distinguishability between the difficult area and the background area.
[0071] (2) Construction of Perceptual Difficulty Loss:
[0072] By incorporating the previous difficulty probability map into the loss function, the pixels in the difficult area will be given a larger weight, so that the model can focus on learning this type of pixel knowledge during fine-tuning training and perceive the difficulty loss:
[0073]
[0074]
[0075] in, represents the size of the smallest batch of labeled images. represents the cross entropy loss, is the loss function based on the weights of the difficulty probability map and the labeled samples, w i is the importance weight matrix, y i is the initial prediction result of SAM for the i-th image, is the i-th sample label, is the pixel entropy value of the i-th image segmentation mask, is the probability value of the difficulty probability map of the i-th image.
[0076] (3) Contrast loss in difficult areas:
[0077] The contrast loss of the difficulty mean and the difficulty feature is calculated. The contrast loss of the difficult area will be given a larger weight, so that the model can focus on learning the discriminative features of the difficult area, thereby enhancing the distinguishability of the difficult area from the background area.
[0078]
[0079] in, represents a positive sample pair, represents a negative sample pair, represents the exponential function for calculating the cosine similarity between two features, It is the temperature parameter that controls the clarity of the contrast loss function. , N represents the number of pixels in the overlapping area, represents the sum of the difficulty probability values of the positive sample pairs, S represents the number of pixels in the non-overlapping area, Represents the sum of the difficulty probability values of the negative sample pairs, is the feature of the overlapping area in image A after nonlinear projection mapping, is the feature of the overlapping area in the B image after nonlinear projection mapping, is the feature of the non-overlapping area in the B image after nonlinear projection mapping, and e is a natural constant.
[0080] Step 6: Based on the loss function that integrates the perceptual difficulty loss and the difficult area contrast loss, the image segmentation model is trained to obtain a trained image segmentation model.
[0081] Specifically, the perceived difficulty loss Comparison loss with difficult areas The fusion is performed as the total loss of the network , The loss function L will repeatedly perform back propagation training during the network learning process, and the loss value will slowly decrease as the number of training rounds increases. When the loss value reaches the minimum value, the obtained network model is the best training result.
[0082] In this embodiment, the parameters of the SAM model are updated using LoRA fine-tuning technology to minimize the parameters. In order to optimize efficiency and performance, the rank of LoRA is set to 4. In order to better train SAM to learn knowledge in difficult areas from multiple aspects, the initial learning rate of the network training is set to 0.005 and the AdamW optimizer is used for network training. Iterative training is continuously performed until the convergence condition is reached and the network model with the minimum loss value is saved.
[0083] At the same time, the image to be tested is input into the trained network model for prediction, and the DiceSimilarity Coefficient, Hausdorff distance and F1 Score indicators are used to test the segmentation performance of the trained network.
[0084] like Figure 2 As shown, the overall network architecture of the solution described in this embodiment is demonstrated.
[0085] Embodiment 2:
[0086] The purpose of this embodiment is to provide an image segmentation method.
[0087] An image segmentation method, comprising:
[0088] Obtain an image to be segmented;
[0089] Based on the image, an image segmentation result is obtained using a pre-trained image segmentation model; wherein the image segmentation model adopts the above-mentioned training method of an image segmentation model.
[0090] Embodiment three:
[0091] The purpose of this embodiment is to provide an image segmentation system.
[0092] like Figure 3 As shown, an image segmentation system comprises:
[0093] An image acquisition unit, used for acquiring an image to be segmented;
[0094] An image segmentation unit is used to obtain an image segmentation result based on the image using a pre-trained image segmentation model; wherein the image segmentation model adopts the above-mentioned training method of an image segmentation model.
[0095] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. A training method for an image segmentation model, characterized in that: include: Based on the training sample images, using the image segmentation model to be trained, an initial prediction result of the segmentation is obtained; Based on the initial prediction results, an uncertainty estimation map is obtained by calculating the pixel-by-pixel difference between the segmentation mask and its label; and an entropy map is obtained by calculating the entropy value of the segmentation mask pixels; Based on the obtained uncertainty estimation map and entropy map, a difficulty probability map is obtained by fusion; Based on the obtained difficulty probability map, a local difficulty image based on the Gaussian distribution context is generated; Based on the obtained difficulty probability map, a perceptual difficulty loss is constructed; based on the local difficulty image including the same difficult area, a difficult area contrast loss is constructed; Based on the loss function of integrating the perceptual difficulty loss and the difficult area contrast loss, the image segmentation model is trained to obtain a trained image segmentation model; The method of using the image segmentation model to be trained to obtain an initial prediction result of the segmentation is specifically as follows: based on the sample image, feature extraction is performed by an image encoder of the image segmentation model; based on the extracted features and the output of the prompt encoder in the image segmentation model, an initial prediction result of the segmentation is obtained by a mask decoder of the image segmentation model; The method generates a local difficult image based on the Gaussian distribution context based on the obtained difficulty probability map, specifically: based on the probability value of each pixel point in the difficulty probability map, a weighted average method is used to calculate the center point of the difficult area; based on the center point of the difficult area, another center point of the difficult area is obtained in combination with a random offset generated by a Gaussian distribution; and two local difficult images of the same size are generated according to preset lengths and widths using the two center points as starting points of rectangular frames.
2. The method for training an image segmentation model according to claim 1, characterized in that: The difficulty probability map is obtained by fusing the obtained uncertainty estimation map and entropy map, specifically: the uncertainty estimation map and the entropy map are normalized respectively, and the difficulty probability map is obtained by weighted summation based on the normalized uncertainty estimation map and entropy map.
3. The method for training an image segmentation model according to claim 1, wherein: The perceived difficulty loss is specifically expressed as: ; ; in, represents the size of the smallest batch of labeled images. represents the cross entropy loss, is the loss function based on the weights of the difficulty probability map and the labeled samples, w i is the importance weight matrix, y i is the initial prediction result of SAM for the i-th image, is the i-th sample label, is the pixel entropy value of the i-th image segmentation mask, is the probability value of the difficulty probability map of the i-th image.
4. The method for training an image segmentation model according to claim 1, characterized in that: The difficult area contrast loss is specifically expressed as: ; in, represents a positive sample pair, represents a negative sample pair, represents an exponential function that calculates the cosine similarity between two features. is the temperature parameter that controls the clarity of the contrast loss function, , N represents the number of pixels in the overlapping area, represents the sum of the difficulty probability values of the positive sample pairs, S represents the number of pixels in the non-overlapping area, Represents the sum of the difficulty probability values of the negative sample pairs. For two local difficult images A and B generated based on the same difficult area, is the feature of the overlapping area in image A after nonlinear projection mapping, is the feature of the overlapping area in the B image after nonlinear projection mapping, is the feature of the non-overlapping area in the B image after nonlinear projection mapping, and e is a natural constant.
5. The method for training an image segmentation model according to claim 1, wherein: The image segmentation model adopts a SAM network model, which includes an image encoder, a hint encoder and a mask decoder.
6. The method for training an image segmentation model according to claim 5, characterized in that: The image encoder adopts a Vision Transformer structure based on an attention mechanism.
7. An image segmentation method, characterized in that: include: Obtain an image to be segmented; Based on the image, an image segmentation result is obtained using a pre-trained image segmentation model; wherein the image segmentation model adopts a training method for an image segmentation model as described in any one of claims 1-6.
8. An image segmentation system, characterized in that: include: An image acquisition unit, used for acquiring an image to be segmented; An image segmentation unit is used to obtain an image segmentation result based on the image using a pre-trained image segmentation model; wherein the image segmentation model adopts a training method for an image segmentation model as described in any one of claims 1-6.
Citation Information
Patent Citations
Image processing method and device and computer readable storage medium
CN114691912A
Sample data determination method and device of matting model, equipment and storage medium
CN115578411A
Night streetscape semantic segmentation method based on difficult category perception mechanism
CN116883667A
Construction method of coal rock microstructure grouping automatic segmentation model
CN117011646A
Chip weld joint cavity defect detection method and system
CN117173120A
