Image-level weak supervision semantic segmentation method and system based on multi-dimensional information comparison

Abstract

The embodiment of the invention discloses an image-level weak supervision semantic segmentation method and system based on multi-dimensional information comparison, and the method comprises the steps: constructing an improved image segmentation model based on multi-dimensional information comparison, and the improved image segmentation model comprises a mixed token comparison mechanism, a self-supervision multi-view comparison mechanism, a memory bank comparison mechanism and a segmentation mechanism; optimizing the original CAM based on the patch token level contrast learning loss Lptc, the class level token contrast learning loss Lctc, the self-supervised multi-view contrast learning loss Lssm, the memory bank contrast loss Lmmb, the main classifier loss Lcls, the auxiliary classifier loss Lmcls and the segmentation loss Lseg to obtain an optimal improved image segmentation model; and obtaining a to-be-segmented image, and inputting the to-be-segmented image into the optimal improved image segmentation model to obtain a segmentation result. The problem that in the prior art, multi-source supervision information cannot be fully mined, and therefore accurate segmentation of an image segmentation model cannot be guaranteed is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image-level weakly supervised semantic segmentation system, method, electronic device and storage medium based on multi-dimensional information comparison. Background Art

[0002] Semantic segmentation, as one of the core tasks in the field of computer vision, aims to predict the corresponding semantic category for each pixel in an image. With the rise of deep learning technology, semantic segmentation has made significant progress in key application areas such as medical image processing and autonomous driving. However, although fully supervised learning methods have made breakthrough progress in semantic segmentation, this method still relies on a large amount of manpower to complete pixel-level fine annotation, which is not only time-consuming but also costly. According to statistics, it may take up to 90 minutes to perform pixel-level annotation on a high-resolution complex scene image (2048×1024 pixels).

[0003] In the prior art, by using simpler annotation forms, such as graffiti, point annotation, bounding box or image-level label, the annotation cost can be effectively reduced; in particular, the image-level label, which is simple and fast to label, has attracted many scholars to conduct research; at present, most weakly supervised semantic segmentation methods based on image-level labels mainly use neural networks to generate class activation maps (CAM) to locate the target in the image, and obtain pseudo labels based on CAM to optimize the training of the semantic segmentation network. The accuracy of CAM greatly determines the performance of weakly supervised segmentation, so how to obtain high-quality CAM has become the core issue of weakly supervised segmentation.

[0004] At present, most weakly supervised semantic segmentation methods use convolutional neural networks (CNNs) as the backbone network of the model; however, since convolution cannot model the global feature relationship of the image, it greatly restricts the improvement of the generation quality of CAM. Recently, the Vision Transformer (ViT) has achieved performance breakthroughs in many visual applications. Unlike convolutional neural networks, the Vision Transformer can capture long-range semantic dependencies through the self-attention mechanism and provide a global understanding of the scene, which makes it more conducive to completing weakly supervised semantic segmentation tasks; based on this advantage, some methods have begun to try to optimize the model by designing different Vision Transformer structures to mine image information from the network, but this strategy often focuses on extracting limited supervision information from the semantic dimension of the model, which limits the accuracy improvement of the segmentation model.

[0005] Therefore, there is an urgent need for a method that can fully mine multi-source supervision information to ensure that the image segmentation model performs accurate segmentation. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide an image-level weakly supervised semantic segmentation system, method, electronic device and storage medium based on multi-dimensional information comparison, so as to solve the problem in the prior art that it is impossible to fully mine multi-source supervision information and thus cannot ensure that the image segmentation model performs accurate segmentation.

[0007] To achieve the above object, an embodiment of the present invention provides an image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison, and the method specifically includes:

[0008] Obtaining a certain number of sample images and image-level classification labels corresponding to the sample images, and performing cropping and data enhancement processing on the sample images to obtain local sample images and multi-view images;

[0009] Constructing an improved image segmentation model based on multi-dimensional information comparison, wherein the improved image segmentation model includes a hybrid token comparison mechanism, a self-supervised multi-view comparison mechanism, a memory library comparison mechanism and a segmentation mechanism;

[0010] The sample image is input into the improved image segmentation model, the patch token features of the sample image in the intermediate auxiliary layer are obtained through the hybrid token contrast mechanism, the auxiliary CAM is generated based on the patch token features, and the patch token level contrast learning loss L is calculated based on the original CAM and the auxiliary CAM. ptc ;

[0011] The local semantics corresponding to the local sample image and the global semantics corresponding to the sample image are obtained through a hybrid token comparison mechanism, and the class-level token comparison learning loss L is calculated based on the local semantics and the global semantics. ctc ;

[0012] The multi-view image corresponding to the sample image is obtained through the self-supervised multi-view contrast mechanism, and a self-supervised CAM corresponding to the multi-view image is generated. The self-supervised multi-view contrast learning loss L is calculated based on the self-supervised CAM and the original CAM. ssm ;

[0013] The memory prototype corresponding to the image target is extracted through the memory mechanism, and the memory contrast loss L is calculated based on the memory prototype and the image target. mmb ;

[0014] Calculate the main classifier loss L cls ;

[0015] Calculate the auxiliary classifier loss L mcls ;

[0016] Calculate the segmentation loss L seg ;

[0017] Based on the patch token-level contrastive learning loss L ptc, class-level token contrastive learning loss L ctc , self-supervised multi-view contrastive learning loss L ssm 、Memory library contrast loss L mmb , the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg Optimizing the original CAM to obtain an optimal improved image segmentation model;

[0018] An image to be segmented is obtained, and the image to be segmented is input into the optimal improved image segmentation model to obtain a segmentation result.

[0019] On the basis of the above technical solution, the present invention can also be improved as follows:

[0020] Furthermore, the patch token features of the sample image in the intermediate auxiliary layer are obtained through the hybrid token comparison mechanism, an auxiliary CAM is generated based on the patch token features, and a patch token level comparison learning loss L is calculated based on the original CAM and the auxiliary CAM. ptc ,include:

[0021] The patch token-level contrastive learning loss L is calculated by formula (1): ptc ;

[0022]

[0023] Where, L ptc is the patch token-level contrastive learning loss, Mask + is the matrix R mid The positive mask of - is the matrix R mid The negative mask, N + is the number of pairs, N - is the number of negative pairs, is the Hadamard product matrix operation, G is the two-dimensional feature vector after the patch token sequence is reshaped, G i and G j Sim cos (·,·) is the calculation of cosine similarity.

[0024] Furthermore, the local semantics corresponding to the local sample image and the global semantics corresponding to the sample image are obtained through the hybrid token comparison mechanism, and the class-level token comparison learning loss L is calculated based on the local semantics and the global semantics. ctc ,include:

[0025] The class-level token contrastive learning loss L is calculated by formula (2): ctc ;

[0026]

[0027] Where, L ctc is the class-level token contrastive learning loss, N + t + The number of τ is a temperature factor, ε is a positive number to avoid the denominator being 0, t + is a positive token, t l is the global token, t - is a negative token, t l A global token.

[0028] Furthermore, the multi-view image corresponding to the sample image is obtained through the self-supervised multi-view contrast mechanism, and a self-supervised CAM corresponding to the multi-view image is generated, and a self-supervised multi-view contrast learning loss L is calculated based on the self-supervised CAM and the original CAM. ssm ,include:

[0029] The self-supervised multi-view contrastive learning loss L is calculated by formula (3): ssm ;

[0030] L ssm =L ssm1 +L ssm2 Formula (3);

[0031] Where, L ssm is the self-supervised multi-view contrastive learning loss, L ssm1 is the segmentation difference between the enhanced image and the original image, L ssm2 is the segmentation difference between the enhanced images.

[0032] Furthermore, the memory library prototype corresponding to the image target is extracted through the memory library mechanism, and the memory library contrast loss L is calculated based on the memory library prototype and the image target. mmb ,include:

[0033] The memory library contrast loss L is calculated by formula (4): mmb ;

[0034]

[0035] Where, L mmb is the memory library comparison loss, ρ is the similarity coefficient, Mask + is the matrix R mid The positive mask of - is the matrix R mid The negative mask, N + is the number of pairs, N - is the number of negative pairs, is the Hadamard product matrix operation, F iand F j Sim cos (·,·) is the calculation of cosine similarity.

[0036] Furthermore, the calculation of the main classifier loss L cls ,include:

[0037] The main classifier loss L is calculated by formula (5): cls ;

[0038]

[0039] Where, L cls is the main classifier loss, N is the number of labels, x is the sample image, y is the true label, and f i (x) is the predicted value of the i-th label, and σ is the Sigmoid function.

[0040] Furthermore, the calculation of the segmentation loss L seg ,include:

[0041] The segmentation loss L is calculated by formula (6): seg ;

[0042]

[0043] Where, L seg is the segmentation loss, N' is the total number of pixels in the image, c is the number of categories, if pixel i belongs to category c, then y i,c is 1, if pixel i does not belong to category c, then y i,c is 0, p i,c is the model’s predicted probability that pixel i belongs to category c.

[0044] An image-level weakly supervised semantic segmentation system based on multi-dimensional information comparison, comprising:

[0045] An acquisition module, used to acquire a certain number of sample images and image-level classification labels corresponding to the sample images;

[0046] A preprocessing module, used for performing cropping and data enhancement processing on the sample image to obtain a local sample image and a multi-view image;

[0047] A construction module, used to construct an improved image segmentation model based on multi-dimensional information comparison, wherein the improved image segmentation model includes a hybrid token comparison mechanism, a self-supervised multi-view comparison mechanism, a memory library comparison mechanism and a segmentation mechanism;

[0048] A hybrid token comparison mechanism is used to input a sample image into the improved image segmentation model, obtain patch token features of the sample image in the intermediate auxiliary layer through the hybrid token comparison mechanism, generate an auxiliary CAM based on the patch token features, and calculate a patch token level comparison learning loss L based on the original CAM and the auxiliary CAM ptc ;

[0049] The local semantics corresponding to the local sample image and the global semantics corresponding to the sample image are obtained through a hybrid token comparison mechanism, and the class-level token comparison learning loss L is calculated based on the local semantics and the global semantics. ctc ;

[0050] A self-supervised multi-view comparison mechanism is used to obtain a multi-view image corresponding to the sample image through the self-supervised multi-view comparison mechanism, and generate a self-supervised CAM corresponding to the multi-view image, and calculate the self-supervised multi-view comparison learning loss L based on the self-supervised CAM and the original CAM ssm ;

[0051] The memory library comparison mechanism is used to extract the memory library prototype corresponding to the image target through the memory library mechanism, and calculate the memory library comparison loss L based on the memory library prototype and the image target mmb ;

[0052] Split mechanism, used to calculate the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg ;

[0053] A training module for patch token-level contrastive learning loss L based on the ptc , class-level token contrastive learning loss L ctc , self-supervised multi-view contrastive learning loss L ssm 、Memory library contrast loss L mmb , the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg Optimizing the original CAM to obtain an optimal improved image segmentation model;

[0054] The optimal improved image segmentation model is used to receive the image to be segmented and obtain the segmentation result.

[0055] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method are implemented when the processor executes the computer program.

[0056] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0057] The embodiments of the present invention have the following advantages:

[0058] The image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison in the present invention fully mines multi-source supervision information from three perspectives: image space, historical information, and model semantics.

[0059] The spatial correlation of multiple views of the image is used to mine self-supervisory information. Multiple views are obtained through data enhancement. After the multi-view segmentation results are inversely operated, they are compared with the segmentation results of the image itself, and the useful information implicit in the image is used to provide accuracy gain for segmentation.

[0060] The feature extraction of historical target information in the image is performed through an updateable memory library strategy, and the adaptive memory contrast loss is used to achieve high-precision positioning of the target, which effectively aggregates the context information.

[0061] For the first time, the present invention combines image space information, historical information and model semantic comparison information as additional supervision information to optimize CAM, making up for the lack of supervision information in weakly supervised semantic segmentation. Compared with previous methods that focus on a single image or context image, the image segmentation model of the present invention expands the breadth of exploration to three dimensions: image space, historical information and model semantics, fully mining semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0063] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0064] Figure 1 It is a flow chart of the image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison of the present invention;

[0065] Figure 2 This is a first architecture diagram of the image-level weakly supervised semantic segmentation system based on multi-dimensional information comparison of the present invention;

[0066] Figure 3 This is a second architecture diagram of the image-level weakly supervised semantic segmentation system based on multi-dimensional information comparison of the present invention;

[0067] Figure 4 It is a structural diagram of the image segmentation model of the present invention;

[0068] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided by the present invention.

[0069] The accompanying drawings are denoted as follows:

[0070] Acquisition module 10, preprocessing module 20, construction module 30, training module 40, image segmentation model 50, hybrid token comparison mechanism 501, self-supervised multi-view comparison mechanism 502, memory comparison mechanism 503, segmentation mechanism 504, electronic device 60, processor 601, memory 602, bus 603. DETAILED DESCRIPTION

[0071] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] Figure 1 FIG. 1 is a flow chart of an embodiment of an image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison of the present invention. Figure 1 As shown, an image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison provided by an embodiment of the present invention includes the following steps:

[0073] S101, obtaining a certain number of sample images and image-level classification labels corresponding to the sample images, and performing cropping and data enhancement processing on the sample images to obtain local sample images and multi-view images;

[0074] Specifically: CAM is widely used in pixel-level weakly supervised semantic segmentation due to its effectiveness and simplicity. The present invention uses CAM to generate initial pseudo labels for images.

[0075] Specifically, given a sample image H and W represent the height and width of the image, respectively. The visual transformer divides the input sample image into patches of fixed size. Assuming the patch size is P×P, the image is divided into (H / P)×(W / P) patches. Each patch is then flattened and mapped to a vector space to form a patch token. Where D is the length of each patch after being flattened; then, the initial patch token is combined with a learnable class token and passed through the encoder of the visual transformer to obtain the final patch and token Then, after the convolution operation, the feature map is obtained Where C represents the output channel, that is, the number of categories. By performing weighted summation on the feature map, applying the ReLU function and the maximum normalization operation, the initial CAM for C categories can be obtained as shown in the following formula.

[0076]

[0077] In the formula, F represents the feature map and W represents the weight;

[0078] The generation of CAM is optimized by using hybrid token comparison mechanism, self-supervised multi-view comparison mechanism, memory comparison mechanism and segmentation mechanism. The improved pseudo-labels are used to supervise the segmentation decoder, thereby optimizing the segmentation results.

[0079] S102, constructing an improved image segmentation model based on multi-dimensional information comparison, wherein the improved image segmentation model includes a hybrid token comparison mechanism, a self-supervised multi-view comparison mechanism, a memory library comparison mechanism and a segmentation mechanism;

[0080] S103, input the sample image into the improved image segmentation model, obtain the patch token features of the sample image in the intermediate auxiliary layer through the hybrid token comparison mechanism, generate the auxiliary CAM based on the patch token features, and calculate the patch token level contrast learning loss L based on the original CAM and the auxiliary CAM. ptc ;

[0081] Patch token comparison uses the similarity of different transformer blocks to provide supervision information for weakly supervised semantic segmentation. In the visual transformer architecture, the basic unit is called a transformer block, which consists of a self-attention mechanism and a feedforward neural network. After the transformer block, 12 patch tokens are generated, which can be expressed as T = [T1, T2, ..., T 12 ]. After research, relevant scholars have found that the patch tokens generated by early blocks are more different than those generated by later blocks. Therefore, the patch tokens generated in the early layers contain more diverse semantic information, while the patch tokens in the later layers are highly similar, which causes the problem of over-smoothing. Inspired by this, the layer with the richest semantic information can be extracted as an intermediate auxiliary layer, and the patch tokens output by this layer are used to generate an auxiliary CAM, which is compared with the original CAM. The knowledge of the intermediate layer is used to supervise the last layer of patch tokens to solve the over-smoothing problem and reduce their gap, so as to optimize the segmentation effect. It is known from relevant research that the semantic information of the 10th layer is the richest. Therefore, the present invention uses the 10th layer as the intermediate layer.

[0082] Specifically, the patch token output by the middle layer can be expressed as Where n and d represent the number of tokens and feature dimensions, respectively. The patch tokens of the middle layer are integrated using the global maximum pooling operation to obtain the feature vector, which is projected into the high-dimensional space through the fully connected layer to obtain the projected feature vector, and the feature vector is normalized to generate the auxiliary CAM, denoted as M mid Then set the thresholds α and β (0<α<β<1) to mid The pixels in the CAM whose responses are higher than the threshold β are marked as foreground, the pixels in the CAM whose responses are lower than the threshold β and higher than the threshold α are marked as uncertain areas, and the pixels in the CAM whose responses are lower than the threshold α are marked as background. The pseudo-label of the auxiliary CAM is obtained, which is recorded as

[0083] The present invention designs a loss function by reducing the difference between the original CAM and the auxiliary CAM. First, the pseudo label L mid Flatten to sequence The label matrix is ​​constructed by repeating h×w times of stacking, and then transposed and compared with the original feature matrix. Pixels with the same semantic information are marked as positive, otherwise they are marked as negative, and the relationship matrix R is obtained. mid In particular, in order to ensure reliability, the uncertain region is ignored here and only the semantic information of the foreground and background is compared. Then the matrix R is extracted mid The positive mask Mask + and negative mask - , so as to maximize the similarity of positive token pairs and minimize the similarity of negative token pairs, thus improving the over-smoothing problem.

[0084] The patch token-level contrastive learning loss L is calculated by formula (1): ptc ;

[0085]

[0086] Where, L ptc is the patch token-level contrastive learning loss, Mask + is the matrix R mid The positive mask of - is the matrix R mid The negative mask, N + is the number of pairs, N - is the number of negative pairs, is the Hadamard product matrix operation, G is the two-dimensional feature vector after the patch token sequence is reshaped, G i and G j Sim cos (·,·) is the calculation of cosine similarity.

[0087] S104, obtaining the local semantics corresponding to the local sample image and the global semantics corresponding to the sample image through a hybrid token comparison mechanism, and calculating the class-level token comparison learning loss L based on the local semantics and the global semantics. ctc .

[0088] Specifically, one of the major drawbacks of CAM is that it does not fully activate the target. Promoting the consistency of representation between local non-salient regions and global objects can further promote CAM to activate more object regions, thereby alleviating this problem. First, pseudo labels are generated using auxiliary CAM, and then the uncertain regions and background regions are found in the pseudo labels. The local image is obtained by cropping the uncertain regions and background regions, which is expressed as The local image is then fed into the visual transformer encoder to obtain local tokens. By minimizing the difference between the global token and the uncertain local region token and maximizing the difference between the global token and the background local region token, the CAM region can be effectively activated, making the representation of the entire object region more consistent.

[0089] Two sets of tokens are used to guide model training, and tokens from the background area are recorded as negative tokens (t - ), and the tokens from the uncertain region are recorded as positive tokens (t + ), in order to ensure the consistency of local semantics and global semantics, two projection heads consisting of linear layers and L2 norm normalization are used to receive the two sets of tokens. During the training process, the gradient of the global projection head is stopped, and the parameters of the global projection head are updated using the exponential moving average method, so that the global projection head can adapt to the changes of data more smoothly and improve the stability of training. The loss function is designed to minimize the positive token t + With the global token t l and maximize the difference of negative tokens t - With the global token t l differences;

[0090] The class-level token contrastive learning loss L is calculated by formula (2): ctc ;

[0091]

[0092] Where, L ctc is the class-level token contrastive learning loss, N + t + The number of τ is a temperature factor, ε is a positive number to avoid the denominator being 0, t + is a positive token, t l is the global token, t - is a negative token, t l A global token.

[0093] S105, obtaining a multi-view image corresponding to the sample image through a self-supervised multi-view contrast mechanism, and generating a self-supervised CAM corresponding to the multi-view image, and calculating the self-supervised multi-view contrast learning loss L based on the self-supervised CAM and the original CAM ssm .

[0094] The goal of self-supervised learning is to mine supervisory information from its own data. Inspired by this, the present invention increases data samples by processing the image itself and uses the spatial consistency of the input image and output label to provide more semantic information for weakly supervised semantic segmentation. In the present invention, the self-supervisory mechanism is embedded in the visual transformer framework, and the differences in the segmentation results of the enhanced data are used to learn the self-supervisory information.

[0095] Specifically, we use scale transformation, position transformation and angle transformation to increase the number of samples, using C scale , C position , C angle Represents these operations; secondly, the transformed image is input into the visual transformer semantic segmentation network V θ In the above example, we obtain the respective CAM pseudo-labels, denoted as M scale 、M position 、M angle , the process is expressed by formula (7); then these pseudo-label images are passed through C scale , C position , C angle The inverse operation Restore it to the image with the same size, position and angle as the original CAM, and then get the multi-view label, recorded as As shown in formula (8), they have the same size as the original pseudo-label M, as shown in formula (9).

[0096]

[0097] Ideally, when the network is fully trained and optimized, these multi-view pseudo labels should be consistent with the pseudo labels of the original images, and these multi-view pseudo labels should also be consistent with each other. As shown in formula (10), the inverse operation generates They should be the same as the pseudo-label M generated from the original image. Therefore, implicit constraints are used to reduce the error between these labels and provide self-supervisory information for the model.

[0098]

[0099] Specifically, Perform L2 norm normalization and calculate the absolute distance between each other and between them and the original pseudo-label. Construct the loss function by reducing the difference between them. The loss function is divided into two parts. One part measures the segmentation difference between the enhanced image and the original image, denoted by L ssm1 , part of which measures the segmentation difference between the enhanced images, denoted as L ssm2 , the total loss is their sum, denoted as L ssm .

[0100]

[0101]

[0102] The self-supervised multi-view contrastive learning loss L is calculated by formula (3): ssm ;

[0103] L ssm =L ssm1 +L ssm2 Formula (3);

[0104] Where, L ssm is the self-supervised multi-view contrastive learning loss, L ssm1 is the segmentation difference between the enhanced image and the original image, L ssm2 is the segmentation difference between the enhanced images.

[0105] S106, extracting the memory prototype corresponding to the image target through the memory mechanism, and calculating the memory comparison loss L based on the memory prototype and the image target mmb .

[0106] Most existing weakly supervised semantic segmentation schemes based on visual transformers generally only utilize limited contextual clues in a single image, without combining useful information from other images in the dataset, and cannot fully mine semantic knowledge. In order to make historical images provide reference and guidance for subsequent segmentation, we established a dynamic memory library that uses prototypes to save the semantic features of each target class as supervision information to guide subsequent images to obtain prior knowledge, thereby optimizing the segmentation effect.

[0107] Specifically, first, an empty memory list is established to store the prototype information of each category. The memory library consists of c prototype dictionaries, that is, m = {m1, m2, ..., m c}, representing the prototype of each category, and c is the number of categories in the data set. Next, obtain the feature map and CAM of the image, and divide the CAM into background, foreground, and uncertain areas according to the threshold set above. Then, use the pseudo-label generated by the CAM to record the category information in the image and make a mask for each target class. Taking the category with label l as an example, the formula for obtaining the mask is shown in formula (11). For the pixel at position (i, j), if the pseudo-label is l, the corresponding position of the mask is set to 1, and the rest is set to 0. Then, use the mask to extract the mask features in the feature map corresponding to each target class element, and add them to the corresponding elements in the prototype list as the initial prototype information. The formula for extracting mask features is shown in formula (12), where F mask represents the extracted mask features, Represents the feature value at channel c, position (i, j) in the feature map, Mask i,j Represents the value of position (i, j) in the mask. ε is a very small positive number to avoid the invalidation of the formula due to the denominator being 0. This formula can be used to extract the features corresponding to each class by multiplying the mask of each class with the feature map, and then perform mask average pooling to convert the features into a one-dimensional vector. The one-dimensional mask feature obtained at this time is the prototype of the corresponding class, and finally it is added to the prototype list to form a memory library.

[0108]

[0109] In addition, an update mechanism is designed to update the prototype of each category, that is, to update the memory bank. The memory bank will be continuously updated in the back propagation stage to absorb useful semantic information and enhance the semantic understanding of the target. The momentum factor γ is set to control the smoothness of the update, thereby ensuring the stability of the update. The prototype in the memory bank will be updated according to the following formula: m'←γm+(1-γ)f, m' represents the updated memory bank, m represents the memory bank before the update, and f is the feature information of the target category input during the training process. In order to prevent the noise of the features from affecting the accuracy of the memory bank, the frequency of updating the memory bank is also set in the present invention. During the training process, the memory bank is updated every certain number of pictures, and the frequency is determined according to the data set.

[0110] Based on the above settings, the present invention designs a memory loss to obtain more supervisory information by measuring the difference between the input image and the historical image.

[0111] Specifically, the input image features are first compared with the prototype information in the memory bank by cosine similarity to generate a pseudo label, and then the pseudo label is used to calculate the patch token-level contrastive learning loss L by comparing it with the input image features. ptc (Formula (1)) is used to mine supervisory information from historical information to further optimize the model.

[0112] To further refine the optimization process of the memory loss, we add a similarity coefficient ρ to the memory loss to make it adaptively change its weight.

[0113] Specifically, the cosine similarity graph between the prototype of the memory library and the input image can be obtained by formula (1), and then the similarity between the similarity graph and the CAM and the auxiliary CAM is calculated respectively, thereby obtaining two probability distribution graphs. If the similarity between the two distribution graphs is high, it means that the memory library is consistent with the CAM and the auxiliary CAM, and the memory library can effectively guide image segmentation. At this time, the weight of the loss is increased, otherwise the weight is reduced.

[0114] The present invention combines the Bhattacharyya distance to measure the similarity of two probability distribution graphs. First, the probability graph is normalized by the L2 norm, and 10 bin intervals are set in the range of 0-1 to calculate the probability histogram of two cosine similarities. Then, the Bhattacharyya distance is used to convert it into a similarity metric. The specific calculation formula is shown in formula (13), where i represents a certain interval in the histogram, and h1 and h2 represent two histograms. Formula (13) measures the similarity of two histograms. The larger the value, the more similar the two histograms are, and the range of the value is between 0 and 1.

[0115]

[0116] Combining the similarity coefficient ρ and the PTC loss, the memory library comparison loss L is calculated by formula (4): mmb ;

[0117]

[0118] Where, L mmb is the memory library comparison loss, ρ is the similarity coefficient, Mask + is the matrix R mid The positive mask of - is the matrix R mid The negative mask, N + is the number of pairs, N - is the number of negative pairs, is the Hadamard product matrix operation, F i and F j Sim cos (·,·) is the calculation of cosine similarity.

[0119] S107, calculate the main classifier loss L cls , calculate the auxiliary classifier loss L mcls , calculate the segmentation loss L seg .

[0120] Specifically, the main classifier loss L is calculated by formula (5): cls ;

[0121]

[0122] Where, L cls is the main classifier loss, N is the number of labels, x is the sample image, y is the true label, and f i (x) is the predicted value of the i-th label, and σ is the Sigmoid function.

[0123] A pixel-adaptive refinement module (PAR) is introduced to refine the pseudo-labels. The improved pseudo-labels will be used to supervise the training of the segmentation decoder.

[0124] The segmentation loss L is calculated by formula (6): seg ;

[0125]

[0126] Where, L seg is the segmentation loss, N' is the total number of pixels in the image, c is the number of categories, if pixel i belongs to category c, then y i,c is 1, if pixel i does not belong to category c, then y i,c is 0, p i,c is the model’s predicted probability that pixel i belongs to category c.

[0127] An additional regularization loss term L is used reg To enhance the spatial consistency of the predicted segmentation mask, as shown in formula (17), L reg It prevents the weight from being too large, smoothes the model parameters, and reduces overfitting. The loss of this method is the weighted sum of these loss terms, as shown in formula (17), where λ1, λ2, λ3, λ4, λ5, and λ6 represent the weights of each loss.

[0128]

[0129] L=λ1L ptc +λ2L ctc +λ3L ssm +λ4L mmb +L cls +L mcls +λ5L seg +λ6L reg Formula (17);

[0130] S108, based on patch token-level contrastive learning loss L ptc , class-level token contrastive learning loss L ctc , self-supervised multi-view contrastive learning loss L ssm、Memory library contrast loss L mmb , the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg Optimize the original CAM to obtain the optimal improved image segmentation model;

[0131] S109, obtaining an image to be segmented, inputting the image to be segmented into an optimal improved image segmentation model, and obtaining a segmentation result.

[0132] like Figure 4 As shown, the image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison obtains a certain number of sample images and image-level classification labels corresponding to the sample images, and performs cropping and data enhancement processing on the sample images to obtain local sample images and multi-view images; an improved image segmentation model is constructed based on multi-dimensional information comparison, wherein the improved image segmentation model includes a hybrid token comparison mechanism, a self-supervised multi-view comparison mechanism, a memory library comparison mechanism and a segmentation mechanism; the sample image is input into the improved image segmentation model, the patch token features of the sample image in the intermediate auxiliary layer are obtained through the hybrid token comparison mechanism, an auxiliary CAM is generated based on the patch token features, and the patch token-level comparison learning loss L is calculated based on the original CAM and the auxiliary CAM. ptc ; Obtain the local semantics corresponding to the local sample image and the global semantics corresponding to the sample image through a hybrid token comparison mechanism, and calculate the class-level token comparison learning loss L based on the local semantics and the global semantics ctc ; Obtain a multi-view image corresponding to the sample image through a self-supervised multi-view contrast mechanism, and generate a self-supervised CAM corresponding to the multi-view image, and calculate the self-supervised multi-view contrast learning loss L based on the self-supervised CAM and the original CAM ssm ; Extract the memory prototype corresponding to the image target through the memory mechanism, and calculate the memory contrast loss L based on the memory prototype and the image target mmb ; Calculate the main classifier loss L cls ; Calculate the auxiliary classifier loss L mcls ; Calculate the segmentation loss L seg ; Based on the patch token-level contrastive learning loss L ptc , class-level token contrastive learning loss L ctc , self-supervised multi-view contrastive learning loss L ssm 、Memory library contrast loss L mmb , the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L segThe original CAM is optimized to obtain an optimal improved image segmentation model; an image to be segmented is obtained, and the image to be segmented is input into the optimal improved image segmentation model to obtain a segmentation result. This solves the problem in the prior art that it is impossible to fully mine multi-source supervision information and thus it is impossible to ensure that the image segmentation model performs accurate segmentation.

[0133] This application fully exploits the context modeling capability of the visual transformer model itself for images, introduces auxiliary class activation map CAM, and uses patch token comparison strategy and class token comparison strategy to promote image semantic feature alignment; designs a self-supervised multi-view module, uses the spatial correlation of multiple views of the image to mine self-supervised information, and provides the model with useful information implicit in the image space itself; designs a memory prototype comparison module, uses an updateable memory library strategy to extract features from historical target information, and uses adaptive memory comparison loss to guide subsequent images to locate targets. Combining image spatial information, historical information and model semantic comparison information, it mines supervision information in multiple dimensions and effectively improves segmentation performance.

[0134] Figure 2-Figure 3 FIG. 1 is an architecture diagram of an embodiment of an image-level weakly supervised semantic segmentation system based on multi-dimensional information comparison of the present invention; Figure 2 As shown, an image-level weakly supervised semantic segmentation system based on multi-dimensional information comparison provided by an embodiment of the present invention includes the following steps:

[0135] An acquisition module 10 is used to acquire a certain number of sample images and image-level classification labels corresponding to the sample images;

[0136] A preprocessing module 20, configured to perform cropping and data enhancement processing on the sample image to obtain a local sample image and a multi-view image;

[0137] A construction module 30 is used to construct an improved image segmentation model 50 based on multi-dimensional information comparison, wherein the improved image segmentation model 50 includes a hybrid token comparison mechanism 501, a self-supervised multi-view comparison mechanism 502, a memory library comparison mechanism 503 and a segmentation mechanism 504;

[0138] The hybrid token comparison mechanism 501 is used to input the sample image into the improved image segmentation model 50, obtain the patch token features of the sample image at the intermediate auxiliary layer through the hybrid token comparison mechanism 501, generate the auxiliary CAM based on the patch token features, and calculate the patch token level comparison learning loss L based on the original CAM and the auxiliary CAM. ptc ;

[0139] The hybrid token comparison mechanism 501 is also used to:

[0140] The patch token-level contrastive learning loss L is calculated by formula (1):ptc ;

[0141]

[0142] Where, L ptc is the patch token-level contrastive learning loss, Mask + is the matrix R mid The positive mask of - is the matrix R mid The negative mask, N + is the number of pairs, N - is the number of negative pairs, is the Hadamard product matrix operation, G is the two-dimensional feature vector after the patch token sequence is reshaped, G i and G j Sim cos (·,·) is the calculation of cosine similarity.

[0143] The local semantics corresponding to the local sample image and the global semantics corresponding to the sample image are obtained by the hybrid token comparison mechanism 501, and the class-level token comparison learning loss L is calculated based on the local semantics and the global semantics. ctc ;

[0144] The class-level token contrastive learning loss L is calculated by formula (2): ctc ;

[0145]

[0146] Where, L ctc is the class-level token contrastive learning loss, N + t + The number of τ is a temperature factor, ε is a positive number to avoid the denominator being 0, t + is a positive token, t l is the global token, t - is a negative token, t l A global token.

[0147] A self-supervised multi-view contrast mechanism 502 is used to obtain a multi-view image corresponding to the sample image through the self-supervised multi-view contrast mechanism 502, and generate a self-supervised CAM corresponding to the multi-view image, and calculate a self-supervised multi-view contrast learning loss L based on the self-supervised CAM and the original CAM ssm ;

[0148] The self-supervised multi-view comparison mechanism 502 is also used for:

[0149] The self-supervised multi-view contrastive learning loss L is calculated by formula (3): ssm ;

[0150] L ssm =L ssm1 +L ssm2 Formula (3);

[0151] Where, L ssm is the self-supervised multi-view contrastive learning loss, L ssm1 is the segmentation difference between the enhanced image and the original image, L ssm2 is the segmentation difference between the enhanced images.

[0152] The memory comparison mechanism 503 is used to extract the memory prototype corresponding to the image target through the memory mechanism, and calculate the memory comparison loss L based on the memory prototype and the image target. mmb ;

[0153] The memory comparison mechanism 503 is also used for:

[0154] The memory library contrast loss L is calculated by formula (4): mmb ;

[0155]

[0156] Where, L mmb is the memory library comparison loss, ρ is the similarity coefficient, Mask + is the matrix R mid The positive mask of - is the matrix R mid The negative mask, N + is the number of pairs, N - is the number of negative pairs, is the Hadamard product matrix operation, F i and F j Sim cos (·,·) is the calculation of cosine similarity.

[0157] Split mechanism 504, used to calculate the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg ;

[0158] The segmentation mechanism 504 is also used to:

[0159] The main classifier loss L is calculated by formula (5): cls ;

[0160]

[0161] Where, L clsis the main classifier loss, N is the number of labels, x is the sample image, y is the true label, and f i (x) is the predicted value of the i-th label, and σ is the Sigmoid function.

[0162] The segmentation loss L is calculated by formula (6): seg ;

[0163]

[0164] Where, L seg is the segmentation loss, N' is the total number of pixels in the image, c is the number of categories, if pixel i belongs to category c, then y i,c is 1, if pixel i does not belong to category c, then y i,c is 0, p i,c is the model’s predicted probability that pixel i belongs to category c.

[0165] A training module 40 is configured to train a patch token-level contrastive learning loss L ptc , class-level token contrastive learning loss L ctc , self-supervised multi-view contrastive learning loss L ssm 、Memory library contrast loss L mmb , the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg Optimizing the original CAM to obtain an optimal improved image segmentation model 50;

[0166] The optimal improved image segmentation model 50 is used to receive the image to be segmented and obtain a segmentation result.

[0167] Figure 5 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 5 As shown, the electronic device 60 includes: a processor 601 (processor), a memory 602 (memory) and a bus 603;

[0168] The processor 601 and the memory 602 communicate with each other via the bus 603;

[0169] The processor 601 is used to call the program instructions in the memory 602 to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining a certain number of sample images and image-level classification labels corresponding to the sample images, and performing cropping and data enhancement processing on the sample images to obtain local sample images and multi-view images; constructing an improved image segmentation model based on multi-dimensional information comparison, wherein the improved image segmentation model includes a hybrid token comparison mechanism, a self-supervised multi-view comparison mechanism, a memory library comparison mechanism and a segmentation mechanism; inputting the sample image into the improved image segmentation model, obtaining the patch token features of the sample image in the intermediate auxiliary layer through the hybrid token comparison mechanism, generating an auxiliary CAM based on the patch token features, and calculating the patch token-level comparison learning loss L based on the original CAM and the auxiliary CAM. ptc ; Obtain the local semantics corresponding to the local sample image and the global semantics corresponding to the sample image through a hybrid token comparison mechanism, and calculate the class-level token comparison learning loss L based on the local semantics and the global semantics ctc ; Obtain a multi-view image corresponding to the sample image through a self-supervised multi-view contrast mechanism, and generate a self-supervised CAM corresponding to the multi-view image, and calculate the self-supervised multi-view contrast learning loss L based on the self-supervised CAM and the original CAM ssm ; Extract the memory prototype corresponding to the image target through the memory mechanism, and calculate the memory contrast loss L based on the memory prototype and the image target mmb ; Calculate the main classifier loss L cls ; Calculate the auxiliary classifier loss L mcls ; Calculate the segmentation loss L seg ; Based on the patch token-level contrastive learning loss L ptc , class-level token contrastive learning loss L ctc , self-supervised multi-view contrastive learning loss L ssm 、Memory library contrast loss L mmb , the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg The original CAM is optimized to obtain an optimal improved image segmentation model; an image to be segmented is acquired, and the image to be segmented is input into the optimal improved image segmentation model to obtain a segmentation result.

[0170] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining a certain number of sample images and image-level classification labels corresponding to the sample images, and performing cropping and data enhancement processing on the sample images to obtain local sample images and multi-view images; constructing an improved image segmentation model based on multi-dimensional information comparison, wherein the improved image segmentation model includes a hybrid token comparison mechanism, a self-supervised multi-view comparison mechanism, a memory library comparison mechanism and a segmentation mechanism; inputting the sample image into the improved image segmentation model, obtaining the patch token features of the sample image in the intermediate auxiliary layer through the hybrid token comparison mechanism, generating an auxiliary CAM based on the patch token features, and calculating the patch token-level comparison learning loss L based on the original CAM and the auxiliary CAM. ptc ; Obtain the local semantics corresponding to the local sample image and the global semantics corresponding to the sample image through a hybrid token comparison mechanism, and calculate the class-level token comparison learning loss L based on the local semantics and the global semantics ctc ; Obtain a multi-view image corresponding to the sample image through a self-supervised multi-view contrast mechanism, and generate a self-supervised CAM corresponding to the multi-view image, and calculate the self-supervised multi-view contrast learning loss L based on the self-supervised CAM and the original CAM ssm ; Extract the memory prototype corresponding to the image target through the memory mechanism, and calculate the memory contrast loss L based on the memory prototype and the image target mmb ; Calculate the main classifier loss L cls ; Calculate the auxiliary classifier loss L mcls ; Calculate the segmentation loss L seg ; Based on the patch token-level contrastive learning loss L ptc , class-level token contrastive learning loss L ctc , self-supervised multi-view contrastive learning loss L ssm 、Memory library contrast loss L mmb , the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg The original CAM is optimized to obtain an optimal improved image segmentation model; an image to be segmented is acquired, and the image to be segmented is input into the optimal improved image segmentation model to obtain a segmentation result.

[0171] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0172] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0174] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. An image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison, characterized in that: The image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison specifically includes: Obtaining a certain number of sample images and image-level classification labels corresponding to the sample images, and performing cropping and data enhancement processing on the sample images to obtain local sample images and multi-view images; Constructing an improved image segmentation model based on multi-dimensional information comparison, wherein the improved image segmentation model includes a hybrid token comparison mechanism, a self-supervised multi-view comparison mechanism, a memory library comparison mechanism and a segmentation mechanism; The sample image is input into the improved image segmentation model, the patch token features of the sample image in the intermediate auxiliary layer are obtained through the hybrid token contrast mechanism, the auxiliary CAM is generated based on the patch token features, and the patch token level contrast learning loss L is calculated based on the original CAM and the auxiliary CAM. ptc ; The local semantics corresponding to the local sample image and the global semantics corresponding to the sample image are obtained through a hybrid token comparison mechanism, and the class-level token comparison learning loss L is calculated based on the local semantics and the global semantics. ctc ; The multi-view image corresponding to the sample image is obtained through the self-supervised multi-view contrast mechanism, and a self-supervised CAM corresponding to the multi-view image is generated. The self-supervised multi-view contrast learning loss L is calculated based on the self-supervised CAM and the original CAM. ssm ; The memory prototype corresponding to the image target is extracted through the memory mechanism, and the memory contrast loss L is calculated based on the memory prototype and the image target. mmb ; Calculate the main classifier loss L cls ; Calculate the auxiliary classifier loss L mcls ; Calculate the segmentation loss L seg ; Based on the patch token-level contrastive learning loss L ptc , class-level token contrastive learning loss L ctc , self-supervised multi-view contrastive learning loss L ssm 、Memory library contrast loss L mmb , the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg Optimizing the original CAM to obtain an optimal improved image segmentation model; An image to be segmented is obtained, and the image to be segmented is input into the optimal improved image segmentation model to obtain a segmentation result.

2. The image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison according to claim 1, characterized in that: The patch token features of the sample image in the intermediate auxiliary layer are obtained by a hybrid token comparison mechanism, an auxiliary CAM is generated based on the patch token features, and a patch token level comparison learning loss L is calculated based on the original CAM and the auxiliary CAM. ptc ,include: The patch token-level contrastive learning loss L is calculated by formula (1): ptc ; Where, L ptc is the patch token-level contrastive learning loss, Mask + is the matrix R mid The positive mask of - is the matrix R mid The negative mask, N + is the number of pairs, N - is the number of negative pairs, is the Hadamard product matrix operation, G is the two-dimensional feature vector after the patch token sequence is reshaped, G i and G j Sim cos (·,·) is the calculation of cosine similarity.

3. The image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison according to claim 1, characterized in that: The local semantics corresponding to the local sample image and the global semantics corresponding to the sample image are obtained through a hybrid token comparison mechanism, and the class-level token comparison learning loss L is calculated based on the local semantics and the global semantics. ctc ,include: The class-level token contrastive learning loss L is calculated by formula (2): ctc ; Where, L ctc is the class-level token contrastive learning loss, N + t + The number of τ is a temperature factor, ε is a positive number to avoid the denominator being 0, t + is a positive token, t l is the global token, t - is a negative token, t l A global token.

4. The image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison according to claim 1, characterized in that: The multi-view image corresponding to the sample image is obtained through the self-supervised multi-view contrast mechanism, and a self-supervised CAM corresponding to the multi-view image is generated, and the self-supervised multi-view contrast learning loss L is calculated based on the self-supervised CAM and the original CAM. ssm ,include: The self-supervised multi-view contrastive learning loss L is calculated by formula (3): ssm ; L ssm =L ssm1 +L ssm2 Formula (3); Where, L ssm is the self-supervised multi-view contrastive learning loss, L ssm1 is the segmentation difference between the enhanced image and the original image, L ssm2 is the segmentation difference between the enhanced images.

5. The image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison according to claim 1, characterized in that: The memory library prototype corresponding to the image target is extracted through the memory library mechanism, and the memory library contrast loss L is calculated based on the memory library prototype and the image target. mmb ,include: The memory library contrast loss L is calculated by formula (4): mmb ; Where, L mmb is the memory library comparison loss, ρ is the similarity coefficient, Mask + is the matrix R mid The positive mask of - is the matrix R mid The negative mask, N + is the number of pairs, N - is the number of negative pairs, and is the Hadamard product matrix operation, F i and F j Sim cos (·,·) is the calculation of cosine similarity.

6. The image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison according to claim 1, characterized in that: The calculation of the main classifier loss L cls ,include: The main classifier loss L is calculated by formula (5): cls ; Where, L cls is the main classifier loss, N is the number of labels, x is the sample image, y is the true label, and f i (x) is the predicted value of the i-th label, and σ is the Sigmoid function.

7. The image-level weakly supervised semantic segmentation method based on multi-dimensional information comparison according to claim 1, characterized in that: The segmentation loss L is calculated seg ,include: The segmentation loss L is calculated by formula (6): seg ; Where, L seg is the segmentation loss, N' is the total number of pixels in the image, c is the number of categories, if pixel i belongs to category c, then y i,c is 1, if pixel i does not belong to category c, then y i,c is 0, p i,c is the model’s predicted probability that pixel i belongs to category c.

8. An image-level weakly supervised semantic segmentation system based on multi-dimensional information comparison, characterized in that: include: An acquisition module, used to acquire a certain number of sample images and image-level classification labels corresponding to the sample images; A preprocessing module, used for performing cropping and data enhancement processing on the sample image to obtain a local sample image and a multi-view image; A construction module, used to construct an improved image segmentation model based on multi-dimensional information comparison, wherein the improved image segmentation model includes a hybrid token comparison mechanism, a self-supervised multi-view comparison mechanism, a memory library comparison mechanism and a segmentation mechanism; A hybrid token comparison mechanism is used to input a sample image into the improved image segmentation model, obtain patch token features of the sample image in the intermediate auxiliary layer through the hybrid token comparison mechanism, generate an auxiliary CAM based on the patch token features, and calculate a patch token level comparison learning loss L based on the original CAM and the auxiliary CAM ptc ; The local semantics corresponding to the local sample image and the global semantics corresponding to the sample image are obtained through a hybrid token comparison mechanism, and the class-level token comparison learning loss L is calculated based on the local semantics and the global semantics. ctc ; A self-supervised multi-view comparison mechanism is used to obtain a multi-view image corresponding to the sample image through the self-supervised multi-view comparison mechanism, and generate a self-supervised CAM corresponding to the multi-view image, and calculate the self-supervised multi-view comparison learning loss L based on the self-supervised CAM and the original CAM ssm ; The memory library comparison mechanism is used to extract the memory library prototype corresponding to the image target through the memory library mechanism, and calculate the memory library comparison loss L based on the memory library prototype and the image target mmb ; Split mechanism, used to calculate the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg ; A training module for patch token-level contrastive learning loss L based on the ptc , class-level token contrastive learning loss L ctc , self-supervised multi-view contrastive learning loss L ssm 、Memory library contrast loss L mmb , the main classifier loss L cls , auxiliary classifier loss L mcls and segmentation loss L seg Optimizing the original CAM to obtain an optimal improved image segmentation model; The optimal improved image segmentation model is used to receive the image to be segmented and obtain the segmentation result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images