Image semantic segmentation method, model and equipment based on multi-granularity prototype, and medium

Abstract

The invention provides an image semantic segmentation method based on a multi-granularity prototype, a model, equipment and a medium. The method comprises the following steps: constructing a multi-granularity label space comprising a coarse-granularity label and a fine-granularity label; establishing a structured logic relationship and logic constraints of the coarse-grained labels and the fine-grained labels; a coarse-grained prototype and a fine-grained prototype of the granularity label space are learned and updated, the constructed structured logic is learned through a logic constraint loss function, and a trained segmentation model is obtained and comprises a first backbone network, a fine-grained segmentation head and a coarse-grained segmentation head; and performing segmentation and fusion on the target image through the trained segmentation model to output a segmentation result. The model comprises a multi-granularity label space construction module, a first backbone network, a fine granularity segmentation head, a coarse granularity segmentation head and a fusion module. According to the method provided by the invention, two human cognitive key capabilities of abstraction and structuring can be better integrated into the segmentation model, and the performance of the existing segmentation model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and relates to image processing technology, and in particular to an image semantic segmentation method, model, device and medium based on multi-granularity prototype (MGLP). Background Art

[0002] Convolutional Neural Networks (CNNs) and Visual Transformers (Vi Ts) both use advanced network structures to mine potentially beneficial deep features from massive data under the framework of "model + data + label". This data-driven paradigm has brought considerable breakthroughs and significantly improved the accuracy and efficiency of pixel-level semantic segmentation in various fields. However, this paradigm is actually very different from the human visual cognitive process.

[0003] The visual cognitive process of humans from perceptual space to cognitive space has two inherent characteristics: abstraction and structuring. Among them, the former can compress similar instances into a shared abstract representation, which can restore all common features but is considered very concise. For example, different types of motorcycles are abstracted into a single representation through the retention and removal of information. This abstraction process occurs simultaneously at different granularity levels. For example, motorcycles, trucks, and cars can be uniformly abstracted as "vehicles" at a coarser granularity level. The latter refers to the ability to organize categories at different granularity levels in a way that follows logical structures. These logical structures usually have obvious hierarchical characteristics. For example, vehicles, infrastructure, and people can be distinguished at a coarse granularity level, and then different types of vehicles, different types of facilities, and people with different behaviors can be further identified at a fine granularity level. At each granularity level, different categories are fully separable. The above two inherent characteristics promote each other and together constitute the significant advantage of human vision over machine vision.

[0004] Obviously, the learning paradigm that only fits the "data-label" relationship is difficult for the machine to acquire the ability of abstraction and structure, which hinders the further improvement of the segmentation performance of existing methods. Therefore, in recent years, people have proposed the following strategies:

[0005] (1) For abstraction capabilities, the most widely used method is to build prototypes. Prototypes can be understood as abstract representations of various instance objects with the same concept, usually calculated in the form of feature vectors. A large number of studies have shown that using prototypes to train deep models can effectively improve their ability to distinguish different categories with similar features. However, existing methods all learn prototypes at a fixed granularity level, ignoring the potential semantic hierarchical dependencies between categories.

[0006] (2) Regarding structuring capabilities, existing methods make drastic modifications to the network structure, pay too much attention to the acquired category hierarchy, and lack the expression of the logical relationship between categories at different levels.

[0007] Furthermore, existing methods study the two inherent characteristics of abstraction and structuring separately without generalizing them to different models and scenarios, which hinders the further leap of machine vision to human vision.

[0008] In general, existing image semantic segmentation methods only fit the data-label correspondence in a data-driven manner, which cannot fully conform to the abstract and structured characteristics of the human visual cognitive process, limiting their performance ceiling. Summary of the invention

[0009] In order to solve the technical problem that the existing image semantic segmentation method only fits the data-label correspondence in a data-driven manner and cannot fully meet the abstract and structured characteristics of the human visual cognitive process, the present invention discloses an image semantic segmentation method based on a multi-granularity prototype, and the method comprises the following steps:

[0010] S1. Based on the feature similarity clustering method, a coarse-grained label of the target image is generated according to the fine-grained label, and a multi-grained label space is constructed using the coarse-grained label and the fine-grained label;

[0011] S2, dividing the target image into a training set and a test set, and performing prototype learning on the coarse-grained labels and the fine-grained labels in the multi-granularity label space through the training set and the established structured logic to obtain a trained first backbone network, a fine-grained segmentation head, and a coarse-grained segmentation head;

[0012] S3, using the trained first backbone network and the fine-grained segmentation head to predict the test set to obtain a first prediction result, and using the trained first backbone network and the coarse-grained segmentation head to predict the test set to obtain a second prediction result;

[0013] S4. Fusing the first prediction result and the second prediction result to obtain a segmentation result of the target image, wherein the segmentation result includes a category of each pixel in the target image.

[0014] An embodiment of the present invention also provides an image semantic segmentation model based on a multi-granularity prototype, including a multi-granularity label space construction module, a first backbone network, a fine-grained segmentation head, a coarse-grained segmentation head and a fusion module.

[0015] The multi-granularity label space construction module is used to generate coarse-grained labels of the target image according to fine-grained labels based on a feature similarity clustering method, and construct a multi-granularity label space using the coarse-grained labels and the fine-grained labels;

[0016] The first backbone network is used to extract a feature map of the target image;

[0017] The fine-grained segmentation head is used to predict the feature map to obtain a first prediction result;

[0018] The coarse-grained segmentation head is used to predict the feature map to obtain a second prediction result;

[0019] The fusion module is used to fuse the first prediction result and the second prediction result to obtain a segmentation result of the target image, where the segmentation result includes a category of each pixel in the target image.

[0020] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned image semantic segmentation methods based on multi-granularity prototypes when executing the computer program, so as to solve the technical problem that the existing image semantic segmentation methods only fit the data-label correspondence in a data-driven manner and cannot fully meet the abstract and structured characteristics of the human visual cognitive process.

[0021] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned image semantic segmentation methods based on multi-granularity prototypes, so as to solve the technical problem that the existing image semantic segmentation methods only fit the data-label correspondence in a data-driven manner and cannot fully meet the abstract and structured characteristics of the human visual cognitive process.

[0022] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: the image semantic segmentation method based on multi-granularity prototypes of the present invention, first, generates coarse-grained labels through a feature clustering method on the basis of the original fine-grained labels, and constructs a multi-granularity label space using coarse-grained labels and fine-grained labels; secondly, extracts segmentation features from the feature maps of the training set through a coarse-grained segmentation head and a fine-grained segmentation head to obtain fine-grained features and coarse-grained features; then, through the fine-grained features and coarse-grained features, a loss function, the logical relationship between the coarse-grained labels and the fine-grained labels, and the designed structured constraints, the coarse-grained label prototypes and the fine-grained label prototypes in the multi-granularity label space are iteratively calculated and learned to obtain the trained coarse-grained segmentation head and the fine-grained segmentation head; finally, the trained coarse-grained segmentation head and the fine-grained segmentation head are used to predict the test set respectively to obtain two prediction results, and the two prediction results are fused to obtain the final segmentation result.

[0023] The method and model of the present invention can better integrate the two key capabilities of abstraction and structuring into the segmentation model, further improving the performance of the existing segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flowchart of an image semantic segmentation method based on multi-granularity prototypes disclosed in an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of an image semantic segmentation method based on multi-granularity prototypes disclosed in an embodiment of the present invention;

[0027] Figure 3 The multi-granularity label space construction process disclosed in the embodiment of the present invention;

[0028] Figure 4 The horizontal logical relationship in the multi-granularity label space established under the premise of considering the semantic level dependency disclosed in the embodiment of the present invention;

[0029] Figure 5 The segmentation map obtained by semantically segmenting Cityscapes using the method of the present invention and traditional SegFormer, OCRNet, and UPerNet (ConvNeXt) disclosed in the embodiment of the present invention;

[0030] Figure 6 It is an architecture diagram of an image semantic segmentation model based on multi-granularity prototypes disclosed in an embodiment of the present invention;

[0031] Figure 7 A schematic diagram of a computer device disclosed in an embodiment of the present invention;

[0032] Among them, 601, multi-granularity label space construction module; 602, first backbone network; 603, fine-grained segmentation head; 604, coarse-grained segmentation head; 605, fusion module; 701, memory; 702, processor. DETAILED DESCRIPTION

[0033] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0034] The following describes the implementation methods of the present application through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and the features of the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0035] The embodiment of the present invention discloses a method for image semantic segmentation based on multi-granularity prototypes, see Figure 1 and Figure 2 As shown, the method comprises the following steps:

[0036] S1. Based on the feature similarity clustering method, a coarse-grained label of the target image is generated according to the fine-grained label, and a multi-grained label space is constructed using the coarse-grained label and the fine-grained label;

[0037] S2, dividing the target image into a training set and a test set, and performing prototype learning on the coarse-grained labels and the fine-grained labels in the multi-granularity label space through the training set and the established structured logic to obtain a trained first backbone network, a fine-grained segmentation head, and a coarse-grained segmentation head;

[0038] S3, using the trained first backbone network and the fine-grained segmentation head to predict the test set to obtain a first prediction result, and using the trained first backbone network and the coarse-grained segmentation head to predict the test set to obtain a second prediction result;

[0039] S4. Fusing the first prediction result and the second prediction result to obtain a segmentation result of the target image, wherein the segmentation result includes a category of each pixel in the target image.

[0040] Generally, traditional segmentation models usually learn semantic features based on the label space provided by a specific data set. For example, the widely used COCO-Stuff data set contains 171 classes (specific subclasses, corresponding to the fine-grained labels of the present invention), many of which actually belong to the same superclass (corresponding to the coarse-grained labels of the present invention). For example, the three classes of bananas, apples and oranges obviously belong to the same superclass, and the three classes of cats, dogs and birds obviously belong to the same superclass. In the existing traditional segmentation methods, it is simply assumed that the relationship between labels is non-intersecting and mutually exclusive, and pixel-level predictions are performed based on these labels, ignoring the hierarchical structure between different classes. Therefore, the present invention automatically and flexibly generates coarse-grained labels based on the original fine-grained labels through the above-mentioned step S1, so as to better model the hierarchical semantic relationship between different categories.

[0041] In an optional embodiment of step S1, based on a feature similarity clustering method, a coarse-grained label of the target image is generated according to the fine-grained label, and a multi-grained label space is constructed using the coarse-grained label and the fine-grained label. Figure 3 As shown, the following steps are included:

[0042] S11. Use the second backbone network pre-trained on the ImageNet image dataset to convert the collected original images into feature maps.

[0043] S12: using the fine-grained labels provided by the set data set as a guide, clustering the feature graphs through a feature similarity clustering algorithm to generate coarse-grained labels.

[0044] Specifically, under the guidance of the original fine-grained labels, the pixel regions belonging to the same category can be averaged and pooled. The feature vectors corresponding to the obtained different fine-grained categories are clustered according to the similarity metric to generate coarse-grained labels. The generation process of coarse-grained labels can be expressed by the following formula (1):

[0045]

[0046] Among them, C and represent the original fine-grained labels and the generated coarse-grained labels respectively, and Y represent feature maps and labels, and V is the feature vector obtained by label-guided average pooling. c is the number of images containing category c, |Y c| is the number of pixels belonging to category c, and H·W is the spatial size of the feature map. In theory, the clustering algorithm can be arbitrary. The present invention preferably uses a hierarchical clustering algorithm, which can automatically determine the number of clusters (coarse-grained labels) according to similarity, which greatly reduces the workload of setting hyperparameters by trial and error.

[0047] S13: Generate a multi-granularity label space with a two-layer tree structure by using the coarse-grained label and the fine-grained label Among them, in the multi-granularity label space, one coarse-grained label corresponds to multiple fine-grained labels.

[0048] In an optional embodiment of step S2, the coarse-grained labels and the fine-grained labels in the multi-granularity label space are prototyped by the training set and the established structured logic to obtain a trained first backbone network, a fine-grained segmentation head, and a coarse-grained segmentation head, including:

[0049] S21. Use the initial first backbone network to extract the feature map of the test set, use the initial fine-grained segmentation head to predict the feature map to obtain fine-grained features, and use the initial coarse-grained segmentation head to predict the feature map to obtain coarse-grained features.

[0050] S22. In the multi-granularity label space, based on the fine-grained features and the coarse-grained features, iteratively generate a coarse-grained label prototype and a fine-grained label prototype by online weighted fusion.

[0051] When the steps S21 and S22 are specifically implemented, see Figure 2 As shown, in form, the input target image is recorded as The first backbone network is denoted as G, and the feature maps generated by the fine-grained segmentation head H1 and the coarse-grained segmentation head H2 are denoted by Z1 and Z2 respectively, where For each fine-grained class c∈C, the calculation method of the fine-grained prototype is shown in the following formula (2):

[0052]

[0053] Among them, in formula (2) Represents the learned prototype vector; represents the weight matrix, obtained using the Sinkhorn-Knopp iterative algorithm, N′ c represents the number of pixels belonging to category c; represents the feature vector corresponding to category c, represents the mask of pixels that are correctly classified. The symbol ⊙ represents element-wise multiplication.

[0054] During the entire training process, the prototype is continuously updated online based on the latest weighted fusion results of the feature vectors. Specifically, after the kth training iteration, the prototype is updated as shown in the following formula (3):

[0055]

[0056] Among them, μ∈[0,1] controls the update momentum. It should be noted that the establishment and update process of the coarse-grained label prototype is the same as that of the fine-grained label prototype. The difference between the two lies in the feature Z2 output by the segmentation head H2 and the generated coarse-grained label The present invention will not describe it in detail.

[0057] S23. Through the cross entropy loss function, the fine-grained features and the coarse-grained features are used to respectively calculate the losses of the coarse-grained label prototype, the fine-grained label prototype and the constructed structured logic, and the coarse-grained label prototype, the fine-grained label prototype and the constructed structured logic are iteratively learned according to the calculated losses to obtain the trained first backbone network, fine-grained segmentation head and coarse-grained segmentation head.

[0058] In an optional embodiment of step S23, the cross entropy loss function is used to calculate the losses of the coarse-grained label prototype, the fine-grained label prototype and the constructed structured logic respectively using the fine-grained features and the coarse-grained features, and the coarse-grained label prototype, the fine-grained label prototype and the constructed structured logic are iteratively learned according to the calculated losses, including:

[0059] S231, using the cross entropy loss function and the nearest retrieval principle, using the fine-grained features to calculate the loss of the fine-grained label prototype The loss of the coarse-grained label prototype is calculated using the coarse-grained features Based on total loss The objective of minimizing is to iteratively learn the coarse-grained label prototype and the fine-grained label prototype.

[0060] In the specific implementation, the fine-grained and coarse-grained prototypes are represented by P1 and P2 respectively. In each training iteration, the distance between all pixel features and the prototypes at the same granularity level is calculated, and the category prediction is realized based on the nearest retrieval principle. The prediction result is expressed as shown in the following formula (4):

[0061]

[0062] Among them, the distance metric is the negative cosine similarity. The first and second rows represent the pixel prediction results at the fine-grained layer and the coarse-grained layer, respectively. Under the supervision of fine-grained and coarse-grained labels, the total loss of multi-granular learning is It can be expressed as the following formula (5):

[0063]

[0064] Among them, the z，c = <z,p c > represents the distance between the feature vector and the assigned prototype, c z represents the true label, and They are all standard cross entropy losses, which minimize the total loss The learned multi-granularity prototypes are continuously optimized. For the same category, learning fine-grained label prototypes can enhance its discriminability in the fine-grained semantic space, while learning coarse-grained label prototypes can establish the relationship between it and other similar categories, making the description of the category richer and more complete.

[0065] S232. Calculate the loss of the structured logic using the fine-grained features and the coarse-grained features through a cross entropy loss function, and iteratively learn the structured logic according to the calculated loss and the established structured logic constraints.

[0066] In an optional embodiment of step S232, the step of calculating the loss of the structured logic using the fine-grained features and the coarse-grained features through a cross entropy loss function, and iteratively learning the structured logic according to the calculated loss and the established structured logic constraints includes:

[0067] S2321. The structured logic includes horizontal logical relationships and vertical logical relationships, and horizontal logical constraints and vertical logical constraints are established.

[0068] Among them, the horizontal logical relationship refers to the discriminability of different prototypes at each granularity level. By increasing the distance between different prototypes, the ability to distinguish similar categories can be effectively improved. In order to increase the discriminability between prototypes, it is necessary to fully consider the semantic hierarchical dependency between coarse-grained prototypes and fine-grained prototypes.

[0069] S2322. According to the fine-grained features and the coarse-grained features, the horizontal total loss of the multi-granularity label space is calculated by a cross entropy loss function, and the horizontal logical relationship is iteratively learned according to the horizontal logical constraints and the horizontal total loss.

[0070] S2323. According to the fine-grained features and the coarse-grained features, the vertical total loss of the multi-granularity label space is calculated by a cross entropy loss function, and the vertical logical relationship is iteratively learned according to the vertical logical constraints and the vertical total loss.

[0071] In the above step S2322, the horizontal total loss of the multi-granularity label space is calculated by a cross entropy loss function according to the fine-grained features and the coarse-grained features, including:

[0072] S23221, for all the fine-grained labels, according to the fine-grained features of all the fine-grained labels belonging to the same coarse-grained label, a metric relationship operation is performed through a cross entropy loss function to obtain a fine-grained layer loss

[0073] S23222: For all the coarse-grained labels in the multi-granularity label space, a metric relationship operation is performed based on the coarse-grained features through a cross entropy loss function to obtain a coarse-grained layer loss.

[0074] S23223. Calculate the fine-grained layer loss With the coarse-grained layer loss The total horizontal loss is obtained by summing

[0075] In the specific implementation, the metric relationship operation is performed on the fine-grained feature Z1 and the coarse-grained feature Z2 respectively, and the metric relationship between the prototypes is automatically learned in a data-driven manner without using any specific metric. In order to perform the metric relationship on both the fine-grained layer and the coarse-grained layer at the same time, a deep network E is calculated, and the loss on the fine-grained layer The calculation is shown in the following formula (7);

[0076]

[0077] Where Con represents the connection operation, and the output of E is the relationship score r∈[0, 1]. Figure 4 As shown, when two eigenvectors belong to the same class, that is, c i =c j When , the relationship score r is optimized to be close to 1, so that the two feature vectors are pulled closer, making the fine-grained prototype more compact. On the contrary, when c i ≠c j When , the relationship score r is optimized to be close to zero, which means that the two feature vectors are pushed away from each other and different fine-grained prototypes become more separable. It should be noted that the metric relationship operation of Formula 7 is only performed in subclass clusters belonging to the same superclass, that is,

[0078] At the coarse-grained level, the loss The metric relation operation of is performed as shown in equation (8):

[0079]

[0080] According to equations (7) and (8), for a coarse-grained label and its fine-grained label, the “pulling closer” operation at the coarse-grained level is in conflict with the “pushing away” operation at the fine-grained level to a certain extent. Therefore, the “pulling closer” operation at the coarse-grained level is discarded, and the coefficient α can be set to be greater than 1, doubling the “pushing away” operation at the coarse-grained level, so that the prototype clusters of the fine-grained labels corresponding to different coarse-grained labels can be fully separable.

[0081] In the above step S2323, according to the fine-grained features and the coarse-grained features, the vertical total loss of the multi-granularity label space is calculated by a cross entropy loss function, and the vertical logical relationship is iteratively learned according to the vertical logical constraint and the vertical total loss, including:

[0082] S23231, according to the fine-grained features and the coarse-grained features, calculating the forward vertical loss of the multi-granularity label space by a cross entropy loss function

[0083] S23232: Calculate the negative vertical loss of the multi-granularity label space by a cross entropy loss function according to the fine-grained features and the coarse-grained features.

[0084] S23233, Calculate the forward vertical loss and negative vertical loss The sum of the vertical losses is The vertical logical relationship is iteratively learned according to the vertical logical constraint and the vertical total loss.

[0085] In specific implementation, the vertical logical constraint includes a negative constraint from the coarse-grained layer to the fine-grained layer and a positive constraint from the fine-grained layer to the coarse-grained layer. The negative constraint includes that pixels not assigned to the coarse-grained label prototype must not be assigned the fine-grained label prototype it contains, and the positive constraint includes that pixels assigned to the fine-grained label prototype must be assigned the coarse-grained label prototype corresponding to it.

[0086] The positive constraint can be expressed by the following formula (9):

[0087]

[0088] Among them, S C ∈[0, 1] h×w×|C| is the softmax matrix at the fine-grained level, Based on multi-granular label space The coarse-grained pseudo labels generated, is the negative cosine similarity between the coarse-grained feature and the prototype. Threshold t vEnsure that only predictions with high confidence are included. Obviously, formula (9) first generates coarse-grained pseudo-labels from the fine-grained prediction results based on τ. Subsequently, corresponding to the superclass in the pseudo-label, the distance between the coarse-grained features and the superclass prototype is reduced to ensure the bottom-up positive constraint of the prototype assignment.

[0089] The negative constraint can be expressed by the following formula (10):

[0090]

[0091] Among them, is the softmax score matrix at the coarse-grained level, Y C is the pseudo label of the subclass that cannot be assigned fine-grained features. In short, formula (10) first obtains high-confidence fine-grained categories that cannot be assigned fine-grained features based on the coarse-grained prediction results, and then increases the distance between the prototypes and features corresponding to these categories.

[0092] In the above step S23, according to the calculated 6 total losses, the coarse-grained label prototype, the fine-grained label prototype and the constructed structured logic can be iteratively learned respectively by the following formula (11) to obtain the trained first backbone network, the fine-grained segmentation head and the coarse-grained segmentation head, which specifically include:

[0093]

[0094] Among them, i indexes different losses, σ is a learnable weight coefficient, and logσ is used to alleviate the negative impact of too small weights. Compared with the method of using trial and error and fixing the weights during training, dynamically optimizing the weights of different losses not only simplifies the training process, but also makes the training more stable and the performance is further improved.

[0095] In an optional embodiment of the above step S4, the first prediction result and the second prediction result are fused to obtain a segmentation result of the target image, and the segmentation result includes a category of each pixel in the target image, including:

[0096] S41, calculating the product or sum of all paths between the coarse-grained labels and the fine-grained labels in the first prediction result and the second prediction result;

[0097] S42, for the paths involved in each pixel point of the target image, select the path corresponding to the maximum product or sum as the optimal path, and use the result of the fine-grained label corresponding to the optimal path as the final category of the pixel point.

[0098] In specific implementation, by combining the prediction results of the coarse-grained segmentation map and the fine-grained segmentation map for pixel reasoning, the softmax score matrix generated by the nearest prototype retrieval at different granularity levels can be uniformly expressed as The final label of each pixel is determined by the highest-scoring path in the two-layer hierarchical tree structure of the multi-granularity label space using the following formula (6):

[0099]

[0100] in represents all feasible paths between superclasses and subclasses, which actually corresponds to the edge set ε. Symbols v2 and v1 represent coarse-grained and fine-grained nodes, respectively. Formula (6) provides two optional calculation methods, namely, sum and product. The final label of each pixel is the category of the fine-grained node in the path calculated in Formula (6).

[0101] The principle of the image semantic segmentation method based on multi-granularity prototypes of the present invention is: first, based on the original fine-grained labels, coarse-grained labels are generated by a feature clustering method, and a multi-granularity label space is constructed by using coarse-grained labels and fine-grained labels; secondly, segmentation features are extracted from the feature map of the training set by using a coarse-grained segmentation head and a fine-grained segmentation head to obtain fine-grained features and coarse-grained features; then, the coarse-grained label prototypes and the fine-grained label prototypes in the multi-granularity label space are iteratively calculated and learned by using the fine-grained features and the coarse-grained features, the logical relationship between the coarse-grained labels and the fine-grained labels, and the designed structured constraints to obtain the trained coarse-grained segmentation head and the fine-grained segmentation head; finally, the trained coarse-grained segmentation head and the fine-grained segmentation head are used to predict the test set respectively to obtain two prediction results, and the two prediction results are fused to obtain the final segmentation result.

[0102] The present invention compares the traditional SegFormer, OCRNet, and UPerNet (ConvNeXt) methods with the MGLP method of the present invention. Compared with the traditional methods, the method of the present invention improves mIoU by 1.69%, 1.42%, and 1.04%, respectively. Figure 5 As shown, the method of the present invention has significant improvements on the segmentation map of the Cityscapes dataset. Specifically, MGLP can further optimize the segmentation effect of scene details. The third column shows a typical example, in which MGLP helps to identify small targets such as ground pillars. Similar results can also be found in the first and second columns. In summary, the method and model of the present invention can better integrate the two key capabilities of abstraction and structuring into the segmentation model, which can reduce the phenomenon of incorrect segmentation, effectively improve the recognition effect of small targets that are prone to class confusion, and further improve the performance of existing segmentation models.

[0103] Based on the same inventive concept, an image semantic segmentation model based on multi-granularity prototypes is also provided in an embodiment of the present invention, as described in the following embodiments. Since the principle of solving the problem by the image semantic segmentation model is similar to the above-mentioned image semantic segmentation method based on multi-granularity prototypes, the implementation of the image semantic segmentation model can refer to the implementation of the above-mentioned image semantic segmentation method based on multi-granularity prototypes, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the model described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0104] Figure 6 is a structural block diagram of an image semantic segmentation model based on multi-granularity prototypes disclosed in an embodiment of the present invention, such as Figure 6 As shown, the image semantic segmentation model includes a multi-granularity label space construction module 601, a first backbone network 602, a fine-grained segmentation head 603, a coarse-grained segmentation head 604 and a fusion module 605. The structure is described below.

[0105] The multi-granularity label space construction module 601 is used to generate coarse-grained labels of the target image according to fine-grained labels based on a feature similarity clustering method, and construct a multi-granularity label space using the coarse-grained labels and the fine-grained labels;

[0106] The first backbone network 602 is used to extract the feature map of the target image;

[0107] The fine-grained segmentation head 603 is used to predict the feature map to obtain a first prediction result;

[0108] The coarse-grained segmentation head 604 is used to predict the feature map to obtain a second prediction result;

[0109] The fusion module 605 is used to fuse the first prediction result and the second prediction result to obtain a segmentation result of the target image, where the segmentation result includes a category of each pixel in the target image.

[0110] In this embodiment, a computer device is provided, such as Figure 7 As shown, it includes a memory 701, a processor 702, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned image semantic segmentation methods based on multi-granularity prototypes is implemented.

[0111] Specifically, the computer device may be a computer terminal, a server or a similar computing model.

[0112] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned image semantic segmentation methods based on multi-granularity prototypes.

[0113] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include transitory media such as modulated data signals and carrier waves.

[0114] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented with a general computing model, they can be concentrated on a single computing model, or distributed on a network composed of multiple computing models, and optionally, they can be implemented with program codes executable by the computing model, so that they can be stored in a storage model and executed by the computing model, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for image semantic segmentation based on multi-granularity prototypes, characterized in that: include: Based on the feature similarity clustering method, a coarse-grained label of the target image is generated according to the fine-grained label, and a multi-grained label space is constructed using the coarse-grained label and the fine-grained label; The target image is divided into a training set and a test set, and prototype learning is performed on the coarse-grained labels and the fine-grained labels in the multi-granularity label space through the training set and the established structured logic to obtain a trained first backbone network, a fine-grained segmentation head, and a coarse-grained segmentation head; Using the trained first backbone network and the fine-grained segmentation head to predict the test set to obtain a first prediction result, and using the trained first backbone network and the coarse-grained segmentation head to predict the test set to obtain a second prediction result; The first prediction result and the second prediction result are fused to obtain a segmentation result of the target image, wherein the segmentation result includes a category of each pixel in the target image.

2. The image semantic segmentation method based on multi-granularity prototypes according to claim 1, characterized in that: The feature similarity-based clustering method generates coarse-grained labels of target images according to fine-grained labels, and constructs a multi-grained label space using the coarse-grained labels and the fine-grained labels, including: Use the second backbone network pre-trained on the ImageNet image dataset to convert the collected raw images into feature maps; Using the fine-grained labels provided by the set data set as a guide, clustering the feature graphs through a feature similarity clustering algorithm to generate coarse-grained labels; The coarse-grained labels and the fine-grained labels are used to generate a multi-granularity label space with a two-layer tree structure, wherein one coarse-grained label in the multi-granularity label space corresponds to a plurality of the fine-grained labels.

3. The image semantic segmentation method based on multi-granularity prototypes according to claim 1, characterized in that: The method of performing prototype learning on the coarse-grained labels and the fine-grained labels in the multi-granularity label space through the training set and the established structured logic to obtain a trained first backbone network, a fine-grained segmentation head and a coarse-grained segmentation head comprises: Extracting a feature map of the test set using an initial first backbone network, predicting the feature map using an initial fine-grained segmentation head to obtain fine-grained features, and predicting the feature map using an initial coarse-grained segmentation head to obtain coarse-grained features; In the multi-granularity label space, based on the fine-grained features and the coarse-grained features, a coarse-grained label prototype and a fine-grained label prototype are iteratively generated by online weighted fusion; The fine-grained features and the coarse-grained features are used to respectively calculate the losses of the coarse-grained label prototype, the fine-grained label prototype and the constructed structured logic through the cross entropy loss function. The coarse-grained label prototype, the fine-grained label prototype and the constructed structured logic are iteratively learned according to the calculated losses to obtain the trained first backbone network, the fine-grained segmentation head and the coarse-grained segmentation head.

4. The image semantic segmentation method based on multi-granularity prototypes according to claim 3 is characterized in that: The coarse-grained label prototype, the fine-grained label prototype and the constructed structured logic are respectively calculated using the fine-grained features and the coarse-grained features through a cross entropy loss function, and the coarse-grained label prototype, the fine-grained label prototype and the constructed structured logic are respectively iteratively learned according to the calculated losses, including: The fine-grained features are used to calculate the loss of the fine-grained label prototype by using a cross entropy loss function and a nearest retrieval principle; the coarse-grained features are used to calculate the loss of the coarse-grained label prototype, and the coarse-grained label prototype and the fine-grained label prototype are iteratively learned according to the goal of minimizing the total loss; The loss of the structured logic is calculated using the fine-grained features and the coarse-grained features through a cross entropy loss function, and the structured logic is iteratively learned according to the calculated loss and the established structured logic constraints.

5. The image semantic segmentation method based on multi-granularity prototypes according to claim 3 is characterized in that: The loss of the structured logic is calculated using the fine-grained features and the coarse-grained features through a cross entropy loss function, and the structured logic is iteratively learned according to the calculated loss and the established structured logic constraints, including: The structured logic includes horizontal logic relations and vertical logic relations, and establishes horizontal logic constraints and vertical logic constraints; According to the fine-grained features and the coarse-grained features, the horizontal total loss of the multi-granularity label space is calculated by a cross entropy loss function, and the horizontal logical relationship is iteratively learned according to the horizontal logical constraint and the horizontal total loss; According to the fine-grained features and the coarse-grained features, the vertical total loss of the multi-granularity label space is calculated by a cross entropy loss function, and the vertical logical relationship is iteratively learned according to the vertical logical constraints and the vertical total loss.

6. The image semantic segmentation method based on multi-granularity prototypes according to claim 5, characterized in that: The vertical logical constraints include negative constraints from the coarse-grained layer to the fine-grained layer and positive constraints from the fine-grained layer to the coarse-grained layer. The negative constraints include that pixels not assigned to the coarse-grained label prototype must not be assigned the fine-grained label prototype it contains, and the positive constraints include that pixels assigned to the fine-grained label prototype must be assigned the coarse-grained label prototype corresponding to it.

7. The image semantic segmentation method based on multi-granularity prototypes according to claim 1, characterized in that: The fusing of the first prediction result and the second prediction result to obtain a segmentation result of the target image, wherein the segmentation result includes a category of each pixel in the target image, including: Calculating the product or sum of all paths between the coarse-grained labels and the fine-grained labels in the first prediction result and the second prediction result; For the paths involved in each pixel point of the target image, the path corresponding to the maximum product or sum is selected as the optimal path, and the result of the fine-grained label corresponding to the optimal path is used as the final category of the pixel point.

8. An image semantic segmentation model based on multi-granularity prototypes, characterized in that: include: A multi-granularity label space construction module, wherein the multi-granularity label space construction module is used to generate coarse-grained labels of target images according to fine-grained labels based on a feature similarity clustering method, and construct a multi-granularity label space using the coarse-grained labels and the fine-grained labels; A first backbone network, wherein the first backbone network is used to extract a feature map of the target image; A fine-grained segmentation head, wherein the fine-grained segmentation head is used to predict the feature map to obtain a first prediction result; A coarse-grained segmentation head, wherein the coarse-grained segmentation head is used to predict the feature map to obtain a second prediction result; A fusion module is used to fuse the first prediction result and the second prediction result to obtain a segmentation result of the target image, wherein the segmentation result includes a category of each pixel in the target image.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image semantic segmentation method based on multi-granularity prototypes described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the image semantic segmentation method based on multi-granularity prototypes according to any one of claims 1 to 7.

Images