Graph matching strategy-based interpretable image segmentation method

Abstract

The invention relates to an interpretable image segmentation method based on a graph matching strategy, and the method comprises the steps: selecting an image segmentation backbone network, and training the image segmentation backbone network; visual words with semantic information are extracted from the input image through the image segmentation backbone network; according to the visual words with the semantic information, establishing a feature unit-level semantic component relation graph and an overall category-level semantic component relation graph one by one spatial position; and measuring the similarity between the feature unit-level semantic component relation graph and the category-level semantic component relation graph and predicting a segmentation result of the image. The method has the beneficial effects that the reasoning of the deep neural network is converted into an interpretable deductive reasoning process through an interactive matching process between the instance level graph and the category level graph, so that the transparency of model decision is enhanced, and the credibility of a user to model prediction is improved in the key application field.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and more specifically, to an interpretable image segmentation method based on a graph matching strategy. Background Art

[0002] Image segmentation is a key task in the field of computer vision, which aims to group pixels in an image into regions with similar features or semantics. Starting from the success of image segmentation methods in the field of medical imaging, such as DeepLab, U-Net, etc., the application of image segmentation methods can provide richer data and deeper understanding, and has been widely used in various fields, such as autonomous driving, object recognition and tracking, virtual reality, etc.

[0003] However, while deep learning-based models have achieved promising performance on challenging benchmarks, their decisions remain ambiguous due to a lack of interpretability. This problem can be particularly impactful in critical applications such as medical diagnosis, autonomous driving, and other fields, potentially leading to reduced safety.

[0004] Most explainable AI approaches focus on classification or regression tasks. Therefore, explainable segmentation is still considered an open problem, with only a few preliminary studies at the intersection of XAI and segmentation. One is the symbolic semantic framework, where along with segmentation, the model generates symbolic statements derived from a categorical distribution. The other approach generalizes the Grad-CAM approach to the segmentation problem. However, both approaches have significant drawbacks. The former requires a predefined symbolic vocabulary, while the latter may be unreliable and introduce additional bias to the results.

[0005] Existing interpretable methods for image segmentation mainly include gradient-based methods, back-propagation-based methods, and integral-based methods. Gradient-based methods generate heat maps of the focus area by calculating the relationship between the gradient of the feature map and the category score, thereby revealing the classification decision of the model. Back-propagation-based methods retain positive gradients for positively activated pixels and negative gradients for negatively activated pixels, and generate visual heat maps by propagating these gradients back to the input image. Integral-based methods quantify the contribution of each pixel to the model output by calculating the gradient difference between the input image and the reference image. These interpretable methods mainly focus on explaining the areas, features, and contextual information that the model pays attention to in the image.

[0006] In addition, existing image segmentation methods mainly use convolutional neural networks as the core architecture of image segmentation methods. Through methods such as skip links, hole convolutions, and attention mechanisms, the segmentation accuracy is continuously improved, and it has advantages in feature extraction, spatial information retention, and multi-scale feature fusion. In other non-convolutional neural network architectures, traditional methods focus on using local features, connectivity, and edge information of images for segmentation. Because of this, these methods lack the interpretability and comprehensibility of the model, and even cannot correct problems when making wrong decisions, and are not suitable for certain critical applications that require decision-making processes. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the prior art and to propose an interpretable image segmentation method based on a graph matching strategy.

[0008] In the first aspect, an interpretable image segmentation method based on a graph matching strategy is provided, including: S1, selecting an image segmentation backbone network and training the image segmentation backbone network; S2, extracting visual words with semantic information from the input image through the image segmentation backbone network; S3, establishing a feature unit-level semantic component relationship diagram and an overall category-level semantic component relationship diagram according to the visual words with semantic information in each spatial position; S4. Measure the similarity between the feature unit-level semantic component relationship graph and the category-level semantic component relationship graph and predict the image segmentation result.

[0009] Preferably, S2 comprises: S201, given a pre-trained backbone and an input image, obtain intermediate features, and discretize the intermediate features into a sequence of feature components with specific semantics; the feature components are indexes of visual words; S202, mapping characteristic components to weighted vertices of a component relationship graph; S203. Assign a weighted edge to each vertex pair to represent the interaction between them, and obtain a component relationship graph.

[0010] Preferably, S3 includes: S301, for the discretized feature component sequence, establish a unit-level component relationship graph at each spatial position with each pixel point as the center; S302. According to the number of categories in the final segmentation result, a corresponding number of overall category-level semantic component relationship graphs, i.e., category-level semantic component relationship graphs, are established; for each overall category-level semantic component relationship graph, several visual word indexes with the most corresponding feature components in the category in the segmentation result are selected as vertices of each overall category-level semantic component relationship graph, and they are initialized as a fully connected graph.

[0011] Preferably, in S4, a matcher is used to find a category-level semantic component relationship graph with the highest similarity to the component relationship graph corresponding to the input image in the category-level semantic component relationship graph, and finally the image segmentation result is obtained pixel by pixel; the matcher is composed of a GCN module and a similarity calculation module.

[0012] Preferably, in S4, the complexity of the component relationship graph is constrained, including: averaging the component relationship graph instances of each category to initialize the component relationship graph, and deleting edges connected to vertices with weights lower than a given threshold from the graph of a specific category.

[0013] Preferably, in S1, the image segmentation backbone network is Deeplabv2 or U-net.

[0014] In a second aspect, an interpretable image segmentation system based on a graph matching strategy is provided, which is used to execute any of the methods described in the first aspect, including: A selection module is used to select an image segmentation backbone network and train the image segmentation backbone network; An extraction module, used to extract visual words with semantic information from an input image through an image segmentation backbone network; An establishment module is used to establish a feature unit-level semantic component relationship diagram and an overall category-level semantic component relationship diagram according to the visual words with semantic information, spatially; The prediction module is used to measure the similarity between the feature unit level semantic component relationship graph and the category level semantic component relationship graph and predict the image segmentation result.

[0015] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0016] In a fourth aspect, an electronic device is provided, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement any method as described in the first aspect.

[0017] The beneficial effects of the present invention are: 1. This paper transforms the reasoning of deep neural networks (DNNs) into an interpretable deductive reasoning process through the interactive matching process between instance-level graphs and category-level graphs. This approach not only enhances the transparency of model decisions, but also improves user trust in model predictions in key application areas such as medical diagnosis and autonomous driving.

[0018] 2. By converting the image segmentation problem into a graph matching problem, the present invention can accurately identify and explain the classification decision of each pixel, thereby reducing the risk of wrong decisions in critical applications. This is especially important for applications that require extremely high accuracy and reliability, such as medical image analysis and autonomous driving systems.

[0019] 3. The present invention does not rely on a specific convolutional neural network architecture, but can be used in combination with any existing deep image segmentation model (such as Deeplabv2, U-net). This compatibility and flexibility allows the method to be easily integrated into existing computer vision systems without large-scale modifications to the underlying model architecture.

[0020] 4. The present invention can more accurately identify details and features in images through the use of precise graphic matching strategies and component relationship maps, thereby improving segmentation accuracy. This provides significant advantages for applications that require highly accurate image segmentation (such as precise medical diagnosis and fine target tracking). BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of an interpretable image segmentation method based on a graph matching strategy; Figure 2 A schematic diagram of the reasoning process of the interpretable image segmentation method based on the graph matching strategy; Figure 3 Schematic diagram of the structure of the interpretable image segmentation system based on graph matching strategy. DETAILED DESCRIPTION

[0022] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0023] Embodiment 1: Aiming at the problem that the process of computing visual representation and the specific embedding learned are opaque, this paper designs an interpretable image segmentation method based on graph matching strategy. The core idea of ​​this method is to transform DNN reasoning into an interactive matching process between instance-level graph and category-level graph, so as to achieve interpretable deductive reasoning.

[0024] Specifically, Figure 1 As shown, the method includes: S1. Select an image segmentation backbone network and train the image segmentation backbone network.

[0025] The present invention uses image segmentation based on deep image segmentation models (such as Deeplabv2, U-net). Deeplabv2 used in the present invention is a convolutional neural network architecture for image segmentation tasks. It uses techniques such as dilated convolution, multi-scale processing, and conditional random fields to efficiently segment each pixel in the image into parts of different semantic categories. For the backbone network, For RGB images, The output of the backbone network for this image, i.e., the intermediate feature .

[0026] S2. Extract visual words with semantic information from the input image through the image segmentation backbone network.

[0027] Specifically, the step includes: Given a pre-trained backbone and an input image, intermediate features are obtained and discretized into a sequence of feature components with specific semantics; the feature components are the indices of visual words.

[0028] The main purpose of feature discretization is to mine the most common patterns that appear in the data set, each of which corresponds to a semantic meaning (visual word) that can be understood by humans. However, local areas with the same semantics may show different features due to scaling, rotation, or even distortion. The present invention uses the DNN backbone to extract relatively uniform features instead of using traditional SIFT features. Specifically, the visual marker set extracted from the detection data set is Run on Clustering to construct a visual word of size M In addition, by Replace it with the index of the nearest visual word and discretize the intermediate feature X into a sequence of feature components. The formula is:

[0029] Feature components refer to visual words The index of the entire component set When , the discrete sequence is expressed as ,in Calculate using the above formula.

[0030] S3. Establish an instance-level feature unit-level semantic component relationship graph and an overall category-level semantic component relationship graph by spatial position.

[0031] First, we define the component relationship graph and component relationship map mathematically: The component relationship graph is an undirected fully connected graph , where the vertex set It is a set of characteristic components of an instance or a specific category, and the edge set Encodes the interactions between pairs of vertices. Each vertex is assigned a non-negative weight To express its importance, is the set of all vertex weights. In addition, the vertex and The interaction is represented by a non-negative weight Quantitative Definition is the weighted adjacency matrix.

[0032] The component relationship graph is defined as an abstract representation of all features, where Class component relationship graph is a set of category-level component relationship graphs, where Elements of a class With learnable vertex weights and edge weight .

[0033] S3 includes: S301. The discretized feature component sequence can correspond to the pixels of the image one by one, so a feature unit-level semantic component relationship diagram is established spatially with each corresponding pixel as the center.

[0034] It should be noted that the feature unit-level semantic component relationship graph corresponds to each input image instance, so it is instance-level. , whose shape is , taking each pixel point as the center and establishing the spatial position A 5-vertex unit-level component relationship graph , where the vertex set , is the center pixel, , , , The pixels above, below, left and right of the center pixel are the edges. .

[0035] S302. According to the number of categories in the final segmentation result, a corresponding number of overall category-level semantic component relationship graphs, i.e., category-level semantic component relationship graphs, are established; for each overall category-level semantic component relationship graph, several visual word indexes with the most corresponding feature components in the category in the segmentation result are selected as vertices of each overall category-level semantic component relationship graph, and they are initialized as a fully connected graph.

[0036] For example, the final segmentation result contains categories, so we need to create For each category-level semantic component relationship diagram, select the one with the most feature components corresponding to the category in the segmentation result. Visual word index, the formula is:

[0037] Will The visual word indexes are used as vertices of each category graph and it is initialized as a fully connected graph.

[0038] S4. Measure the similarity between the feature unit-level semantic component relationship graph and the category-level semantic component relationship graph and predict the image segmentation result.

[0039] Embodiment 2: Based on Example 1, Example 2 of the present application provides a more specific interpretable image segmentation method based on a graph matching strategy, including: S1. Select an image segmentation backbone network and train the image segmentation backbone network.

[0040] S2. Extract visual words with semantic information from the input image through the image segmentation backbone network.

[0041] S3. According to the visual words with semantic information, a feature unit-level semantic component relationship diagram and an overall category-level semantic component relationship diagram are established spatially.

[0042] S4. Measure the similarity between the feature unit-level semantic component relationship graph and the category-level semantic component relationship graph and predict the image segmentation result.

[0043] In S4, after the input image is converted into a feature unit-level semantic component relationship graph, the matcher finds the most similar category-level semantic component relationship graph in the category-level semantic component relationship graph, and finally obtains the image segmentation result pixel by pixel. The matcher consists of a GCN module and a similarity calculation module that generates the final prediction.

[0044] To feed the component relationship graph s to the GCN module, each component Each vertex in the graph is assigned a trainable embedding vector, where each embedding vector is derived from a dimensional random vectors, which are independently drawn from a multivariate Gaussian distribution Extracted from .

[0045] In the GCN module, the present invention adopts , with a slight modification to the weighted edges: yes The input features of all vertices in the layer, and its output is calculated as:

[0046] in represents a nonlinear activation function, represents feature normalization, is a learnable projection matrix. All After the layer, the weighted average pooling layer embeds the vertices , and press Weighted, graphical representation Now for have , and for All the pictures in By calculating the inner product similarity, the final predicted logarithm is defined as .

[0047] Given an instance graph and a category graph , first analyze the similarity score ,in and are the weighted sums of vertex embeddings respectively. For further discussion, let Represents a vertex The embedding vector of indivual The output of the layer, express The weight of . The original vertex embedding of is defined as , which is the same for the same component in any two graphs. In addition, let yes and The set of shared vertices in . The calculation can be expanded as:

[0048] For a shallow GCN module, s can be approximated as:

[0049] like and , the method of the present invention is equivalent to .

[0050] In addition, the interpretability of the graph matcher with shallow GCN is expressed as follows: (1) the final prediction score of a class is the sum of all existing class evidences represented by the vertex weights; (2) the connections to the shared vertices The shared neighbors (local structure) of also contribute to the final prediction.

[0051] During the inference phase, we use bilinear interpolation combined with the input image size to match the size of the segmented image, while during the training phase, we reduce the resolution of the underlying ground-truth segmentation to fit the size of the output feature map.

[0052] Furthermore, in S4, the present invention constrains the complexity of the component relationship graph to facilitate model training.

[0053] The present invention defines the complexity of the edge and vertex weights of all graphs in the component relationship graph as:

[0054] The function Calculate the input vector The entropy of The sum of the components, Representation and Diagram Categories in Vertex The final optimization goal is:

[0055] in and is a hyperparameter.

[0056] Each graph in the component relationship graph is initialized as a fully connected graph with random vertices and edge weights. However, this requires To alleviate this problem, the present invention initializes the component relationship graph by averaging the component relationship graph instances of each class and removes the instances with weights below a given threshold from the graph of a specific class. Such a process can not only significantly reduce the learnable parameters, but also improve the final performance.

[0057] The present invention conducted experiments on Pascal VOC 2012 (PASCAL Visual Object Classes Challenge 2012). Pascal VOC2012 is a very influential dataset in the field of computer vision, widely used for model training and evaluation of tasks such as object detection, image segmentation, and image classification. It is part of the Pascal VOC series of challenges, and its goal is to promote the research and development of image understanding.

[0058] The present invention achieves an average interaction ratio of 74.5% on the validation set of the Pascal VOC 2012 dataset, which is only 2.2% lower than that of Deeplabv2, as shown in Table 1. However, the interpretability of this method is greatly improved, which helps to improve the reliability of semantic segmentation and has higher usability in key areas such as medical diagnosis and autonomous driving.

[0059] Table 1 Comparison of the effects of the present invention and other methods

[0060] The present invention also visualizes the reasoning process to illustrate the interpretability of the present invention. Figure 2 As shown in , first discretize the features of the elephant map and visualize the components, as Figure 2 As shown in the lower left corner, it can be seen that the image mainly includes components 3, 46, 107, 121, 133, 150, and 198. Different colors represent different components. Component 198 mainly represents the elephant's body, and component 150 mainly represents the elephant's limbs. In addition, there are components representing the sky, grass, etc. The outlines of the elephant's body and limbs are basically consistent with the visualized components. Then, instance-level relationship graphs and category-level relationship graphs are established respectively. The instance-level relationship graph is obtained from the image, and the category-level relationship graph is obtained from training. Finally, the pixel-by-pixel semantic component relationship is performed. Figure 1 Through one-to-one matching, we get the probability that the pixel corresponds to each category, thus achieving interpretable classification of each pixel.

[0061] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0062] Embodiment 3: Based on embodiments 1 and 2, embodiment 3 of the present application provides an interpretable image segmentation system based on a graph matching strategy, including: An interpretable image segmentation system based on a graph matching strategy, including: A selection module is used to select an image segmentation backbone network and train the image segmentation backbone network; An extraction module, used to extract visual words with semantic information from an input image through an image segmentation backbone network; An establishment module is used to establish a feature unit-level semantic component relationship diagram and an overall category-level semantic component relationship diagram according to the visual words with semantic information, spatially; The prediction module is used to measure the similarity between the feature unit-level semantic component relationship graph and the category-level semantic component relationship graph and predict the image segmentation result.

[0063] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiments 1 and 2. Therefore, the parts in this embodiment that are the same or similar to Embodiments 1 and 2 can be referenced to each other and will not be repeated in this application.

Claims

1. An interpretable image segmentation method based on a graph matching strategy, characterized in that: include: S1, selecting an image segmentation backbone network and training the image segmentation backbone network; S2, extracting visual words with semantic information from the input image through the image segmentation backbone network; S3, establishing a feature unit-level semantic component relationship diagram and an overall category-level semantic component relationship diagram according to the visual words with semantic information in each spatial position; S4. Measure the similarity between the feature unit-level semantic component relationship graph and the category-level semantic component relationship graph and predict the image segmentation result.

2. The interpretable image segmentation method based on the graph matching strategy according to claim 1, characterized in that S2 include: Given a pre-trained backbone and an input image, obtain intermediate features and discretize them into a sequence of feature components with specific semantics; The feature component is an index of the visual word.

3. The interpretable image segmentation method based on the graph matching strategy according to claim 2, characterized in that S3 include: S301, for the discretized feature component sequence, establish a unit-level component relationship graph at each spatial position with each pixel point as the center; S302. According to the number of categories in the final segmentation result, a corresponding number of overall category-level semantic component relationship graphs are established; for each overall category-level semantic component relationship graph, several visual word indexes with the most corresponding feature components of the category in the segmentation result are selected as vertices of each overall category-level semantic component relationship graph, and they are initialized as a fully connected graph.

4. The interpretable image segmentation method based on the graph matching strategy according to claim 3 is characterized in that: In S4, a matcher is used to find a category-level graph with the highest similarity to a component relationship graph corresponding to an input image in a component relationship graph, and finally obtain the image segmentation result pixel by pixel; the matcher is composed of a GCN module and a similarity calculation module.

5. The interpretable image segmentation method based on the graph matching strategy according to claim 4 is characterized in that: In S4, the complexity of the component relationship graph is constrained, including: averaging the component relationship graph instances of each category to initialize the component relationship graph, and removing edges connected to vertices with weights lower than a given threshold from the graph of a specific category.

6. The interpretable image segmentation method based on the graph matching strategy according to claim 5, characterized in that: In S1, the image segmentation backbone network is Deeplabv2 or U-net.

7. An interpretable image segmentation system based on a graph matching strategy, characterized in that: Used to perform the method according to any one of claims 1 to 6, comprising: A selection module is used to select an image segmentation backbone network and train the image segmentation backbone network; An extraction module, used to extract visual words with semantic information from an input image through an image segmentation backbone network; An establishment module is used to establish a feature unit-level semantic component relationship diagram and an overall category-level semantic component relationship diagram according to the visual words with semantic information, spatially; The prediction module is used to measure the similarity between the feature unit-level semantic component relationship graph and the category-level semantic component relationship graph and predict the image segmentation result.

8. A computer storage medium, characterized in that The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 6.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 6.

Images