An image segmentation method based on sparse annotation with pixel-pair relationship modeling

Abstract

This invention discloses an image segmentation method based on pixel-pair relationship modeling under sparse annotation. The method includes the following steps: First, a training dataset is constructed, comprising images with sparse annotations. Next, a weakly supervised image segmentation model is established, consisting of a basic image segmentation network and a pixel-pair relationship propagation module. Then, the basic image segmentation network is trained end-to-end using high-quality pseudo-labels generated by the pixel-pair relationship propagation module, resulting in a well-trained basic image segmentation network under sparse annotation. Finally, a test image from a natural scene is input into the trained basic image segmentation network to obtain the target category and pixel-level mask in the test image. This method can be seamlessly embedded into existing image segmentation network frameworks, achieving accurate pseudo-label generation under sparse or unannotated conditions, thereby significantly reducing the annotation cost of image segmentation data.

Description

Technical Field

[0001] This invention relates to an image segmentation method under sparse annotation in the fields of computer vision, image processing and analysis, specifically to an image segmentation method based on pixel pair relationship modeling under sparse annotation with global and local propagation. Background Technology

[0002] Image segmentation is a widely studied problem in computer vision, aiming to generate mask predictions for a given image, such as grouping each pixel into object instances (instance segmentation) or assigning a class label to each pixel (semantic segmentation). While encouraging performance has been achieved, most existing methods are trained in a fully supervised manner, heavily relying on pixel-level mask annotations and incurring cumbersome annotation costs. Weakly supervised image segmentation techniques employ efficiently labeled sparse annotations, such as dots, doodles, bounding boxes, and image-level labels, to reduce the reliance on dense pixel-level labels. These methods make dense segmentation of new categories or scene types more feasible with less annotation cost.

[0003] Most existing weakly supervised segmentation methods employ local appearance kernel functions to model adjacent pairwise affinity relationships, where pixels in similar color spaces (such as LAB and RGB color spaces) are likely to belong to the same category. While proven effective, these methods suffer from two main limitations. First, local operations fail to capture global contextual cues and long-distance affinity dependencies. Second, appearance kernel functions cannot capture the intrinsic topology of objects and lack the ability to preserve details.

[0004] One of the existing techniques, as described in the paper "Learning pixel-level semanticaffinity with image-level supervision for weakly supervised semanticsegmentation" by Ahn et al., involves constructing a small-radius kernel function to model local relationships and generate preliminary predictions. A probabilistic random walk method is used to construct long-distance pixel relationships. However, this technique cannot model fine-grained pixel pair relationships, resulting in inaccurate pseudo-labels.

[0005] The second existing technology, Liang et al., in their paper "Tree energy loss: Towards sparsely annotated semantic segmentation", uses a global minimum spanning tree to construct the topological structure between image pixels under sparse annotation. However, its affinity function for the affinity interaction between distant pixel pairs will rapidly decay as the path increases. This technology still focuses on a small local range and cannot achieve the modeling of global image relationships.

[0006] The third existing technology, Song et al., in their paper "Rethinking learnable tree filter for generic feature transform", uses Markov random fields to establish pixel pair relationships with distant pixel pairs. However, the predictions they construct are not dense and often focus on the sparse semantic response of the image, resulting in most of the generated pseudo-labels being inaccurate and unable to effectively achieve weakly supervised image segmentation tasks under sparse annotation. Summary of the Invention

[0007] To effectively model global long-distance relationships and obtain the topological structure of targets, and to address the problem of inaccurate pseudo-labels in existing methods under sparse annotation, this invention constructs a general and efficient image segmentation method under annotation. This method can generate high-quality pseudo-labels for images under sparse annotation, thereby improving image segmentation performance under weak supervision and significantly reducing the annotation cost required for image segmentation.

[0008] The technical solution adopted in this invention is as follows:

[0009] I. An Image Segmentation Method Based on Pixel Pair Relationship Modeling under Sparse Labeling

[0010] S1: Construct the training dataset, which includes images with sparse annotations;

[0011] S2: Establish a weakly supervised image segmentation model, which consists of a basic image segmentation network and a pixel pair relationship propagation module connected together;

[0012] S3: Use the training dataset to train the weakly supervised image segmentation model end-to-end to obtain a well-trained weakly supervised image segmentation model, thereby obtaining a well-trained basic image segmentation network.

[0013] S4: Input the test image from the natural scene into the trained base image segmentation network to obtain the category of the target in the test image and the pixel-level mask.

[0014] In step S2, the pixel pair relationship propagation module performs affinity propagation on the output of the basic image segmentation network to obtain fine-grained pseudo-labels.

[0015] In S2, affinity propagation includes global affinity propagation and local affinity propagation. The basic image segmentation network is supervised based on the fine-grained pseudo-labels obtained from global affinity propagation and local affinity propagation. The specific formula is as follows:

[0016]

[0017]

[0018]

[0019] in, ψ(x) represents the sparse initial prediction results output by the basic image segmentation network. i ,x j ) represents pixel x i to pixel x j Global affinity propagation or local affinity propagation; τ represents the receptive field; z i For global affinity propagation or local affinity propagation ψ(x) i ,x j In pixel x j Summation of y i These are pseudo-labels generated after global affinity propagation or local affinity propagation; ψ g (x i ,x j ) and ψ s (x i ,x j (x) represents pixels. i to pixel x j Global affinity propagation or local affinity propagation; w k,l E represents the edge weight between adjacent nodes k and l. i,j To represent the tree from node j to node i The set of edges in the path above; I i and I j , i and j are the RGB pixel values ​​at positions i and j, respectively; || represents the absolute value; ζ g With ζ s These are the global similarity parameter and the local similarity parameter, respectively.

[0020] The global affinity propagation process utilizes a delayed update propagation method for efficient global tree structure relation calculation, specifically including the following steps:

[0021] Each pixel in the output of the basic image segmentation network is used as a node in the global minimum spanning tree, and each node in the global minimum spanning tree is denoted as a different set U. Then, the nodes are sorted in ascending order according to the weight w between them. Two nodes are connected in sequence, and the sets containing the current two nodes are merged into a new set. In the new set, only the value of its root node is updated, and the update information is retained in the delay label. The new set is merged with the set containing the next node in sequence until all nodes are merged into a single set, thus obtaining the final tree. The specific update formula for the root node value in the new set is as follows:

[0022]

[0023]

[0024] in, Represents set U k The value of the merged root node, Represents set U k The value of the root node before merging, S(δ) l Represents set U k The sum or set U of the predicted values ​​corresponding to all nodes in the set. k Summation of the quantities of all points; δ j U represents the initial network prediction value or 1 for each node j. k .rank represents set U k The height of the middle tree, U l .rank represents set U l The height of the middle tree, Represents set U l The delayed label of the root node;

[0025] Once all sets U are connected, based on the delay label Propagating down from the root node of the final tree to its derived nodes, the relationship update for each node is shown in the following formula:

[0026]

[0027] Wherein, LProp(δ) i This represents the weighted sum of the propagation values ​​of the remaining nodes on node i. Indicates the ancestor node of node i. This represents the delayed update label for node r.

[0028] In the training of S3, the basic image segmentation network is trained end-to-end based on the generated high-quality pseudo-labels.

[0029] In the training of S3, the corresponding loss value is optimized and gradient backpropagation is performed. Finally, the parameters are updated to achieve end-to-end training. The training is iterated until the network converges, and finally a weakly supervised image segmentation model file under sparse annotation is obtained.

[0030] In step S4, a test image of the natural scene to be input is first loaded and used as input to the trained weakly supervised image segmentation model. The forward propagation process of the basic image segmentation network is then calculated to obtain the object category and pixel-level mask prediction in the image. Note that the pseudo-label generation module is only used during the training of the basic segmentation model and is not required during inference.

[0031] II. A computer device

[0032] The computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the method.

[0033] III. A computer-readable storage medium

[0034] A computer program is stored on a computer-readable storage medium, characterized in that the computer program, when executed by a processor, implements the steps of the method.

[0035] The beneficial effects of this invention are:

[0036] By adopting the above technical solutions, the method of the present invention enables end-to-end training of image segmentation models under sparse annotation, which can significantly save the annotation cost in the field of image segmentation and solve the problem of large workload of image segmentation annotation data.

[0037] The delayed update method in this invention enables efficient pseudo-label calculation, which can save the training cost of training models and accelerate the application of image segmentation in new scenarios.

[0038] The method of this invention can be seamlessly embedded into existing image segmentation network frameworks. Based on images acquired by image sensors, it focuses on pixel-level recognition and classification of targets to achieve instance segmentation, semantic segmentation, etc. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating an efficient image segmentation method based on pixel-pair relationship propagation, according to an embodiment of the present invention.

[0040] Figure 2 This is a weakly supervised image segmentation model under end-to-end sparse labeling in an embodiment of the present invention.

[0041] Figure 3This is a task scenario framework diagram under sparse annotation in an embodiment of the present invention.

[0042] Figure 4 This is a pixel pair relationship diagram of global affinity propagation and local affinity propagation when testing natural scene images according to an embodiment of the present invention.

[0043] Figure 5 This is a visualization of the final image segmentation result.

[0044] Figure 6 This is an example image of a test image segmented using the method of this invention. Detailed Implementation

[0045] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0046] like Figure 1 As shown, the embodiments and implementation process of the present invention are as follows:

[0047] S1: Construct the training dataset, which includes images with sparse annotations. These images can be from general datasets containing sparse annotations, images with sparse annotations generated from existing images with pixel mask annotations, or images with manually sparse annotations of natural scenes by annotators. This collection of image data collectively constitutes the training data. Sparse annotation refers to sparse pixel annotations in an image, such as points, lines, and boxes. Compared to full mask annotation, sparse annotation significantly reduces annotation time and cost.

[0048] S1-1: In this embodiment, two general datasets, Pascal VOC and COCO, are selected. Both datasets have sparsely labeled data, that is, only a small number of pixels of specific objects on the image are labeled, including sparse labeling information such as points, boxes, and graffiti.

[0049] S1-2: Construct corresponding training datasets from different types of image sets.

[0050] S2: Establish a weakly supervised image segmentation model with sparse labels, such as Figure 2 As shown, the weakly supervised image segmentation model consists of a basic image segmentation network and a pixel pair relation propagation module connected together;

[0051] First, the basic image segmentation network extracts features and makes initial predictions on the input image, obtaining sparse initial prediction results. Next, for regions with sparse pixel labeling, supervised learning is performed using the corresponding cross-entropy loss function; in this example, L1 distance is used. For generating pseudo-labels for unlabeled regions, the image and initial predictions are input into the pixel pair propagation module to calculate the global affinity propagation process and the local affinity propagation process. An efficient delayed update propagation algorithm accelerates the calculation of the global affinity propagation process, thereby generating high-quality pseudo-labels for unlabeled regions. These high-quality pseudo-labels are then used as supervisory information for unsupervised regions, enabling end-to-end network parameter training.

[0052] This invention designs a weakly supervised image segmentation model, which includes a basic image segmentation network and a pixel pair relation propagation module. Then, a pseudo-label is generated by an efficient delay propagation algorithm to supervise the basic image segmentation network, thus forming an end-to-end efficient labeled image segmentation framework.

[0053] The basic image segmentation network can be an instance segmentation network model such as SOLOv2 or a semantic segmentation network model LTF.

[0054] like Figure 3 As shown, global affinity propagation and local affinity propagation together constitute affinity propagation. The fine-grained pseudo-labels after global affinity propagation and local affinity propagation are used together to supervise the basic image segmentation network.

[0055] For global affinity propagation, the input image is represented as a 4-connected planar graph G, where each node is adjacent to 4 neighboring nodes, and the edge weights measure the pixel distance between adjacent nodes. The minimum spanning tree algorithm (MST) is used to remove edges with large distances, thus obtaining a tree-based sparse graph of pixel relationships. Global pairwise potential is modeled by iterating through each node. Specifically, the current node is taken as the root node of the spanning tree, and long-distance affinity is propagated to other nodes. The distance-insensitive max affinity function is used to alleviate the geometric constraints on affinity decay between long-distance nodes, thereby relaxing the affinity decay of long-distance nodes.

[0056] For local affinity propagation, a defined local Gaussian kernel function is used to calculate the similarity between color and intensity in local regions, thus maintaining local spatial consistency.

[0057] The specific formula is as follows:

[0058]

[0059]

[0060]

[0061] in, ψ(x) represents the unary term, i.e., the sparse initial prediction result output by the basic image segmentation network; i ,x j ) represents a binary term used to model pairwise relationships between pixels, i.e., pixel x i to pixel x j Global affinity propagation or local affinity propagation; τ represents different receptive fields; z i For global affinity propagation or local affinity propagation ψ(x) i ,x j In pixel x j Summation over y is used for regularization. i These are pseudo-labels generated after global affinity propagation or local affinity propagation; ψ g (x i ,x j ) and ψ s (x i ,x j (x) represents pixels. i to pixel x j Global affinity propagation or local affinity propagation, K(I) represents the global minimum spanning tree. i ,I j ) represents a local Gaussian kernel. It is a set containing locally adjacent pixels. w represents all nodes in the global minimum spanning tree; k,l The edge weight between adjacent nodes k and l is specifically the Euclidean distance between the pixel values ​​of the two adjacent nodes, i.e., w. k,l =|I k -I l | 2 I k and I l These are the RGB pixel values ​​of neighboring nodes k and l, respectively; E i,j To represent the tree from node j to node i The set of edges in the path above; I i and I j , where i and j are the RGB pixel values ​​at node i and node j respectively; || is for calculating their absolute values; ζ g With ζ s These are the global similarity parameter and the local similarity parameter, respectively.

[0062] In the global affinity propagation process, the delayed update propagation method is used to efficiently calculate the relationships of the global tree structure, specifically including the following steps:

[0063] Each pixel in the output of the basic image segmentation network is used as a node in the global minimum spanning tree. A disjoint-set data structure is introduced, and each node in the global minimum spanning tree is denoted as a different set U. Then, the nodes are sorted in ascending order according to the weight w between them. Two nodes are connected sequentially, and the sets containing the current two nodes are merged into a new set. In the new set, only the value of the root node is updated, and the values ​​of the derived nodes of the root node are not updated. The updated information is retained in the delay label. The new set is merged with the set containing the next node in sequence until all nodes are merged into a single set, thus obtaining the final tree. The specific update formula for the root node value in the new set is as follows:

[0064]

[0065]

[0066] in, Represents set U k The value of the merged root node, Represents set U k The value of the root node before merging, S(δ) l Represents set U k The sum or set U of the predicted values ​​corresponding to all nodes in the set. k Summation of the quantities of all points; δ j U represents the initial network prediction value for each node j, or 1 (i.e., the number of nodes j is 1). k .rank represents set U k The height of the middle tree, U l .rannk represents set U l The height of the middle tree, Represents set U l The delayed label of the root node;

[0067] Once all sets U are connected, based on the delay label Propagating down from the root node of the final tree to its derived nodes, the relationship update for each node is shown in the following formula:

[0068]

[0069] Wherein, LProp(δ) i This represents the weighted sum of the propagation values ​​of the remaining nodes on node i. Indicates the ancestor node of node i. This represents the delayed update label for node r.

[0070] Therefore, the global affinity propagation of this invention uses minimum spanning tree for modeling, effectively capturing long-distance relationships and object topology, and efficiently generating high-quality pseudo-labels, thereby significantly reducing the computational cost in the global affinity propagation method and ensuring computational efficiency.

[0071] S3: Use the training dataset to train the weakly supervised image segmentation model end-to-end to obtain a well-trained weakly supervised image segmentation model, thereby obtaining a well-trained basic image segmentation network.

[0072] S3-1: Randomly select an image from the training set as the original input image;

[0073] S3-2: The original input image is scaled to 800×800 resolution and then fed into the basic image segmentation network to obtain the network's initial prediction. Figure 4 It is a basic image segmentation network model for different tasks.

[0074] S3-3: The original image and the initial predictions from the network are processed by the pseudo-label generation module to obtain high-quality pseudo-labels.

[0075] S3-4: Then calculate the corresponding loss to optimize the base image segmentation network by backpropagating the gradient, and finally update the parameters to achieve end-to-end training.

[0076] S3-5: Repeat the above steps until the network converges, and finally obtain the image segmentation model file under sparse annotation.

[0077] S4: Input the test image from the natural scene into the trained base image segmentation network. After the model performs forward inference, it obtains the category of the target in the test image and the pixel-level mask.

[0078] S4-1: Load the final weakly supervised image segmentation model;

[0079] S4-2: Images acquired through an image acquisition system in a natural scene are input into the model;

[0080] S4-3: The basic image segmentation network extracts image features and generates pixel-level mask predictions for the target of interest;

[0081] S4-4: Output the obtained segmentation results to the user display system.

[0082] Figure 5 Examples of testing different natural scene images demonstrate that the relation propagation method in this invention can effectively establish global and local affinity relationships between pixels and generate high-quality pseudo-labels. Figure 6 Here are some examples of images for testing.

[0083] This invention proposes a general and efficient image segmentation method under annotation, which effectively achieves global long-distance relationship modeling and obtains the topological structure of the target. It can generate high-quality pseudo-labels for images with sparse annotation, thereby improving image segmentation performance under weak supervision and significantly reducing the annotation cost required for image segmentation. This invention will further facilitate the practical application of low-annotation-cost image segmentation methods.

[0084] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for image segmentation under sparse annotation based on modeling pixel pair relationship, characterized in that, Includes the following steps: S1: Construct the training dataset, which includes images with sparse annotations; S2: Establish a weakly supervised image segmentation model, which consists of a basic image segmentation network and a pixel pair relationship propagation module connected together; The pixel-to-relationship propagation module performs affinity propagation on the output of the basic image segmentation network to obtain fine-grained pseudo-labels. Affinity propagation includes global affinity propagation and local affinity propagation. The fine-grained pseudo-labels obtained from both global and local affinity propagation are used to supervise the basic image segmentation network. The specific formula is as follows: in, This represents the sparse initial prediction results output by the basic image segmentation network; Represents pixels To pixel Global affinity propagation or local affinity propagation; Indicates the feeling field; Is it global affinity propagation or local affinity propagation? In pixels Summation on, These are pseudo-labels generated after global affinity propagation or local affinity propagation. and Pixels To pixel Global affinity propagation or local affinity propagation; Adjacent nodes and Edge weights between them; To represent the node To the node tree The set of edges in the path above; and They are respectively at the node With nodes The RGB pixel value of the location; This indicates the absolute value; and These are the global similarity parameter and the local similarity parameter, respectively. S3: Use the training dataset to train the weakly supervised image segmentation model end-to-end to obtain a trained weakly supervised image segmentation model, thereby obtaining a trained basic image segmentation network. S4: Input the test image from the natural scene into the trained base image segmentation network to obtain the category of the target in the test image and the pixel-level mask.

2. The image segmentation method based on pixel-pair relationship modeling under sparse annotation according to claim 1, characterized in that, The global affinity propagation process utilizes a delayed update propagation method for efficient global tree structure relation calculation, specifically including the following steps: Each pixel in the output of the basic image segmentation network is used as a node in the global minimum spanning tree, and each node in the global minimum spanning tree is denoted as a different set. Then, according to the weights between each node... Sort all nodes in ascending order, connect two nodes in sequence, and merge the sets containing the two nodes into a new set. In the new set, only update the value of the root node, and retain the update information in the deferred label. The new set is merged with the set containing the next node in sequence until all nodes are merged into a single set, thus obtaining the final tree. The specific update formula for the root node value in the new set is as follows: in, Represents a set The value of the merged root node, Represents a set The value of the root node before the merge. Represents a set The sum or set of the predicted values ​​for all nodes in the set. Summation of the quantities of all points; Represents each node The initial network prediction value or 1, Represents a set The height of the middle tree, Represents a set The height of the middle tree, Represents a set The delayed label of the root node; When all sets After the connection is established, based on the delay tag The process propagates downwards from the root node of the final tree to its derived nodes, with the relationship update for each node as shown in the following formula: in, This indicates that the remaining nodes are in the node Weighted sum of propagation values ​​on the surface, Represents a node Ancestor node, Represents a node Delayed update tags.

3. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 2.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 2.

Images