A panorama segmentation method based on semantic-instance-panorama association constraint

Abstract

The present application relates to a kind of panorama segmentation methods based on semantic-instance-panorama association constraint, comprising: receiving the picture to be panorama segmented;The picture to be panorama segmented is input to panorama segmentation model, and panorama segmentation result is obtained;Wherein, panorama segmentation model includes: feature extraction part, for the multi-scale feature extraction of input picture, obtain semantic multi-scale feature and instance multi-scale feature;Gating coding module, for enhancing semantic multi-scale feature and instance multi-scale feature by bidirectional information interaction, obtain reinforced semantic multi-scale feature and reinforced instance multi-scale feature;Semantic head module, for predicting reinforced semantic multi-scale feature, obtain semantic prediction result;Instance head module, for predicting reinforced instance multi-scale feature, obtain instance prediction result;Fusion module, for fusing semantic prediction result and instance prediction result to obtain panorama segmentation result.The present application improves the segmentation precision and efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a panoramic segmentation method based on semantic-instance-panoramic association constraints. Background Technology

[0002] Panoptic segmentation is an important task in computer vision, aiming to assign a semantic category label to each pixel in an image and then assign a corresponding instance number to each instance class. This task combines the concepts of semantic segmentation and instance segmentation, combining their advantages to achieve a more comprehensive scene understanding. Existing panoptic segmentation models can be roughly divided into convolutional neural network (CNN)-based models and Transformer-based models. The former segments the scene by extracting multi-scale features and capturing local patterns, achieving excellent segmentation accuracy with low computational burden. The latter improves segmentation accuracy through attention mechanisms, but such models suffer from high complexity due to the use of global sequence encoding, slow inference speed, and long training time, making it difficult to meet real-world needs in terms of practicality.

[0003] Most CNN-based panoptic segmentation models perform semantic segmentation and instance segmentation independently, then fuse the results to generate a panoptic image. Few panoptic segmentation models incorporate information interaction, even though experiments have shown that appropriate semantic-instance information interaction improves the overall segmentation accuracy. Furthermore, most models that do engage in interaction only consider one-way interaction. For example, AUnet only considers interacting semantic information to the instance branch to optimize instance segmentation results, failing to fully explore the potential of two-way interaction.

[0004] In summary, current CNN-based panoptic segmentation models rarely engage in bidirectional interaction, and there are currently no segmentation models that utilize global feature information. Furthermore, panoptic segmentation, as a combination of semantic and instance segmentation, involves more and more edges, making it more challenging, and edge segmentation has always been a difficult aspect of segmentation. Currently, there are almost no models in the panoptic segmentation field that focus on optimizing edge segmentation results. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a panoramic segmentation method based on semantic-instance-panoramic association constraints, which can improve the segmentation accuracy and efficiency of panoramic segmentation.

[0006] The technical solution adopted by this invention to solve its technical problem is: to provide a panoramic segmentation method based on semantic-instance-panoramic association constraints, including the following steps:

[0007] Receive the image to be segmented into a panorama;

[0008] The image to be segmented in panorama is input into the panorama segmentation model to obtain the panorama segmentation result; wherein, the panorama segmentation model includes:

[0009] The feature extraction section is used to extract features from the input image at multiple scales, resulting in semantic multi-scale features and instance multi-scale features.

[0010] The gated coding module is used to enhance semantic multi-scale features and instance multi-scale features by merging structured, channel attention and residual structures based on the idea of ​​bidirectional information interaction, so as to obtain enhanced semantic multi-scale features and enhanced instance multi-scale features.

[0011] The semantic head module is used to predict the enhanced semantic multi-scale features and obtain the semantic prediction results;

[0012] The instance header module is used to predict the multi-scale features of enhanced instances and obtain the instance prediction results;

[0013] The fusion module is used to fuse semantic prediction results and instance prediction results to obtain panoramic segmentation results.

[0014] The gated encoding module includes:

[0015] The semantic enhancement unit is used to generate instance-aware semantic features based on semantic multi-scale features and instance multi-scale features, and then enhance the instance-aware semantic features through channel attention and residual structure to obtain enhanced semantic multi-scale features.

[0016] The instance enhancement unit is used to generate semantically aware instance features based on enhanced semantic multi-scale features and instance multi-scale features, and then to obtain enhanced instance multi-scale features from the semantically aware instance features through channel attention and residual structure.

[0017] The expression for the semantic enhancement unit is: Among them, F′ sem To enhance semantic multi-scale features, F i-sem For instance-aware semantic features, F sem For semantic multi-scale features, F ins For instance multi-scale features, σ[] represents the sigmoid activation function, BN() represents batch normalization, GAP() represents global average pooling, ReLU() represents the ReLU activation function, and Conv() represents a 3×3 convolution kernel. This indicates multiplication of the primary element. This indicates element-wise addition, and the superscript [n] indicates repetition n times.

[0018] The expression for the instance enhancement unit is: Among them, F′ insTo enhance the multi-scale features of instances, F s-ins For semantically aware instance features, F′ sem To enhance semantic multi-scale features, F ins For instance multi-scale features, σ[] represents the sigmoid activation function, BN() represents batch normalization, GAP() represents global average pooling, ReLU() represents the ReLU activation function, and Conv() represents a 3×3 convolution kernel. This indicates multiplication of the primary element. This indicates element-wise addition, and the superscript [n] indicates repetition n times.

[0019] The panoramic segmentation model employs a multi-level edge optimization method, as detailed below:

[0020] The semantic prediction results are extracted using the Laplacian operator to obtain the predicted semantic edges, and the predicted semantic edges are supervised.

[0021] Extract multiple individual instance masks of a preset size from the mask branch of the instance prediction results, then use the Laplacian operator to extract the predicted mask edges on the instance masks, and supervise the predicted mask edges.

[0022] The m highest-confidence individual instance masks are regressed to the original image size through region of interest sampling. Then, the mask edges corresponding to the m highest-confidence individual instance masks are logically ORed to obtain the predicted instance edges. The predicted instance edges are ORed with the predicted semantic edges to obtain the predicted panoramic edges. The predicted panoramic edges are then supervised.

[0023] The loss function used when supervising the predicted semantic edges is: L sem-edge =L bce (P sem ,T sem )+L f (P sem ,T sem ), where L sem-edge Let P be the semantic edge loss function. sem For the predicted semantic edges, T sem L represents the truth value of semantic margins. bce The binary cross-entropy function is expressed as: L bce = -[T*Log(P)+(1-T)*Log(1-P)], where P is the predicted value and T is the true value; L f The geometric edge loss between the predicted and ground truth values ​​is calculated by passing through a pre-trained fully connected layer.

[0024] The loss function used when supervising the predicted mask edges is: L ins-edge =Lbce (P ins ,T ins )+L f (P ins ,T ins ), where L ins-edge Let P be the instance edge loss function. ins For the predicted mask edges, T ins L is the truth value of the mask edge. bce The binary cross-entropy function is expressed as: L bce = -[T*Log(P)+(1-T)*Log(1-P)], where P is the predicted value and T is the true value; L f The geometric edge loss between the predicted and ground truth values ​​is calculated by passing through a pre-trained fully connected layer.

[0025] The loss function used when supervising the predicted panoramic edges is: L pan-edge =L bce (P pan ,T pan )+L f (P pan ,T pan ), where L pan-edge Let P be the panoramic edge loss function. pan The predicted panoramic edge is represented as: P sem For the predicted semantic edges, P ins [ i] T represents the edge of the i-th mask, Roismpling represents the region of interest sampling operation, and T pan L is the ground truth value of the panoramic edge. bce The binary cross-entropy function is expressed as: L bce = -[T*Log(P)+(1-T)*Log(1-P)], where P is the predicted value and T is the true value; L f The geometric edge loss between the predicted and ground truth values ​​is calculated by passing through a pre-trained fully connected layer.

[0026] Beneficial effects

[0027] By adopting the above-mentioned technical solution, this invention has the following advantages and positive effects compared with the prior art: This invention adds a gating encoding module between the semantic head module and the instance head module. The gating encoding module seeks the commonalities of the tasks of the semantic head module and the instance head module and retains the differences, enabling the two to cooperate and thus improve their respective expressive capabilities. In order to enhance the perception of edges, this invention proposes an edge-aware consistency loss. This loss function integrates considerations from the perspectives of panorama, semantics, and instance, while emphasizing the consistency of geometric edges. It makes full use of the fact that the edges of the semantic segmentation result and the edges of the instance segmentation result are logically ORed to equal the edges of the panorama segmentation result, thereby improving the segmentation accuracy of the panorama segmentation boundary part. Attached Figure Description

[0028] Figure 1 This is a flowchart of the panoramic segmentation method based on semantic-instance-panoramic association constraints according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the panoramic segmentation model in an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the gated coding module in an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of panoramic edge optimization in an embodiment of the present invention;

[0032] Figure 5 This is a comparison diagram of the segmentation results using the embodiments of the present invention and the segmentation results using existing technologies;

[0033] Figure 6 This is a graph showing the results of an ablation experiment. Detailed Implementation

[0034] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0035] The embodiments of the present invention relate to a panoramic segmentation method based on semantic-instance-panoramic association constraints, such as... Figure 1 As shown, it includes the following steps:

[0036] Step 1: Receive the image to be segmented into a panorama;

[0037] Step 2: Input the image to be segmented into the panoramic segmentation model to obtain the panoramic segmentation result.

[0038] The panoramic segmentation model in this embodiment first acquires multi-scale features through a backbone network and a dual-path feature pyramid. Then, the semantic head and instance head learn useful information for their respective tasks. A novel gated coding module (GE Module) is designed between these two heads. By seeking commonalities in their tasks while preserving differences, this collaboration enhances their respective expressive capabilities. Furthermore, the panoramic segmentation result is obtained through a fusion process for the semantic and instance prediction results. To enhance edge perception, a proposed edge-aware consistency loss is utilized. This loss function integrates considerations from panoramic, semantic, and instance perspectives, while emphasizing the consistency of geometric edges. It works in conjunction with the basic loss function to supervise the network training. Figure 2 As shown, the panoramic segmentation model includes:

[0039] The feature extraction section performs multi-scale feature extraction on the input image, obtaining semantic multi-scale features and instance multi-scale features. In this implementation, the backbone network of the feature extraction section employs a dual-path feature pyramid.

[0040] The gated coding module is used to enhance semantic multi-scale features and instance multi-scale features by merging structured, channel attention and residual structures based on the idea of ​​bidirectional information interaction.

[0041] While instance segmentation and semantic segmentation tasks differ, they also exhibit correlations. For example, pixels with different semantics must belong to different instances. Existing methods often suffer from inaccuracies due to neglecting this correlation. For instance, a semantic head might predict a person and a bicycle in an image, while an instance head might identify them both as one person, leading to a consistency problem of "pixels with different semantics belonging to the same instance," ultimately resulting in incorrect predictions. To mitigate this problem, this implementation proposes a gated coding module to integrate this correlation.

[0042] The gating coding module in this embodiment includes:

[0043] The semantic enhancement unit is used to generate instance-aware semantic features based on semantic multi-scale features and instance multi-scale features, and then enhance the instance-aware semantic features through channel attention and residual structure to obtain enhanced semantic multi-scale features.

[0044] The instance enhancement unit is used to generate semantically aware instance features based on enhanced semantic multi-scale features and instance multi-scale features, and then to obtain enhanced instance multi-scale features from the semantically aware instance features through channel attention and residual structure.

[0045] F sem For semantic multi-scale features, Fins This involves multi-scale features for instances. The instance header module identifies instances belonging to the "object" category, which are relatively fine-grained, localized, and structured. This implementation first attempts to enhance semantic information by incorporating structured "object" information, generating instance-aware semantic features F. i-sem Then, enhanced semantic multi-scale features F are obtained through channel attention and residual structure. s ′ em This avoids adding too much computational burden to the model. The process can be expressed as:

[0046]

[0047]

[0048] ConvNormAct(x)=ReLu(BN(Conv(x)))

[0049] Where Conv() represents a 3×3 convolution kernel, σ[] represents the sigmoid activation function, BN() represents batch normalization, GAP() represents global average pooling, and ReLU() represents the ReLU activation function. This indicates multiplication of the primary element. The superscript [3] indicates that the elements are added one by one, and the superscript [3] indicates that it is repeated 3 times.

[0050] Next, we use the enhanced semantic multi-scale feature F′ sem Optimizing instance features is a process similar to semantic enhancement, and can be expressed as follows:

[0051]

[0052]

[0053] Among them, F′ ins To enhance the multi-scale features of instances, F s-ins These are instance features for semantic awareness. The overall structure diagram of the gating coding module is as follows: Figure 4 As shown. Through the semantic multi-scale feature F in the gated coding module... sem and instance multi-scale features F ins The two-way information interaction and mutual optimization between the two enable a tight connection between semantic information and instance information, allowing the model to more comprehensively understand the semantic structure and instance distribution of the image, thereby improving the performance of panoptic segmentation.

[0054] The semantic head module is used to enhance the semantic multi-scale features F′. sem Make predictions to obtain semantic prediction results.

[0055] The instance header module is used to enhance the multi-scale features F′ of instances. ins Make predictions and obtain the instance prediction results.

[0056] The fusion module is used to fuse semantic prediction results and instance prediction results to obtain panoramic segmentation results.

[0057] The segmentation quality at the boundary regions of the segmentation domain has consistently been less than ideal. Panoramic segmentation tasks require more accurate segmentation of numerous boundaries, significantly increasing the difficulty. Currently, there are few optimization methods for panoramic segmentation boundaries. This implementation optimizes the boundaries from three perspectives: semantics, instance, and panorama. Ultimately, this invention successfully improves the segmentation accuracy of the panoramic segmentation boundary regions.

[0058] Edge optimization is mainly achieved through the most basic binary cross-entropy function L. bce Supervision is required, but there are too many edges to segment in the panoptic segmentation task, so using only L... bce Edge optimization cannot be achieved in panoramic tasks, and as... Figure 5 As shown, L bce It is impossible to determine the difference in geometric transformation between edges. This implementation incorporates the Inverseformer loss L. f It assists in edge supervision to optimize the segmentation boundary. f The geometric edge loss between predicted edges and ground truth values ​​is calculated using a pre-trained fully connected layer. The loss function L... bce The expression is shown below, where P is the predicted value and T is the true value, usually 0 or 1.

[0059] L bce = -[T*Log(P)+(1-T)*Log(1-P)]

[0060] Loss function L f The expression is shown below, where FC refers to a pre-trained fully connected layer. In this embodiment, the fully connected layer can calculate the geometric transformation difference between two binary edge maps.

[0061] L f =FC(P,T)

[0062] When the panoramic segmentation model in this embodiment adopts a multi-level edge optimization method, it specifically includes the following steps:

[0063] The first step is to optimize semantic edges. This involves extracting the predicted semantic edges using the Laplacian operator and then supervising the process.

[0064] L sem-edge =L bce (P sem ,Tsem )+L f (P sem ,T sem )

[0065] Among them, L sem-edge Let P be the semantic edge loss function. sem For the predicted semantic edges, T sem This represents the truth value of the semantic edge.

[0066] The second step is to optimize instance edges. First, multiple individual instance masks of size 28×28 are extracted from the mask branch of the instance prediction results. Then, the Laplacian operator is used to extract the predicted mask edges on the instance masks. The predicted mask edges are supervised. The process is as follows:

[0067] L ins-edge =L bce (P ins ,T ins )+L f (P ins ,T ins )

[0068] Among them, P ins For the predicted mask edges, T ins This is the truth value for the mask edge.

[0069] The third step optimizes the panoramic edge. Based on the relationship between semantics, instances, and panoramic segmentation, the inventors of this invention discovered that the merged edge output after a logical OR operation between semantic edges and instance edges is consistent with the panoramic edge. Based on this theory, a correlation constraint loss function is proposed among the three. Specifically, the following steps are implemented: First, the m instances with the highest confidence from the 28×28 individual instance masks in the instance header's mask branch are regressed to the original image size through region of interest sampling. Then, the mask edges corresponding to the m instances with the highest confidence are logically ORed to obtain the predicted instance edges. Subsequently, the predicted instance edges are ORed with the predicted semantic edges to obtain the predicted panoramic edge. Finally, the panoramic edge ground truth is used for supervision. The process is as follows:

[0070]

[0071] L pan-edge =L bce (P pan ,T pan )+L f (P pan ,T pan )

[0072] Among them, L pan-edge Let P be the panoramic edge loss function. panFor the predicted panoramic edges, Roismpling represents the region of interest sampling operation.

[0073] Through the semantic-instance-panoramic three-angle edge optimization in this embodiment, the edge segmentation quality has been significantly improved, which is reflected in both quantitative analysis and visualization results.

[0074] like Figure 5 and Figure 6 As shown, the results obtained using this embodiment achieve the highest accuracy among panoptic segmentation models based on convolutional neural networks, with a PQ of 64%, comparable to many Transformer-based panoptic segmentation models. Furthermore, the segmentation speed of this embodiment is significantly faster than Transformer-based panoptic segmentation models, exceeding the representative Transformer method CMT by 25%. This embodiment also underwent quite detailed ablation experiments, which verified its effectiveness and demonstrated that the invention combines excellent segmentation accuracy and efficiency.

Claims

1. A panoramic segmentation method based on semantic-instance-panoramic association constraints, characterized in that, Includes the following steps: Receive the image to be segmented into a panorama; The image to be segmented in the panorama is input into the panorama segmentation model to obtain the panorama segmentation result; wherein, the panorama segmentation model includes: The feature extraction section is used to extract features from the input image at multiple scales, resulting in semantic multi-scale features and instance multi-scale features. A gated coding module is used to enhance semantic multi-scale features and instance multi-scale features based on the idea of ​​bidirectional information interaction by merging structured, channel attention, and residual structures to obtain enhanced semantic multi-scale features and enhanced instance multi-scale features; the gated coding module includes: The semantic enhancement unit is used to generate instance-aware semantic features based on semantic multi-scale features and instance multi-scale features, and then enhance the instance-aware semantic features through channel attention and residual structure to obtain enhanced semantic multi-scale features. An instance enhancement unit is used to generate semantically aware instance features based on enhanced semantic multi-scale features and instance multi-scale features, and then obtain enhanced instance multi-scale features from the semantically aware instance features through channel attention and residual structure; the expression of the semantic enhancement unit is: The expression for the instance reinforcement unit is: ,in, To enhance semantic multi-scale features, To enhance the multi-scale features of instances, Semantic features that are perceptible to instances. These are semantically aware instance features. For semantic multi-scale features, For instance multi-scale features, This represents the sigmoid activation function. This indicates a batch normalization operation. This indicates a global average pooling operation. Represents the ReLU activation function. This represents a 3×3 convolution kernel. This indicates multiplication of the primary element. Indicates element-wise addition, superscript This indicates that the number of repetitions is n. The semantic head module is used to predict the enhanced semantic multi-scale features and obtain the semantic prediction results; The instance header module is used to predict the multi-scale features of enhanced instances and obtain the instance prediction results; The fusion module is used to fuse semantic prediction results and instance prediction results to obtain panoramic segmentation results.

2. The panoramic segmentation method based on semantic-instance-panoramic association constraints according to claim 1, characterized in that, The panoramic segmentation model employs a multi-level edge optimization method, as detailed below: The semantic prediction results are extracted using the Laplacian operator to obtain the predicted semantic edges, and the predicted semantic edges are supervised. Extract multiple individual instance masks of a preset size from the mask branch of the instance prediction results, then use the Laplacian operator to extract the predicted mask edges on the instance masks, and supervise the predicted mask edges. The m highest-confidence individual instance masks are regressed to the original image size through region of interest sampling. Then, the mask edges corresponding to the m highest-confidence individual instance masks are logically ORed to obtain the predicted instance edges. The predicted instance edges are ORed with the predicted semantic edges to obtain the predicted panoramic edges. The predicted panoramic edges are then supervised.

3. The panoramic segmentation method based on semantic-instance-panoramic association constraints according to claim 2, characterized in that, The loss function used when supervising the predicted semantic edges is: ,in, For semantic edge loss function, For predicted semantic edges, For the truth value of semantic margins, The binary cross-entropy function is expressed as: , For predicted values, It is true; The geometric edge loss between the predicted and ground truth values ​​is calculated by passing through a pre-trained fully connected layer.

4. The panoramic segmentation method based on semantic-instance-panoramic association constraints according to claim 2, characterized in that, The loss function used when supervising the predicted mask edges is: ,in, For instance edge loss function, For the predicted mask edges, The truth value for the mask edge. The binary cross-entropy function is expressed as: , For predicted values, It is true; The geometric edge loss between the predicted and ground truth values ​​is calculated by passing through a pre-trained fully connected layer.

5. The panoramic segmentation method based on semantic-instance-panoramic association constraints according to claim 2, characterized in that, The loss function used when supervising the predicted panoramic edges is: ,in, For panoramic edge loss function, The predicted panoramic edge is represented as: , For predicted semantic edges, This represents the edge of the i-th mask. This indicates a region of interest sampling operation. For the true value of the panoramic edge, The binary cross-entropy function is expressed as: , For predicted values, It is true; The geometric edge loss between the predicted and ground truth values ​​is calculated by passing through a pre-trained fully connected layer.

Images