A single-stage method for end-to-end panoramic narrative segmentation

Abstract

A single-stage method of end-to-end panoramic narrative segmentation relates to panoramic narrative segmentation. It comprises the following steps: 1) multi-scale extraction and fusion of image features by using a convolutional neural network; extraction of each word vector in the text by using a transformer-based bidirectional encoder representation technique, and filtering of the word vectors of non-noun parts by using dataset annotations, and for the case that multiple word vectors correspond to one noun, an average value method is used as the feature corresponding to the current noun; 2) embedding local spatial prior into attention modeling to enhance visual features of different scales, thereby improving the segmentation effect; 3) using the text features extracted in step 2) as a convolution kernel to perform convolution on the obtained feature map to obtain a mask M; using a bidirectional contrastive learning loss to standardize the semantic consistency between modalities; and adding other loss functions to make the value and distribution of the predicted mask close to the real mask.

Description

Technical Field

[0001] This invention relates to panoramic narrative segmentation, and more particularly to an end-to-end single-stage method for panoramic narrative segmentation that incorporates local spatial prior embedding into attention modeling and bidirectional contrastive learning. Background Technology

[0002] Panoramic narrative segmentation [1] is a new and challenging task that combines panoramic segmentation and directional target segmentation in multimodal tasks. It locates image target instances corresponding to text descriptions using binary pixel masks. The difficulty of this task lies not only in understanding multimodal information, but also in aligning the same semantic features of language and vision according to the many-to-many relationship in language and vision, and matching all relevant instances and background region names mentioned in the text description with their corresponding regions in the image; making it different from directional target segmentation tasks [4-7].

[0003] The goal of panoptic segmentation [8] is to fully understand the scene containing objects and background. Its task is to assign semantic labels and instance IDs to each pixel in the image to generate a global, uniform segmentation image. Here, semantic labels refer to the category of objects, while instance IDs correspond to different numbers of objects of the same category. Early methods

[16] regarded it as a combination of object mask and background mask. Later, it was hoped that a unified framework could be used to segment objects and background at the same time, such as the methods PFCN

[17] , K-Net

[18] and Panoptic SegFormer

[19] , which achieved significant performance. Inspired by these methods, the model of this invention uses the features of nouns in the text as convolution kernels and obtains the corresponding mask by convolution of multimodal features.

[0004] Directional object segmentation (RES) is a common task in the multimodal domain, segmenting the corresponding object based on the understanding of relevant phrases. Previous models [9-11] obtain a set of candidate regions through general segmentation methods and select the candidate region with the highest matching degree to the given phrase description. However, with the increasing use of visual information, these models are limited by the performance of segmentation methods. On this basis, a method of refining the segmentation mask with a single-level network has been proposed [12-15]. This increases the possibility of misclassifying incorrect segmentation as correct segmentation. In summary, RES is an incomplete task, ignoring background factors and the many-to-many relationship between natural language and images. In addition, the many-to-many relationship between objects and background should also be considered in panoramic narrative segmentation.

[0005] Reference [1] proposes a two-stage method for panoramic narrative segmentation (PNG). First, a set of candidate masks for a given image is provided using a pre-trained panoramic segmentation model, such as PFPN [2]. Second, these masks are further transformed into convolutional features, which are then matched with text features through cross-modal semantic matching. Finally, the candidate mask with the highest similarity is selected by ranking similarity from high to low. With the help of the panoramic segmentation model, this two-stage solution defines panoramic narrative segmentation as a mask-text matching problem, which greatly reduces the difficulty of prediction. However, this solution still has two limitations. On the one hand, this two-stage method requires offline feature extraction, storage and alignment, which is very time-consuming. Moreover, this limitation poses a huge obstacle to real-time applications such as text-based image retrieval and video matting. On the other hand, the pre-trained panoramic segmentation model requires a large number of mask annotations, which adds a greater burden to the already expensive panoramic narrative segmentation (PNG). More importantly, the performance of these panoramic segmentation models also limits the performance of panoramic narrative segmentation (PNG) models. To solve the above problems, designing an effective end-to-end single-stage network from scratch is a suitable approach. However, this solution also faces two crucial challenges. First, in panoptic narrative segmentation (PNG), each pixel can belong to a different mask, which is very different from panoptic segmentation [2]. This makes the model need to capture the semantics of the image from a macroscopic to a microscopic view. However, existing methods only focus on global modeling and ignore local information, resulting in limited performance. Second, panoptic narrative segmentation (PNG) involves more complex relationships than detection or segmentation tasks [3]. In a descriptive paragraph, multiple nouns in a sentence may correspond to the same mask, or a noun may correspond to multiple masks. This situation further increases the difficulty of visual-language alignment. To solve the above problems, this invention proposes a novel end-to-end panoptic narrative grounded network (EPNG) for real-time panoptic narrative segmentation. The model introduces spatial prior-based position-aware attention (LPA) to enhance image features through neighborhood interaction. To ensure the semantic consistency of language and visual many-to-many relationships in panoptic narrative segmentation (PNG), a new bidirectional semantic alignment loss (SAL) is designed, which uses features of one modality as a tool to eliminate feature bias of corresponding similar semantics in another modality. These innovative designs enable the End-to-End Panoramic Narrative Segmentation Network (EPNG) to maintain real-time reasoning while also offering advantages in cross-modal reasoning.

[0006] References:

[0007] [1].González,C.;Ayobi,N.;Hernández,I.;Hernández,J.;PontTuset,J.;andArbeláez,P.2021.Panoptic Narrative Grounding.In Proceedings of the IEEE / CVFInternational Conference on Computer Vision,1364–1373.

[0008] [2].Kirillov,A.;Girshick,R.;He,K.;and Dollár,P.2019a.Panoptic featurepyramid networks.In Proceedings of the IEEE / CVF conference on computer visionand pattern recognition,6399–6408.

[0009] [3].Liu,J.;Wang,L.;and Y ang,M.-H.2017.Referring expressiongeneration and comprehension via attributes.In Proceedings of the IEEEInternational Conference on Computer Vision,4856–4864.

[0010] [4].Hu,R.;Rohrbach,M.;and Darrell,T.2016.Segmentation from naturallanguageexpressions.In European Conference on Computer Vision,108–124.Springer.

[0011] [5].Y u,L.;Lin,Z.;Shen,X.;Y ang,J.;Lu,X.;Bansal,M.;and Berg,T.L.2018.Mattnet:Modular attention network for referring expressioncomprehension.In Proceedings of the IEEE Conference on Computer Vision andPattern Recognition,1307–1315.

[0012] [6].Y u,Z.;Y u,J.;Cui,Y.;Tao,D.;and Tian,Q.2019.Deep modular co-attention networks for visual question answering.In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition,6281–6290.

[0013] [7].Shi,H.;Li,H.;Meng,F.;and Wu,Q.2018.Key-word-aware network forreferring expression image segmentation.In Proceedings of the EuropeanConference on Computer Vision(ECCV),38–54.

[0014] [8].Kirillov,A.;He,K.;Girshick,R.;Rother,C.;and Dollár,P.2019b.Panoptic segmentation.In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition,9404–9413.

[0015] [9].Li,R.;Li,K.;Kuo,Y.-C.;Shu,M.;Qi,X.;Shen,X.;and Jia,J.2018.Referring image segmentation via recurrent refinement networks.InProceedings of the IEEE Conference on Computer Vision and PatternRecognition,5745–5753.

[0016]

[10] .Liu,C.;Lin,Z.;Shen,X.;Y ang,J.;Lu,X.;and Y uille,A.2017.Recurrent multimoda linteraction for referring image segmentation.InProceedings of the IEEE International Conference on Computer Vision,1271–1280.

[0017]

[11] .Margffoy-Tuay,E.;Pérez,J.C.;Botero,E.;and Arbeláez,P.2018.Dynamic multimodal instance segmentation guided by natural languagequeries.In Proceedings of the European Conference on Computer Vision(ECCV),630–645.

[0018]

[12] .Luo,G.;Zhou,Y.;Sun,X.;Cao,L.;Wu,C.;Deng,C.;and Ji,R.2020.Multi-task collaborative network for joint referring expression comprehension andsegmentation.In Proceedings of the IEEE / CVF Conference on computer vision andpattern recognition,10034–10043.

[0019]

[13] .Y e,L.;Rochan,M.;Liu,Z.;and Wang,Y.2019.Cross modal self-attention network for referring image segmentation.In Proceedings of theIEEE / CVF conference on computer vision and pattern recognition,10502–10511.

[0020]

[14] .Liu,D.;Zhang,H.;Wu,F.;and Zha,Z.-J.2019.Learning to assembleneural module tree networks for visual grounding.In Proceedings of the IEEE / CVF International Conference on Computer Vision,4673–4682.

[0021]

[15] .Zhou,Y.;Ji,R.;Luo,G.;Sun,X.;Su,J.;Ding,X.;Lin,C.-W.;and Tian,Q.2021.Areal-time global inference network for one-stage referring expressioncomprehension.IEEE Transactions on Neural Networks and Learning Systems.

[0022]

[16] .De Geus,D.;Meletis,P.;and Dubbelman,G.2018.Panoptic segmentationwith a joint semantic and instance segmentation network.arXiv preprint arXiv:1809.02110.

[0023]

[17] . Li, Y.; Zhao, H.; Qi,

[0024]

[18] . Zhang, W.; Pang, J.; Chen, K.; and Loy, CC2021. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems, 34: 10326–10338.

[0025]

[19] . Li, Z.; Wang, W.; Summary of the Invention

[0026] The purpose of this invention is to address the slow speed of traditional two-stage panoramic narrative segmentation models by utilizing the local information of the image itself to capture visual semantics from macroscopic to microscopic views, and to design a new bidirectional semantic alignment loss to provide an end-to-end single-stage method for panoramic narrative segmentation.

[0027] This invention includes the following steps:

[0028] 1) Multimodal coding: For images, convolutional networks are used to extract and fuse image features at multiple scales; for text, a transformer-based bidirectional encoder representation (BERT) model is used to extract features of each noun or noun group in the text.

[0029] 2) Multimodal interaction: Utilizing spatial prior-based position-aware attention (LPA) and combining the distance relationship of feature points, the image features are processed using their own multi-head attention mechanism; text features and image features are linearly transformed and mapped to the same dimension, and the attention mechanism is then combined between the two; this process is repeated S times.

[0030] 3) Dense prediction: Use text features as a check against the image feature map obtained in step 2) for mask prediction; then calculate Dice loss and BCE loss, and use the proposed bidirectional contrastive learning model to make different modal features with the same semantics continuously align closer together.

[0031] In step 1), the specific method of multimodal encoding can be as follows: extract each word vector in the text using a bidirectional encoder representation technique based on a transformer, filter out the word vectors of non-noun parts using dataset annotation, and use the averaging method as the feature corresponding to the current noun when multiple word vectors correspond to the same noun.

[0032] In step 2), the multimodal interaction embeds local spatial priors into attention modeling, enhancing visual features at different scales, thereby improving segmentation performance;

[0033] In step 3), the text features are used as the kernel to perform mask prediction on the image feature map obtained in step 2). This is done by using the text features extracted in step 2 as the convolution kernel and performing convolution on the obtained feature map to obtain the mask M. Then, a bidirectional contrastive learning loss is used to normalize the semantic consistency between modalities. At the same time, other loss functions are added to make the predicted mask's value and distribution close to the real mask.

[0034] The present invention has the following outstanding advantages:

[0035] 1) This invention proposes a real-time end-to-end panoramic narrative segmentation network (EPNG), which unifies multimodal alignment and mask prediction in a single forward structure, greatly reducing computational overhead.

[0036] 2) This invention proposes two novel designs: Local Perceptual Attention (LPA) and Bidirectional Semantic Alignment Loss (SAL). LPA enhances visual features at different scales to understand complex cross-modal relationships. SAL normalizes the semantic consistency problem through contrastive learning between pixels and noun phrases.

[0037] 3) On benchmark datasets, the end-to-end panoramic narrative segmentation network (EPNG) performs comparably to, or even better than, existing two-stage panoramic narrative segmentation network (PNG) methods. Its inference speed is 10 times faster than existing methods. Furthermore, it does not require additional mask annotations for pre-training. Attached Figure Description

[0038] Figure 1 This is a flowchart of a single-stage, end-to-end panoramic narrative segmentation (EPNG) method. Detailed Implementation

[0039] The following embodiments will further illustrate the present invention with reference to the accompanying drawings.

[0040] The purpose of this invention is to address the slow speed of traditional two-stage panoramic narrative segmentation models. It considers utilizing the local information within the image itself to capture visual semantics from a macroscopic to a microscopic view, and designs a novel bidirectional semantic alignment loss to provide an end-to-end single-stage method for panoramic narrative segmentation. The specific method flow is as follows: Figure 1 As shown.

[0041] Each module is as follows:

[0042] 1. Multimodal coding

[0043] Convolutional networks are used to extract and fuse image features at multiple scales, while BERT is used to extract features from nouns or noun phrases in the text. The semantic information of both is initially extracted.

[0044] F v =concat[Down(F v1 );F v2 Up(F) v3 (1)

[0045] Among them, F v1 F v2 F v3 These are features at different scales extracted from the original image pixel matrix after passing through a convolutional network; Down() and Up() represent upsampling and downsampling, respectively. Upsampling and downsampling are used to bring all three to the same resolution, and then concatenating them along the feature dimension yields the fused image feature F. v .

[0046] For a text sequence T, all word vectors F are obtained using the transformer-based bidirectional encoder representation technique (BERT) [9]. t The annotations provided in reference [1] are used to filter the word vectors of non-noun parts. For the case where a noun corresponds to multiple word vectors, average pooling is used to take the average of the multiple word vectors as the feature of the noun. Then, the dimension of each noun feature is changed by linear transformation, and finally the feature set F of N nouns is obtained. N .

[0047]

[0048] Where L is the number of nouns, C is the mapped dimension, and f nlDefined as the feature corresponding to each noun.

[0049] 2. Multimodal interaction

[0050] Based on visual features F v and text features F N A multi-mode communicator was designed for cross-modal interaction and fusion. For example... Figure 1 As shown, it consists of S consecutive identical layers, each of which consists of two modules: Local Perceptual Attention (LPA) and Cross-Attention (CA).

[0051]

[0052] Using formula (3), any two points (x, y) in the image features can be obtained. m ,y m ), (x n ,y n D between ) m,n , where D m,n When the value is greater than the set threshold, it is uniformly regarded as a predetermined value K, resulting in matrix D.

[0053] R j =W j D. (4)

[0054] Matrix D is passed through matrix W j Linear transformation yields matrix R j This is used to reweight the attention matrix for subsequent attention head calculations.

[0055]

[0056]

[0057] LPA(F i ,F i ,F i = concat(Head) 1 ,,Head h W o (7)

[0058] F i′ =LN(LPA(F i ,F i ,F i )+F i (8)

[0059] According to formula (5-8), the F-value combining LPA local perceptual attention and the original image features is obtained. i′ Among them, F i Initially, F was obtained from formula (1).v Following F i yes Figure 1 The output of the previous layer in the same S consecutive layers.

[0060] F i+1 =FFN(LN(MHA(F i′ ,F N ,F N )+F i′ (9)

[0061] According to formula (9), such as Figure 1 The diagram illustrates the interaction between different modal features, ultimately yielding a fused new feature F. i +1 Repeating the steps of formula (3-9) S times will yield the final desired feature F.

[0062] 3. Dense Forecasting

[0063] M = Up(Sigmoid(F) N *F)) (10)

[0064] As shown in formula (10), the features F corresponding to N nouns in the text are used. N As the convolution kernel, a convolution operation is performed on the new feature F obtained in step 2. Then, the result is processed with the sigmoid activation function, and finally upsampled to the required resolution to obtain the mask M. M represents the predicted result mask, where the pixel value of each point is within [0, 1], where the closer it is to 1, the greater the probability that the point belongs to a certain category. G represents the true mask.

[0065]

[0066] As shown in formula (11), the loss function L BCE In this context, M represents the prediction mask. Let y represent the value of the i-th point on the prediction mask. i This represents the value of the i-th point on the true mask. This loss function reflects the loss at each point between the predicted and true mask values, without considering the distribution area of ​​the mask points.

[0067]

[0068] As shown in formula (12), M is the predicted mask region and G is the real mask region. This loss is mainly to reflect the overlap between the predicted region and the real region.

[0069]

[0070]

[0071] As shown in formula (13), where L represents the number of filtered nouns in the sentence, G + Let τ represent the points in the real image mask where the mask value is 1, k represent all points in the image, and τ be a coefficient. The purpose of this loss is to maximize the sum of the products of the feature vectors corresponding to each noun and the feature vectors corresponding to the points in the real image mask where the value is 1, while minimizing the sum of the products of these products and the feature vectors corresponding to the points in the real image mask where the value is 0. The goal is to make the mask kernel, i.e., the feature vectors corresponding to the nouns, semantically close to the real image mask.

[0072] As shown in formula (14), G represents all points on the actual mask image, and T + This represents the set of nouns corresponding to a point in the real mask image. The goal of this step is to maximize the product of the feature vector corresponding to each point and the feature vector of the noun to which it belongs, while minimizing the product of the feature vector of the noun to which it belongs.

[0073] L=λ1L BCE +λ2L Dice +λ3L SAL (15)

[0074] As shown in formula (15), the three losses involved in formulas (11-14) are processed with different weight ratios and then summed to obtain the final loss of the method.

[0075] The specific experimental results are as follows:

[0076] This invention compares with state-of-the-art PNG methods on a panoramic narrative dataset, where Beseline(ours) represents the model with LPA and SAL removed from the EPNG model.

[0077] Table 1.

[0078]

[0079] The formula is explained below: (For the defined formula variables and symbols, please refer to the specific formula expression description.)

[0080] Formula (1) defines the result of extracting and fusing multi-scale features of an image using a convolutional network.

[0081] Formula (2) shows that features are extracted using BERT, and word vectors of non-noun parts are filtered by annotation. The features corresponding to each noun are obtained by average pooling. The feature dimension is changed by linear transformation to obtain the result of the features corresponding to each noun.

[0082] Formulas (3) and (4) define the local perception coefficient between two feature points in local perception attention.

[0083] Formula (5-8) defines the use of Local Perceptual Attention (LPA) to obtain feature results containing local perceptual information.

[0084] Formula (9) indicates that the features obtained by using the LPA module are modally interacted with the text features to obtain a feature result that combines the information of both.

[0085] Formula (10) defines the process of using text features as convolution kernels to perform convolution on the last features extracted by the multimodal interaction module to obtain the mask M corresponding to each class of nouns.

[0086] Formulas (11-15) define three types of loss, the third of which is the bidirectional semantic alignment loss proposed in this invention.

[0087] The definitions of proper nouns in English are as follows:

[0088] Average Recall is defined as the integral of the IoU-regression curve.

[0089] Inference Time is defined as the time it takes for a model to go from input data to output inference results.

[0090] Params are defined as the total number of all trainable parameters in the model, where M represents millions.

[0091] Training Data is defined as the number of labels required for model training, where each label represents a segmentation mask, and M represents millions.

Claims

1. A single-stage method for end-to-end panoramic narrative segmentation, characterized in that... Includes the following steps: 1) Multimodal coding: For images, convolutional networks are used to extract and fuse image features at multiple scales; for text, the BERT model, a bidirectional encoder representation technique based on transformers, is used to extract features of each noun or noun group in the text. 2) Multimodal interaction: Utilize location-aware attention (LPA) based on spatial priors, combined with the distance relationship of feature points, to perform its own multi-head attention mechanism on image features; map text features and image features to the same dimension through linear transformation, and combine the two to perform attention mechanism operations; repeat S times; 3) Dense prediction: Use text features as a check against the image feature map obtained in step 2) for mask prediction; then calculate Dice loss and BCE loss, and use the proposed bidirectional contrastive learning model to make different modal features with the same semantics continuously align closer together. The step of using text features as a kernel to perform mask prediction on the image feature map obtained in step 2) involves using the text features extracted in step 2) as a convolution kernel to perform convolution on the obtained feature map to obtain the mask M; then, a bidirectional contrastive learning loss is used to normalize the semantic consistency between modalities; at the same time, other loss functions are added to make the predicted mask's value and distribution close to the true mask, as detailed below: M=Up(Sigmoid(F N *F)) ((10) As shown in formula (10), the features F corresponding to N nouns in the text are used. N As the convolution kernel, a convolution operation is performed on the new feature F obtained in step 2. Then, the result is processed with the activation function Sigmoid, and finally upsampled to the required resolution to obtain the mask M. M represents the predicted result mask, and the pixel value of each point is within [0, 1]. The closer the value is to 1, the greater the probability that the point belongs to a certain category. G represents the true mask. As shown in formula (11), the loss function L BCE In this context, M represents the prediction mask. Let y represent the value of the i-th point on the prediction mask. i This represents the value of the i-th point on the true mask; the loss function reflects the loss of each point between the predicted mask value and the true mask value, without considering the distribution area of ​​the mask points. As shown in formula (12), M is the predicted mask region and G is the real mask region. This loss is mainly to reflect the overlap between the predicted region and the real region. As shown in formula (13), where L represents the number of filtered nouns in the sentence, G + τ represents the point in the real mask image that corresponds to a real mask value of 1, k represents all points in the image, and τ is a coefficient. As shown in formula (14), G represents all points on the actual mask image, and T + It represents the set of nouns corresponding to a point on the actual mask image; L=λ1L BCE +λ2L Dice +λ3L SAL (15) As shown in formula (15), the three losses involved in formulas (11-14) are processed with different weight ratios and then summed to obtain the final loss of the method.

2. The single-stage method for end-to-end panoramic narrative segmentation as described in claim 1, characterized in that... In step 1), the specific method of multimodal encoding is as follows: extract each word vector in the text using a bidirectional encoder representation technique based on a transformer, and filter out the word vectors of non-noun parts using dataset annotations. At the same time, for the case where multiple word vectors correspond to the same noun, the average method is used as the feature corresponding to the current noun.

3. The single-stage method for end-to-end panoramic narrative segmentation as described in claim 1, characterized in that... In step 2), the multimodal interaction embeds local spatial priors into attention modeling, enhancing visual features at different scales and thus improving segmentation performance.

Images