Unified cascade panoramic narrative detection and segmentation method

Abstract

A unified cascaded approach for panoramic narrative detection and segmentation involves 1) multimodal encoding; 2) multimodal interaction; 3) coordinate-guided aggregation (CGA); 4) centroid-driven localization (BDL); and 5) training loss. A unified framework based on dynamic kernels is constructed, where a learnable kernel is built for each noun phrase to predict its corresponding mask and bounding box. To address the prediction conflict issue, two new cascaded modules are proposed to continuously process segmentation and detection to achieve cross-task alignment: a coordinate-guided aggregation (CGA) module and a centroid-driven localization (BDL) module. Using the centroid of the segmentation mask as an anchor, segmentation and detection are connected in series, naturally aligning the two tasks. The combination of the two modules mutually promotes their respective performance: the position information of the mask drives the BDL module to produce accurate boxes, while the backward guidance of the BDL module promotes the CGA module's ability to distinguish different instances during training.

Description

Technical Field

[0001] The present invention relates to panoramic narrative detection and segmentation, and in particular to a panoramic narrative detection and segmentation method based on a unified cascade of coordinate-guided aggregation and centroid-driven positioning. Background Art

[0002] As common tasks in the multimodal community, directed object segmentation (RES) and directed object detection (REC) tasks aim to localize an object based on the understanding of related phrases. For the REC task, early methods [1 -3] Through RCNN [4] and YOLO [5-7] The detection model obtains a large number of candidate regions in advance, and then compares the features of these regions with the text features to select the most appropriate box as the output. [8,9] The multimodal features are integrated in advance and then detected to achieve end-to-end training. For RES tasks, the segmentation model method was previously used. [10,11] Similar to the two-stage REC method, a number of one-stage methods for refining segmentation masks have emerged. [12,13] , which achieved better performance. However, both RES and REC can only locate one target, ignoring the background factors and the many-to-many relationship between natural language and images.

[0003] Panoramic Narrative Segmentation (PNS) and Panoramic Narrative Detection (PND) are two highly related tasks. Given an image and an associated narrative text, it segments and detects multiple panoramic object instances in the image corresponding to the noun phrases contained in the text. For the PND task, early methods [14-18] Following the two-stage approach, it is considered as a detection task first and then a matching task, which makes the results overly dependent on the results of object detection. After this, end-to-end methods have emerged, which usually first perform feature fusion of text and image, and then use the detection head to detect the object.

[19] In addition, there are some methods that rely on graph neural networks to model the relationship between objects and also rely on the existing multi-anchor detection paradigm.

[20] The biggest difference between the PNS task and the PND task is that it is a pixel-level positioning task that requires classification of each pixel in the image to generate a global and unified segmentation image. Gonzalez et al.

[21] We first explore this task and propose a two-stage approach that uses a segmentation model

[22] Generate a mask, and then match the mask to the phrase. After that, a KNet-like

[23] PPMN

[24] , achieving good performance.

[0004] Multi-task learning is a common paradigm for solving two highly related tasks. The essence of segmentation and detection tasks lies in the understanding of semantic information, so multi-task learning is a very natural idea. For REC and RES tasks, the multi-task joint learning method [19,25] It is not uncommon that these multi-task models usually adopt a dual-branch paradigm to handle detection and segmentation respectively. Although they share a multimodal backbone network, their reasoning branches are still relatively separated, which leads to the problem of prediction conflict, that is, the positioning box predicted by the joint model and the segmented content do not point to the same target instance. Therefore, in the MCN method

[19] Tedious additional cross-task alignment and post-processing operations are required to solve this problem.

[0005] However, for PNS and PND tasks, there is a many-to-many cross-modal alignment, and it is not a simple matter to directly conduct multi-task learning. If we directly adopt a paradigm like the MCN method, we will encounter two problems. On the one hand, in PNS and PND tasks, the predicted targets are not only singular nouns, but also plural and uncountable nouns, which makes post-processing difficult; on the other hand, due to the existence of multiple target instances, directly using the existing multi-task paradigm means that the inference cost of multimodal prediction increases linearly with the number of targets. Previous methods

[26] Usually, the positioning frame is obtained through the detection algorithm first, and then pixel-level segmentation is performed within the positioning frame. This will cause the segmentation result to be overly dependent on the performance of the detector.

[0006] References:

[0007] [1]. Ronghang Hu, et al. Modeling relationships inreferential expressions with compositional modular networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1115–1124, 2017.

[0008] [2].Jingyu Liu,et al.Referringexpression generation and comprehensionvia attributes.InProceedings ofthe IEEE International Conference on ComputerVision,pages 4856–4864,2017.

[0009] [3].Ruotian Luo,et al.Comprehension-guided referring expressions.InProceedings of the IEEEConference on Computer Vision and Pattern Recognition,pages 7102–7111,2017.

[0010] [4].Ross Girshick,et al.Rich feature hierarchies for accurate objectdetectionand semantic segmentation.In Proceedings of the IEEE conference oncomputer vision and pattern recognition,pages580–587,2014.

[0011] [5].Joseph Redmon,et al.You only look once:Unified,real-time objectdetection.In Proceedings of the IEEE conference on computervision and patternrecognition,pages 779–788,2016.

[0012] [6].Joseph Redmon and Ali Farhadi.Yolo9000:better,faster,stronger.InProceedings of the IEEE conference on computervision andpattern recognition,pages 7263–7271,2017.

[0013] [7].Joseph Redmon and Ali Farhadi.Yolov3:An incrementalimprovement.arXivpreprint arXiv:1804.02767,2018.

[0014] [8].Arka Sadhu,et al.Zero-shot grounding of objects from naturallanguage queries.In Proceedingsof the IEEE / CVF International Conference onComputer Vision,pages 4694–4703,2019.

[0015] [9].Zichao Yang,et al.Hierarchical attention networksfor documentclassification.In Proceedings of the 2016 conference of the North Americanchapter of the associationfor computational linguistics:human languagetechnologies,pages 1480–1489,2016.

[0016]

[10] .Ruiyu Li,et al.Referring image segmentation via recurrentrefinement networks.In Proceedings of the IEEE Conference on Computer Visionand PatternRecognition,pages 5745–5753,2018.

[0017]

[11] .Chenxi Liu,et al.Recurrent multimodal interaction for referringimage segmentation.In Proceedings ofthe IEEE International Conference onComputer Vision,pages 1271–1280,2017.

[0018]

[12] .Daqing Liu,et al.Learning to assemble neural module treenetworks for visualgrounding.In Proceedings ofthe IEEE / CVFInternationalConference on Computer Vision,pages 4673–4682,2019

[0019]

[13] .Yiyi Zhou,et al.A real-timeglobal inference network for one-stage referring expressioncomprehension.IEEE Transactions on Neural NetworksandLearning Systems,2021.

[0020]

[14] .Hassan Akbari,et al.Multi-level multimodal common semantic spacefor image-phrase grounding.In Proceedings of the IEEE / CVF Conference onComputerVision and Pattern Recognition,pages 12476–12486,2019.

[0021]

[15] .Andrej Karpathy,et al.Deepfragment embeddings for bidirectionalimage sentence mapping.Advances in neural information processing systems,27,2014.

[0022]

[16] .Varun K Nagaraja,et al.Modeling context between objects forreferring expression understanding.In European Conference on Computer Vision,pages 792–807.Springer,2016.

[0023]

[17] .Bryan A Plummer,et al.Conditional image-text embeddingnetworks.In Proceedings of theEuropean Conference on Computer Vision(ECCV),pages249–264,2018.

[0024]

[18] .Liwei Wang,et al.Learning two-branch neural networks for image-text matching tasks.IEEE Transactions on PatternAnalysis and MachineIntelligence,41(2):394–407,2018.

[0025]

[19] .Gen Luo,et al.Multi-task collaborative network for jointreferring expression comprehensionand segmentation.In Proceedings of theIEEE / CVF Conferenceon computer vision and pattern recognition,pages10034–10043,2020.

[0026]

[20] .Mohit Bajaj,et al.G3raphground:Graph-based language grounding.InProceedings of the IEEE / CVF International Conference on Computer Vision,pages4281–4290,2019.

[0027]

[21] .Cristina Gonz′alez,et al.Panoptic narrative grounding.InProceedings ofthe IEEE / CVF International Conference on Computer Vision,pages1364–1373,2021.

[0028]

[22] . Alexander Kirillov, et al. Panoptic feature pyramid networks. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 6399–6408, 2019.

[0029]

[23] . Wenwei Zhang, et al. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems, 34:10326–10338, 2021.

[0030]

[24] . Zihan Ding, et al. Ppmn: Pixel-phrase matching network for one-stage panoptic narrative grounding. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5537–5546, 2022.

[0031]

[25] . Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. Advances in Neural Information Processing Systems, 34:19652–19664, 2021.

[0032]

[26] . Kaiming He, et al. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. Summary of the Invention

[0033] The purpose of this invention is to provide a unified cascaded panoramic narrative detection and segmentation method (NICE) to solve the prediction conflict problem in panoramic narrative detection and segmentation tasks and the many-to-many cross-modal alignment problem unique to PNS and PND. A learnable kernel is constructed for each noun phrase, and two cascaded modules are proposed to continuously process segmentation and detection to achieve cross-task alignment. They are the coordinate guided aggregation (CGA) module and the centroid driven localization (BDL) module. The centroid of the segmentation mask is used as an anchor point to connect segmentation and detection in series, so that the two tasks are naturally aligned. This is a unified cascaded panoramic narrative detection and segmentation method.

[0034] The present invention comprises the following steps:

[0035] 1) Multimodal encoding: Given a narrative text and an image pair, the image is passed through a feature pyramid network to extract visual features as F v The narrative text is extracted from each word vector in the text through the pre-trained Bidirectional Encoder Based on Trasnformer (BERT) representation technology, and then the word vector belonging to the noun phrase is filtered and obtained as the text feature F p ;

[0036] 2) Multimodal interaction: using text features F P As the convolution kernel in the image feature F v Convolution is performed on the mask M of the target instance 0 ;

[0037] 3) Coordinate Guided Aggregation (CGA): For the PNS task, there are L layers of CGA modules. For the lth layer of CGA, the output K is obtained through the multi-head MaskedCrossAttention l , and then with the image feature F v Do convolution to get mask M l ; Among them, the specific algorithm of attention is to use the text feature K l-1 As Query, use image feature F v As the key, in the mask M l-1 The attention weight is calculated under the constraint of l-1 As the value multiplied by the attention weight, the final output K is obtained l ;

[0038] 4) Centroid-driven Localization (BDL): For the PND task, there are L layers of BDL modules. For the I-th layer of BDL, the image features F vAfter a BottleNeck module and Sigmoid function, the number of channels of the image feature map is changed to 4. Then, the centroid coordinates of the mask map obtained in step 3) are used to index the pixel values ​​of the corresponding points on these four feature maps. These values ​​are used as the offsets of the left, top, right, and bottom relative to the centroid of the positioning box to obtain the final prediction box.

[0039] 5) Training Loss: Since the NICE method is a cascade framework that unifies panoramic narrative segmentation and panoramic detection tasks, it is necessary to consider both segmentation loss and detection loss when calculating the loss. For segmentation loss, the BCE loss function and the Dice loss function are used as the objective function of the segmentation task; for detection loss, the Smooth L1 loss function and the gIOU loss function are used as the objective function of the detection task.

[0040] Since there is currently no joint learning method for PNS and PND, in order to solve the prediction conflict problem and the many-to-many cross-modal alignment problem unique to PNS and PND, the present invention proposes a unified cascade panoramic narrative detection and segmentation method (NICE). Inspired by KNet and PPMN, a dynamic kernel-based approach is first adopted to construct a unified framework, building a learnable kernel for each noun phrase and directly predicting its corresponding mask and bounding box at the same time. This method can get rid of the original dense prediction and align each phrase for a visual area, while also avoiding tedious feature pre-fusion and post-processing operations, thereby improving the efficiency of multi-target prediction at the same time. To address the prediction conflict problem, two new cascade modules are proposed to continuously process segmentation and detection to achieve cross-task alignment, namely the coordinate guided aggregation (CGA) module and the centroid driven localization (BDL) module. The centroid of the segmentation mask is used as an anchor point to connect segmentation and detection in series, so that the two tasks are naturally aligned. The combination of the two modules also promotes their respective performance. The position information of the mask drives the BDL module to produce accurate boxes, while the backward guidance of the BDL module also greatly promotes the CGA module's ability to distinguish different instances during training. Based on the efficient mechanism of PNS, even with a cascade structure, PND is hardly slower than the single-task model.

[0041] Compared with the prior art, the present invention has the following outstanding advantages:

[0042] 1) This paper proposes a unified cascaded panoramic narrative detection and segmentation method (NICE), which is the first collaborative multi-task framework proposed for PND and PNS tasks, which performs segmentation and detection in a cascade manner.

[0043] 2) This paper proposes two novel module designs: the Coordinate Guided Aggregation (CGA) module and the Centroid Driven Localization (BDL) module, responsible for segmentation and detection, respectively. The positional information of the mask centroid drives the BDL module to produce accurate boxes, while the backward guidance provided by the BDL module significantly enhances the CGA module's ability to distinguish different instances during training. The two modules are cleverly cascaded to resolve prediction conflicts and achieve collaborative reasoning.

[0044] 3) The proposed Unified Cascade Panoramic Narrative Detection and Segmentation (NICE) method achieves the best performance among all existing methods on both PND and PNS tasks, improving segmentation performance by 1.8% and detection performance by 7.8%. Furthermore, its inference speed is 22.5% faster than that of the single-task network PPMN. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Flowchart of the unified cascaded panoramic narrative detection and segmentation method (NICE).

[0046] Figure 2 For comparison and NICE on the same samples.

[0047] Figure 3 To show the visual comparison between the prediction results of the NICE method on some samples and the true labels. DETAILED DESCRIPTION

[0048] The following embodiments will further illustrate the present invention with reference to the accompanying drawings.

[0049] The purpose of this invention is to solve the problem of prediction conflict in panoramic narrative detection and segmentation tasks and the problem of many-to-many cross-modal alignment unique to PNS and PND. A learnable kernel is constructed for each noun phrase, and two cascade modules are proposed to continuously process segmentation and detection to achieve cross-task alignment. They are the coordinate guided aggregation (CGA) module and the centroid driven localization (BDL) module. The centroid of the segmentation mask is used as the anchor point to connect segmentation and detection in series, so that the two tasks are naturally aligned. This is a unified cascade panoramic narrative detection and segmentation method. The specific algorithm flow is as follows: Figure 1 shown.

[0050] The Unified Cascaded Panoptic Narrative Detection and Segmentation Framework (NICE) is a unified, simple and effective method that simultaneously solves two highly related tasks: Panoptic Narrative Detection (PND) and Panoptic Narrative Segmentation (PND). Both PND and PNS aim to locate multiple targets in a given image based on a longer narrative description, but existing methods cannot solve the prediction conflict problem of PND and PNS well, that is, the localization box and segmentation mask predicted by the model do not point to the same object. Taking into account the prediction conflict problem, two innovative modules are proposed, namely the Coordinate Guided Aggregation (CGA) module and the Barycenter Driven Localization (BDL) module. The two modules collaborate to improve the performance of the two tasks. The entire framework includes the following steps:

[0051] 1) Multimodal encoding: Given a narrative text and an image pair, the image is passed through a feature pyramid network to extract visual features as F v The narrative text is extracted from each word vector in the text through the pre-trained Bidirectional Encoder Based on Trasnformer (BERT) representation technology, and then the word vector belonging to the noun phrase is filtered and obtained as the text feature F p ;

[0052] 2) Multimodal interaction: using text features F P As the convolution kernel in the image feature F v Convolution is performed on the mask M of the target instance 0 ;

[0053] 3) Coordinate Guided Aggregation (CGA): For the PNS task, there are L layers of CGA modules. For the lth layer of CGA, the output K is obtained through the multi-head MaskedCrossAttention l , and then with the image feature F v Do convolution to get mask M l ; Among them, the specific algorithm of attention is to use the text feature K l-1 As Query, use image feature F v As the key, in the mask M l-1 The attention weight is calculated under the constraint of l-1 As the value multiplied by the attention weight, the final output K is obtained l ;

[0054] 4) Centroid-driven Localization (BDL): For the PND task, there are L layers of BDL modules. For the I-th layer of BDL, the image features F v After a BottleNeck module and Sigmoid function, the number of channels of the image feature map is changed to 4. Then, the centroid coordinates of the mask map obtained in step 3) are used to index the pixel values ​​of the corresponding points on these four feature maps. These values ​​are used as the offsets of the left, top, right, and bottom relative to the centroid of the positioning box to obtain the final prediction box.

[0055] 5) Training Loss: Since the NICE method is a cascade framework that unifies panoramic narrative segmentation and panoramic detection tasks, it is necessary to consider both segmentation loss and detection loss when calculating the loss. For segmentation loss, the BCE loss function and the Dice loss function are used as the objective function of the segmentation task; for detection loss, the Smooth L1 loss function and the gIOU loss function are used as the objective function of the detection task.

[0056] The specific modules are as follows:

[0057] 1. Multimodal encoding

[0058] For the input image, a feature pyramid network with ResNet-50 as the backbone network is used to extract the multi-scale features of the image. Considering the importance of location information, Add position encoding separately, and then follow the approach of panoramic feature pyramid network to fuse multi-scale features to obtain the final visual feature F v ∈R H×W×C ,in,

[0059] For the input text, a pre-trained bidirectional encoder representation technology based on Trasnformer (BERT) is used to extract each word vector in the text, and then filter out the word vectors belonging to noun phrases. In the case where a noun corresponds to multiple word vectors, an average pooling operation is performed, and the average value of multiple word vectors is used as the feature of the noun. The features corresponding to these noun phrases are then projected through a linear layer to align their dimensions with the visual features to obtain the final text features. Among them, f n Represents the features of the nth noun phrase, N is the number of phrases, and C is the dimension of the word vector corresponding to each phrase after projection.

[0060] 1. Multimodal interaction

[0061] Using text features F P As the convolution kernel in the image feature F vConvolution is performed on the mask M of the target instance 0 .

[0062] M 0 =Sigmoid(K 0 *F v ) (1)

[0063] 2. Coordinate Guided Aggregation (CGA)

[0064] For the PNS task, there are L layers of CGA modules. For the lth layer of CGA, the output K is obtained through multi-head Masked CrossAttention. l , then in the image feature F v Do convolution to get mask M l Among them, the specific algorithm of attention is to use the text feature K l-1 As Query, use image feature F v As the key, in the mask M l-1 The attention weight A is calculated under the constraint of j .

[0065]

[0066]

[0067] Where τ is the control The threshold value, The scope of attention interaction between text features and visual features is limited, thus avoiding irrelevant semantics. l-1 ∈[0,1] N×H×W Indicates the The layer mask, and The projection is the weight matrix, is the scale factor, j represents the jth head h in the multi-head attention, and there are 8 heads in total.

[0068] Then, based on these attention weights, K l-1 As the multiplication of Value and attention weight, and aggregate the output of multiple heads to update the kernel feature K of layer l l .

[0069]

[0070] K l =FFN(LN([Head 1 ,...,Head h ]W o +K l-1 ) (5)

[0071] Finally, the text-guided dynamic feature kernel K l Will be in the visual feature F v The final mask is obtained by convolution.

[0072] M l =Sigmoid(K l *F v ) (6)

[0073] In addition, we further utilize the bounding boxes generated by the detections to constrain the segmentation to help distinguish different instances. Specifically, we decouple the bounds by using a soft constraint, i.e., the centroid of the mask, to connect the two modules in series, as described in the next subsection.

[0074] 3. Mass Driven Positioning (BDL)

[0075] For the PND task, since the mask generated by the CGA module in the previous section can roughly mark the location of the target, this can be regarded as a very practical hint for PND. Generally speaking, the center of the mask is likely to be the center of the target entity. Therefore, it is necessary to first find the center of mass of the mask. For the mask M generated by the l-th layer CGA, l , the position of its center of mass is calculated as follows:

[0076]

[0077]

[0078] Among them, n represents the nth noun phrase, D l ∈R N×2 are the 2D coordinates of the centroid of the mask.

[0079] At the same time, the image feature F of the l-1 layer v After a BottleNeck module and a Sigmoid function, the number of channels of the image feature map is changed to 4 to predict the offset value of each prediction box relative to the mask centroid.

[0080] O=Sigmoid(BottleNeck(F v )) (9)

[0081] Among them, BottleNeck(·) is a cascade convolutional network that can change the dimension of the feature map, O∈[0,1] H×W×4 .

[0082] The final prediction box is obtained by combining the centroid coordinates of the mask image to index the pixel values ​​of the corresponding points on the four feature maps as the offsets of the left l, right r, top t, and bottom b relative to the centroid of the positioning box.

[0083]

[0084] Among them, l, r, t, and b come from O x,y , represents the distance between the center of mass and the bounding box of the target. are the coordinates of the center of mass, x 1 、y 1 is the coordinate of the upper left point of the bounding box, x 2 、y 2 is the coordinate of the lower right point of the bounding box. Finally, the specific position of the target positioning box can be predicted based on the obtained offset and center of mass coordinates.

[0085] 4. Training loss

[0086] Since the NICE method is a cascade framework that unifies panoramic narrative segmentation and panoramic detection tasks, it is necessary to consider both segmentation loss and detection loss when calculating the loss.

[0087] For segmentation loss, the BCE loss function and Dice loss function are used as the objective function of the segmentation task. Among them, the BCE loss is shown in formula (11) and the Dice loss is shown in formula (12).

[0088]

[0089]

[0090] For detection loss, the Smooth L1 loss function and gIOU loss function are used as the objective function of the detection task. The Smooth L1 loss function for each noun phrase is calculated as shown in formula (13).

[0091]

[0092] Where x = B n,i -G n,i ,B n ∈R 4 represents the prediction box of the nth phrase, G n represents the ground-truth box of the nth phrase, i∈{x,y,w,h}. The parameter ξ is set to 0.5 during training.

[0093] For all phrases in the text, the Smooth L1 total loss function is shown in formula (14).

[0094]

[0095] Since the optimization of smooth L1 loss is not equivalent to the optimization of IoU, the gIoU loss function is added as shown in formula (15).

[0096]

[0097] in, It contains B n and G n The minimum closed area.

[0098] Finally, the above four loss functions are weighted summed to obtain the total loss function, which is the final loss of the NICE method.

[0099]

[0100] The specific experimental results are as follows:

[0101] This paper is trained and tested on the Panoramic Narrative Localization Dataset (PNG). Each narrative in the PNG dataset contains an average of 5.1 objects, and the object annotations corresponding to each noun in the narrative consist of segmentation and detection annotations in the COCO format. The training set of the PNG dataset contains 133,103 training images and 8,380 test images, with each image containing 875,073 and 56,531 masks and localization boxes.

[0102] The average recall is used as the metric for segmentation and detection, and the inconsistent error metric (IE) is used to evaluate the prediction conflict between segmentation and detection tasks to compare with other existing methods, as shown in Table 1.

[0103] Table 1. Comparison of the NICE method with other methods

[0104]

[0105] Compared to the two-stage panoptic segmentation method PNG, our method NICE achieves significant improvements in all scenarios, while also doubling the inference speed. Compared to the one-stage panoptic segmentation method PPMN, NICE also achieves the best performance and inference speed. This demonstrates that our method achieves the best results among all currently available methods for panoptic segmentation tasks.

[0106] Furthermore, when comparing our model with the multi-task learning method MCN, since the MCN method is designed for REC and RES tasks, when migrating it to PND and PNS tasks, it can only generate a mask and a box for each noun phrase. Compared to the MCN method, our method achieves a 7.0% improvement on segmentation tasks and a 7.8% improvement on detection tasks, while also significantly increasing inference speed.

[0107] It refers to the text features obtained after MaskedAcross Attention, which are passed through two different fully connected layers to obtain two sets of features. One set is used to convolve with the image features to generate a mask, and the other set is directly converted to four-dimensional features through a linear layer, for the four offsets of the prediction box. In contrast, the cascade segmentation structure and detection method NICE of the present invention brings huge improvements at a small inference time cost, especially for PND tasks, which further proves its effectiveness. Figure 2 In comparison The performance of the NICE method on the same sample shows that the NICE method of the present invention can handle the The problem of unsolvable prediction conflict demonstrates the effectiveness of the collaboration between the CGA module and the BDL module.

[0108] In addition, this paper introduces a strong reference, NICE*, which directly finds the corresponding box based on the shape of the mask. Compared with NICE, its detection performance is significantly reduced. This is because the strong correlation paradigm causes detection to overly rely on unmanageable segmented edges.

[0109] exist Figure 3 In the figure, the NICE method’s prediction results on some samples are compared with the true labels. It can be seen that NICE’s segmentation and detection results are very accurate, which means that the model of the present invention accurately understands the semantic and image information. At the same time, on some samples, the performance of the NICE method is even better than the true labels. The white dotted line frames the parts where the true values ​​are inaccurate, and the model of the present invention gives more accurate results. Figure 3 For the first sample, the true label incorrectly segments the air conditioner as part of the wall; for the second sample, the true value does not accurately segment the beach, water, and hills as the NICE method of our invention.

[0110] In summary, the NICE method proposed in the present invention has good results in terms of performance and inference speed, and from the perspective of IE indicators, it also solves the prediction conflict problem very well.

[0111] The formula is described as follows: (For the defined formula variables and symbols, please refer to the specific formula expression description)

[0112] Formula (1) defines the process of using text features as convolution kernels to convolve image features to obtain the target mask.

[0113] Formula (2-3) defines a fixed threshold to limit the scope of attention interaction between text features and visual features, thereby avoiding irrelevant semantics, and calculating the attention weight under this constraint.

[0114] Formula (4-5) defines the result of multiplying the kernel features of the l-1 layer as Value and the attention weight, and aggregating the outputs of multiple heads to update the kernel features of the l layer.

[0115] Formula (6) defines the text-guided dynamic feature kernel in the I-th layer CGA module to convolve on the visual features to obtain the final mask result.

[0116] Formula (7-8) represents the calculation result of the centroid coordinates of the mask generated by the l-th layer CGA module for the n-th phrase.

[0117] Formula (9) defines that the image features of the I-1 layer are passed through a BottleNeck module and a Sigmoid function, and the number of channels of the image feature map is changed to 4 to predict the offset value of each prediction box relative to the center of mass of the mask.

[0118] Formula (10) defines the pixel values ​​of the corresponding points on the four feature maps in formula (9) indexed by the centroid coordinates of the mask image as the offsets of the four relative centroids of the positioning box, namely, the left l, right r, top t, and bottom b, to obtain the final predicted box.

[0119] Formula (11-12) defines the process of calculating the segmentation loss using the BCE loss function and the Dice loss function as the objective function of the segmentation task.

[0120] Formula (13-16) defines the process of calculating the detection loss using the Smooth L1 loss function and the gIOU loss function as the objective function of the detection task.

[0121] English proper nouns are defined as follows:

[0122] Average Recall is defined as the integral of the iou-regression rate curve.

[0123] Inference Time is defined as the time it takes for a model to input data and output inference results.

[0124] Params is defined as the total number of all trainable parameters in the model, where M represents millions.

[0125] Training Data is defined as the number of annotations required for model training, where each annotation represents a segmentation mask and M represents one million.

Claims

1. Unified cascade panoramic narrative detection and segmentation method, characterized by The following steps are involved: 1) Multimodal encoding: Given a narrative text and an image pair, the image is passed through a feature pyramid network to extract visual features as F v The narrative text is extracted from each word vector in the text through the pre-trained bidirectional encoder BERT representation technology based on Trasnformer, and then the word vector belonging to the noun phrase is filtered and obtained as the text feature F p ; 2) Multimodal interaction: using text features F P As the convolution kernel in the image feature F v Convolution is performed on the mask M of the target instance 0 ; 3) Coordinate guided aggregation CGA: For the PNS task, there are L layers of CGA modules. For the lth layer of CGA, the output K is obtained through multi-head Masked Cross Attention l , and then with the image feature F v Do convolution to get mask M l ; Among them, the specific algorithm of attention is to use the text feature K l-1 As Query, use image feature F v As the key, in the mask M l-1 The attention weight is calculated under the constraint of l-1 As the value multiplied by the attention weight, the final output K is obtained l ; The specific steps of the coordinate-guided aggregation are: For the PNS task, there are L layers of CGA modules. For the lth layer of CGA, the output K is obtained after multi-head Masked Cross Attention l , then in the image feature F v Do convolution to get mask M l ; Among them, the specific algorithm of attention is to use the text feature K l-1 As Query, use image feature F v As the key, in the mask M l-1 The attention weight A is calculated under the constraint of j ; Where τ is the control The threshold value, Limit the scope of attention interaction between text features and visual features to avoid irrelevant semantics; M l-1 ∈[0,1] N×H×W Indicates the The layer mask, and The projection is the weight matrix, is the scale factor, j represents the jth head h in the multi-head attention, with a total of 8 heads; Based on these attention weights, K l-1 As the multiplication of Value and attention weight, aggregate the output of multiple heads to update the kernel feature K of layer l l ; K l =FFN(LN([Head 1 ,...,Head h ]W O )+K l-1 ) Text-guided dynamic feature kernel K l In the visual feature F v Convolution is performed on the final mask; M l =Sigmoid(K l *F v ) Use the boxes generated by detection to constrain segmentation to help distinguish different instances; 4) Centroid-driven localization BDL: For the PND task, there are L layers of BDL modules. For the lth layer of BDL, the image features F of the l-1th layer are v After a BottleNeck module and Sigmoid function, the number of channels of the image feature map is changed to 4. Then, the centroid coordinates of the mask map obtained in step 3) are used to index the pixel values ​​of the corresponding points on the four feature maps. These values ​​are used as the offsets of the left, top, right, and bottom relative to the centroid of the positioning box to obtain the final prediction box. 5) Training Loss: Since the NICE method is a cascade framework that unifies panoramic narrative segmentation and panoramic detection tasks, it is necessary to consider both segmentation loss and detection loss when calculating the loss. For segmentation loss, the BCE loss function and the Dice loss function are used as the objective function of the segmentation task; for detection loss, the Smooth L1 loss function and the gIOU loss function are used as the objective function of the detection task.

2. The unified cascade panoramic narrative detection and segmentation method as claimed in claim 1, characterized in that In step 1), the image is processed through a feature pyramid network to extract visual features F v For the input image, a feature pyramid network with ResNet-50 as the backbone network is used to extract the multi-scale features of the image. Considering the importance of location information, Add position encoding separately, and then follow the approach of panoramic feature pyramid network to fuse multi-scale features to obtain the final visual feature F v ∈R H×W×C ,in, 3. The unified cascade panoramic narrative detection and segmentation method as claimed in claim 1, characterized in that In step 1), for narrative text, a pre-trained BERT (Bidirectional Transformer-based Encoder Representation) is used to extract word vectors from the text. The vectors for noun phrases are then filtered. For nouns with multiple word vectors, an average pooling operation is performed, with the average of the multiple word vectors used as the feature of the noun. The features corresponding to these noun phrases are then projected through a linear layer to align their dimensions with the visual features to obtain the final text features Among them, f n Represents the features of the nth noun phrase, N is the number of phrases, and C is the dimension of the word vector corresponding to each phrase after projection.

4. The unified cascade panoramic narrative detection and segmentation method as claimed in claim 1, characterized in that In step 4), the specific steps of the centroid driven positioning BDL are: For the PND task, we need to find the center of mass of the mask. For the mask M generated by the l-th layer CGA, l , the position of its center of mass is calculated as follows: Among them, n represents the nth noun phrase, D l ∈R N×2 is the 2D coordinate of the centroid of the mask; The image feature F of the l-1 layer v After a BottleNeck module and Sigmoid function, the number of channels of the image feature map becomes 4 to predict the offset value of each prediction box relative to the mask centroid; O=Sigmoid(BottleNeck(F v )) Among them, BottleNeck(·) is a cascaded convolutional network, O∈[0,1] H×W×4 ; The pixel values ​​of the corresponding points on the four feature maps are indexed by combining the centroid coordinates of the mask image as the offsets of the four relative centroids of the positioning box, namely the left l, right r, top t, and bottom b, to obtain the final prediction box. Among them, l, r, t, and b come from O x,y , represents the distance between the center of mass and the bounding box of the target; are the coordinates of the center of mass, x 1 、y 1 is the coordinate of the upper left point of the bounding box, x 2 、y 2 is the coordinate of the lower right point of the bounding box; based on the obtained offset and center of mass coordinates, the specific position of the target positioning box can be predicted.

5. The unified cascade panoramic narrative detection and segmentation method as claimed in claim 1, characterized in that In step 5), the training loss includes segmentation loss and detection loss; For segmentation loss, the BCE loss function and Dice loss function are used as the objective function of the segmentation task; among them, the BCE loss is: The Dice loss is: For detection loss, the Smooth L1 loss function and gIOU loss function are used as the objective functions of the detection task; the Smooth L1 loss function for each noun phrase is calculated as shown below: Where x = B n,i -G n,i ,B n ∈R 4 represents the prediction box of the nth phrase, G n represents the ground-truth box of the nth phrase, i∈{x,y,w,h}; the parameter ξ is set to 0.5 during training; For all phrases in the text, the Smooth L1 total loss function is as follows: Since the optimization of smooth L1 loss is not equivalent to the optimization of IoU, the gIoU loss function is added as shown below: in, It contains B n and G n The minimum enclosed area; The above four loss functions are weighted summed to obtain the total loss function, which is the final loss of the NICE method:

Images