A group and predicate parallel prediction group visual relationship detection method

Abstract

A method for detecting group visual relationships by parallel prediction of groups and predicates first extracts objects from the input image. Object similarity is calculated based on object type and visual features. Objects are then divided into multiple candidate groups based on similarity, and these candidate groups are paired to form candidate group pairs. Next, visual, semantic, and positional features are extracted at the object level, group level, and group pair level. Then, cross-attention encoding of object features within a group is used to predict whether an object is in that group. Object features are weighted with object confidence and concatenated with group features and group pair features to predict predicates. Finally, group visual relationships are generated based on the confidence of objects forming a group and the probability of predicates. This invention is based on the discovery that group range and group visual relationship predicates influence each other. By predicting group range and predicate categories in parallel, it identifies group visual relationships where predicates and group ranges accurately match, exhibiting good broad applicability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to visual relationship detection in images, specifically a group visual relationship detection method that uses parallel prediction of groups and predicates. Background Technology

[0002] Visual relationship detection aims to explore visual relationships between objects, represented by a triple of <subject, relational verb, object>. The subject and object are represented by the object's category and bounding box, while the relational verb includes verbs, verb phrases, and locative words. Visual relationship detection in images can parse image content and further support applications such as visual retrieval, visual question answering, and image-based question answering.

[0003] like Figure 2 (a) and Figure 2 As shown in (c), current visual relationship detection methods can only handle visual relationships where both the subject and object are single objects, such as <person, cut, cake>, <cup, on, table>, <chair, beside, table>. These are one-to-one relationships, meaning they can only detect individual visual relationships. However, in images with complex content, there are not only relationships between single objects, but also more complex relationships between individuals and groups, and between groups themselves, such as... Figure 2 (b) and Figure 2 As shown in (d), it is hoped that visual relationship detection can reflect visual relationships within a group, such as <multiple people, cutting, cake>, <multiple cups, on, table>. Therefore, existing individual visual relationship detection methods greatly limit the ability of visual relationship detection to represent complex image content.

[0004] Group visual relations include <subject*, relational verb, object>, <subject, relational verb, object*>, or <subject*, relational verb, object*>, where "*" indicates that the subject or object is a group. For individual visual relation detection, the number of combinations between subject, object, and verb increases cubically with the number of objects and the verb category. For group visual relation detection, the increase in the number of triple combinations is further based on the increase in the number of group components, leading to an explosive increase in the possibilities of group visual relation triples. Therefore, group visual relation detection is an extremely challenging problem.

[0005] Currently, no research has attempted to detect visual relationships in groups. Current studies on visual relationship detection only focus on visual relationships between individual objects, while group-related research is limited to crowd behavior and has not expanded to the form of relationship triples or to general objects. Therefore, current work cannot solve the problem of visual relationship detection in groups. Summary of the Invention

[0006] The problem this invention aims to solve is that visual relationships in images are too complex. In addition to the relationships between individuals, there are also relationships between individuals and groups, and between groups. Existing visual relationship detection technologies do not have a detection scheme for group-related relationship triples. It is necessary to study methods for detecting group-related visual relationships to help machines understand and analyze images more comprehensively, and further solve the problem of computational complexity.

[0007] The technical solution of this invention is as follows: a group visual relationship detection method based on parallel prediction of groups and predicates. A group-predicate parallel prediction network is established to output the group visual relationships in the input image. The group-predicate parallel prediction network includes an entity construction module, a feature extraction module, and a group relationship prediction module. First, the entity construction module extracts objects from the input image, calculates object similarity based on object type and visual features, and then divides the objects into multiple candidate groups based on similarity, pairing each candidate group to form a candidate group pair. Next, in the feature extraction module, visual features and positions at the entity level are extracted. As object features, visual features, positional features, and semantic features at the group level are extracted as group features, and visual features at the group pair level are extracted as group pair features. Then, in the group relationship prediction module, through the cross-attention of the object group to the subject group or the subject group to the object group, the positional features and visual features of objects in the candidate group are jointly encoded to predict the confidence that an object belongs to the group. The object features are then weighted with the object confidence and concatenated with the group features and group pair features to predict the predicate. Finally, based on the confidence that an object belongs to the group and the probability of the predicate, the final group visual relationship is generated in parallel.

[0008] Furthermore, the present invention includes the following steps:

[0009] 1) For the input image, detect all objects and extract the object bounding boxes and categories. Calculate the similarity between objects using object categories and visual features. Divide the objects into multiple candidate groups based on the similarity and then pair the candidate groups to form candidate group pairs.

[0010] 2) Extract individual characteristics, population characteristics, and population pair characteristics;

[0011] 3) For each candidate group pair, the candidate group is either the subject group or the object group. Based on the visual and positional features of the object, calculate the cross-attention matrix of the object group to the subject group and the cross-attention matrix of the subject group to the object group. Based on the obtained attention matrix, encode the object features of the corresponding candidate group through cross-attention to predict the confidence of the object in the candidate group.

[0012] 4) For each candidate group pair, weight the object visual features and position features with the object confidence predicted in step 3), and concatenate them with the candidate group visual features, position features, semantic features, and candidate group pair visual features to predict the predicate probability of the candidate group pair.

[0013] 5) For each candidate group pair, combining steps 3) and 4), perform parallel predictions of the candidate groups based on the confidence level of the object belonging to the group and the predicate probability to generate visual group relationships.

[0014] Step 1) is implemented in the entity construction module, step 2) is implemented in the feature export module, and steps 3)-5) are implemented in the group relationship prediction module.

[0015] The present invention has the following improvements:

[0016] 1. Existing technologies only perform triplet detection for relationships where both the subject and object are single objects. This invention, however, focuses on groups and proposes a method for detecting visual relationships within groups. First, candidate groups are defined, and then the relationships between the groups and objects within them are detected based on these candidate groups, thus achieving group visual detection.

[0017] 2. The method of first dividing "candidate groups" in this invention significantly reduces the number of groups that need to be involved in subsequent calculations, which is beneficial for predicting group relationships and avoids the problem that the number of combinations of individual visual relationships increases cubically with the increase of the number of objects and predicate categories.

[0018] 3. In the scheme of the present invention, the parallel prediction of the group and the predicate can improve the accuracy of the group relationship detection. The reason is that (1) the confidence of the objects in the group participates in the prediction of the predicate. During the training optimization process, when the predicate loss is backpropagated, it will affect the update of the parameters involved in the prediction of the object confidence, thereby optimizing the prediction of the object confidence. (2) the prediction of the predicate does not depend on the final determination of the group, but controls the influence weight of the features of different objects through the confidence of the objects, thereby improving the robustness of the predicate prediction.

[0019] The beneficial effects of this invention are as follows: This invention proposes a group visual relationship detection method for the first time, providing a solution to the problem that overly complex visual relationships in images lead to insufficient machine understanding of images. It achieves parallel prediction of predicate categories for group range and group relationships, and simultaneously considers the mutual influence between group range and group relationship predicates during the detection of group visual relationships, thus identifying group visual relationships in the image. This method can be widely used in various image visual understanding scenarios and has strong practicality. Attached Figure Description

[0020] Figure 1This describes the architecture of the group and predicate parallel prediction network and the process of group visual relationship detection in this invention.

[0021] Figure 2 A comparison of the definitions of individual visual relationships and group visual relationships.

[0022] Figure 3 This is a comparison between the results of traditional individual visual relationship detection and the results of this invention. Detailed Implementation

[0023] The group visual relationship detection method of the group and predicate parallel prediction network involved in this invention provides a solution to the problem that the visual relationships in the image are too complex and the machine cannot fully understand them. It realizes the identification of group visual relationships in the image by considering the mutual influence between the group range and the group relationship predicate.

[0024] The implementation of this invention is described in detail below.

[0025] like Figure 1 As shown, this invention establishes a group and predicate parallel prediction network. The input image is a group visual relationship, and the output image contains group visual relationships. The group and predicate parallel prediction network includes an entity construction module, a feature extraction module, and a group relationship prediction module: the entity construction module extracts objects from the input image, constructs candidate groups, and forms candidate group pairs; the feature extraction module extracts features at the entity level, group level, and group pair level, respectively; the group relationship prediction module simultaneously predicts the confidence level of an object belonging to a group and the group relationship predicate; finally, the final group visual relationship is generated based on the confidence level of an object belonging to a group and the probability of the predicate. The candidate group is equivalent to an intermediate state. This invention first obtains the candidate group, and then uses the prediction of the candidate group to finally determine whether the object belongs to the subject or object, and the predicate relationship between them, thus obtaining the group visual relationship.

[0026] Building upon the aforementioned parallel prediction network for groups and predicates, this invention first extracts objects from the input image through an entity construction module. Object similarity is calculated based on object type and visual features, and then objects are categorized into multiple candidate groups based on similarity. These candidate groups are then paired to form candidate group pairs. Next, in the feature extraction module, visual and positional features at the entity level are extracted as object features, while visual, positional, and semantic features at the group level are extracted as group features. Visual and positional features at the group pair level are extracted as group pair features. Then, in the group relationship prediction module, object features within the group are encoded using cross-attention to predict whether an object belongs to the group (i.e., the confidence level of the object belonging to the group). The object features are weighted using this confidence level and concatenated with the group features and group pair features to predict the predicate. Finally, the confidence level of the object belonging to the group and the predicate probability are combined in parallel to generate the final group visual relationship.

[0027] The implementation of this invention is described in detail below. This invention includes the following steps:

[0028] 1) For the input image, the entity building module is used to extract entities, construct candidate groups, and form candidate group pairs.

[0029] 1.1) Detect all objects and extract their borders and categories.

[0030] 1.2) Calculate the similarity between objects in step 1.1). The calculation method is as follows:

[0031]

[0032]

[0033] in, and S is the normalized global position feature of object i and object j. ij It is the cosine similarity between object i and object j. S is the normalized cosine similarity between object i and object j. min It is the minimum value in the similarity matrix, that is, the minimum S. ij .

[0034] 1.3) Based on the object similarity in step 1.2), classify the objects into multiple candidate groups according to multi-level thresholds.

[0035] 1.4) Pair the candidate groups from step 1.3) to form candidate group pairs.

[0036] 2) For the objects, candidate groups and candidate group pairs in step 1), the feature extraction module is used to extract the visual features, positional features and semantic features of the object level, group level and group pair level.

[0037] 2.1) For each object in step 1.1), extract the visual and positional features of the object.

[0038] 2.2) For each candidate group in step 1.3), extract the visual features, location features and semantic features of the region corresponding to the candidate group.

[0039] 2.3) For each candidate group pair in step 1.4), extract the visual features of the corresponding region of the candidate group pair.

[0040] 3) For each candidate group pair in step 1.4), the group prediction branch of the group relationship prediction module is used to predict the confidence of the object in the candidate group.

[0041] 3.1) For each candidate group pair in step 1.4), each group may be used as a subject or an object. The final group relationship is determined as a subject or an object based on the final result score. Here, cross-attention is calculated for all candidate groups.

[0042] The cross-attention matrix of the object candidate group with respect to the subject candidate group is calculated as follows:

[0043]

[0044] in, It is the object candidate group g o For the subject candidate group g s Attention matrix, It is the subject candidate group g s The matrix is ​​a combination of visual and positional features of all objects in the dataset, obtained by directly concatenating visual and positional features and then inputting the result into the formula. It is the subject candidate group g o A matrix of joint visual and positional features of all objects in the matrix. yes The transpose of , where d is the number of objects in each candidate group after filling, and filling is to keep the number of objects in the group consistent.

[0045] 3.2) For each candidate group pair in step 1.4), calculate the cross-attention matrix of the subject candidate group for the object candidate group, using the same method as in step 3.1).

[0046] 3.3) For each candidate group pair in step 1.4), the object features of the subject candidate group are encoded using cross-attention, calculated as follows:

[0047]

[0048]

[0049] Where η is the normalization operation, τ is the activation function, and ι is the linear transformation. It is the encoded feature of all objects in the subject candidate group after cross-attention encoding.

[0050] 3.4) For each candidate group pair in step 1.4), the object features of the object candidate group are encoded using cross-attention, and the calculation method is the same as in step 3.3).

[0051] 3.5) For each candidate group pair in step 1.4), calculate the confidence γ of the object in the candidate group. i|g The calculation method is as follows:

[0052]

[0053] Where, f″ i|g It is a matrix The matrix elements represent the encoded features of object i in the candidate population g, γ i|g is the confidence level that object i belongs to candidate group g, and μ is the linear transformation that reduces the dimension to 1.

[0054] During training, the population range prediction loss is calculated using the following method:

[0055]

[0056] in, Let p be the set of positive samples from the population, and g be the candidate population in the candidate population pair p. Let g represent the set of objects in the candidate group. These are objects in the actual annotations. This represents the group in the real-world annotations, ξ is the cross-entropy loss, and |·| denotes the modulus of the set. When real objects... Belongs to a real group It is 1 if it is true, otherwise it is 0. The goal is to minimize the average value of the binary cross-entropy between the confidence that an object belongs to a candidate group and the confidence that an object belongs to a real group (0 or 1).

[0057] 4) For each candidate group pair in step 1.4), the predicate prediction branch of the group relationship prediction module is used to predict the probability of the group relationship.

[0058] 4.1) For each candidate group pair in step 1.4), the object's visual features are weighted using the object's confidence level within the group to obtain... The calculation method is as follows:

[0059]

[0060] in, This indicates the visual features generated by the feature export module.

[0061] 4.2) For each candidate group pair in step 1.4), the object location feature is weighted using the confidence level of the object in the group, and the calculation method is the same as in 4.1).

[0062] 4.3) For each candidate group pair in step 1.4), the weighted object visual features and positional features from steps 4.1) and 4.2) are concatenated with the candidate group visual features, positional features, semantic features, and candidate group pair visual features to predict the predicate probability of the candidate group pair. This invention utilizes a multilayer perceptron to predict predicate probability.

[0063] During training, the group visual relation predicate prediction loss is calculated using the following method:

[0064]

[0065] Where P is the set of all candidate population pairs, β and Let β represent the predicted predicate probability vector and the true predicate probability vector for a candidate group, respectively. k It is the kth element of β. The function minimizes the average difference between the predicted probability of a relation and the actual probability of a relation (0 or 1), and increases the penalty for loss when the predicted probability is 0.

[0066] 5) For each candidate group pair in step 1.4), generate group visual relationships by predicting the confidence and predicate probability of the objects forming a group in parallel according to steps 3) and 4).

[0067] 5.1) Remove objects with a confidence level below 0.5 from step 3.5).

[0068] 5.2) For objects that are in both the subject and object candidate groups after filtering in step 5.1), retain the object with the higher confidence level and remove the object with the lower confidence level. For example, if an object has a confidence level of 0.8 in group A and a confidence level of 0.6 in group B, then it is considered that the object is in group A but not in group B, and the object is deleted from group B.

[0069] 5.3) Combining the predicate probability score from step 4.3) with the groups obtained from steps 5.1) and 5.2) to generate the final group visual relationship.

[0070] The parallel prediction described in step 5) is implemented by the group prediction module in the group-predicate parallel prediction network, and its total training loss is the group range prediction loss calculated based on the confidence that the object belongs to the group. With predicate probability prediction loss sum:

[0071]

[0072] The final constructed group-predicate parallel prediction network outputs the group visual relationships in the input image. The entity construction module extracts objects from the input image, constructs candidate groups and forms candidate group pairs, and the feature derivation module extracts features at the entity level, group level, and group pair level. The group relationship prediction module simultaneously predicts the confidence of objects belonging to a group and the group relationship predicate, thus deriving the group visual relationships.

[0073] The present invention was implemented on the ViROI image dataset, and the results were compared with those of traditional individual visual relationship detection methods. Figure 3 This paper compares the results of traditional individual visual relationship detection with those of the present invention. Data shows that the traditional individual visual relationship detection method performs significantly worse than the present invention, demonstrating the effectiveness of the present invention. The following is a comparison of the corresponding methods. Figure 3 Literature on existing detection technologies in China:

[0074] [1]Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. 2017. Scenegraph generation by iterative message passing. In IEEE Conference on ComputerVision and Pattern Recognition. 5410–5419.

[0075] [2] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. 2018. Neuralmotifs: scene graph parsing with global context. In IEEE Conference on ComputerVision and Pattern Recognition. 5831–5840.

[0076] [3]Jianwei Yang,Jiasen Lu,Stefan Lee,Dhruv Batra,and DeviParikh.2018.Graphr-cnn for scene graph generation.In European Conference onComputer Vision.670–685.

[0077] [4] Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. 2019. Knowledge embedded routing network for scene graph generation. In IEEE Conference on Computer Vision and Pattern Recognition. 6163–6171.

[0078] [5]Kaihua Tang,Hanwang Zhang,Baoyuan Wu,Wenhan Luo,and WeiLiu.2019.Learning to compose dynamic tree structures for visual contexts.InIEEEConference on Computer Vision and Pattern Recognition.6619–6628.

[0079] [6]Kaihua Tang,Yulei Niu,Jianqiang Huang,Jiaxin Shi,and HanwangZhang.2020.Unbiased scene graph generation from biased training.In IEEEConference on Computer Vision and Pattern Recognition.3716–3725.

[0080] [7]Yibing Zhan,Jun Yu,Ting Yu,and Dacheng Tao.2019.On exploringundeterminedrelationships for visual relationship detection.In IEEEConference on Computer Vision and Pattern Recognition.5128–5137.

[0081] [8]Chao Chen,Yibing Zhan,Baosheng Yu,Liu Liu,Yong Luo,and BoDu.2022.Resistance training using prior bias:toward unbiased scene graphgeneration.AAAI Conference on Artificial Intelligence.

[0082] [9]Meng-Jiun Chiou,Henghui Ding,Hanshu Yan,Changhu Wang,RogerZimmermann,and Jiashi Feng.2021.Recovering the unbiased scene graphsfrom the biased ones.ACM International Conference on Multimedia.

[0083]

[10] Mohammed Suhail,Abhay Mittal,Behjat Siddiquie,Chris Broaddus,JayanEledath,GerardMedioni,and Leonid Sigal.2021.Energy-based learning forscene sraph generation.IEEEConference on Computer Vision and PatternRecogition.

[0084]

[11] Rongjie Li,Songyang Zhang,Bo Wan,and Xuming He.2021.Bipartitegraphnetwork withadaptive message passing for unbiased scene graphgeneration.InIEEE Conference on ComputerVision and Pattern Recognition.11109–11119.

[0085]

[12] Xingning Dong,Tian Gan,Xuemeng Song,Jianlong Wu,Yuan Cheng,andLiqiang Nie.2022.Stacked hybrid-attention and group collaborativelearningfor unbiased scene graph generation.IEEEConference on Computer Visionand Pattern Recognition.

Claims

1. A group visual relationship detection method based on parallel prediction of group and predicate, characterized by establishing... A group-predicate parallel prediction network outputs group visual relationships in an input image. The network comprises an entity construction module, a feature extraction module, and a group relationship prediction module. First, the entity construction module extracts objects from the input image, calculates object similarity based on object type and visual features, and then classifies objects into multiple candidate groups based on similarity, pairing each candidate group to form a candidate group pair. Next, in the feature extraction module, visual and positional features at the entity level are extracted as object features, visual, positional, and semantic features at the group level are extracted as group features, and visual features at the group pair level are extracted as group pair features. Then, in the group relationship prediction module, through cross-attention between the object group and the subject group or vice versa, the positional and visual features of objects in the candidate groups are jointly encoded to predict the confidence that an object belongs to that group. The object features are weighted by the object confidence and concatenated with the group features and group pair features to predict the predicate. Finally, the network generates the final group visual relationships by parallel prediction based on the confidence that an object belongs to a group and the probability of the predicate.

2. The group visual relationship detection method based on parallel prediction of groups and predicates according to claim 1, characterized in that: Includes the following steps: 1) For the input image, detect all objects and extract the object bounding boxes and categories. Calculate the similarity between objects using object categories and visual features. Divide the objects into multiple candidate groups based on the similarity and then pair the candidate groups to form candidate group pairs. 2) Extract individual characteristics, population characteristics, and population pair characteristics; 3) For each candidate group pair, the candidate group is either the subject group or the object group. Based on the visual and positional features of the object, calculate the cross-attention matrix of the object group to the subject group and the cross-attention matrix of the subject group to the object group. Based on the obtained attention matrix, encode the object features of the corresponding candidate group through cross-attention to predict the confidence of the object in the candidate group. 4) For each candidate group pair, weight the object visual features and position features with the object confidence predicted in step 3), and concatenate them with the candidate group visual features, position features, semantic features, and candidate group pair visual features to predict the predicate probability of the candidate group pair. 5) For each candidate group pair, combining steps 3) and 4), perform parallel predictions of the candidate groups based on the confidence level of the object belonging to the group and the predicate probability to generate visual group relationships. Step 1) is implemented in the entity construction module, step 2) is implemented in the feature export module, and steps 3)-5) are implemented in the group relationship prediction module.

3. The group visual relationship detection method based on parallel prediction of group and predicate as described in claim 2, characterized in that: In step 1), the method for calculating object similarity is as follows: in, and S is the normalized global position feature of object i and object j. ij It is the cosine similarity between object i and object j. S is the normalized cosine similarity between object i and object j. min It is the minimum value in the similarity matrix.

4. The group visual relationship detection method based on parallel prediction of groups and predicates according to claim 2, characterized in that: In step 3), the method for calculating the cross-attention matrix of the object candidate group for the subject candidate group is as follows: in, It is the object candidate group g o For the subject candidate group g s Attention matrix, It is the subject candidate group g s A matrix of joint visual and positional features of all objects in the matrix. It is the subject candidate group g o A matrix of joint visual and positional features of all objects in the matrix. yes The transpose of , where d is the number of objects after each candidate group is filled; The cross-attention matrix for the subject candidate group and the object candidate group can be obtained similarly; The calculation method for cross-attention encoding of object features in the subject candidate group is as follows: in, For matrix multiplication, η is the normalization operation, τ is the activation function, and ι is the linear transformation. It is the encoded feature of all objects in the subject candidate group after cross-attention encoding; Similarly, the object features of the candidate object group can be obtained through cross-attention encoding. The confidence γ of the object i in the candidate group g i|g The calculation method is: gamma i|g = sigmoid(mu(f i|g )) where f" (x) = f'(x) - f'(0) and f'(x) = f(x) - f(0) i|g denotes the encoded feature of object i in candidate group g, γ i|g is the confidence that object i belongs to candidate group g, and μ is a linear transformation that reduces the dimensionality to 1.

5. The group visual relationship detection method based on parallel prediction of groups and predicates according to claim 4, characterized in that: In step 3), when training the group relationship prediction module, for the object confidence score, the group range prediction loss is calculated based on the confidence score that the object belongs to the group. in, Let p be the set of positive samples from the population, and g be the candidate population in the candidate population pair p. Let g represent the set of objects in the candidate group. These are objects in the actual annotations. It is the group in the real annotation, ξ is the cross-entropy loss, and |·| represents the modulus of the set. Belongs to a real group It is 1 if it is true, otherwise it is 0.

6. The group visual relationship detection method based on parallel prediction of group and predicate as described in claim 4, characterized in that: In step 4), the method for calculating the object confidence weighted visual features is as follows: in, The visual features generated by the feature export module represent the same as the calculation of the object confidence weighted object position features.

7. The group visual relationship detection method based on parallel prediction of groups and predicates according to claim 2, characterized in that: In step 4), when training the group relationship prediction module, the loss function for predicting predicate probability is: Where P is the set of all candidate population pairs, β and Let β represent the predicted predicate probability vector and the true predicate probability vector for a candidate group, respectively. k It is the kth element of β.

8. The group visual relationship detection method based on parallel prediction of group and predicate as described in claim 2, characterized in that: In step 5), the method for generating group visual relations is as follows: based on the confidence of the object in the candidate group calculated in step 3), remove objects in the candidate group with a confidence of less than 0.

5. For objects that are in both the subject candidate group and the object candidate group, retain the objects with higher confidence and remove the objects with lower confidence. The final score of the group visual relations is the probability of the predicate.

9. The group visual relationship detection method based on parallel prediction of group and predicate as described in claim 2, characterized in that: In step 5), the total loss of the group prediction module is the group range prediction loss calculated based on the confidence level that an object belongs to the group. With predicate probability prediction loss sum:

Images