A visual relationship detection method based on relationship label hierarchy

Abstract

The present invention belongs to the field of computer vision, and specifically relates to a visual relationship detection method based on a relationship label hierarchy. The method comprises the following steps: Step 1, obtaining a probability distribution of relationship examples; Step 2, constructing a sub-hierarchy; Step 3, aggregating the sub-hierarchy; Step 4, constructing a loss function based on the relationship label hierarchy; Step 5, obtaining visual features; Step 6, obtaining relative position features; Step 7, obtaining language features; and Step 8, performing feature fusion for relationship prediction. Visual relationship detection based on a relationship label hierarchy is a key technology for visual understanding and reasoning, which further detects the relationships between objects based on object recognition. It is used to improve computers' ability to understand images.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a visual relationship detection method based on a relationship label hierarchy. Background Art

[0002] Visual relationship detection plays an important role in image understanding, such as image captioning and visual question answering. Its specific task is to detect and locate objects in an image and predict the relationships between them, typically using subject-relation-object triples, such as "person-walk-on-street."

[0003] Currently, classic models for visual relationship detection all employ an object-relationship detection framework. The object detection module detects and locates objects in an image, while the relationship detection module extracts visual, label, and positional features of the objects. By fusing these three modalities, the relationship between pairs of objects is predicted. Existing methods all build on this object-relationship detection framework. While they have achieved some success in visual relationship detection, the long-tail distribution of relationship samples hinders low-sample relationship detection results. To address this issue, some studies have proposed leveraging prior knowledge of relationships to improve the ability to learn relationships and the accuracy of relationship prediction in low-shot learning. For example, they use the semantic word vectors of objects to predict the prior probability of relationships. This prior probability is then used to fine-tune the relationship prediction of the visual model, thereby improving the accuracy of tail relationship prediction. Alternatively, they have collected a large amount of text data from an external knowledge base like Wikipedia and, combined with internal knowledge of text triples, calculated the prior probability distribution of visual relationship triplets between pairs of objects to improve relationship prediction accuracy.

[0004] The aforementioned methods all focus on the prior probability of relationships to improve relationship detection efficiency. However, these methods require additional collection and processing of relationship text data and ignore the visual similarity between relationships, leading to tail relationships with a small number of samples being mistakenly predicted as head relationships with a large number of samples. Due to the visual similarity between the tail relationship "walk on" and the head relationship "on," as well as the fact that head relationships have abundant training samples while tail relationships have only a small number of training samples, these methods incorrectly predict the relationship in the image as "person-on-street." Furthermore, further analysis of the prediction results of these methods reveals that they are not very effective in distinguishing tail relationships such as "walk on," "stand on," and "across."

[0005] To address the above issues, the present invention first divides relationship labels into coarse-grained and fine-grained categories. Coarse-grained relationships are head relationships containing rich samples, such as "on," "next to," and "above." Fine-grained relationships are other relationships within the same concept as the coarse-grained relationship. Fine-grained relationships typically have fewer samples and richer information. For example, compared to the coarse-grained relationship "on," fine-grained relationships include "stand on," "walk on," and "park on." Then, based on the visual similarity between the coarse-grained and fine-grained relationships and the inherent bias of the data, the coarse-grained and fine-grained relationship labels are represented in a hierarchical structure, and a visual relationship detection method based on the relationship label hierarchy is proposed. Summary of the Invention

[0006] In order to improve the detection effect of tail relations, the present invention divides the relationship labels into coarse and fine granularity to construct a hierarchical representation of the labels, and proposes a visual relationship detection model based on the relationship label hierarchy. The model uses the similarity between visual relationships and the bias of the data to construct a hierarchical representation of the relationship labels, thereby distinguishing the relationships into coarse-grained relationships and fine-grained relationships, so that the tail relations receive more attention in the structure from coarse-grained to fine-grained. At the same time, the loss function is designed according to the properties of the label hierarchy. The loss function learns the differences between different categories of relationships layer by layer through structured information, so that the model can better detect the tail fine-grained relationships.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A visual relationship detection method based on a relationship label hierarchy comprises the following steps:

[0009] Step 1, obtain the probability distribution of relationship samples;

[0010] Step 2, sub-hierarchy construction;

[0011] Step 3, sub-hierarchy aggregation;

[0012] Step 4: Construct a loss function based on the relationship label hierarchy;

[0013] Step 5: Obtain visual features;

[0014] Step 6, obtaining relative position features;

[0015] Step 7: Obtain language features;

[0016] Step 8: Feature fusion for relationship prediction.

[0017] Furthermore, the probability distribution of the relationship samples obtained in step 1 is specifically:

[0018] For all relationship classes c i The sample, using the bias of the visual relationship detection model itself, calculates each relationship class c i The label probability distribution after relationship prediction of all relationship samples is:

[0019]

[0020] Among them, n i Represents the relationship class c i The number of samples, m k Represents the relationship class c i The sample prediction of is the number of the k-th relationship.

[0021] Due to the long-tail distribution of relationship samples and the visual similarity between some tail relationships and head relationships, the probability distribution of tail relationship samples is often head relationships.

[0022] Furthermore, the process of constructing the sub-hierarchy in step 2 is to divide the relationships into coarse-grained relationships and fine-grained relationships.

[0023] For relationship class c i For all samples, the highest relationship class c in the label probability distribution of relationship prediction j As a relationship class c i The conceptual relationship of the aggregated categories, if the relationship class c i The relationship class with the highest label probability predicted by all sample relationships is also c i , then the relationship class c i The conceptual relationship of a large category is to itself.

[0024] Furthermore, the sub-hierarchy aggregation in step 3 is specifically as follows: each layer summarizes the relationships into relationships that are coarser than the relationships in the next layer, wherein the virtual nodes do not represent any relationship labels but are only a vague concept.

[0025] Furthermore, the loss function formula of the relationship label hierarchy in step 4 is:

[0026] L=λLP+CB sigmoid

[0027] Among them, λ is the balance weight, LP represents the loss based on the label path, and CB sigmoid represents class-balanced sigmoid loss;

[0028] For the label path-based loss, the correct path L in the label hierarchy is used path Tracking the path from the virtual root node to the relationship node, the correct path of the label structure represents the correct classification path for a training example in the label hierarchy. For a given training example with a correct path Lpath For the sample, we calculate the cross entropy loss of each layer of the label hierarchy and the path loss of the correct sample, and get the loss (LP) formula based on the label path:

[0029]

[0030] Among them, B(i) represents the brother node of node i, and the path probability Correct label D label , the probability formula for each node is:

[0031]

[0032] Among them, for a sample of a given relation class r, the probability predicted by the biased visual relation detection model is p r , L(i) represents all leaf nodes of node i, leaf(i) = T means node i is a leaf node, class(i) = r means node i is of relationship class r, leaf(i) = F means node i is not a leaf node;

[0033] The probability calculation formula for each node i’s child node j∈C(i) is:

[0034] P(j|i)=Soft max(Z i )[j];

[0035] A leaf node represents a relationship class r, and the path from the root node to the leaf node of class r is L path , it is known that node i belongs to path L path , traverse to the next node C r (i)∈L path ∩C(i), probability is expressed as P L (C r (i)|i), finally, the path probability formula for each relationship class r is:

[0036]

[0037] LP loss enables the network to transcend the noise of relationships between major categories, separate the classes represented by leaf nodes, and separate each internal node in the hierarchy to learn the embedding of specific major categories, thereby producing more fine-grained predictions.

[0038] The class balance weight represents the balance weight of each node in the label hierarchy. The weight factor is used to calculate the class balance weight of each node, and the class balance weight of each node is uniformly defined as follows:

[0039]

[0040] Where the hyperparameter β∈[0,1);

[0041] Class balance loss is used to solve the problem of training in imbalanced data. For a given example, the correct relation r is based on the biased prediction probability P. pre The formula for calculating the class-balanced sigmoid loss function is:

[0042]

[0043] Among them, w i is the weight factor of relationship class i.

[0044] Furthermore, the visual features obtained in step 5 are specifically obtained by passing the image through the convolutional neural network with Faster-RCNN as the backbone network to obtain the feature map of the entire image, and then giving a bunch of bounding boxes B of the detection objects s ​​and o s and B o , and the minimum bounding box containing the two objects, ROI Pooling is used to obtain the feature map of the two objects and the relationship between the two objects, and the feature map of the objects and the relationship is passed through the convolution layer and the fully connected layer to obtain the visual features of the relationship, and then the visual features of the objects are fused through the fully connected layer to form the final visual features. The visual features mainly include the visual features of the subject s and the object o, as well as the visual features of the relationship, which is expressed as R viso (s, o).

[0045] Furthermore, the relative position feature in step 6 represents the relative position between two objects in the relationship. The relative position feature is expressed as shown in the formula:

[0046]

[0047] in, i=s, o, u represent the bounding box coordinates of the subject and object, and the joint area of ​​subject and object respectively. u and H u Represent the width and height of the bounding box of the joint area of ​​the subject and object respectively.

[0048] Furthermore, the language features obtained in step 7 are specifically:

[0049] In the visual relationship detection model, word vectors are used as the linguistic features of objects. First, the object category is obtained from the object detection module. Then, external linguistic features are extracted and pre-trained word2vec is used to obtain the semantic representation of the subject s and object o in the object pair. The linguistic features are expressed as:

[0050] R l (s, o) = [word2vec(s), word2vec(o)].

[0051] Furthermore, the feature fusion in step 8 for relationship prediction is specifically as follows: the features of each modality are converted to the same dimension, and then spliced ​​into multimodal features. The representation of feature fusion is expressed as:

[0052] R r =R vis ·f(R l ,θ1)·f(R loc ,θ2)

[0053] Among them, R r is the representation after fusion of multimodal features, θ1 and θ2 represent the learning weights of the fully connected layer, and the fused features are used to make the final relationship prediction through the fully connected network.

[0054] This alleviates the problem that high-dimensional features (such as 4096-dimensional visual features) can easily mask low-dimensional features (such as 8-dimensional position features) due to the large difference in dimensions of features of different modalities.

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] Visual relationship detection, as described in this invention, is a crucial task in computer vision and plays a crucial supporting role in downstream tasks such as visual common sense generation, visual question answering, and image description. Previous techniques for detecting visual relationships in images were limited to a few coarse-grained head relationships, such as "on," "next to," and "near." While these techniques detected some relationships, they contained relatively little information, providing little benefit for downstream tasks. The present invention, however, can detect more fine-grained relationships that contain rich information. For example, while previous techniques could only predict the relationship in an image as "person-on-street," the present invention can predict relationships in an image as "person-walk on-street" or "person-stand on-street," which contain clear information. These rich and clear relationships can significantly benefit downstream tasks such as visual question answering, visual common sense generation, and image description. They can, to a certain extent, address the long-tail problem of uneven sample distribution in downstream tasks. Furthermore, through continuous improvement, visual relationship detection can detect a more comprehensive relationship system, thereby improving machines' ability to understand images. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a visual relationship detection model diagram based on the relationship label hierarchy;

[0058] Figure 2 It is a diagram of the construction process of the relationship label hierarchy;

[0059] Figure 3 It is a comparison of the effects of different λ and β in the loss function;

[0060] Figure 4 It is the prediction result of some relation samples of RLM and VRDMLH on VRD;

[0061] Figure 5 This is the performance comparison between VRDMLH and RLM. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below in conjunction with specific embodiments (but not limited to the embodiments) and the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0063] The research method of the present invention is described in further detail below:

[0064] A visual relationship detection method based on a relationship label hierarchy comprises the following steps:

[0065] Step 1, obtain the probability distribution of relationship samples;

[0066] Step 2, sub-hierarchy construction;

[0067] Step 3, sub-hierarchy aggregation;

[0068] Step 4: Construct a loss function based on the relationship label hierarchy;

[0069] Step 5: Obtain visual features;

[0070] Step 6, obtaining relative position features;

[0071] Step 7: Obtain language features;

[0072] Step 8: Feature fusion for relationship prediction.

[0073] Furthermore, the probability distribution of the relationship samples obtained in step 1 is specifically:

[0074] For all relationship classes c i The sample, using the bias of the visual relationship detection model itself, calculates each relationship class c i The label probability distribution after relationship prediction of all relationship samples is:

[0075]

[0076] Among them, ni Represents the relationship class c i The number of samples, m k Represents the relationship class c i The sample prediction of is the number of k-th relations;

[0077] Due to the long-tail distribution of relationship samples and the visual similarity between some tail relationships and head relationships, the probability distribution of tail relationship samples is often head relationships.

[0078] Furthermore, the process of constructing the sub-hierarchy in step 2 is to divide the relationships into coarse-grained relationships and fine-grained relationships.

[0079] For relationship class c i For all samples, the highest relationship class c in the label probability distribution of relationship prediction j As a relationship class c i The conceptual relationship of the aggregated categories, if the relationship class c i The relationship class with the highest label probability predicted by all sample relationships is also c i , then the relationship class c i The conceptual relationship of a large category is to itself.

[0080] Furthermore, the sub-hierarchy aggregation in step 3 is specifically as follows: each layer summarizes the relationships into relationships that are coarser than the relationships in the next layer, wherein the virtual nodes do not represent any relationship labels but are only a vague concept.

[0081] Furthermore, the loss function formula of the relationship label hierarchy in step 4 is:

[0082] L=λLP+CB sigmoid

[0083] Among them, λ is the balance weight, LP represents the loss based on the label path, and CB sigmoid represents class-balanced sigmoid loss;

[0084] For the label path-based loss, the correct path L in the label hierarchy is used path Tracking the path from the virtual root node to the relationship node, the correct path of the label structure represents the correct classification path for a training example in the label hierarchy. For a given training example with a correct path L path For the sample, we calculate the cross entropy loss of each layer of the label hierarchy and the path loss of the correct sample, and get the loss (LP) formula based on the label path:

[0085]

[0086] Among them, B(i) represents the brother node of node i, and the path probability Correct label Dlabel , the probability formula for each node is:

[0087]

[0088] Among them, for a sample of a given relation class r, the probability predicted by the biased visual relation detection model is p r , L(i) represents all leaf nodes of node i, leaf(i) = T means node i is a leaf node, class(i) = r means node i is of relationship class r, leaf(i) = F means node i is not a leaf node;

[0089] The probability calculation formula for each node i’s child node j∈C(i) is:

[0090] P(j|i)=Soft max(Z i )[j];

[0091] A leaf node represents a relationship class r, and the path from the root node to the leaf node of class r is L path , it is known that node i belongs to path L path , traverse to the next node C r (i)∈L path ∩C(i), probability is expressed as P L (C r (i)|i), finally, the path probability formula for each relationship class r is:

[0092]

[0093] LP loss enables the network to transcend the noise of relationships between major categories, separate the classes represented by leaf nodes, and separate each internal node in the hierarchy to learn the embedding of specific major categories, thereby producing more fine-grained predictions.

[0094] The class balance weight represents the balance weight of each node in the label hierarchy. The weight factor is used to calculate the class balance weight of each node, and the class balance weight of each node is uniformly defined as follows:

[0095]

[0096] Where the hyperparameter β∈[0,1);

[0097] Class balance loss is used to solve the problem of training in imbalanced data. For a given example, the correct relation r is based on the biased prediction probability P. pre The formula for calculating the class-balanced sigmoid loss function is:

[0098]

[0099] Among them, w i is the weight factor of relationship class i.

[0100] Furthermore, the visual features obtained in step 5 are specifically obtained by passing the image through the convolutional neural network with Faster-RCNN as the backbone network to obtain the feature map of the entire image, and then giving a bunch of bounding boxes B of the detection objects s ​​and o s and B o , and the minimum bounding box containing the two objects, ROI Pooling is used to obtain the feature map of the two objects and the relationship between the two objects, and the feature map of the objects and the relationship is passed through the convolution layer and the fully connected layer to obtain the visual features of the relationship, and then the visual features of the objects are fused through the fully connected layer to form the final visual features. The visual features mainly include the visual features of the subject s and the object o, as well as the visual features of the relationship, which is expressed as R viso (s, o).

[0101] Furthermore, the relative position feature in step 6 represents the relative position between two objects in the relationship. The relative position feature is expressed as shown in the formula:

[0102]

[0103] in, i=s, o, u represent the bounding box coordinates of the subject and object, and the joint area of ​​subject and object respectively. u and H u Represent the width and height of the bounding box of the joint area of ​​the subject and object respectively.

[0104] Furthermore, the language features obtained in step 7 are specifically:

[0105] In the visual relationship detection model, word vectors are used as the linguistic features of objects. First, the object category is obtained from the object detection module. Then, external linguistic features are extracted and pre-trained word2vec is used to obtain the semantic representation of the subject s and object o in the object pair. The linguistic features are expressed as:

[0106] R l (s, o) = [word2vec(s), word2vec(o)].

[0107] Furthermore, the feature fusion in step 8 for relationship prediction is specifically as follows: the features of each modality are converted to the same dimension, and then spliced ​​into multimodal features. The representation of feature fusion is expressed as:

[0108] R r =R vis ·f(R l ,θ1)·f(R loc ,θ2)

[0109] Among them, R r is the representation after fusion of multimodal features, θ1 and θ2 represent the learning weights of the fully connected layer, and the fused features are used to make the final relationship prediction through the fully connected network.

[0110] This alleviates the problem that high-dimensional features (such as 4096-dimensional visual features) can easily mask low-dimensional features (such as 8-dimensional position features) due to the large difference in dimensions of features of different modalities.

[0111] like Figure 1 、 Figure 2 The method of the present invention is experimented on two public datasets, Visual Relationship Detection (VRD) and Visual Genome (VG). The VRD dataset consists of 5,000 images, 100 objects and 70 relationships. It contains a total of 37,993 pairs of relationship triplets. The dataset is divided into 4,000 training images and 1,000 test images. The VG dataset is one of the largest visual relationship detection datasets. The VG dataset has multiple versions. In the present invention, the VG dataset provided contains 150 common object classes and 50 common relationship classes. The method divides the VG dataset into 94,854 training images and 13,223 test images. At the same time, 5,000 images are sampled from the training set as a validation set.

[0112] The present invention uses three tasks for evaluating visual relationship detection, namely predicate detection, phrase detection and relation detection.

[0113] (1) In predicate detection, an image is input along with a set of object pairs with ground-truth bounding boxes and object labels. The model predicts the relationship between the object pairs. This metric evaluates the model's ability to predict relationships given an object pair.

[0114] (2) In phrase detection, a picture is input and the model outputs a triplet <object 1, relation, object 2>. If the intersection over union (IOU) of the bounding box of the entire located triplet (the joint bounding box of the object pair) and the bounding box of the true triplet is greater than 0.5, the prediction is correct.

[0115] (3) In relation detection, a picture is input and the model outputs a set of triples <object 1, relation, object 2> and the bounding boxes of object 1 and object 2. If the IOU of the output bounding boxes with the real bounding boxes of object 1 and object 2 is greater than 0.5, it means that the prediction is correct.

[0116] The present invention uses the average recall rate as the evaluation indicator.

[0117] Mean Recall@K (mR@K): This performance metric calculates the recall of each relation category independently and then averages the recall of all relation categories. Therefore, each relation category contributes equally. This evaluation metric reduces the influence of less informative head relations, such as "on" and "next to", and gives equal attention to less common tail relation categories, such as "walk on" and "stand on". This paper defines the mean recall as shown in the formula:

[0118]

[0119] Among them, τ p Recall@(p) represents the recall rate of relation p. The method is evaluated using mR@K with K = 20, 50, and 100.

[0120] This method is compared with two visual relationship detection models, namely MF-URLN and RLM, on the VRD dataset; and with three visual relationship detection models, namely MF-URLN, RLM and VRD-DM, on the VG dataset.

[0121] MF-URLN: This method proposes that uncertain relations have a certain impact on visual relationship detection and constructs an uncertain relationship learning network based on multimodal features. It has achieved significant improvements in visual relationship detection.

[0122] RLM: This method deeply mines and utilizes the relative positions of each stage of the object pair visual relationship detection process, extracts the relative position information of each object pair and encodes it as auxiliary features to improve the ability of object pair proposal and relationship detection.

[0123] VRD-DM: This method proposes to use depth maps to provide valuable information about object relationships. It fuses depth map features, RGB image features, label features, and position features for visual relationship detection.

[0124] To verify the effectiveness of the label hierarchy, this paper conducted three experiments: a comparative test with mainstream visual relationship detection methods, an ablation experiment, and a performance analysis. In the comparative experiment with existing methods, this paper tested the visual relationship detection model based on the relationship label hierarchy (VRDMLH) on the public datasets VRD and VG datasets. In the ablation experiment, first, to demonstrate that the combination of the label hierarchy loss and the class-balanced sigmoid loss has complementary advantages and enables the model to achieve optimal results, the visual relationship detection model was trained with different loss functions and experiments were conducted on the VRD dataset. Second, to demonstrate the effectiveness of the proposed label hierarchy construction method, different forms of hierarchical structures were experimented on VRD data and. Finally, experiments were conducted on VRD data and the values ​​of parameters λ and β. In the performance analysis, the detection effect of VRDMLH and the existing model RLM on coarse-grained tail relationships is demonstrated.

[0125] Table 1 Results on VRD dataset

[0126]

[0127] As shown in Table 1, on the VRD dataset, our proposed visual relationship detection method based on a relational label hierarchy outperforms existing models in average recall across predicate detection, phrase detection, and relation detection. In predicate detection, compared to the state-of-the-art model RLM, VRDMLH improves mR@20, mR@50, and mR@100 by 1.69%, 4.29%, and 5.35%, respectively. This demonstrates that, given an object pair, VRDMLH outperforms existing models in detecting fine-grained tail relations and can detect a wider variety of relations. In the relation detection task, VRDMLH achieves superior average recall compared to the state-of-the-art model RLM.

[0128] Table 2 Results on the VG dataset

[0129]

[0130]

[0131] Table 3 Detection results of different loss models on VRD dataset

[0132]

[0133] As shown in Table 2, on the VG dataset, our proposed visual relationship detection method based on a relational label hierarchy outperforms existing models in average recall for predicate detection, phrase detection, and relation detection. In predicate detection, compared to the state-of-the-art model VRD-DM, VRDMLH improves mR@20, mR@50, and mR@100 by 1.82%, 2.32%, and 2.51%, respectively. This demonstrates that, given an object pair, VRDMLH outperforms existing models in detecting fine-grained tail relations and can detect a wider variety of relations.

[0134] In order to verify that the combination of class-balanced loss based on label hierarchy and class-balanced sigmoid loss has complementary advantages and that the combination of the two can achieve the best effect, this paper studies the relationship detection effect of the visual relationship detection model using different loss functions on the VRD dataset. As shown in Table 3, LP represents the use of only class-balanced loss based on label hierarchy, CB represents the use of sigmoid Indicates that only class-balanced sigmoid loss is used, CB softmax "Full Loss" indicates the use of only the class-balanced softmax loss, while "Full Loss" indicates the use of both the class-balanced loss based on the label hierarchy and the class-balanced sigmoid loss. Experiments show that combining the class-balanced loss based on the label hierarchy and the class-balanced loss has complementary advantages, resulting in optimal model training results.

[0135] To further demonstrate that the relationship label hierarchy partially addresses the problem of relationship detection models only detecting coarse-grained relationships at the head due to the long-tail distribution of data, this paper experiments on the VRD dataset using a visual relationship detection model with a relationship label hierarchy (VRDMLH) and a visual relationship detection model without a relationship label hierarchy (VRDMNLH). As shown in Table 4, compared with VRDMNLH, VRDMLH improves mR@20, mR@50, and mR@100 by 3.08%, 4.58%, and 5.44%, respectively, in predicate detection. In phrase detection, VRDMLH improves mR@20, mR@50, and mR@100 by 2.00%, 2.98%, and 3.16%, respectively. In relationship detection, VRDMLH improves mR@20, mR@50, and mR@100 by 0.70%, 2.08%, and 2.46%, respectively. Experiments show that the label relationship hierarchy can effectively enable the model to distinguish coarse-grained relationships from fine-grained relationships, solving the problem of the model's incorrect prediction of fine-grained relationship samples.

[0136] Table 4 Results of different construction methods of label hierarchy on VRD dataset

[0137]

[0138] At the same time, in order to verify the effectiveness of constructing relationship categories and distinguishing coarse-grained and fine-grained relationships in the label hierarchy constructed in this paper, this paper provides a construction method for a hierarchy that fuses all subtrees, that is, the second layer of the label hierarchy is all relationships, and a construction method that fuses coarse-grained and fine-grained relationships, that is, the third and fourth layers of the label hierarchy are fused. As shown in Table 4, FuseSubtree represents a construction method for a hierarchy that fuses all subtrees, and FuseLayer represents a construction method that fuses coarse-grained and fine-grained relationships. The label hierarchy constructed by utilizing the bias of the visual relationship detection model proposed in this paper is mainly based on organizing relationships belonging to the same category and dividing the relationships of a sub-hierarchy into different levels from coarse-grained to fine-grained. From Table 4, it can be seen that when the regulations for constructing a hierarchy are violated, the detection effect of the model will drop significantly.

[0139] like Figure 3 As shown in the figure, when β takes a fixed value, the model effect is best when λ=1; when λ takes a fixed value, the model effect is best when β=0.999. It is obvious from the figure that when λ=1 and β=0.999, the model indicators achieve the best performance.

[0140] In order to demonstrate the detection effect of the visual relationship detection model based on the relationship label hierarchy on the tail fine-grained relationship, such as Figure 4 As shown, Figure 4Figure 2 shows the detection performance of RLM and VRDMLH on samples with tail fine-grained relations in the VRD dataset. Black represents the RLM model, and gray represents the VRDMLH model. It is clear that RLM is unable to detect some tail fine-grained relations. For example, its detection performance is zero on samples with "sleep on," "lying on," and "drive," while VRDMLH can successfully detect some samples with tail fine-grained relations. Furthermore, VRDMLH shows improvement on some tail relations that RLM can detect, such as "hold," "ride," and "fly." However, VRDMLH's detection performance declines on some samples with head coarse-grained relations. This is primarily because VRDMLH trains its model based on a label hierarchy. When a large category has many fine-grained relations, the calculation method of virtual node probabilities in the hierarchy causes the model to prioritize detecting fine-grained relations over coarse-grained ones. Detecting coarse-grained relationship samples as fine-grained relationships is simply a mismatch between the detection result and the correct one, and does not indicate that the visual relationship is completely incorrect. RLM, on the other hand, improves head relationship detection by predicting more head relationships using a relative position estimation module. While this improves detection, detection of tail relationships remains suboptimal. Overall, RLM can detect more relationships in coarse-grained head samples, while VRDMLH can more evenly detect various relationships in the image.

[0141] In addition, if Figure 5 As shown, Figure 5 The performance of VRDMLH is also demonstrated. In relation detection, RLM cannot detect tail fine-grained relations such as "car-park on-street" and "person-stand on-street." Furthermore, RLM often predicts coarse-grained head relations for object pairs, such as "on," "next to," and "wear." VRDMLH, on the other hand, has better detection results for tail relation samples. It can predict fine-grained relations such as "car-park on-street" and "car-behind-car." Furthermore, VRDMLH's predictions for certain object pairs, such as "Car-has-Wheel," are often more logical.

[0142] Any matters not described in detail in this specification are prior art known to those skilled in the art. Although the above description of the present invention is based on specific embodiments to facilitate understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the specific embodiments. As long as various modifications are within the spirit and scope of the present invention as defined and determined by the appended claims, such modifications will be obvious to those skilled in the art, and all inventions and creations utilizing the concepts of the present invention are protected.

Claims

1. A visual relationship detection method based on a relationship label hierarchy, characterized in that: The following steps are included: Step 1, obtain the probability distribution of relationship samples; Step 2, sub-hierarchy construction; Step 3, sub-hierarchy aggregation; Step 4: Construct a loss function based on the relationship label hierarchy; Step 5: Obtain visual features; Step 6, obtaining relative position features; Step 7: Obtain language features; Step 8: Feature fusion for relationship prediction; The process of constructing the sub-hierarchy in step 2 is to distinguish the relationships into coarse-grained relationships and fine-grained relationships; The sub-hierarchy aggregation in step 3 is specifically as follows: each layer summarizes the relationships into relationships that are coarser than the relationships in the next layer, where the virtual nodes do not represent any relationship labels but are just a vague concept; The loss function formula of the relationship label hierarchy in step 4 is: L=λLP+CB sigmoid Among them, λ is the balance weight, LP represents the loss based on the label path, and CB sigmoid Represents class balance sigmoid loss; For the label path-based loss, the correct path L in the label hierarchy is used path Tracking the path from the virtual root node to the relationship node, the correct path of the label structure represents the correct classification path for a training example in the label hierarchy. For a given training example with a correct path L path For the sample, we calculate the cross entropy loss of each layer of the label hierarchy and the path loss of the correct sample, and get the loss (LP) formula based on the label path: Among them, B(i) represents the brother node of node i, and the path probability Correct label D lobel , the probability formula for each node is: Among them, for a sample of a given relation class r, the probability predicted by the biased visual relation detection model is p r , L(i) represents all leaf nodes of node i, leaf(i) = T means node i is a leaf node, class(i) = r means node i is of relationship class r, leaf(i) = F means node i is not a leaf node; The probability calculation formula for each node i’s child node j∈C(i) is: [(j|i)=Softmax(Z i )[j]: A leaf node represents a relationship class r, and the path from the root node to the leaf node of class r is L path , it is known that node i belongs to path L path , traverse to the next node C r (i)∈L path ∩C(i), probability is expressed as P L (C r (i)|i), finally, the path probability formula for each relationship class r is: The class balance weight represents the balance weight of each node in the label hierarchy. The weight factor is used to calculate the class balance weight of each node, and the class balance weight of each node is uniformly defined as follows: Where the hyperparameter β∈[0,1); For a given example, the correct relation r is based on the biased prediction probability P pre The formula for calculating the class-balanced sigmoid loss function is: Among them, w i is the weight factor of relationship class i.

2. The visual relationship detection method based on relationship label hierarchy according to claim 1, characterized in that: The probability distribution of the relationship samples obtained in step 1 is specifically: For all relationship classes c i The sample, using the bias of the visual relationship detection model itself, calculates each relationship class c i The label probability distribution after relationship prediction of all relationship samples is: Among them, n i Represents the relationship class c i The number of samples, m k Represents the relationship class c i The sample prediction of is the number of the k-th relationship.

3. The visual relationship detection method based on relationship label hierarchy according to claim 1, characterized in that: The visual features obtained in step 5 are specifically obtained by passing the image through the convolutional neural network with Faster-RCNN as the backbone network to obtain the feature map of the entire image, and then giving a bunch of bounding boxes B of the detection objects s, o s and B o , and the minimum bounding box containing the two objects, ROIPooling is used to obtain the feature map of the two objects and the relationship between the two objects, and the feature map of the objects and the relationship is passed through the convolution layer and the fully connected layer to obtain the visual features of the relationship, and then the visual features of the objects are fused through the fully connected layer to form the final visual features. The visual features mainly include the visual features of the subject s and the object o and the visual features of the relationship, which are expressed as R viso (s, o).

4. The visual relationship detection method based on relationship label hierarchy according to claim 1, characterized in that: The relative position feature in step 6 represents the relative position between two objects in the relationship. The relative position feature is expressed as shown in the formula: in, Represent the bounding box coordinates of the subject and object, as well as the joint area of ​​subject and object respectively; W u and H u Represent the width and height of the bounding box of the joint area of ​​the subject and object respectively.

5. The visual relationship detection method based on relationship label hierarchy according to claim 1, characterized in that: The language features obtained in step 7 are specifically as follows: In the visual relationship detection model, word vectors are used as the linguistic features of objects. First, the object category is obtained from the object detection module. Then, external linguistic features are extracted and pre-trained word2vec is used to obtain the semantic representation of the subject s and object o in the object pair. The linguistic features are expressed as: R l (s,o)=[word2vec(s),word2vec(o)]。 6. The visual relationship detection method based on relationship label hierarchy according to claim 1, characterized in that: The feature fusion for relationship prediction in step 8 is specifically as follows: the features of each modality are converted to the same dimension, and then spliced ​​into multimodal features. The representation of feature fusion is expressed as: R r =R vis ·f(R l ,θ1)·f(R loc ,θ2) Among them, R r is the representation after fusion of multimodal features, θ1 and θ2 represent the learning weights of the fully connected layer, and the fused features are used to make the final relationship prediction through the fully connected network.

Images