Method for enhancing geometric dimension and semantic dimension in unbiased scene graph generation

Abstract

The present application relates to the technical field of computer vision, and in particular to a method for generating an unbiased scene graph by enhancing geometric dimensions and semantic dimensions, comprising: constructing a predicate class tree structure; inputting a data set picture into a target detection model and outputting a processing result; inputting the processing result into a semantic module and a geometric module and outputting new joint region features; defining a loss function to obtain a designed class balance swing loss function; inputting the new joint region features into a transformer model, using the designed class balance swing loss function to calculate the probability value of each predicate class in the object to obtain a first probability value; substituting the probability value into the predicate class tree structure, using a tree-based class balance swing loss function to calculate the probability value to obtain a second probability value; weighting and summing the first probability value and the second probability value to obtain a final probability value; and outputting a triple corresponding to the picture based on the final probability value, thereby improving the problem of long-tail distribution of the data set hindering deep learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a method for enhancing geometric dimension and semantic dimension in unbiased scene graph generation. BACKGROUND

[0002] The field of computer vision has been working hard to enable machines to understand images, videos or other forms of content as humans do, and the foundation of this field is cognitive tasks such as image classification and instance detection. Tasks requiring higher-level semantic understanding have gradually attracted attention in recent years with the rapid development of visual cognition and natural language processing. These tasks combine computer vision and natural language processing fields, such as image description generation and image question answering. However, the above basic visual cognitive tasks treat instances in visual information in isolation, ignoring the visual relationship information between instances, so these tasks cannot fully understand visual content. The field of scene graph generation (SGG) generates triples <subject-predicate-object> by analyzing the relationship between object pairs, which builds rich semantic information for the above tasks.

[0003] In the natural world, most of the relationships in the dataset are only distributed in a few categories, while other relationships are scattered in most categories. This is called class imbalance, i.e. the long tail problem. For example, the use of the predicate "on" will be much more than the use of "parkon". Due to this common natural phenomenon, deep learning faces great obstacles, thereby reducing the effect of scene graph generation. SUMMARY

[0004] The present application aims to provide a method for enhancing geometric dimension and semantic dimension in unbiased scene graph generation, which aims to improve the problem that the long tail distribution of the dataset hinders deep learning.

[0005] To achieve the above-mentioned purpose, the present application provides a method for enhancing geometric dimension and semantic dimension in unbiased scene graph generation, comprising the following steps:

[0006] Constructing a predicate class tree structure;

[0007] Inputting the dataset picture into the target detection model and outputting the processing result;

[0008] Inputting the processing result into the semantic module and the geometric module to output new joint region features;

[0009] Defining a loss function to obtain a designed class balance swing loss function;

[0010] inputting the new joint region feature into the transformer model, calculating a probability value of each predicate class in the object using a designed class-balanced swing loss function to obtain a first probability value;

[0011] substituting the probability value into the predicate class tree structure, calculating a probability value using a tree-based class-balanced swing loss function to obtain a second probability value;

[0012] weighting and summing the first probability value and the second probability value to obtain a final probability value;

[0013] outputting a triple corresponding to the picture based on the final probability value.

[0014] The target detection model is a Faster R-CNN model.

[0015] The transformer model is a Transformer model.

[0016] The inputting the processing result into the semantic module and the geometric module to output a new joint region feature comprises:

[0017] obtaining a word embedding vector in the processing result and position information of the object;

[0018] inputting the word embedding vector into the semantic module for processing to obtain a first output result;

[0019] inputting the position information of the object into the geometric module for processing to obtain a second output result;

[0020] performing a full connection operation on the first output result, the second output result and the joint region feature to obtain a connection matrix;

[0021] processing the connection matrix through a linear rectifier activation function, a random deactivation layer and a linear layer to obtain a new joint region feature.

[0022] The inputting the word embedding vector into the semantic module for processing to obtain a first output result comprises:

[0023] inputting the word embedding vector into the semantic module;

[0024] The semantic module performs a dot product operation on the word embedding vector to obtain an operation result;

[0025] The semantic module performs adaptive average pooling on the operation result after using a 5x5 convolution, a linear rectifier activation function, average pooling and two 3x3 convolutions to obtain a pooling result;

[0026] The semantic module performs two 1x1 convolutions on the pooling result to obtain a first output result.

[0027] The position information of the object is input into a geometry module for processing to obtain a second output result, comprising:

[0028] The position information of the object is input into a geometry module.

[0029] The geometry module obtains the overlap ratio between the object and adjacent objects in each picture based on the position information of the object, and obtains each object and the object pair with the highest overlap ratio to obtain the second output result.

[0030] The method for enhancing geometric dimension and semantic dimension in unbiased scene graph generation provided by the application comprises the following steps: constructing a predicate class tree structure; inputting a data set picture into a target detection model to output a processing result; inputting the processing result into a semantic module and a geometry module to output a new joint region feature; defining a loss function to obtain a designed class balance swing loss function; inputting the new joint region feature into a transformer model, using the designed class balance swing loss function to calculate the probability value of each predicate class in the object to obtain a first probability value; substituting the probability value into the predicate class tree structure, using a tree-based class balance swing loss function to calculate the probability value to obtain a second probability value; weighting and summing the first probability value and the second probability value to obtain a final probability value; and outputting a triple corresponding to the picture based on the final probability value. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0032] Figure 1 is a flowchart of the method for enhancing geometric dimension and semantic dimension in unbiased scene graph generation provided by the application.

[0033] Figure 2 is a method structure diagram of the method for enhancing geometric dimension and semantic dimension in unbiased scene graph generation provided by the application.

[0034] Figure 3 is a structure diagram of the geometry module and the semantic module.

[0035] Figure 4 is a structure diagram of the geometry module.

[0036] Figure 5 is a structure diagram of the semantic module.

[0037] Figure 6 is a loss function graph. DETAILED DESCRIPTION

[0038] Embodiments of the present application are described below in detail with reference to examples illustrated in the accompanying drawings, in which like or similar elements or components are denoted throughout by like reference numerals, and of which embodiments are not limited, which are described by way of example with reference to the drawings.

[0039] Referring to Figures 1 to 6 The present application provides a method for enhancing geometric dimension and semantic dimension in unbiased scene graph generation, comprising the following steps:

[0040] S1, constructing a predicate class tree structure;

[0041] S2, inputting a data set picture into a target detection model, and outputting a processing result;

[0042] Specifically, the target detection model is a Faster R-CNN model. The Faster R-CNN model can obtain object-related information. However, this model uses an existing model (a pre-trained model).

[0043] S3, inputting the processing result into a semantic module and a geometric module to output a new joint region feature;

[0044] The specific manner is:

[0045] S31, obtaining a word embedding vector and position information of an object in the processing result;

[0046] S32, inputting the word embedding vector into the semantic module for processing to obtain a first output result;

[0047] Specifically, the semantic module performs a dot product operation on the word embedding vector to obtain an operation result; the semantic module performs 5x5 convolution, linear rectifier activation function, average pooling, and two 3x3 convolutions on the operation result, and then performs adaptive average pooling to obtain a pooling result; the semantic module performs two 1x1 convolutions on the pooling result to obtain the first output result.

[0048] S33, inputting the position information of the object into the geometric module for processing to obtain a second output result;

[0049] Specifically, the position information of the object is input into the geometric module; the geometric module obtains the overlap ratio between objects and adjacent objects in each picture based on the position information of the object, and obtains each object and the object pair with the highest overlap ratio to obtain the second output result.

[0050] The coincidence ratio s can be 0.80.

[0051] Initial joint region feature vector Corresponding to the object pair information in each batch of pictures. The improved geometric feature module first determines the joint region feature of the object pair in each picture Where k represents the number of object pairs in each picture. Second, we calculate the position information of the joint region between the object pairs in U to obtain the joint region feature position matrix Third, we calculate the IOU value of the joint region position The input of this step is C_Uij and Ui, where i represents the current object pair and j represents another object pair in the same image.

[0052] Matrix F ij IOU is stored in F ij The maximum value of each i. We take F ij , generate The formula is shown in equation 3.2. Since the maximum value of IOU is mostly between 0.75 and 0.95, we set s to 0.80.

[0053]

[0054] If only the geometric dimension enhancement module is used, the matrix F and M are fully connected to obtain new joint region features As shown in equation 3.3.

[0055] F' = G drop (σ(G FC (G drop (σ(G FC (F, M))))), (3.3)

[0056] F' improved joint feature, we will use it as the input of the Transformer model to ensure that the model obtains rich geometric features.

[0057] S34 fully connects the first output result, the second output result and the joint region feature to obtain a connection matrix;

[0058] S35 processes the connection matrix through a linear rectification activation function, a random inactivation layer and a linear layer to obtain new joint region features.

[0059] S4 defines a loss function to obtain a designed class-balanced swing loss function;

[0060] Specifically, a class balanced loss function (CBLoss) is used to solve the influence of the long-tail problem of data on the image segmentation task. For the long-tail distribution problem of the data set, resampling and reweighting methods are often used. The class balanced loss function is a weighting strategy based on the number of effective samples of each class, which aims to solve the training problem of unbalanced data by introducing a weighting factor inversely proportional to the number of effective samples.

[0061] However, the class balanced loss function ignores a problem: when the predicted relationship is inconsistent with the true value label, the tail relationship that is predicted wrong is not always misclassified as the head relationship. To address this situation, we define a new loss function, the class balanced swing loss function. When the true value is T i , the predicted probability value is T j , and T i ≠ T j , the loss function weight W gi is as shown in equation (3.8).

[0062]

[0063] where n i represents the number of leaf nodes in the subtree, and the hyperparameter β is set to 0.999. q is set to 3.00.

[0064] The class balanced swing loss function is defined as shown in equation (3.9). Where P pred is the predicted probability value, g i is the true value label of node i. The loss function contains a normalized exponential function and a weight value W gi .

[0065]

[0066] S5 inputs the new joint region feature into the transformer model, calculates the probability value of each predicate class in the object using the designed class balanced swing loss function, and obtains a first probability value;

[0067] Specifically, the transformer model is a Transformer model.

[0068] S6 substitutes the probability value into the predicate class tree structure and calculates the probability value using a tree-based class balanced swing loss function, and obtains a second probability value;

[0069] Specifically, the tree-based class balanced loss function is:

[0070] The tree-based class balanced loss function is a loss function constructed for leaf nodes, and the weight is as shown in equation 3.8, and the overall loss function is as shown in equation (3.10).

[0071]

[0072] where K represents the number of nodes outside the initial leaf node, W sk The calculation is shown in formula (3.8). The input of the Softmax function is the probability value of the parent node that is, the average value of the sum of the probabilities of all leaf nodes of the node. The probability value of the leaf node is calculated by the class-balanced swing loss function.

[0073] The total loss function is composed of the class-balanced swing loss function and the tree-based class-balanced loss function, as shown in formula (3.11). The hyperparameter λ is set to 0.999.

[0074]

[0075] S7 weights and sums the first probability value and the second probability value to obtain a final probability value.

[0076] S8 outputs a triple corresponding to the picture based on the final probability value.

[0077] Specifically, the composition of the triple is <subject-predicate-object>. The subject and the object are judged according to the object category predicted by the Faster R-CNN model, and the method mainly judges the predicate (also referred to as the relationship).

[0078] The picture of the data set is not input into the model at one time, but is input into the model round by round according to the number of pictures that can be calculated and processed by the performance of the graphics card (4 in this experiment). The data set contains a total of 108077 pictures.

[0079] The method for enhancing geometric dimension and semantic dimension in unbiased scene graph generation provided by the application can further alleviate the problem caused by the long-tail distribution of the data set on the basis of the original method. The experimental results show that the application can effectively suppress the influence caused by the serious imbalance of the long-tail distribution of the data set. Not only the prediction accuracy of the tail data is improved, but also the overall relationship prediction is improved. GSI does not focus only on the head or tail relationship, but improves from the perspective of relationship prediction, which is more practical and closer to the actual application scene compared with some scene graph methods that only focus on the head relationship and some unbiased scene graph methods that only focus on the tail relationship.

[0080] The above disclosed is only a preferred embodiment of the method for enhancing the geometric dimension and semantic dimension in the unbiased scene graph generation, and of course cannot limit the scope of the present application. Those skilled in the art can understand that all or part of the processes of the above embodiment are implemented, and equivalent changes made according to the claims of the present application still belong to the scope covered by the present application.

Claims

1. A method of geometric and semantic dimension augmentation on unbiased scene graph generation, characterized in that, The method comprises the following steps: constructing a predicate class tree structure; inputting a data set picture into a target detection model and outputting a processing result; inputting the processing result into a semantic module and a geometric module to output new joint region features; defining a loss function to obtain a designed class balance swing loss function; inputting the new joint region features into a transformer model, using the designed class balance swing loss function to calculate a probability value of each predicate class in the object to obtain a first probability value; substituting the probability value into the predicate class tree structure, using a tree-based class balance swing loss function to calculate a probability value to obtain a second probability value; weighting and summing the first probability value and the second probability value to obtain a final probability value; outputting a corresponding triple of the picture based on the final probability value; the step of inputting the processing result into the semantic module and the geometric module to output new joint region features comprises: obtaining word embedding vectors in the processing result and position information of the object; inputting the word embedding vectors into the semantic module for processing to obtain a first output result; inputting the position information of the object into the geometric module for processing to obtain a second output result; performing full connection operation on the first output result, the second output result and joint region features to obtain a connection matrix; processing the connection matrix through a linear rectifier activation function, a random inactivation layer and a linear layer to obtain new joint region features; the step of inputting the word embedding vectors into the semantic module for processing to obtain a first output result comprises: inputting the word embedding vectors into the semantic module; the semantic module performs dot product operation on the word embedding vectors to obtain an operation result; the semantic module performs adaptive average pooling on the operation result after using a 5*5 convolution, a linear rectifier activation function, average pooling and two 3*3 convolutions to obtain a pooling result; the semantic module performs two 1*1 convolutions on the pooling result to obtain a first output result; the step of inputting the position information of the object into the geometric module for processing to obtain a second output result comprises: inputting the position information of the object into the geometric module; the geometric module obtains the overlap ratio between the object and adjacent objects in each picture based on the position information of the object, and obtains each object and the object pair with the highest overlap ratio to obtain a second output result the class balance swing loss function is defined as: ， wherein, is a predicted probability value, is a true value label for node i, the loss function comprises a normalized exponential function and a weight value ; the tree-based class balance loss function is a loss function constructed for leaf nodes, and the weight formula is: wherein, represents the number of leaf nodes in the sub-tree, a hyperparameter is set to 0.999, q is set to 3.00, the true value is , the predicted probability value is ; the overall loss function is: where K represents the number of nodes outside the initial leaf node, The calculation and weight formula are consistent, and the input of the Softmax function is the probability value of the parent node , that is, the average value of the sum of the probabilities of all leaf nodes of the node, leaf node The probability value is calculated by the class balanced swing loss function; the overall loss function is composed of the class balance swing loss function and the tree-based class balance loss function, and is ， where the hyperparameters is set to 0.

999.

2. The method of claim 1, wherein the target detection model is a Faster R-CNN model; and the transformer model is a Transformer model. ​ ​

Images