An unbiased scene graph generation method based on effective feature representation

Abstract

The application discloses a kind of unbiased scene graph generation methods based on effective feature representation, it is related to computer vision technology.Adopt the training strategy of feature extraction network and classification network decoupling, utilize pre-trained backbone network to extract the visual feature of object;Target detection is carried out, the visual feature of object, the position coding of object and the class coding of object are pairwise matched again to be combined coding, and the coding feature suitable for predicate classification is obtained;Predicate classification is carried out through a fully connected layer;Train feature extraction network, do not adopt the classification network of fully connected layer form in inference, by calculating the mean of the coding feature for predicate classification of each class predicate, predicate classification is carried out according to the cosine similarity of the coding feature of the sample to be classified and the mean of each class predicate feature.The full connection layer classifier is abandoned, and classification is directly carried out based on predicate feature, the problem that full connection layer parameter is susceptible to long tail data can be excluded, so as to improve the performance of scene graph generation task.

Description

TECHNICAL FIELD

[0001] The present application relates to computer vision technology, and in particular to an unbiased scene graph generation method based on effective feature representation. BACKGROUND

[0002] In recent years, scene graph generation algorithms based on deep learning have made great progress. However, the dataset used for training the scene graph generation model has a serious long-tail problem, that is, the number of a few head predicates is much larger than that of the middle and tail predicates. The extreme imbalance between sample categories leads to a large bias in the prediction results of the model, that is, there is a strong tendency to predict head predicates, which does not allow the model to learn relationship prediction well. Therefore, it is particularly important to solve the problem of low algorithm performance caused by long-tail data in the scene graph generation task.

[0003] To improve the predicate classification accuracy of the scene graph generation task, one natural idea is to improve the network, increase its complexity so that the network can extract better predicate features. Therefore, early SGG methods focus on building better feature extraction networks. Guojun Yin et al. (Yin, G., Sheng, L., Liu, B., Yu, N., Wang, X., Shao, J., & Loy, C. C. (2018). Zoom-net: Mining deep feature interactions for visual relationship recognition. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 322-338).) use the interaction of local features to improve the performance of scene graph generation. Kaihua Tang et al. (Tang, K., Zhang, H., Wu, B., Luo, W., & Liu, W. (2019). Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 6619-6628).) learn global visual context information by using standard recurrent neural network structures or their variants for message passing. Jianwei Yang et al. (Yang, J., Lu, J., Lee, S., Batra, D., & Parikh, D. (2018). Graph r-cnn for scene graph generation. In Proceedings of the European conference on computer vision (ECCV) (pp. 670-685).) propose to prune the original dense complete scene graph to generate the final sparse scene graph. The above focuses on improving the structure of the feature extraction network, while ignoring the long-tail problem, which is a huge factor affecting the training effect.Therefore, Kaihua Tang et al. (Tang, K., Niu, Y., Huang, J., Shi, J., & Zhang, H. (2020). Unbiased scene graph generation from biased training. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 3716-3725).) eliminate bias by counterfactual causal analysis during training. Yuyu Guo et al. (Guo, Y., Gao, L., Wang, X., Hu, Y., Xu, X., Lu, X., … & Song, J. (2021). From general to specific: Informative scene graph generation via balance adjustment. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 16383-16392).) adopt a two-step training strategy to alleviate the long-tail problem by fine-tuning some parameters of the network in the constructed balanced training domain. SUMMARY

[0004] The present application aims to solve the problem of poor relationship detection performance caused by long-tail training data in the scene graph generation task, and provides an unbiased scene graph generation method based on effective feature representation, which adopts a training strategy of decoupling feature extraction network and classification network to handle the long-tail problem in the scene graph generation task.

[0005] The present application comprises the following steps:

[0006] A. Collect the scene graph generation dataset, divide it into training set, validation set and test set, and perform image preprocessing;

[0007] B. Use the pre-trained backbone network to extract the visual features of the object, input them into the object detection branch, and obtain the position and category of the object;

[0008] C. Encode the object position and category obtained in step B respectively to obtain the object position encoding feature and the object category encoding feature;

[0009] D. Concatenate the object visual feature obtained in step B and the object position encoding feature and the object category encoding feature obtained in step C to obtain the effective feature representation of the object;

[0010] E. The effective feature representation of all objects in each image obtained from step D is input into the relational fusion feature encoder, and the encoding results are paired two by two to obtain a series of effective feature representations of relations;

[0011] F. The effective feature representation of relations obtained in step E is input into the fully connected layer network for classification, and the classification loss is calculated to update the parameters of the network;

[0012] G. After the training converges, for each predicate class, the average value of the effective features of the relations of the samples containing the predicate class in the training set is calculated using steps A-E. In inference, the cosine similarity between the effective features of the relations of the sample to be classified and the average value of the effective features of each class of relations calculated is calculated, and the class with the largest similarity is taken as the classification result.

[0013] In step A, the scene graph generation dataset uses the public dataset VG-150 (Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L. (2017). Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5410-5419).); the dataset collects 108077 pictures, containing 150 classes of objects and 50 classes of predicates; the dataset is divided into a training set and a test set in a ratio of 7:3, and the first 5000 pictures of the training set are used as a validation set; when training the model, the pictures are preprocessed by random cropping, random flipping, normalization, etc. to further enrich the richness of the training samples.

[0014] In step B, the backbone network adopts ResNeXt-101-FPN network (Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1492-1500), Lin, T. Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2117-2125).), and the target detection branch adopts Faster R-CNN network (Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.); the visual features of the object are a 4096-dimensional vector learned by the neural network; the position of the object is a four-dimensional vector representing the horizontal and vertical coordinates of the upper left and lower right corners of the object box; and the category of the object is an integer in the range of [0, C O ), where C O represents the number of object categories in the data set.

[0015] In step C, the position encoding feature of the object is: first, calculate a nine-dimensional vector, which represents the width of the object frame / image, the height of the object frame / image, the horizontal coordinate of the object frame center point / image width, the vertical coordinate of the object frame center point / image height, the horizontal coordinate of the object frame top left corner / image width, the vertical coordinate of the object frame top left corner / image height, the horizontal coordinate of the object frame bottom right corner / image width, the vertical coordinate of the object frame bottom right corner / image height, and (the width of the object frame * the height of the object frame) / (the width of the image * the height of the image), respectively; then linear transformation to a 128-dimensional vector; the category encoding of the object is a 200-dimensional vector learned by the neural network embedding layer.

[0016] In step D, the effective feature representation of the object is: first, concatenate the object visual feature obtained in step B, the object position encoding feature obtained in step C, and the object category encoding feature, and then input the concatenated result into a fully connected layer to convert it into a 768-dimensional vector.

[0017] In step E, the following sub-steps are further included:

[0018] E1. The relationship fusion encoder is composed of a series of Transformer encoders (Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N.,... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.) and two fusion strategies are added;

[0019] E2. The first part of the two fusion strategies is the fusion operation on the input of the Transformer encoder; specifically, the input of the first Transformer encoder is the effective feature representation of the object obtained in step D, and the input of the subsequent Transformer encoder is the output of the previous Transformer encoder; in order to prevent the effective feature representation of the object from being forgotten during the encoding process, the input of the M+1th Transformer encoder is changed to the fusion result of the output of the Mth Transformer encoder and the effective feature representation of the object, and the specific fusion method is as follows:

[0020] X M+1 =(X1+Y M )W in +b in (Formula One)

[0021] where X M+1 is the input of the M+1th Transformer encoder; X1 is the input of the 1st Transformer encoder, i.e., the effective feature representation of the object; Y M is the output of the Mth Transformer encoder, W in and b in are the matrix and vector required for linear transformation.

[0022] E3. The second part of the two fusion strategies is the fusion operation on the output of each Transformer encoder, which can make the encoding result contain multi-level features, and the calculation method is as follows:

[0023]

[0024] where Y represents the fusion result of the encoding result of each Transformer encoder, and M+N is the number of Transformer encoders.

[0025] E4. The fusion encoding result of the Transformer encoder is calculated for each object, and the effective feature representation of a series of relationships is obtained by splicing two by two, specifically, for a pair of objects <s, o>, the effective feature representation F s，o of the relationship between the two objects is calculated by the following formula:

[0026] F s，o = cat(Y s W out +b out , Y o W out +b out ) (Formula Three)

[0027] where cat(·, ·) represents the splicing operation of vectors, Y s and Y o represent the fusion encoding results of the Transformer encoder for the object s and the object o, W out and b out are the matrix and vector required for linear transformation, and the final effective feature representation F s，o of the relationship is a 768-dimensional vector.

[0028] In step F, in order to enable the network to update the parameters, the effective feature representation of the relationship obtained in step E is input into a fully connected layer network for predicate classification, and the cross-entropy loss of predicate classification is used for backward gradient propagation, so as to update the parameters of the feature extraction network part.

[0029] In step G, the following sub-steps are further included:

[0030] G1. In order to avoid the problem that the parameters of the full connection layer tend to optimize the classification effect of the class with a large number of samples, all parameters of the entire network are frozen when training in steps A-F to convergence; and for each class predicate, the mean value of the effective feature representation of the relationship containing the class predicate in the training set is calculated by using steps A-E to obtain C R relationship feature means, C R is the number of classes of predicates in the data set, and the relationship feature mean μ i of the i-th class predicate r i is calculated as follows:

[0031]

[0032] Where n i is the number of training samples containing the i-th class predicate, p s，o is the predicate true value between the object pair <s, o>, The definition of the function is as follows:

[0033]

[0034] G2. In the model inference stage, the effective feature of the relationship of the sample to be classified is first calculated according to steps A-E, and the cosine similarity is calculated with all the relationship feature means calculated in step G1, and the class with the largest similarity is taken as the classification result in the inference The details are as follows:

[0035]

[0036] The present application proposes an unbiased scene graph generation method based on effective feature representation, adopts a training strategy of decoupling feature extraction network and classification network, and considers that long-tail data mainly affects the classification network in the model. Therefore, the overall training of the scene graph generation network is first performed, and then the classifier in the form of the full connection layer in the original network is discarded in the inference, and a method of cosine similarity matching of features is used for predicate classification. This method requires that the predicate features learned by the network can accurately represent the relationship between the subject, predicate and object in each relationship, and a relationship feature fusion encoder is proposed to perform multi-level fusion operation on the predicate features, so as to obtain more effective relationship feature representation. This method can effectively avoid the problem of deviation of the classifier learning caused by long-tail data, and effectively improve the performance of scene graph generation. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is the overall network structure diagram of the embodiment of the present application.

[0038] Figure 2 For random extraction of several pictures on the scene graph generation data set VG-150, the contrast chart of the scene graph generated by the benchmark method and the scene graph generated by the method of the application. DETAILED DESCRIPTION

[0039] The following examples will further illustrate the present application with reference to the accompanying drawings. The examples are implemented on the premise of the technical scheme of the present application, and the implementation modes and specific operation processes are given, but the protection scope of the present application is not limited to the following examples.

[0040] Referring to Figure 1 The implementation mode of the embodiment of the present application includes the following steps:

[0041] A. Collect the scene graph generation data set, and divide it into a training set, a validation set and a test set, and then perform image preprocessing. The specific method is as follows: the present application uses a public data set VG-150 (Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L. (2017). Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5410-5419).). The data set collects 108077 pictures, including 150 classes of objects and 50 classes of predicates. The data set is divided into a training set and a test set according to a ratio of 7:3, and the first 5000 pictures of the training set are used as a validation set. When training the model, the pictures are randomly cropped, randomly flipped, normalized and other preprocessing operations to further enrich the richness of the training samples.

[0042] B. With the pre-trained backbone network, the visual features of the object are extracted and sent to the target detection branch to obtain the position and category of the object. The specific method is as follows: the backbone network adopts ResNeXt-101-FPN network (Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1492-1500), Lin, T. Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2117-2125).), and the target detection branch adopts Faster R-CNN network (Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.). The visual features of the object are a 4096-dimensional vector learned by the neural network. The position of the object is a four-dimensional vector representing the horizontal and vertical coordinates of the upper left and lower right corners of the object frame, and the category of the object is an integer in the range of [0, C O ) represents the number of object categories in the data set. O

[0043] ​C. Encode the object position and class respectively to get the position encoding feature and class encoding feature. The position encoding feature is calculated as follows: first, a nine-dimensional vector is calculated, each position of which represents the width of the object frame / image, the height of the object frame / image, the horizontal coordinate of the object frame center / image width, the vertical coordinate of the object frame center / image height, the horizontal coordinate of the object frame top left corner / image width, the vertical coordinate of the object frame top left corner / image height, the horizontal coordinate of the object frame bottom right corner / image width, the vertical coordinate of the object frame bottom right corner / image height, and (width of object frame*height of object frame) / (width of image*height of image). Then linear transformation is performed to become a 128-dimensional vector. The class encoding of the object is a 200-dimensional vector learned by the neural network embedding layer.

[0044] D. Concatenate the object visual feature obtained in step B and the object position encoding feature and object class encoding feature obtained in step C to obtain the effective feature representation of the object. The effective feature representation of the object is calculated as follows: first, the object visual feature obtained in step B and the object position encoding feature and object class encoding feature obtained in step C are concatenated, and then the concatenated result is input into a fully connected layer to become a 768-dimensional vector.

[0045] E. Input the effective feature representation of all objects in each image obtained in step D into the relationship fusion feature encoder, and pair the encoding results two by two to obtain a series of effective feature representations of relationships. The specific method includes the following sub-steps:

[0046] E1. The relationship fusion encoder is composed of a series of Transformer encoders (Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N.,... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.) and two fusion strategies are added.

[0047] E2. The first part of the two fusion strategies is the fusion operation on the input of the Transformer encoder. Specifically, the input of the first Transformer encoder is the effective feature representation of the object obtained in step D, and the input of the subsequent Transformer encoder is the output of the previous Transformer encoder. In order to prevent the effective feature representation of the object from being forgotten in the encoding process, the input of the M+1th Transformer encoder is changed to the fusion result of the output of the Mth Transformer encoder and the effective feature representation of the object, and the specific fusion method is as follows:

[0048] X M+1 = (X1+ Y M )W in +b in (Formula One)

[0049] Where X M+1 is the input of the M+1th Transformer encoder; X1 is the input of the first Transformer encoder, i.e. the effective feature representation of the object; Y M is the output of the Mth Transformer encoder, W in and b in are the matrix and vector required for linear transformation.

[0050] E3. The second part of the two fusion strategies is the fusion operation on the output of each Transformer encoder, which can make the encoding result contain multi-level features, and the calculation method is as follows:

[0051]

[0052] Where Y represents the fusion result of the encoding result of each Transformer encoder, and M+N is the number of Transformer encoders.

[0053] E4. The fusion encoding result of the Transformer encoder is calculated for each object, and a series of effective feature representations of relationships are obtained by splicing two by two, specifically, for a pair of objects <s, o>, the effective feature representation F s，o of the relationship between the two objects is calculated by the following formula:

[0054] F s，o = cat(Y s W out +b out , Y o W out +b out ) (Formula Three)

[0055] where cat(·, ·) denotes the concatenation operation of vectors, Y s and Y o denote the fusion encoding results of object s and object o by the Transformer encoder, W out and b out are the matrix and vector required for linear transformation, and the final effective feature representation F s，o is a 768-dimensional vector.

[0056] F. The effective feature representation of the relation obtained in step E is input into a fully connected layer network for classification, and the classification loss is calculated to update the parameters of the network. The specific method is as follows: in order to enable the network to update the parameters, the effective feature representation of the relation obtained in step E is input into a fully connected layer network for predicate classification, and the cross-entropy loss of predicate classification is used for back propagation of the gradient, so as to update the parameters of the feature extraction network part.

[0057] G. After the training converges, for each type of predicate, the average value of the effective feature of the relation of the sample containing the predicate in the training set is calculated by steps A-E, and the cosine similarity between the effective feature of the relation of the sample to be classified and the average value of the effective feature of each type of relation is calculated during inference, and the category with the largest similarity is taken as the classification result. The specific method includes the following sub-steps:

[0058] G1. In order to avoid the problem that the parameters of the fully connected layer tend to optimize the classification effect of the category with more samples, all the parameters of the network are frozen when training to convergence in steps A-F. And for each type of predicate, the average value of the effective feature representation of the relation of the sample containing the predicate in the training set is calculated by steps A-E, to obtain C R effective feature mean values, C R is the number of categories of predicates in the data set, and the effective feature mean value μ i of the i-th type of predicate r i is calculated as follows:

[0059]

[0060] where n i is the number of training samples containing the i-th type of predicate, p s，o is the true value of the predicate between the object pair <s, o>, The definition of the function is as follows:

[0061]

[0062] G2. In the model inference stage, the effective feature of the relation of the sample to be classified is first calculated according to steps A-E And all the relationship features calculated by step G1 are calculated with the cosine similarity, and the category with the largest similarity is taken as the classification result in reasoning The specific implementation is as follows:

[0063]

[0064] As shown in Figure 2 The method of the present application can generate a more meaningful scene graph compared with the baseline method, avoids the problem of blindly predicting high-frequency predicates of the baseline method, effectively alleviates the long-tail problem in the scene graph generation task, and can predict low-frequency but more informative predicates.

[0065] Table 1 is the comparison of the predicate average recall (mR) of the method of the present application and some other existing scene graph generation methods on three common sub-tasks of the test data of VG-150.

[0066] As can be seen from Table 1, the present application has achieved the highest predicate average recall (mR) on the three common sub-tasks of the scene graph generation model evaluated on the VG-150 dataset.

[0067] Table 1

[0068]

[0069] IMP corresponds to the method proposed by Danfei Xu et al. (Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L. (2017). Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5410-5419).);

[0070] MotifNet corresponds to the method proposed by Rowan Zeller et al. (Zellers, R., Yatskar, M., Thomson, S., & Choi, Y. (2018). Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5831-5840).);

[0071] VCTree corresponds to the method proposed by Kaihua Tang et al. (Tang, K., Zhang, H., Wu, B., Luo, W., & Liu, W. (2019). Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 6619-6628).);

[0072] TDE corresponds to the method proposed by Kaihua Tang et al. (Tang, K., Niu, Y., Huang, J., Shi, J., & Zhang, H. (2020). Unbiased scene graph generation from biased training. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 3716-3725).);

[0073] PUM corresponds to the method proposed by Gengcong Yang et al. (Yang, G., Zhang, J., Zhang, Y., Wu, B., & Yang, Y. (2021). Probabilistic modeling of semantic ambiguity for scene graph generation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 12527-12536).);

[0074] The BGNN corresponds to the method proposed by Rongjie Li et al. (Li, R., Zhang, S., Wan, B., & He, X. (2021). Bipartite graph network with adaptive message passing for unbiased scene graph generation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 11109-11119.);

[0075] The BA-SGG corresponds to the method proposed by Yuyu Guo et al. (Guo, Y., Gao, L., Wang, X., Hu, Y., Xu, X., Lu, X.,... & Song, J. (2021). From general to specific: Informative scene graph generation via balance adjustment. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 16383-16392).

[0076] The scene graph generation aims to detect objects in an image and the relationship between the objects, which is represented by a triple such as <subject, predicate, object>. The present application adopts a training strategy of decoupling feature extraction network and classification network, first extracts the visual features of the objects by using a pre-trained backbone network; then performs object detection, and uses the extracted visual features of the objects, the position encoding of the objects and the class encoding of the objects to pair and combine the encoding again, to obtain the encoding features suitable for predicate classification; then classifies the predicate through a fully connected layer; the feature extraction network is trained through the above steps, and a classification network in the form of a fully connected layer is not used during inference, but the mean value of the encoding features for predicate classification of each class of predicate is calculated, and the cosine similarity between the encoding features of the sample to be classified and the mean value of the predicate features of each class is calculated to classify the predicate. This method of abandoning the fully connected layer classifier and directly classifying based on the predicate features can eliminate the problem that the parameters of the fully connected layer are easily affected by the long-tail data, thereby improving the performance of the scene graph generation task.

Claims

1. An unbiased scene graph generation method based on effective feature representation, characterized by The following steps are involved: A. Collect scene graphs to generate a dataset, divide it into training set, validation set, and test set, and perform image preprocessing; B. Use the pre-trained backbone network to extract the visual features of the object and send them to the target detection branch to obtain the location and category of the object; The backbone network uses the ResNeXt-101-FPN network, and the target detection branch uses the Faster R-CNN network. The visual feature of the object is a 4096-dimensional vector learned by the neural network. The position of the object is a four-dimensional vector representing the horizontal and vertical coordinates of the upper left corner and the lower right corner of the object box. The category of the object is [0, C O ) in the range of C O Indicates the number of object categories in the dataset; C. Encode the object position and category obtained in step B respectively to obtain the object position coding feature and the object category coding feature; D. Concatenate the object visual features obtained in step B with the object position coding features and object category coding features obtained in step C to obtain a valid feature representation of the object; E. Pass the valid feature representations of all objects in each image obtained in step D into the relational fusion feature encoder, and pair the encoding results to obtain a series of valid feature representations of relations; Obtaining a series of effective feature representations of the relationship includes: E1. The relational fusion encoder consists of a series of Transformer encoders and adds two fusion strategies; The first part of the two fusion strategies is the fusion operation performed on the input of the Transformer encoder. Specifically, the input of the first Transformer encoder is the effective feature representation of the object obtained in step D, and the input of subsequent Transformer encoders except the M+1th Transformer encoder is the output of the previous Transformer encoder. In order to prevent the effective feature representation of the object from being forgotten during the encoding process, the input of the M+1th Transformer encoder is changed to the fusion result of the output of the Mth Transformer encoder and the effective feature representation of the object. The fusion method is as follows: X M+1 =(x1+Y M )W in +b in Among them, X M+1 is the input of the M+1th Transformer encoder; X1 is the input of the first Transformer encoder, that is, the effective feature representation of the object; Y M is the output of the Mth Transformer encoder, W in and b in The matrices and vectors required for linear transformation; The second part of the two fusion strategies is to perform a fusion operation on the output of each Transformer encoder, which enables the encoding result to contain multi-level features. The calculation method is as follows: Where Y represents the fusion result of the encoding results of each Transformer encoder, and M+N is the number of Transformer encoders; E2. Calculate the fusion encoding results of the Transformer encoder for each object, and concatenate them two by two to obtain a series of effective feature representations of the relationship. Specifically, for a pair of objects<s,o> , the effective feature representation F of the relationship between these two objects s,o Calculated by the following formula: F s,o =cat(Y s W out +b out ,Y o W out +b out ) Among them, cat(·,·) represents the concatenation operation of vectors, Y s and Y o Respectively represent the fusion encoding results of Transformer encoder for object s and object o, W out and b out The matrix and vector required for linear change, and the effective feature representation F of the relationship finally obtained s,o is a 768-dimensional vector; F. Pass the effective feature representation of the relationship obtained in step E into the fully connected layer network for classification, calculate the classification loss and update the network parameters; G. After training convergence, for each type of predicate, use step AE to calculate the average value of the effective features of the relationships of the samples containing this type of predicate in the training set. During inference, calculate the cosine similarity between the effective features of the relationships of the samples to be classified and the calculated effective feature means of each type of relationship, and take the category with the largest similarity as the classification result.

2. The unbiased scene graph generation method based on effective feature representation according to claim 1, characterized in that: In step A, the scene graph generation dataset adopts the public dataset VG-150, which collects 108,077 pictures, including 150 types of objects and 50 types of predicates; the dataset is divided into training set and test set in a ratio of 7:3, and the first 5,000 pictures of the training set are used as the validation set; when training the model, the picture preprocessing operation expands the richness of the training samples, and the preprocessing operations include random cropping, random flipping, and normalization.

3. The unbiased scene graph generation method based on effective feature representation according to claim 1, characterized in that: In step C, the object position encoding feature is as follows: a nine-dimensional vector is calculated, in which each position of the vector represents: the width of the object frame / the width of the image, the height of the object frame / the height of the image, the horizontal coordinate of the center point of the object frame / the width of the image, the vertical coordinate of the center point of the object frame / the height of the image, the horizontal coordinate of the upper left corner of the object frame / the width of the image, the vertical coordinate of the upper left corner of the object frame / the height of the image, the horizontal coordinate of the lower right corner of the object frame / the width of the image, the vertical coordinate of the lower right corner of the object frame / the height of the image, (the width of the object frame * the height of the object frame) / (the width of the image * the height of the image), and then a linear transformation is performed to convert it into a 128-dimensional vector; the object category is encoded as a 200-dimensional vector learned by the neural network embedding layer.

4. The unbiased scene graph generation method based on effective feature representation according to claim 1, characterized in that: In step D, the effective features of the object are represented by concatenating the object visual features obtained in step B and the object position coding features and object category coding features obtained in step C, and passing the concatenated result into a fully connected layer to be converted into a 768-dimensional vector.

5. The unbiased scene graph generation method based on effective feature representation according to claim 1, characterized in that: In step F, the effective feature representation of the relationship obtained in step E is passed into the fully connected layer network for classification, and the classification loss is calculated to update the network parameters. In order to enable the network to update the parameters, the effective feature representation of the relationship obtained in step E is passed into a fully connected layer network for predicate classification, and reverse gradient propagation is performed through the cross entropy loss of the predicate classification to update the parameters of the feature extraction network part.

6. The unbiased scene graph generation method based on effective feature representation according to claim 1, characterized in that: In step G, the following sub-steps are included: G1. In order to avoid the problem that the parameters of the fully connected layer tend to optimize the classification effect of the category with a large number of samples, all parameters of the entire network are frozen when step AF is trained to convergence; for each type of predicate, the mean of the effective feature representation of the relationship containing this type of predicate in the training set is calculated using step AE to obtain C R The mean of relationship characteristics, C R is the number of categories of predicates in the dataset, and the i-th category predicate r i The mean μ of the relationship characteristics i The calculation method is as follows: Among them, n i is the number of training samples containing the i-th type of predicate, p s,o For object pairs<s,o> The truth value of the predicate between The function is defined as follows: G2. In the model inference stage, calculate the effective features of the relationship between the samples to be classified according to step AE Calculate the cosine similarity one by one with the mean of all relationship features calculated in step G1, and take the category with the largest similarity as the classification result during inference The formula is as follows:

Images