A bias-free dynamic scene graph generation method based on a multi-task learning framework

Abstract

The application discloses a kind of methods for generating unbiased dynamic scene graph based on multi-task learning framework, it is related to image processing technical field, solve recent unbiased scene graph generation method cannot simultaneously take care of head predicate and tail predicate learning technical problem, including the following steps: video is input into target detector, the visual feature, semantic feature and spatial feature of instance in image in video are obtained;The features of instance in image are sent into global feature learning branch and local feature learning branch, and the scene graph generation model is trained in it by linking local features and global features;Using multi-task learning model, the unreasonable prediction in main task scene graph generation task is corrected by the learning of auxiliary task action recognition task, the gap between the semantics of predicted scene graph and the semantics of labeled scene graph is calculated at the same time, and mean square error loss and focal loss function are optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, more particularly to a bias-free dynamic scene graph generation method based on a multi-task learning framework. BACKGROUND

[0002] In recent years, deep neural networks have been widely used in various fields due to their superior performance. In addition to basic cognitive tasks such as image classification and object detection, some application tasks that require higher-level semantic understanding have also received great attention with the development of visual cognition and natural language processing, such as image description generation and image question answering. However, basic cognitive tasks treat instances in images in isolation, ignoring the visual relationship information between different instances, and thus cannot fully understand the visual content in the image.

[0003] Image scene graph generation is a task that specifically detects the visual relationship between different instances, which uses a graph structure to represent the visual relationship, where nodes represent instances detected in the image, and edges represent the relationship between instances, containing different relationship predicate information. The scene graph generation task can provide a medium-level image abstract feature for downstream tasks, facilitating further processing.

[0004] Because the widely used video scene graph dataset Action Genome has a natural long-tail distribution, if a common model is used for training, most of the predicted results will be some common predicates, such as "writing on", "standing on", and "working on", which cannot provide rich semantics, and thus the generated image scene graph can only provide a small amount of semantic information, which is less helpful for downstream tasks. Therefore, most current methods are solving the long-tail distribution problem to generate bias-free scene graphs in order to predict more predicates with rich information, such as "holding", "touching", etc. However, many bias-free scene graph generation methods overemphasize the importance of tail predicates and underestimate the importance of head predicates during the entire training process, thereby damaging the representation ability of head predicate features. In addition, because the inference of tail predicates is highly dependent on the basic features provided by head predicates, such as "writing on", "standing on" being highly dependent on "on", the damaged head predicate features will also affect the learning of tail predicates.

[0005] Therefore, these bias-free scene graph generation methods have neither achieved very good tail predicate recognition performance nor satisfactory head predicate recognition performance. SUMMARY

[0006] The application aims at solving the above technical problems, and provides a bias-free dynamic scene graph generation method based on a multi-task learning framework.

[0007] To achieve the above purpose, the application specifically adopts the following technical solutions:

[0008] A bias-free dynamic scene graph generation method based on a multi-task learning framework comprises the following steps:

[0009] S1, input a video into a target detector to obtain visual features, semantic features and spatial features of instances in images in the video;

[0010] S2, send the features of the instances in the images into a global feature learning branch and a local feature learning branch;

[0011] S3, train the scene graph generation model by linking the local features and the global features;

[0012] S4, use a multi-task learning model to correct unreasonable predictions in the main task of scene graph generation by learning the auxiliary task of action recognition, calculate the gap between the predicted semantics of the scene graph and the annotated semantics of the scene graph, and optimize by using a mean square error loss and a focal loss function;

[0013] S5, use a standard cross-entropy loss function to optimize the prediction results of the auxiliary task of action recognition, and correct the prediction effect of the scene graph generation network by back propagation;

[0014] S6, under the guidance of the auxiliary task, the model first focuses on learning global semantic information, and gradually places the learning center on difficult-to-predict local semantics, and calculates the overall loss of the model.

[0015] As an optional technical solution, in S1 and S2, the input video V of the target detector is first parsed into a plurality of video frame images {I1, I2,..., I t , and then the images are output through a global feature extraction network and a local feature extraction network, and the output is visual features x v , global features x s and global semantic features x g of the instances in the images.

[0016] As an optional technical solution, the visual features x v , the global features x s and the global semantic features x g are extracted by the following formula, specifically as follows:

[0017]

[0018]

[0019] where <,> is concatenation operation, is flatten operation, is feature addition operation, W s is semantic feature transformation matrix; W g is image feature transformation matrix; W o and W u are target and joint region feature transformation matrix respectively; joint feature u ij is the feature vector of the joint region of the i-th and j-th target extracted by RoIAlign, f b is a function of converting the bounding box of the subject and the object into exclusive features, f resize denotes a function of transforming image size, f vgg denotes VGG-16 pre-trained network model, v i refers to visual feature vector, b i refers to feature vector of target detection framework, s i is semantic feature vector, I t is the picture feature vector at the t-th moment.

[0020] As an optional technical solution, in S3, the extracted global feature x g and local feature x v , x s are trained by a scene graph generation network: x c , x c is an attention mechanism vector, and then it is input into a time-space conversion model to obtain global relationship feature x r :

[0021] x c =<x s , x v , x g >

[0022] x r =sttran(x c )

[0023] Where sttran refers to a time-space conversion model, and then x r is respectively input into a double-branch multi-task learning network for prediction learning.

[0024] As an optional technical solution, in S4, the main task generates a dynamic scene graph using a fully connected neural network and a loss function based on binary cross entropy and attention for scene graph prediction learning.

[0025] As an optional technical solution, in the S4, the auxiliary task action recognition is used to predict the fine-grained action features appearing in the scene graph, so as to guide the main learning model, that is, the scene graph prediction model, and then correct the prediction effect of the main task of generating the dynamic scene graph, and the model network structure is the same as that of the main task.

[0026] As an optional technical solution, in the S5 and S6, first, the neural network with full connection structure is used to reduce the complexity of the features, then the binary cross entropy loss function is used to predict the relationship in the global features, and the attention loss function is used to strengthen the learning of the fine-grained predicate of the local semantic information, so as to improve the prediction effect of the local semantic predicate:

[0027] x action =W2dropout(ReLu(W1x r ))

[0028]

[0029]

[0030] Wherein W1, W2 are dimension transformation matrices, ReLu is a rectified linear unit function, dropout is a random part of parameters discarded in the neural network to enhance robustness; L BCE and L focal are binary cross entropy loss function and attention loss function respectively, wherein y n is the true value of the sample, x n is the prediction value of the model; N is the number of samples in the data set, γ is the attention coefficient, and log is the logarithmic function; x action is the prediction score.

[0031] The beneficial effects of the present application are as follows:

[0032] 1. A novel deep learning framework based on multi-task learning is proposed to generate an unbiased scene graph, which can extract global semantic information in the video to assist the learning task of fine-grained semantic information;

[0033] 2. A global feature extraction model and a multi-task double-flow learning strategy are proposed, which is helpful to solve the problem that the tail predicate is difficult to learn in the generation of the dynamic scene graph;

[0034] 3. A learning strategy based on multi-task prediction is proposed, in which the multi-label classification is applied to the relationship prediction for the first time, and the leading effect is achieved on the action gene data set. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 It is a flow chart of an unbiased dynamic scene graph generation method based on a multi-task learning framework.

[0036] Figure 2 This is a general framework diagram of an unbiased dynamic scene graph generation method based on a multi-task learning framework. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0038] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0039] Example 1

[0040] refer to Figure 1 , 2 The flowchart and overall framework diagram of an unbiased dynamic scene graph generation method based on a multi-task learning framework are shown respectively. This method includes steps S1 to S6. Figure 2 As shown, the main functional modules are divided into a global semantic learning branch, a local semantic learning branch, and a semantic context module.

[0041] In one embodiment of the present invention, steps S1-S6, specifically include the execution steps of the global semantic learning branch and the local semantic learning branch:

[0042] A1. The input video V of the target detector is first parsed into multiple video frame images {I1, I2, ..., I...}. t The image is then processed through a global feature extraction network and a local feature extraction network, and the output is the visual features x of the instances in the image. v Global feature x s and global semantic features x g The specific formula is as follows:

[0043]

[0044] x s = i s j >

[0045] ​

[0046] where <,> is concatenation operation, is flatten operation, is feature addition operation, W s is semantic feature transformation matrix; W g is image feature transformation matrix; W o and W u are target and joint region feature transformation matrix respectively; joint feature u ij is the feature vector of the joint region of the i-th and j-th target extracted by RoIAlign, f b is a function of converting the bounding box of the subject and object into exclusive features, f resize denotes a function of transforming image size, f vgg denotes VGG-16 pre-trained network model, v i refers to visual feature vector, b i refers to feature vector of target detection framework, s i is semantic feature vector, I t is the picture feature vector at the t-th moment.

[0047] A2, the extracted global feature x g and local feature x v , x s is trained by scene graph generation network: x c is obtained by concatenation operation, x c is attention mechanism vector, and then it is input into time-space conversion model to obtain global relationship feature x r :

[0048] x c = <x s , x v , x g >

[0049] x r = sttran(x c )

[0050] where sttran refers to time-space conversion model. Then x r is respectively input into double-branch multi-task learning network for prediction learning.

[0051] A3, the main task generates dynamic scene graph using a neural network with a full connection structure and a loss function based on binary cross entropy and attention for scene graph prediction learning. The auxiliary task action recognition predicts the fine-grained action features appearing in the scene graph, thereby guiding the main learning model, i.e. the scene graph prediction model, to correct the prediction effect of the main task of generating dynamic scene graph. The model network structure is the same as the main task.

[0052] First, the complexity of the features is reduced by a neural network with a full connection structure, and then the relationship in the global features is predicted by a binary cross-entropy loss function, while the local semantic information of the fine-grained predicate is strengthened by an attention loss function to improve the prediction effect of the local semantic predicate:

[0053] x action =W2dropout(ReLu(W1x r ))

[0054]

[0055]

[0056] where W1, W2 are dimension transformation matrices, ReLu is a rectified linear unit function, dropout is a random part of parameters discarded in the neural network to enhance robustness; L BCE and L focal are binary cross-entropy loss function and attention loss function, respectively, where y n is the true value of the sample, x n is the predicted value of the model; N is the number of samples in the data set, and γ is the attention coefficient, which is set to 2 in the experiment; log is the logarithmic function; x action is the prediction score.

[0057] Next, the effect of the present scheme is compared and explained by simulation experiments based on the scene graph generation method of the prior art:

[0058] The experimental conditions are: system: Ubuntu 20.04, software: Python 3.7, processor: Intel(R) Xeon(R) CPU E5-2678 v3@2.50GHz x 2, memory: 256GB.

[0059] Test description: each algorithm is used in turn to train the training set in the data set. After training, each algorithm is used to predict the pictures on the test set of the data set to obtain the predicted scene graph. The specific test of the image scene graph generation task includes three sub-tasks:

[0060] 1) Predicate classification: given the bounding box of all instances in a picture and the instance label, then the relationship between instances is detected and classified;

[0061] 2) Scene graph classification: given the bounding box of all instances in a picture, the instances need to be classified and the relationship between the instances needs to be detected;

[0062] 3) Scene graph detection: given a picture, a scene graph is predicted from zero.

[0063] The evaluation indicators used include: Recall@K (R@K). Wherein Recall@K refers to the proportion of correct predicates in the first K predicted relationship predicates. Both evaluation indicators used are the greater the value, the better the performance. The constraint condition refers to predicting a predicate for each pair of objects only once, and there is no constraint condition, that is, no limit; The test results are shown in the following table. DynSGG-MTL is the method used in the present scheme.

[0064] Table 1

[0065]

[0066] Table 2

[0067]

[0068] By comparing the experimental results of Table 1 and Table 2, it can be known that the performance of the present scheme is better under the use of the same general scene graph generation method, and these results verify the effectiveness of the method proposed by us.

[0069] The above only describes the preferred embodiments of the present application and is not used to limit the present application, and any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating unbiased dynamic scene graphs based on a multi-task learning framework, characterized in that, Includes the following steps: S1. Input the video into the target detector to obtain the visual features of the instances in the images within the video. Global features and global semantic features ; Visual features are extracted using the following formula. Global features and global semantic features The details are as follows: Where <,> represents a cascading operation. It is a flattening operation. It is a feature addition operation. It is a semantic feature transformation matrix; It is the image feature transformation matrix; and These are the target and joint region feature transformation matrices, respectively. Joint features It is the feature vector of the joint region of the i-th and j-th targets extracted by RoIAlign. It is a function that converts the bounding boxes of the subject and objects into exclusive features. A function that represents how to resize an image. This refers to the VGG-16 pre-trained network model. It refers to visual feature vectors. The feature vector of the object detection framework. It is a semantic feature vector. It is the first Image feature vector at time step; S2. Feed the features of instances in the image into the global feature learning branch and the local feature learning branch; S3. Train the scene graph generation model by linking local and global features; S4. A multi-task learning model is adopted. The unreasonable predictions in the main task scene graph generation task are corrected by learning the auxiliary task action recognition task. At the same time, the difference between the semantics of the predicted scene graph and the semantics of the labeled scene graph is calculated and optimized using mean squared error loss and focal loss function. S5. Use the standard cross-entropy loss function to optimize the prediction results of the auxiliary task action recognition task, and correct the prediction effect of the scene graph generation network through backpropagation. S6. Guided by the auxiliary task, the model first focuses on learning global semantic information, and then gradually shifts its learning focus to the difficult-to-predict local semantics, and calculates the overall loss of the model.

2. The method for generating unbiased dynamic scene graphs based on a multi-task learning framework according to claim 1, characterized in that, In S1 and S2, the input video of the target detector is... First, it is parsed into multiple video frame images. The image is then processed through a global feature extraction network and a local feature extraction network, and the output is the visual features of the instances in the image. Global features and global semantic features .

3. The method for generating unbiased dynamic scene graphs based on a multi-task learning framework according to claim 2, characterized in that, In step S3, the extracted visual features are then... Global features and global semantic features Training is performed using a scene graph generation network: This is achieved through cascaded operations. , The attention mechanism vector is then input into the time-space transformation model to obtain global relation features. : in Refers to the time-space transformation model, and then... They are fed into a two-branch multi-task learning network for prediction learning.

4. The method for generating unbiased dynamic scene graphs based on a multi-task learning framework according to claim 1, characterized in that, In S4, the main task generates a dynamic scene graph using a fully connected neural network and a loss function based on binary cross-entropy and attention for scene graph prediction learning.

5. The method for generating unbiased dynamic scene graphs based on a multi-task learning framework according to claim 4, characterized in that, In S4, the auxiliary task action recognition predicts fine-grained action features appearing in the scene graph, thereby guiding the main learning model, i.e., the scene graph prediction model, and thus correcting the prediction effect in the dynamic scene graph generated by the main task. The model network structure is the same as that of the main task.

6. The method for generating unbiased dynamic scene graphs based on a multi-task learning framework according to claim 1, characterized in that, In S5 and S6: First, the complexity of features is reduced by using a fully connected neural network. Then, the relationships in the global features are predicted using a binary cross-entropy loss function. At the same time, an attention loss function is used to strengthen the fine-grained predicates of local semantic information, thereby improving the prediction performance of local semantic predicates. in , The dimension transformation matrix, For rectified linear unit functions, To enhance robustness, some parameters are randomly discarded in the neural network. and These are the binary cross-entropy loss function and the attention loss function, respectively. For the true value of the sample, These are the model's predicted values; The number of samples in the dataset. The attention coefficient was set to 2 in the experiment. It is a logarithmic function; It is the predicted score.

Images