A scene graph generation apparatus and method based on alternating iterative optimization

Abstract

The application belongs to the technical field of computer vision, and discloses a scene graph generation device and method based on alternating iterative optimization, which comprises an image object detection module, a scene graph object recognition module and a scene graph relationship recognition module. The method fully utilizes the context information in the image to optimize the object features by designing a relationship-sensitive message passing network, and alternately iteratively optimizes the object recognition and relationship recognition capabilities of the model, so that high-precision scene graph generation is realized. Compared with the traditional method, the method solves the problem of missing context information and alleviates the error transmission problem of the model architecture. At the same time, since the use of statistical information of the co-occurrence frequency of objects and relationships is avoided, the dependence of the model after training on a specific data set is reduced. In summary, for natural scene images, the application generates high-precision scene graph prediction results by fully utilizing the context and alternately iteratively optimizing different parts of the model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a scene graph generation device and method based on alternating iterative optimization. BACKGROUND

[0002] Scene graph generation is a task of inferring spatial and semantic information of objects and their relationships in an image. In a scene graph, nodes and edges represent objects and their relationships in the image, respectively. This task is a further expansion of object detection, image segmentation and other tasks to deep image understanding, and has attracted a lot of attention in the research field in recent years. Through the generation of a scene graph, many downstream visual applications can be facilitated. At the same time, existing research has shown that through scene graph generation, the model can also extract better image features to help more basic visual tasks such as image classification and object detection.

[0003] Generally, existing scene graph generation methods follow the following process: first, the bounding box coordinates of objects in the image are obtained by using any object detector (usually Faster-RCNN is used for comparison), and the visual features corresponding to the object bounding box are extracted. Then, an object recognition module is constructed to optimize the object features using the context information in the image and obtain the object classification results. Finally, the relationship between each pair of objects is identified by fusing the visual features of the objects and other information, and the scene graph is output by combining all the results. Common extensions include but are not limited to modifying the network structure of the recognition network and introducing text features of objects. Recently, many works have focused on solving the long-tail effect problem of relationship class labeling in scene graph datasets.

[0004] However, the above framework has two major problems. First, the context information is missing. The relationship between objects in the image is an important part of the context information, and existing methods usually represent such relationships with edges in the scene graph. However, since the relationship between objects is unknown, existing methods often use full connection or sequential connection to construct an initial scene graph to perform message passing. Such construction methods cause the context information in the initial scene graph to be inconsistent with the actual scene, affecting the message passing process. Second, error propagation. The above framework divides the scene graph generation into three independent steps, but it is difficult to guarantee the accuracy of each step. Therefore, this framework has a serious error propagation problem. For example, incorrect object recognition results will make it difficult for the model to correctly identify the relationship class between objects. SUMMARY

[0005] The present application aims to provide a scene graph generation device and method based on alternating iterative optimization to solve the above technical problems.

[0006] To solve the above technical problems, the specific technical solutions of the scene graph generation device and method based on alternating iterative optimization are as follows:

[0007] A scene graph generation device based on alternating iterative optimization comprises an image object detection module, a scene graph object recognition module and a scene graph relationship recognition module.

[0008] The image object detection module is responsible for locating the bounding box of the object in the input image and extracting the object visual features corresponding to the bounding box.

[0009] The scene graph object recognition module is responsible for identifying the bounding box output by the image object detection module based on the output of the scene graph relationship recognition module to obtain the corresponding object category.

[0010] The scene graph relationship recognition module is responsible for extracting text features according to the output of the scene graph object recognition module and combining object visual features and spatial features to identify the relationship between objects.

[0011] The application further discloses a scene graph generation method comprising a training stage and a testing stage.

[0012] Step 1: Organize the data set: use natural scene images with scene graph annotations as training data.

[0013] Step 2: Construct the training network:

[0014] Step 2.1: The object recognition module adopts a relationship-sensitive message passing network, trains different conversion matrices for different types of relationships, and aggregates the converted neighbor node messages to update the features of the current node.

[0015] Step 2.2: The relationship recognition module identifies the relationship between the two by combining object features, spatial features and text features of object classification.

[0016] Step 3: Train the network: Before formal model training, first use the pre-trained text feature extraction model to extract the text features of the object and relationship categories in the image annotation set for subsequent training; at the same time, fine-tune the object detector model pre-trained on the external data set on the scene graph data set to obtain the optimal object detection effect.

[0017] Step 4: Alternate step 2 several times and combine the results of object recognition and relationship recognition to obtain the scene graph corresponding to the image.

[0018] Further, the step 1 includes the following specific steps: screening 150 object categories and 50 relationship categories with the highest frequency in the data set as the image annotation set; all images used in training are uniformly scaled to 592x592 pixels, and at least 70,000 and 30,000 images are required for training images and test images, respectively.

[0019] Further, the specific formula of the step 2.1 is as follows:

[0020]

[0021] wherein xi and zi are the node features before and after updating, Wt1 and Wt2 are two conversion matrices shared by all nodes, Wr(i,j) is a group of conversion matrices corresponding to the relationship category, β represents a nonlinear function, LN represents horizontal normalization, and Ni is the neighborhood of node i; at the same time, for each edge in the input scene graph, a reverse conversion matrix is learned, so that for the 50 relationship categories in the data set, the module trains 100 different conversion matrices to fully utilize the relationship constraint to supervise message passing.

[0022] Further, the specific formula of the step 2.2 is as follows:

[0023] p r(i,j) =FC([f1(x s ,x u ,x o ),f2(g s ,g o )])

[0024] wherein Pr(i,j) represents the predicted probability distribution of the relationship between node i and node j, FC is a two-layer fully connected layer, f1 and f2 are two different long short-term memory neural networks, s, u, and o represent the subject object, the minimum bounding box region surrounding the two objects, and the object, respectively, and x and g represent the visual features and text features, respectively.

[0025] Further, the step 3 includes the following specific steps: in training, all parameters in the model are randomly initialized, and the object recognition module and the relationship recognition module are optimized according to the following alternating iterative optimization algorithm:

[0026]

[0027] wherein O is the output of the object recognition network, R is the output of the relation recognition network, F is a function corresponding to the output Pr(i,j) of the relation recognition module, and finally, the probability distribution estimates of the final object recognition and relation recognition are obtained by averaging the probability distribution outputs of the object recognition module and the relation recognition module in multiple iterative optimization steps, and the cross-entropy loss function is used to constrain them respectively, and the stochastic gradient descent method is used to optimize the model by calculating the gradient of the loss function.

[0028] Further, the training stage comprises the following steps: the test image is uniformly scaled to 592x592 pixels, the trained scene graph generation model is used for testing, and a source image is input into the scene graph generation model to obtain the scene graph prediction result corresponding to the image.

[0029] The Visual Genome dataset is used, which includes a total of 108000 images, of which 75651 are training images and 32422 are test images, each image contains an average of 35 objects, 26 attributes, 21 pairs of object relationships and 50 object region descriptions, wherein all concepts correspond to the concepts in WordNet, and the dataset also includes 1773258 pairs of question and answer related to the image content, among the total of 33877 object categories and 42374 relationship categories in the dataset, only the top 150 object categories and 50 relationship categories are selected as the annotation set.

[0030] The scene graph generation task can be divided into three sub-tasks, which are: 1) predicate classification: given an image, the bounding box of the object in the image and the label corresponding to the bounding box, the relationship between the objects in the image is predicted; 2) scene graph classification: given an image and the bounding box of the object in the image, the label corresponding to the bounding box and the relationship between the objects are predicted; 3) scene graph detection: only input the image, the bounding box of the object in the image, the label corresponding to the bounding box and the relationship between the objects are predicted.

[0031] For the object bounding box, the correct prediction criterion is that the overlap between the prediction result and the annotation is greater than 50%, and only one three-tuple relationship including subject, predicate and object is correctly predicted, and the bounding boxes of the subject object and the object object are correctly predicted. This relationship is correctly predicted; for the overall prediction result of a whole image, the topK recall rate is used to judge, wherein K is usually 20, 50 and 100, according to the probability distribution of the model prediction, all possible relationship triplets are sorted, and the topK triplets are taken, if the triplets include the triplets in the image annotation, then it is included in the recall rate, the specific formula is as follows:

[0032]

[0033] Wherein tK represents the number of correct predictions in the top K confidence ranked relationship triplets, n represents the total number of triplets in the image annotation.

[0034] The scene graph generation device and method based on alternating iterative optimization have the following advantages:

[0035] 1. The object features are converted according to the different relationships between objects, the fixed message passing mode of the existing method is changed, and the problem of missing context information is solved.

[0036] 2. The error transmission problem existing in the process of the traditional scene graph generation model is found, and an alternating iterative optimization algorithm is designed by utilizing the dependency relationship between the object recognition module and the relationship recognition network, so that the error transmission problem is alleviated.

[0037] 3. The statistical information of the co-occurrence frequency of objects and objects, and objects and relationships is not introduced, and the scene graph is predicted only according to the information contained in the image, so that the dependence of the model after training on a specific data set is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The error transmission problem of the existing scene graph generation method is shown in the figure. DETAILED DESCRIPTION

[0039] In order to better understand the purpose, structure and function of the present application, the present application based on alternating iterative optimization of scene graph generation device and method will be further described in detail below in combination with the drawings.

[0040] The scene graph generation device based on alternating iterative optimization of the present application comprises an image object detection module, a scene graph object recognition module and a scene graph relationship recognition module,

[0041] The image object detection module is responsible for locating the bounding box of the object in the input image, and extracting the object visual features corresponding to the bounding box;

[0042] The scene graph object recognition module is responsible for identifying the bounding box output by the image object detection module based on the output of the scene graph relationship recognition module, and obtaining the corresponding object category;

[0043] The scene graph relationship recognition module is responsible for extracting text features according to the output of the scene graph object recognition module, and combining object visual features and spatial features to identify the relationship between objects.

[0044] The scene graph generation method based on alternating iterative optimization of the present application comprises two stages of training and testing, and the steps of the training stage are as follows:

[0045] 1) Organize the data set.

[0046] Dataset preparation: The invention uses natural scene images with scene icon annotations as training data. Since there is a serious long-tail effect of objects and relationship categories in the actual scene, the object and relationship categories with high frequency of occurrence are selected as training data in the annotation. Specifically, the method selects the 150 most frequently occurring object categories and 50 relationship categories in the dataset as the image annotation set.

[0047] Further, all images used in training need to be uniformly scaled to 592x592 pixels. The model needs enough data for training, and at least 70,000 and 30,000 images are needed for training images and test images, respectively.

[0048] 2) Construct the training network.

[0049] The training network mainly includes an object detector, an object recognition module, and a relationship recognition module, wherein the object detector uses an existing pre-trained model, and the model parameters are fixed during the training process.

[0050] 2.1) The object recognition module uses a relationship-sensitive message passing network, trains different transformation matrices for different categories of relationships, and aggregates the messages of the adjacent nodes after transformation to update the features of the current node. The specific formula is as follows:

[0051]

[0052] where x i , z i are the node features before and after updating, Wt1, Wt2 are two transformation matrices shared by all nodes, Wr(i,j) is a group of transformation matrices corresponding to the relationship category, б represents a nonlinear function (such as ReLU function), LN represents Layer Normalization, N i is the neighborhood of node i. At the same time, for each edge in the input scene graph, a reverse transformation matrix is learned. Therefore, for the 50 relationship categories in the dataset, the module trains 100 different transformation matrices to fully utilize the relationship constraint to supervise message passing.

[0053] 2.2) The relationship recognition module identifies the relationship between the two by combining object features, spatial features, and text features of object classification, and its formula definition is as follows:

[0054] p r(i,j) = FC ([f1(x s ,x u ,x o ), f2(g s ,g o )])

[0055] where P r(i,j) represents the predicted probability distribution of the relationship between node i and node j, FC is a two-layer fully connected layer, f1 and f2 are two different long short-term memory neural networks respectively, s, u and o represent the subject object, the minimum bounding box region surrounding the two objects and the object, x and g represent the visual features and text features respectively.

[0056] 3) training using the network.

[0057] Before formally training the model, first, the pre-trained text feature extraction model is used to extract the text features of the objects and relationship categories in the image annotation set for subsequent training. At the same time, for the object detector model pre-trained on the external data set, fine-tuning training is performed on the scene graph data set to obtain the optimal object detection effect.

[0058] In the training, all parameters in the model are first randomly initialized. The object recognition module and the relationship recognition module are optimized according to the following alternating iterative optimization algorithm:

[0059]

[0060] where O is the output of the object recognition network, R is the output of the relationship recognition network, F is the output P r(i,j) of the relationship recognition module described above, and the corresponding functions. Finally, the probability distribution outputs of the object recognition module and the relationship recognition module in multiple iteration optimization steps are averaged to obtain the final probability distribution estimates of object recognition and relationship recognition respectively, and cross-entropy loss functions are used to constrain them respectively. The model is optimized by calculating the gradient of the loss function.

[0061] The steps of the test phase are as follows:

[0062] The test image is uniformly scaled to 592x592 pixels, and the scene graph generation model obtained by the above training is used for testing. Input a source image into the scene graph generation model to obtain the scene graph prediction result corresponding to the image.

[0063] The test environment and experimental results of the scene graph generation method based on alternating iterative optimization proposed by the present application are as follows:

[0064] (1) Test environment:

[0065] System environment: Ubuntu 18.04;

[0066] Hardware environment: memory: 32GB, GPU: NVIDIA Tesla V100, hard disk: 1TB;

[0067] (2) Experimental data:

[0068] 1) Visual Genome dataset. This dataset includes 108000 images in total, among which 75651 images are used for training and 32422 images are used for testing. Each image contains an average of 35 objects, 26 attributes, 21 pairs of inter-object relationships and 50 object region descriptions. All concepts in this dataset correspond to concepts in WordNet. At the same time, this dataset also includes a large number of question and answer pairs related to image content, totaling 1773258 pairs. Among the total of 33877 object categories and 42374 relationship categories in this dataset, only the top 150 object categories and 50 relationship categories are selected as the annotation set.

[0069] (3) Evaluation criteria and corresponding results:

[0070] The scene graph generation task can be divided into three sub-tasks, which are: 1) predicate classification: given an image, bounding boxes of objects in the image and labels corresponding to the bounding boxes, predict the relationships between objects in the image; 2) scene graph classification: given an image and bounding boxes of objects in the image, predict the labels corresponding to the bounding boxes and the relationships between objects; 3) scene graph detection (generation): only input an image, predict the bounding boxes of objects in the image, the labels corresponding to the bounding boxes and the relationships between objects. As shown in FIG. 1, it is an error propagation diagram of the existing scene graph generation method. Figure 1

[0071] For object bounding boxes, the correct prediction criterion is that the overlap (intersection-over-union) between the prediction result and the annotation is greater than 50%. Only when the subject object and the object object in a three-tuple relationship including subject, predicate and object are all correctly predicted, the relationship is considered to be correctly predicted. For the overall prediction result of an image, topK recall rate is usually used for evaluation, where K is usually 20, 50 and 100. Specifically, according to the probability distribution predicted by the model, all possible relationship triplets are sorted, and the topK triplets are taken, and if they include the triplets in the image annotation, they are included in the recall rate. The specific formula is as follows:

[0072]

[0073] where t K represents the number of correctly predicted relationship triplets in the topK relationship triplets ranked by confidence, and n represents the total number of triplets in the image annotation (since there are an average of 21 annotated triplets per image, K < n rarely occurs).

[0074] ​Correspondingly, the scene graph classification effect of the present application on the Visual Genome dataset is shown in Table 1:

[0075] Table 1. Scene graph classification effect of the present application on the Visual Genome dataset

[0076]

[0077]

[0078] Particularly, the present application adopts a relation-sensitive message passing network as the object recognition module, and an ablation experiment is performed on the module. Based on the formula in 2.1), the transformation matrix Wr(i,j) is changed to a non-relation category sensitive shared transformation matrix W r , and the fully connected scene graph is taken as the input graph of the message passing network, and a benchmark method "fully connected message passing network" can be obtained. The experimental results are as follows:

[0079]

[0080] It can be understood that the present application is described through some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to the features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, the features and embodiments can be modified to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application belong to the scope of protection of the present application.

Claims

1. A scene graph generation method based on an alternating iterative optimization scene graph generation device, the alternating iterative optimization scene graph generation device comprising an image object detection module, a scene graph object recognition module, and a scene graph relationship recognition module; the image object detection module is responsible for locating the bounding boxes of objects in an input image and extracting object visual features corresponding to the bounding boxes; the scene graph object recognition module is responsible for identifying the bounding boxes output by the image object detection module based on the output of the scene graph relationship recognition module to obtain corresponding object categories; the scene graph relationship recognition module is responsible for extracting text features according to the output of the scene graph object recognition module and identifying the relationships between objects in combination with object visual features and spatial features; The scene graph generation method comprises a training phase and a test phase, and the steps of the training phase are as follows: Step 1: Organize the data set: use natural scene images with scene graph annotations as training data; Step 2: Construct the training network: Step 2.1: The object recognition module uses a relationship-sensitive message passing network, trains different transformation matrices for different types of relationships, and aggregates the messages of the adjacent nodes after transformation to update the features of the current node; The specific formula is as follows: where x i , z i are the node features before and after updating, Wt1, Wt2 are two transformation matrices shared by all nodes, Wr(i,j) is a set of transformation matrices corresponding to the relationship category, b represents a nonlinear function, LN represents horizontal normalization, N i is the neighborhood of node i; at the same time, for each edge in the input scene graph, a reverse transformation matrix is learned, so for the 50 categories of relationships in the data set, a total of 100 different transformation matrices are trained to fully utilize the relationship constraint to supervise message passing; Step 2.2: The relationship recognition module identifies the relationship between the two by combining object features, spatial features, and text features of object classification; The specific formula is as follows: p r(i,j) = FC([f1(x s ,x u ,x o ),f2(g s ,g o )]) where P r(i,j) denotes the predicted probability distribution of the relation between node i and node j, FC is a two-layer fully connected layer, f1 and f2 are two different long short-term memory neural networks, s, u, o represent the subject object, the minimum bounding box region surrounding the two objects, and the object, respectively, and x and g represent the visual features and text features, respectively; Step 3: Train the network: Before formally training the model, first use the pre-trained text feature extraction model to extract the text features of the object and relationship categories in the image annotation set for subsequent training; At the same time, fine-tune the object detector model pre-trained on the external data set on the scene graph data set to obtain the optimal object detection effect; Step 4: Alternate step 2 several times and combine the results of object recognition and relationship recognition to obtain the scene graph corresponding to the image.

2. The scene graph generation method of claim 1, wherein, Step 1 comprises the following specific steps: 150 object categories and 50 relationship categories with the highest frequency in the data set are selected as the image annotation set; All images used in training are uniformly scaled to 592x592 pixels, and at least 70,000 and 30,000 images are required for training images and test images, respectively.

3. The scene graph generation method of claim 1, wherein, Step 3 comprises the following specific steps: In the training, all parameters in the model are randomly initialized, and the object recognition module and the relationship recognition module are optimized according to the following alternating iterative optimization algorithm: where O is the output of the object recognition network, R is the output of the relation recognition network, and F is the output P of the relation recognition module described above r(i,j) Corresponding functions, finally, by averaging multiple iterations of the optimization step object recognition module and relation recognition module probability distribution output, respectively, to obtain the final object recognition and relation recognition probability distribution estimates, and using cross-entropy loss function to constrain, by calculating the gradient of the loss function, the model using stochastic gradient descent method for optimization.

4. The scene graph generation method of claim 1, wherein, The training phase comprises the following steps: The test image is uniformly scaled to 592x592 pixels, and the trained scene graph generation model is used for testing. Input a source image into the scene graph generation model to obtain the scene graph prediction result corresponding to the image. The Visual Genome dataset is adopted, which includes 108,000 images in total, 75,651 training images and 32,422 test images, each image contains an average of 35 objects, 26 attributes, 21 pairs of inter-object relationships and 50 object region descriptions, all concepts correspond to the concepts in WordNet, meanwhile, the dataset also includes 1,773,258 pairs of question and answer related to image content, among the total of 33,877 object categories and 42,374 relationship categories in the dataset, only the top 150 object categories and 50 relationship categories are selected as the annotation set; The scene graph generation task can be divided into three sub-tasks, which are: 1) predicate classification: given an image, bounding boxes of objects in the image and labels corresponding to the bounding boxes, predict the relationships between objects in the image; 2) scene graph classification: given an image and bounding boxes of objects in the image, predict the labels corresponding to the bounding boxes and the relationships between objects; 3) scene graph detection: only input the image, predict the bounding boxes of objects in the image, the labels corresponding to the bounding boxes and the relationships between objects; For the object bounding box, the standard for judging the prediction to be correct is that the overlap between the prediction result and the annotation is greater than 50%, only one three-tuple relationship including subject, predicate and object is correctly predicted, and the bounding boxes of the subject object and the object object are correctly predicted, and this relationship is considered to be correctly predicted; for the overall prediction result of an image, topK recall rate is used to judge, K is 20, 50 and 100, according to the probability distribution of model prediction, all possible relationship triplets are sorted, and the topK triplets are taken, if the triplets include the triplets in the image annotation, they are counted into the recall rate, the specific formula is as follows: where t K represents the number of correctly predicted top-K ranked relationship triplets, and n represents the total number of triplets in the image annotation.

Images