A fine-grained scene graph generation method based on overlapping regions and geometric centers

Abstract

The application discloses a fine-grained scene graph generation method based on overlapping areas and geometric centers, relates to the technical field of scene graph generation, and comprises the following steps: acquiring a data set for training and evaluating a scene graph generation model; dividing the data set into a training set and a test set; sampling a specified number of images from the training set as a verification set; training the scene graph generation model by using the training set, verifying the trained scene graph generation model by using the verification set, further adjusting parameters of the scene graph generation model, and obtaining a trained scene graph generation model; and inputting the test data set into the scene graph generation model to obtain a scene graph. The multi-branch feature combination module, the overlapping area Transformer module and the geometric center enhancement module are used to process the overlapping areas and the geometric centers between objects, so that the ability of fine-grained predicate prediction is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of scene graph generation, and particularly relates to a fine-grained scene graph generation method based on overlapping areas and geometric centers. BACKGROUND

[0002] Deconvolution: Deconvolution is also called transposed convolution, but it is not the inverse operation of convolution. Deconvolution can map a small feature map to a large feature map, obtaining the relative position relationship information in the mapping, solving the problem of small feature map resolution after a series of convolution operations. At present, it is widely used in various fields of computer vision, such as object detection, image segmentation, image description, etc.

[0003] Vision receptive field: Vision receptive field is a concept in biology, which refers to a certain area on the retina. When a certain area on the retina is stimulated, it can activate the activity of each layer of nerve cells connected to the area in the visual system, and the area on the retina is the receptive field of these nerve cells.

[0004] Atrous convolution: Atrous convolution is also called dilation convolution. Unlike normal convolution, atrous convolution introduces a hyper-parameter called "dilation rate", which defines the spacing of each value when the convolution kernel processes data. Since adjacent pixels in an image are almost the same, if they all participate in convolution operation, it will produce redundancy, so atrous convolution selects to skip H pixel values and takes one effective value, which can reduce the amount of calculation on the premise of increasing the receptive field. In the problem of image needing global information, atrous convolution can be well applied, such as image segmentation and object detection.

[0005] Vision Transformer: Applying Transformer to computer vision tasks has attracted more and more attention from researchers. Compared with traditional convolutional neural networks and other types of networks such as recurrent neural networks, vision Transformer is becoming a hot topic in the field of computer vision with its superior performance and great potential. The Vision Transformer (ViT) architecture proposed by Google first cuts the image into blocks, and then applies the image block sequence directly to the Transformer architecture to complete the image classification task. Due to its excellent performance, more and more researchers have proposed models based on vision Transformer to improve various visual tasks.

[0006] The ultimate goal of computer vision is to build intelligent systems that can extract valuable information from digital images, videos, or other forms as humans do. Scene graph generation is an essential part of it. Scene graph generation is the task of automatically mapping an image into a semantic graph structure, which is a complex problem because it requires a machine to accomplish multiple computer vision tasks simultaneously, such as object recognition, attribute detection, and relationship detection. Scene graphs have always been the focus of research due to their powerful semantic representation and applications in scene understanding.

[0007] Due to the rapid development of deep learning, recent scene graph generation models have made substantial progress in several accuracy-based evaluation metrics. Existing scene graph generation models predict relationships between objects based on visual information input by a visual feature extraction procedure. Most existing methods strive for better feature extraction networks. Some works focus on improving the context aggregation module, which helps to learn better representations and thus improve performance. Some works identify the shortcomings of training using existing losses and propose handcrafted loss formulas to improve performance. However, existing scene graph generation models typically only use the features of the candidate bounding box regions of individual objects for relationship prediction, without considering the unique importance of the overlap region between two objects for predicate prediction.

[0008] Scene graph generation is the task of automatically mapping an image into a semantic graph structure, which requires correct labeling of detected objects and their relationships. With the continuous development of computer vision technology, people are no longer satisfied with simply detecting and recognizing objects in images, but expect higher-level understanding and reasoning of images. Scene graphs are a powerful tool for image understanding and have attracted widespread attention from many researchers. Scene graphs are a bridge connecting computer vision and natural language processing, and also bring potential revolution to downstream visual reasoning tasks.

[0009] Currently, the challenges in the field of scene graph generation mainly include the following four aspects: First, predicate semantics are not mutually exclusive, and semantic overlap is obvious, such as on (on) and walking on (walk on), standing on (stand on), sitting on (sit on), etc. Second, there is a serious long-tail distribution of predicates describing visual relationships, which is very obvious for commonly used datasets. Third, the image annotation of the dataset is too sparse, and there are obvious relationships between many target groups that are not annotated. Fourth, the dataset is not clean, in addition to the existing long-tail distribution and annotation sparsity problems, the dataset also has many repetitive and unreasonable annotations, and the target annotation bounding box is not accurate.

[0010] To solve the above problems, there are many researches on scene graph generation. On the one hand, researchers start to focus on the main content of the image that the scene graph should show from the perspective of perception. For example, Wang et al. consider that the Visual Genome dataset is densely annotated, and the COCO caption dataset describes the main content of the picture. They think that the visual relationship mentioned in the COCO caption dataset is the key relationship, and thus construct a new dataset for scene graph generation. Yu et al. find the reference of the noun in the description in the image through artificial method, and explicitly label the relationship.

[0011] On the other hand, researchers also consider introducing semantic association between predicate labels. For example, PCPL learns a vector representation for each predicate, and calculates the distance with other predicates according to the vector representation. The farther from other predicates, the stronger the semantic independence. In this way, the loss is reweighted. CogTree not only considers the gap between the output predicate and the labeled predicate when classifying, but also considers the matching degree of the output predicate and the ancestor nodes of the labeled predicate.

[0012] In addition, some researches introduce external information such as common sense and knowledge graph to assist scene graph generation. For example, GB-NET rephrases scene graph generation as the reasoning of the bridge between scene and common sense graph, where each object or predicate in the scene graph must be linked to the corresponding object or predicate class in the common sense graph. One-shot SGG introduces relationship knowledge and common sense knowledge for scene graph generation. Relationship knowledge represents the prior knowledge of the relationship between entities extracted from visual content, and common sense knowledge encodes the meaning construction knowledge.

[0013] The existing models solve the problem of non-mutually exclusive predicate semantics by introducing semantic association between predicate labels. From the perspective of perception, they focus on the main content of the picture that the scene graph should show, and solve the problem of sparse and dirty dataset annotation. From the perspective of external auxiliary knowledge, they solve the problem of long-tail distribution of predicates. However, these models do not perform well in fine-grained relationship prediction. For example, when the overlap area of the object is too small, the model cannot predict the relationship between the objects in a fine-grained manner. When two objects have a large overlap or no overlap, the relationship prediction between the objects will be wrong. The existing technologies and researches ignore the fine-grained features of the overlap area of the object candidate box and the relative geometric center relationship between the objects, which play a key role in fine-grained relationship prediction.

[0014] Therefore, focusing on the influence of the overlap area and the geometric center of the object candidate box on fine-grained scene graph generation, a fine-grained scene graph generation method based on overlap area and geometric center is proposed to solve the problems existing in the prior art, which is a problem that the person skilled in the art needs to solve urgently. SUMMARY

[0015] Therefore, the present application provides an overlap region and geometrical center-based fine-grained scene graph generation method, a multi-branch feature combination module (MFC Module) and an overlap region transformer module (ORT Module) are designed to fully obtain visual feature information contained in the overlap region of the object, the use of the two modules realizes the mutual complementation of the local connection characteristics of convolution and the global connection characteristics of attention. At the same time, a geometrical center augmented module (GCA) is designed to obtain the relative position information of the geometric center between objects, prevent the problem that the relative relationship between objects cannot be accurately judged when the overlap region ratio is too large or there is no overlap region. The ability of fine-grained predicate prediction is enhanced, and the accuracy and efficiency of scene graph generation are improved.

[0016] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0017] An overlap region and geometrical center-based fine-grained scene graph generation method, comprising the following steps:

[0018] S1, obtaining a data set for training and evaluating a scene graph generation model;

[0019] S2, dividing the data set into a training set and a test set; sampling a specified number of images from the training set as a validation set;

[0020] S3, training the scene graph generation model using the training set, obtaining the trained scene graph generation model, verifying the trained scene graph generation model using the validation set, further adjusting the parameters of the scene graph generation model, and obtaining the trained scene graph generation model;

[0021] S4, inputting the test data set into the scene graph generation model to obtain a scene graph.

[0022] The above method, optionally, the scene graph generation model is trained using the cross-entropy loss of object labels and predicate labels.

[0023] The above method, optionally, the scene graph generation model ORGC (Overlap Region and Geometrical Center) includes three branches,

[0024] In the first branch, the input image is subjected to Faster R-CNN to obtain candidate boxes {B=b i|i=1,2,…,n},RoI features{R=r i |i=1,2,…,n} and the initial object label {L=l i |i=1,2,…,n}, then encode the visual content of each object, merge them into a fused representation, and fine-tune the label l' of each object. i

[0025] The visual features x of the object are input to each LSTM unit. i and the initial object label l i Obtain a vector l of a pair of object labels. p =(l' i ,l' j After that, computation is performed through a joint embedding layer;

[0026] In the second branch, a multi-branch feature combination module and an overlapping region Transformer module are used to extract fine-grained overlapping region features v' of the two objects. overlap ;

[0027] In the third branch, the geometry center enhancement module is used to extract the relative geometry center feature R' of the bounding boxes between the two objects. p ;

[0028] Finally, the fusion function uses the input features obtained from the three branches to predict the final predicate Y:

[0029] SUM:Y=W x x' p +l' p +W o v' overlap +R' p ;

[0030] Where, x' p It is a fused representation of paired object features, l' i It is the label for each object, W x and W o These are trainable weights.

[0031] Optionally, in the first branch of the above method, a bidirectional LSTM is used to encode the visual content of each object: Input: {(b i ,r i ,l i Output: {x} i};

[0032] The paired object feature X takes values ​​in {x p =(x i ,x ji≠j; i, j = 1, 2, …, n}, merged into a fusion representation x' p : input: {x p = (x i , x j )} → output: {x' p};

[0033] Using LSTM as the decoder, fine-tune the label l' i : input: {(x i , l i )} → output: {l' i};

[0034] The calculation through the joint embedding layer is:

[0035] Where W l is a learnable weight, and l' i and l' j are the fine-tuned labels of objects i and j, Generate a unique vector R N×N of N-way object label pairs.

[0036] The above method, optionally, the overlap region is calculated as follows: when there is an intersection region between two candidate frames, the coordinates of the frame corresponding to the intersection region are:

[0037] x overlap1 = max(head_proposal(x1), tail_proposal(x1));

[0038] y overlap1 = max(head_proposal(y1), tail_proposal(y1));

[0039] x overlap2 = min(head_proposal(x2), tail_proposal(x2));

[0040] y overlap2 = min(head_proposal(y2), tail_proposal(y2));

[0041] bbox overlap = (x overlap1 , y overlap1 , x overlap2 , y overlap2 );

[0042] wherein head_proposal represents the first candidate box with intersection, tail_proposal represents the second candidate box with intersection, (x1, y1) and (x2, y2) represent the left upper corner coordinates and the right lower corner coordinates of the two candidate boxes respectively; (x overlap1 ,y overlap1 ) and (x overlap2 ,y overlap2 ) represent the left upper corner and the right lower corner coordinates of the overlapping region respectively; bbox overlap represents the obtained overlapping region bounding box.

[0043] The method can further include the following steps.

[0044] The overlapping region is converted into a feature map of 14x14x256 through RoI Align; the multi-branch feature combination module processes the feature map through convolution, deconvolution and multi-branch dilated convolution operation, the feature maps of the three different dilated convolution branches are spliced together to obtain a feature map of 28x28x768; and the feature map is converted from 28x28x768 to 7x7x256 through convolution, and is combined with the 7x7x256 feature map of the original overlapping region RoI obtained by the Faster R-CNN.

[0045] The overlapping region Transformer module and the multi-branch feature combination module are used in parallel, the overlapping region is first converted into a feature map of fixed size 14x14x256 through RoI Align, and then the self-attention of the overlapping region is obtained through the overlapping region Transformer module; the feature extraction structure in the overlapping region Transformer module is as follows: the intersection region feature of 14x14x256 is first flattened into 196x256, a class token is added to represent the global visual feature of the overlapping region, and the position information corresponding to each token is added; then the feature information is input into the encoding block of the scene graph Transformer for encoding to obtain the global visual feature of the overlapping region; the global visual feature 1x256 of the overlapping region is the input of the fine-grained relationship prediction.

[0046] In the geometric center enhancement module, the two-dimensional relative coordinate positions of each candidate box are as follows: wherein is the relative position coordinate of the left upper corner of the candidate box, is the relative position coordinate of the right lower corner of the candidate box; the relative center coordinates (x i ,y i ) of the candidate box, the width w i and the height h i are as follows:

[0047]

[0048]

[0049]

[0050] The relative feature of the geometric center relationship between the candidate boxes i and j is:

[0051]

[0052] R p = FC(r p );

[0053] R' p = ReLU(R p );

[0054] where r p is the relative geometric relationship between two candidate boxes, FC is a fully connected layer with an activation function, R p is a high-dimensional representation of r p , R' p is the relative geometric feature of the candidate boxes, and the ReLU activation function ensures that the model only considers the geometric relationship between the candidate boxes.

[0055] Optionally, the data set obtained in S1 is a Visual Genome (VG) data set.

[0056] Optionally, the data set in S2 is divided into a training set accounting for 70% and a test set accounting for 30%.

[0057] According to the above technical solution, compared with the prior art, the present application provides a fine-grained scene graph generation method based on overlapping regions and geometric centers, which uses a multi-branch feature combination module (MFC) and an overlapping region Transformer module (ORT) to fully obtain the visual feature information contained in the overlapping region of the object. The MFC module uses deconvolution and multi-branch dilated convolution to obtain high-resolution and multi-receptive field feature information in the overlapping region, and the ORT module uses Transformer to obtain self-attention in the overlapping region. The use of the two modules realizes the mutual complementation of the local connection characteristics of convolution and the global connection characteristics of attention.

[0058] Meanwhile, the application also designs a geometrical center augmented module (GCA) to obtain the relative position information of the geometric centers between objects, so as to prevent the problem that the relative relationship between objects cannot be accurately judged when the proportion of the overlapping area is too large or there is no overlapping area. Moreover, the GCA module integrates the relative geometric relationship between the center coordinates of the candidate frame of the object and the size of the candidate frame, and uses the relative geometric relationship between the center coordinates and the size of the candidate frame to improve the performance of the predicate prediction between objects. The spatial relative position relationship between objects is enhanced, and the problem that the relative position information between instances cannot be accurately judged when the candidate frame overlapping area has a large overlap or no overlapping area is solved.

[0059] The MFC module, the ORT module and the GCA module of the application can enhance the ability of fine-grained predicate prediction through the processing of the overlapping area and the geometric center between objects. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only the embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0061] Figure 1 A flow chart of a fine-grained scene graph generation method based on overlapping areas and geometric centers is provided for the application.

[0062] Figure 2 A model architecture diagram of a fine-grained scene graph generation method based on overlapping areas and geometric centers is provided for the application.

[0063] Figure 3 An intersection area and fine-grained relationship prediction graph is provided for the application.

[0064] Figure 4 A multi-branch feature combination module diagram is provided for the application.

[0065] Figure 5 An overlapping area Transformer module diagram is provided for the application.

[0066] Figure 6 A visualization result diagram of the scene graphs generated by the baseline model (orange), the baseline+MFC+ORT model (green, upper) and the baseline+GCA (green, lower) model is provided for the application.

[0067] Figure 7The visualization result figure of the scene graph generated by the baseline model (orange) and the model (green) provided by the present application. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0069] In the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive containing, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0070] REFERENCE Figure 1 As shown in the figure, the present application discloses a fine-grained scene graph generation method based on overlapping region and geometric center, comprising the following steps:

[0071] S1, obtaining a data set for training and evaluating a scene graph generation model;

[0072] S2, dividing the data set into a training set and a test set; sampling a specified number of images from the training set as a validation set;

[0073] S3, training the scene graph generation model using the training set, obtaining the trained scene graph generation model, verifying the trained scene graph generation model using the validation set, further adjusting the parameters of the scene graph generation model, and obtaining the trained scene graph generation model;

[0074] S4, inputting the test data set into the scene graph generation model to obtain a scene graph.

[0075] Further, the scene graph generation model is trained using the cross-entropy loss of object labels and predicate labels.

[0076] Further, referring to Figure 2 As shown in the figure, the scene graph generation model ORGC (Overlap Region and Geometrical Center) contains three branches,

[0077] In the first branch, input image is fed into Faster R-CNN to get candidate boxes {B = b i |i = 1, 2, …, n} and initial object labels {L = l | i = 1, 2, …, n} and initial object labels {L = l i |i = 1, 2, …, n} and initial object labels {L = l i

[0078] In each LSTM unit, input the visual feature x i of the object and the initial object label l i , get a pair of object label vectors l p = (l' i , l' j ) after calculation by joint embedding layer;

[0079] In the second branch, use multi-branch feature combination module and overlapping region Transformer module to extract fine-grained overlapping region features v overlap of the two objects;

[0080] In the third branch, use geometric center enhancement module to extract the relative geometric center features R p of the bounding boxes between the two objects;

[0081] Finally, the fusion function uses the input features obtained from the three branches to predict the final predicate Y:

[0082] SUM: Y = W x x' p + l' p + W o v' overlap + R' p ;

[0083] Where x' p is the fusion representation of the pair of object features, l' i is the label of each object, W x and W o are trainable weights.

[0084] Further, in the first branch, use bidirectional LSTM to encode the visual content of each object: input: {(b i , r i , l i )} → output: {x i};

[0085] The pair of object features X takes values in {x p = (xi ,x j The expression `|i≠j;i,j=1,2,…,n}` is merged into a single fused representation `x'`. p Input: {x p =(x i ,x j Output: {x' p};

[0086] Using LSTM as the decoder, the label l' of each object is fine-tuned. i Input: {(x i ,l i Output: {l' i};

[0087] The computation is performed through the joint embedding layer as follows:

[0088] Among them, W l These are learnable weights, l' i and l' j These are the labels after fine-tuning objects i and j. Generate a unique vector R of N-way object labels paired together. N×N .

[0089] Furthermore, all models are trained using cross-entropy loss between object labels and predicate labels.

[0090] Further, see Figure 3 As shown, in the actual relationship prediction process, the overlapping area of ​​two objects plays a crucial role in the fine-grained prediction of the predicate between the objects. That is, the fine-grained predicate can often be obtained solely through the overlapping area. In the overlapping area of ​​two object candidate boxes, the predicate between the two objects, "person" and "surfboard" (holding or standing on), can be determined in a fine-grained manner solely through the contact area (hand or leg) between the person and the surfboard.

[0091] The overlapping area is calculated as follows: when two candidate boxes have an intersection, the coordinates of the box corresponding to the intersection area are:

[0092] x overlap1 =max(head_proposal(x1),tail_proposal(x1));

[0093] y overlap1 =max(head_proposal(y1),tail_proposal(y1));

[0094] x overlap2 = min(head_proposal(x2), tail_proposal(x2));

[0095] y overlap2 = min(head_proposal(y2), tail_proposal(y2));

[0096] bbox overlap = (x overlap1 ,y overlap1 ,x overlap2 ,y overlap2 );

[0097] where head_proposal denotes the first candidate box with intersection, tail_proposal denotes the second candidate box with intersection, (x1, y1) and (x2, y2) denote the top-left corner coordinates and bottom-right corner coordinates of the two candidate boxes respectively, (x overlap1 ,y overlap1 ) and (x overlap2 ,y overlap2 ) denote the top-left corner and bottom-right corner coordinates of the overlapping region respectively, and bbox overlap denotes the obtained overlapping region bounding box.

[0098] RoI Align converts the overlapping region into a fixed-size feature map (14x14x256), and the multi-branch feature combination module and the overlapping region Transformer module process the intersection region to obtain more fine-grained relationship prediction. In particular, when there is no intersection region between the two candidate boxes, the above processing is not performed.

[0099] Further, the existing scene graph generation method is based on the visual information input by the visual feature extraction program to predict the relationship between objects. However, when the traditional target detection obtains the features after the ROI Align operation for relationship prediction, the spatial structure information contained in the feature map is greatly reduced due to multiple convolution and pooling in advance, and the receptive field of the feature map is also limited.

[0100] The multi-branch feature combination module in the present application processes the overlapping region feature map by using deconvolution and multi-branch dilated convolution to obtain more spatial structure information and receptive field information of the overlapping region. This module deeply mines the visual information in the overlapping region to enhance the contribution of the overlapping region visual information to relationship prediction, so that the relationship prediction pays more attention to the real visual content of the overlapping region, thereby improving the fine-grained relationship prediction ability between objects.

[0101] Referring to Figure 4As shown, the multi-branch feature combination module is specifically: the overlapping region is converted into a feature map of 14x14x256 through RoI Align; the multi-branch feature combination module processes the feature map by using convolution, deconvolution and multi-branch dilated convolution operation, the feature maps of the three different dilated convolution branches are spliced together to obtain a feature map of 28x28x768; and a series of convolutions are used to convert the feature map from 28x28x768 to 7x7x256, and combine it with the 7x7x256 feature map of the original overlapping region RoI obtained by the Faster R-CNN. In this way, the spatial structure information of the image capture and the receptive field contained in the feature map can be greatly improved, thereby further improving the fine-grained relationship prediction capability of the instance overlapping region.

[0102] In the multi-branch feature combination module, "3x3conv" represents a convolution with a kernel size of 3, a step of 1 and a padding width of 1, "two3x3conv" represents a convolution through two layers of 3x3conv, "2x2deconv" represents a deconvolution with a kernel size of 2 and a step of 2, "5x5conv" represents a convolution with a kernel size of 5, a step of 1 and a padding width of 2, "3x3conv rate=1" represents a 3x3conv convolution, "3x3conv rate=2" represents a dilated convolution with a kernel size of 3, a step of 1, a padding width of 2 and an expansion rate of 2, "2x2conv" represents a convolution with a kernel size of 2 and a step of 2, and the channel number of the feature map is 768 when connected, and the channel number at other places is 256.

[0103] Further, the multi-branch feature combination module can obtain the local visual features of the instance overlapping region, while ignoring the global visual information of the overlapping region. The overlapping region Transformer module uses Transformer to obtain the global visual features of the overlapping region of the two object candidate boxes. The parallel use of the overlapping region Transformer module and the multi-branch feature combination module simultaneously realizes the mutual complementation of the local connection characteristics of convolution and the global connection characteristics of attention, and can better improve the performance of relationship prediction in the scene graph generation process.

[0104] RoI Align first converts the overlapping region into a fixed-size 14x14x256 feature map, and then uses the overlapping region Transformer module to obtain the self-attention of the overlapping region; the overlapping region Transformer module can obtain the global visual features of the overlapping region to help the model make more fine-grained relationship prediction. The feature extraction structure in the overlapping region Transformer module is similar to that in the Transformer. See Figure 5As shown, the feature extraction structure in the overlap region Transformer module is as follows: the 14x14x256 overlap features are first flattened into 196x256, a class token (1x256) is added to represent the global visual features of the overlap region, and position information corresponding to each token is added; then the feature information is input into the encoder block of the scene graph Transformer for encoding to obtain the global visual features of the overlap region; the global visual features 1x256 of the overlap region are the input of the fine-grained relationship prediction.

[0105] Each Encoder Block of the overlap region Transformer module includes two sub-layers. Referring to Figure 5 As shown, the first sub-layer is Multi-Head Attention (MHS) used to calculate the self-attention of the input. The second sub-layer is a fully connected feed-forward neural network using a GELU activation function. Normalization operation (LayerNorm, LN) is applied before each sub-layer, and a residual network is applied after each sub-layer. In order to ensure the connection, the dimensions of all sub-layers and the output are the same, i.e. d model = 256.

[0106] Multi-Head Attention can be used to improve the performance of the self-attention layer. Different heads use different query, key and value matrices, which are randomly initialized. Specifically, in one embodiment of the present application, h in the Multi-Head Attention is set to 4, and the input overlap features are converted into three different vector groups, i.e. query, key and value groups, with a dimension of d q = d k = d v = d model / h = 64, and each head performs scaled dot-product attention separately; the process formula of the Multi-Head Attention is as follows:

[0107] MultiHead(Q',K',V') = Concat(head1,head2,head3,head4)W o ;

[0108] head i= Attention(Q i , K i , V i );

[0109]

[0110] where Q', K' and V' are and are weights.

[0111] Further, in one specific embodiment, in order to prevent the occurrence of the problem that the input image spatial information is missed when the overlap area of two object candidate boxes is too large or there is no overlap area, a geometrical center augmented module (GCA) is proposed, which uses the relative geometrical center position information between objects to assist predicate prediction, thereby improving the performance of inter-object fine-grained predicate prediction.

[0112] The GCA module integrates the relative geometric relationship between the center coordinates of the object candidate box and the size of the candidate box, thereby improving the performance of inter-object relationship prediction. Specifically, the two-dimensional relative coordinate position of each candidate box is where is the relative position coordinate of the upper left corner of the candidate box, is the relative position coordinate of the lower right corner of the candidate box; therefore, the relative center coordinates (x i , y i ) of the candidate box, the width w i and the height h i are respectively:

[0113]

[0114]

[0115]

[0116] The relative features of the geometrical center relationship between the candidate boxes of i and j are:

[0117]

[0118] R p = FC(r p ) ;

[0119] R' p = ReLU(R p ) ;

[0120] where r pis the relative geometric relationship between two bounding boxes, FC is a fully connected layer with an activation function, R p is the high-dimensional representation of r p , R' p is the relative geometric feature of the bounding boxes, and the ReLU activation function ensures that the model only considers the geometric relationship between the bounding boxes.

[0121] Further, the dataset obtained in S1 is the Visual Genome (VG) dataset.

[0122] Further, the dataset in S2 is divided as follows: the training set accounts for 70%, and the test set accounts for 30%.

[0123] In one specific embodiment, the dataset: the Visual Genome (VG) dataset is used to train and evaluate the model. The VG dataset consists of 75k object classes and 37k predicate classes, and contains 108k images. Since 92% of the predicates in the VG dataset have no more than 10 instances, the dataset is split following the widely used VG dataset split method, and the split dataset contains the 150 most frequently occurring object classes and 50 predicate classes. The split dataset only has a training set and a test set, with the training set accounting for 70% and the test set accounting for 30%. At the same time, according to the method of Neural Motifs, 5k images are sampled from the training set as a validation set for parameter adjustment.

[0124] Evaluation indicators: two evaluation indicators, Recall@K (R@K) and mean Recall@K (mR@K), are used to evaluate the performance of the scene graph generation model. Recall@K is the earliest and most widely accepted evaluation indicator. Since the Visual Genome dataset does not have complete annotations for relationships, accuracy cannot well reflect the effect of scene graph generation. Therefore, researchers use the Recall (recall rate) indicator from the retrieval field as the evaluation indicator for scene graph generation, which not only requires accurate identification, but also requires better elimination of object pairs without relationships. Considering the long-tail effect of the Visual Genome dataset, mean Recall@K calculates the recall rate of all predicate categories separately and then calculates their mean, so that all categories are equally important. The evaluation is carried out in a setting where the scene graph is generated directly from the image, which is also the most realistic application scenario for the scene graph generation model.

[0125] Target detection: Pretrain Faster R-CNN and freeze it as the underlying target detector of our scene graph generation model. Choose ResNeXt-101-FPN as the backbone network of Faster R-CNN and scale the long side of the input image to 1k pixels. Train the target detector on the VG dataset using 1 3090, with a batch size of 8 and an initial learning rate of 8x10 -3 , decaying to 1 / 10 of the existing learning rate at the 30kth and 40kth iterations. The final detector achieves a detection accuracy of 28.14 mAP on the VG dataset.

[0126] Scene graph generation: The target detector Faster R-CNN has been pre-trained and its parameters are fixed before the training of the scene graph generation model. The scene graph generation model is trained using a 3090. When training the scene graph generation model, the batch size is set to 12 and the initial learning rate is set to 12x10 -3 . Linear learning rate warm-up is applied on the first 1K steps, and then cosine learning rate decay is used. After the validation performance stabilizes, the learning rate will decay to 1 / 10 of the existing learning rate.

[0127] The experimental quantitative results of various existing state-of-the-art models and the model proposed in the embodiment of the present application on the Visual Genome dataset are shown in Table 1, which shows the performance comparison of the model proposed in the embodiment of the present application and various existing state-of-the-art models on various evaluation indicators of scene graph generation. From Table 1, it can be clearly observed that compared with existing models, the method of the embodiment of the present application is significantly improved in six evaluation indicators (R@20, R@50, R@100, mR@20, mR@50 and mR@100), and achieves good results. From the experimental results in Table 1, it can be found that the model reaches 26.89, 33.88 and 38.32 in evaluation indicators R@20, R@50 and R@100, respectively. The model improves by 5.7%, 4.4% and 2.8% respectively in R@20, R@50 and R@100 evaluation indicators compared with the previous best model Causal-TDE; in the case of achieving the current best experimental effect in the evaluation indicator Recall, the model is also better than the corresponding model in the evaluation indicators mR@20, mR@50 and mR@100, and can reach 6.22, 8.38 and 9.45 under the same model setting.

[0128] Table 1 quantitative results.

[0129]

[0130]

[0131] In another embodiment of the present application, in order to study the contribution of each component of the model, extensive ablation experiments are carried out, and the experimental results are shown in Table 2, which shows the experimental results of various modules and combinations of modules. Through the ablation experiment, it can be found that the multi-branch feature combination module (MFC) can improve the performance of the model from 32.45 to 33.29, which is 2.6% higher than the baseline model. The overlapping region Transformer module (ORT) can improve the performance of the model from 32.45 to 32.92, which is 1.4% higher than the baseline model. The geometric center enhancement module (GCA) can improve the performance of the model from 32.45 to 32.89, which is 1.4% higher than the baseline model. The joint use of the MFC module and the ORT module can improve the performance of the model from 32.45 to 33.57, which is 3.5% higher than the baseline model. The joint use of the MFC module, the ORT module and the GCA module can improve the performance of the model from 32.45 to 33.88, which is 4.4% higher than the baseline model. Through the ablation experiment, it is fully proved that the modules proposed in the present model can improve the performance of the scene graph generation model.

[0132] Experiments show that the multi-branch feature combination module can enhance the spatial structure information and receptive field contained in the visual features, improve the dependence of relationship prediction on visual information, make relationship prediction pay more attention to real visual content, and thus improve the performance on the Visual Genome dataset. The R@50 index on the Visual Genome test set is improved by 0.8 percentage points compared with the baseline model.

[0133] The MFC module and the ORT module act on the overlapping region of two objects, and the visualization experimental results of the joint use of the two modules are shown in Figure 6 The orange box in the figure represents an example of predicate prediction results of the baseline model, and the green box represents an example of predicate prediction results after using the MFC module and the ORT module. The joint use of the two modules can generate a new triple <woman, walking on, sidewalk>, instead of the triple <woman, on, sidewalk> generated by the baseline model. Experiments prove that using the MFC module and the ORT module on the overlapping region can obtain fine-grained visual feature information contained in the overlapping region of the objects, and thus obtain more fine-grained predicate prediction results.

[0134] Experimental results show that our geometric position center module can enhance the geometric position relationship between instances, so that relationship prediction pays more attention to the position information between instances, thereby improving the performance on the Visual Genome dataset. The R@50 index on the Visual Genome test set is improved by 0.4 percentage points compared with the baseline model.

[0135] GCA acts on two objects, the visualized experimental results are shown in Figure 7 The model using the GCA module can generate a new triple <car, parked on, street> instead of the triple <car, on, street> generated by the baseline model. Experiments prove that the GCA module is used to obtain the relative position information of the geometric centers between objects, thereby improving the accuracy of predicate prediction in scene graph generation.

[0136] Table 2 ablation experiment

[0137] MODEL R@20 R@50 R@100 mR@20 mR@50 mR@100 Baseline 25.42 32.45 37.26 4.36 5.83 7.08 Baseline+MFC 26.23 33.29 37.91 5.62 7.37 8.49 Baseline+ORT 25.91 32.92 37.62 5.18 6.92 7.96 Baseline+GCA 25.85 32.89 37.48 4.95 6.65 7.81 Baseline+MFC+ORT 26.58 33.57 38.19 6.05 8.09 9.05 Baseline+MFC+ORT+GCA 26.89 33.88 38.32 6.22 8.38 9.45

[0138] In another embodiment of the present application, the same model is used for comparative experiments on overlapping regions and union regions, and the experimental results are shown in Table 3. Overlap Region represents the overlapping region of two objects, Union Region1 represents the union region of two objects when they have overlapping regions, Union Region2 represents the union region of two objects, including two objects with overlapping regions and no overlapping regions. The model used in the comparative experiment is the Baseline+MFC+ORT model. Baseline represents the baseline model, MFC represents the multi-branch feature combination module, and ORT represents the overlapping region Transformer module. By comparing the results of Overlap Region and Union Region1, it can be found that when there is an overlapping region between two objects, the overlapping region of two objects has better performance than the union region for relationship prediction, which proves the special importance of the overlapping region for relationship prediction, and also proves that the proposed module has better effect on the overlapping region than the union region. By comparing the results of Union Region1 and Union Region2, it can be found that when there is no overlapping region between two objects, paying too much attention to the union region of two objects will affect the performance of the model for relationship prediction.

[0139] Table 3 comparison of overlapping regions and union regions

[0140]

[0141] Visualize several scene graph examples generated by the baseline model and the proposed model, and the results are shown in Figure 7The bottom row shows the man without touching the kite. The baseline model produces "holding" which means the original model's prediction is easy to be "blind", while our model successfully predicts "looking at". All the examples show a clear trend that our handling of overlapping regions and geometric centers plays a greater role in fine-grained scene graph generation.

[0142] Experiments show that the model of the present application can significantly enhance the ability of fine-grained relationship prediction in the scene graph generation process. The proposed method does not make any assumptions about the underlying generation model and can be easily used with any model.

[0143] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A fine-grained scene graph generation method based on overlapping regions and geometric centers, characterized in that, The method comprises the following steps: S1, obtaining a data set for training and evaluating a scene graph generation model; S2, dividing the data set into a training set and a test set; S3, training the scene graph generation model using the training set, obtaining the trained scene graph generation model, and verifying the trained scene graph generation model using the verification set, further adjusting the parameters of the scene graph generation model, and obtaining the trained scene graph generation model; S4, inputting the test data set into the scene graph generation model to obtain a scene graph; The scene graph generation model ORGC comprises three branches, Finally, the fusion function predicts the final predicate Y using the input features obtained from the three branches: In the first branch, input image, through Faster R-CNN to get candidate frame , RoI feature and the initial object label , then encode the visual content of each object, merge into a fusion representation, fine-tune the label of each object ; visual features of the input object at each LSTM cell and the initial object label , obtaining a vector of the pair of object labels are then computed by a joint embedding layer; In the second branch, the multi-branch feature combination module and the overlapping region Transformer module are used to extract the fine-grained overlapping region features of the two objects ; In the third branch, the relative geometric center features of the bounding boxes between two objects are extracted using a geometric center enhancement module ; The relative features of the geometric center relationship between the i and j candidate boxes are: ； wherein, is a fused representation of pairs of object features, is a label for each object, and are trainable weights; In the geometric center enhancement module, the two-dimensional relative coordinate position of each candidate box is wherein is the relative position coordinate of the upper left corner of the candidate box, is the relative position coordinate of the lower right corner of the candidate box; the relative center coordinate of the candidate box is , the width and the height , respectively. ； ； ； The scene graph generation model is trained using the cross-entropy loss of the object label and the predicate label. ； ； ； wherein, is the relative geometric relationship between two candidate boxes, is a fully connected layer with an activation function, is a high-dimensional representation of is the relative geometric feature of the candidate boxes, The activation function ensures that the model only considers the geometric relationship between the candidate boxes.

2. The method of claim 1, wherein, 3. The method according to claim 1, wherein, 4. The method according to claim 1, wherein, In the first branch, bidirectional LSTM is utilized to encode the visual content of each object: ; The pair of object features X takes values in , merged into a fused representation : ; Fine-tune the label of each object with LSTM as the decoder : ; The calculation by the joint embedding layer is: ； wherein, is a learnable weight, and is an object and a fine-tuned label, produces a unique vector of N-way object label pairs . The overlapping region is calculated as follows: when there is an intersection region between two candidate boxes, the coordinates of the box corresponding to the intersection region are:

5. The method according to claim 1, wherein, ； ； ； ； ； wherein, represents the first candidate box where there is an intersection, represents the second candidate box where there is an intersection, and respectively represent the top-left corner coordinates and the bottom-right corner coordinates of the two candidate boxes; and respectively represent the top-left corner and the bottom-right corner coordinates of the overlapping region; represents the resulting overlapping region bounding box. The multi-branch feature combination module specifically comprises: converting the overlapping region into a 14x14x256 feature map through RoI Align; processing the feature map through convolution, deconvolution, and multi-branch dilated convolution operations; splicing the feature maps of the three different dilated convolution branches to obtain a 28x28x768 feature map; and converting the feature map from 28x28x768 to 7x7x256 through convolution, and combining it with the 7x7x256 feature map of the original overlapping region RoI obtained by the Faster R-CNN.

6. The method according to claim 1, wherein, The overlapping region Transformer module and the multi-branch feature combination module are used in parallel, RoI Align first converts the overlapping region into a fixed-size 14x14x256 feature map, and then uses the overlapping region Transformer module to obtain the self-attention of the overlapping region; the feature extraction structure in the overlapping region Transformer module is as follows: the 14x14x256 intersection region feature is first flattened into 196x256, a class token is added to represent the global visual feature of the overlapping region, and the position information corresponding to each token is added; then the feature information is input into the encoding block of the scene graph Transformer for encoding to obtain the global visual feature of the overlapping region; The global visual feature 1x256 of the overlapping region is the input of the fine-grained relationship prediction.

7. The method according to claim 1, wherein, ​ The dataset obtained in S1 is a Visual Genome dataset.

8. The method according to claim 1, wherein the method is characterized in that, The division of the dataset in S2 is that the training set accounts for 70%, and the test set accounts for 30%.

Images