Salient scene graph generation method based on important instance pair feature learning

Abstract

The invention discloses a significant scene graph generation method based on important instance pair feature learning. The method comprises the following steps: inputting a training image in a data set into a target detector to generate an instance tag of a to-be-predicted relationship and an object instance tag, and splicing the instance tag and the object instance tag to form an instance pair tag of a prediction relationship; respectively establishing corresponding models based on the instance pair labels of the prediction relation and the instance pair labels of the real relation to respectively extract features; using Manhattan distance constraint to expand the difference between the instance pair features of the prediction relationship to obtain distance loss; an instance pair feature PoolFormer module based on importance scores is adopted to fuse instance pair features of the real relation and important instance pair features of the real relation generated by the relation importance scores and instance pair features of the prediction relation to apply Euclidean distance constraint, and instance pair loss is generated; summing the distance loss and the instance pair loss to obtain total loss, and performing model training to obtain a trained scene graph generation model; and the recognition capability of the model on important relationships is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and computer vision, and particularly to a method for generating a significant scene graph based on feature learning of important instances. Background Art

[0002] Scene Graph Generation (SGG) is a cutting-edge topic in the field of artificial intelligence and one of the key tasks in computer vision. This task combines image recognition and natural language processing technologies, aiming to identify and classify various instances in an image and predict the visual relationships between these instances. Existing scene graph generation methods often attempt to capture all the relationships in the image, resulting in the generated scene graph containing a large amount of redundant information. If such a scene graph is used for image caption generation in downstream tasks, the generated captions are verbose and lack logical hierarchy, making them difficult to read and understand. For readers, it may cause information overload.

[0003] By using the method of fuzzy logic to calculate the importance scores of real relationships in the VG150 dataset, the importance of relationships can be evaluated flexibly and comprehensively without reducing the number of relationship samples. At the same time, a plug-and-play relationship loss weighting module is designed, which uses the relationship importance scores as weights to weight the loss of each relationship sample, enabling the model to pay more attention to important relationships. Although this method can effectively improve the model's ability to identify important relationships, it only applies the relationship importance scores to the loss calculation of relationship features, ignoring the in-depth modeling of instance pair features. Therefore, the role of relationship importance scores is not fully exerted, resulting in the accuracy of important relationship prediction not reaching the ideal effect and not fully meeting the requirements of downstream advanced visual understanding tasks for scene graphs. Summary of the Invention

[0004] Object of the Invention: In order to overcome the deficiencies in the prior art, the present invention provides a method for generating a significant scene graph based on feature learning of important instances. By introducing the instance pair features of relationship importance scores and real relationships and applying the Euclidean distance constraint to the instance pair features of predicted relationships after fusion, the model can learn the instance pair features of important relationships and improve the model's ability to identify important relationships.

[0005] Technical Solution: To achieve the above object, a method for generating a significant scene graph based on feature learning of important instances according to the present invention includes the following steps:

[0006] S1. Concatenate the instance labels of the relationships to be predicted and the object instance labels generated by inputting the training images in the dataset into the target detector to form the instance pair labels of the predicted relationships;

[0007] S2. Based on the instance pairs of the predicted relationships and the instance pairs of the true relationships, establish corresponding models respectively, and extract the instance pair features of the predicted relationships and the instance pair features of the true relationships;

[0008] S3. Use the Manhattan distance constraint to expand the differences between the instance pair features of the predicted relationships to obtain the distance loss;

[0009] S4. Based on the input relationship importance scores, use the instance pair feature PoolFormer module based on the importance scores to fuse the instance pair features of the true relationships and the relationship importance scores to generate the important instance pair features of the true relationships;

[0010] S5. Impose the Euclidean distance constraint between the instance pair features of the predicted relationships and the important instance pair features of the true relationships, so that the instance pair features of the predicted relationships are close to the important instance pair features of the true relationships, and generate the instance pair loss;

[0011] S6. Sum the distance loss obtained in step S3 and the instance pair loss obtained in step S5 to obtain the total loss, and train the scene graph generation model based on the obtained total loss;

[0012] S7. Determine whether the set number of training rounds is reached. If not, return to step S1. If so, complete the training and output the trained scene graph generation model.

[0013] Further, in step S1, the subject instance label of the relationship to be predicted and the object instance label are concatenated to obtain the instance pair label f p of the predicted relationship; at the same time, the instance pair label f t of the true relationship is generated by concatenating the true subject instance label and the object instance label ;

[0014]

[0015] where [.,.] represents column-wise concatenation.

[0016] Further, in step S2, a model is established for the instance pair label f p of the predicted relationship, and a number of instance pair features E p of the predicted relationship are extracted. At the same time, a model is established for the instance pair label f t of the true relationship, and a number of instance pair features E p of the predicted relationship are extracted.

[0017] E p = BN(ReLU(FC(f p ))), Et = BN(ReLU(FC(f t )))

[0018] In the formula, FC, BN, and ReLU respectively represent the fully connected layer, batch normalization operation, and ReLU activation function.

[0019] Furthermore, in step S3, the Manhattan distance constraint method is used to calculate the instance pair feature E of all the predicted relationships extracted by the model pi and E pj The Manhattan distance d between p is obtained, and the Manhattan distances calculated for the instance pair features of each predicted relationship are combined to obtain the distance matrix D p =(d p )∈R (N+1)×(N+1) ;

[0020] The elements in each row of several rows in the distance matrix D p are sorted in ascending order to obtain the sorted distance matrix D' p =(d' p )∈R (N+1)×(N+1) ; and the first k minimum values in each row are selected for summation and averaging to obtain the average Manhattan distance The distance loss L is calculated through the obtained average Manhattan distance p_mah ;

[0021]

[0022] In the formula, γ is a hyperparameter used to adjust the distance margin.

[0023] Furthermore, the instance pair feature PoolFormer module based on importance scores includes a feature extraction layer, a normalization layer, and a multi-layer perceptron; the instance pair label is obtained by concatenating the subject instance label and the object instance label, and the instance pair feature is obtained by extracting features from the instance pair label through the feature extraction layer; the instance pair feature is input into the feature pooling token mixer for operation after batch normalization, and the output of the feature pooling token mixer is added and fused with the instance pair feature to obtain the instance pair feature capturing local features; the instance pair feature capturing local features is successively subjected to batch normalization and multi-layer perceptron operations, and the output of the multi-layer perceptron is added and fused with the instance pair feature capturing local features and then multiplied and fused with the relationship importance score to obtain the important instance pair feature; the important instance pair feature is calculated to obtain the instance pair loss.

[0024] Further, for the instance pair based on the importance score, the feature pooling token mixer FtTokenMixer in the Feature PoolFormer module performs one-dimensional pooling on the instance pair features of the true relationship to capture the changes in local features, and retains the information of the original features through residual connection; the calculation process is as follows:

[0025] E’ t = FtTokenMixer(BN(E t )) + E t

[0026] In the formula, E’ t is the instance pair feature of the true relationship that captures local features, BN is the batch normalization operation, and FtTokenMixer(.) represents the operation of the feature pooling token mixer;

[0027] Use the relationship importance score Rim to weight the instance pair feature E′ t of the true relationship that captures local features to generate the important instance pair feature to capture important instance pair information, and the calculation process is as follows:

[0028]

[0029] In the formula, GeLU represents the GeLU activation function, and w1 and w2 represent learnable weight parameters.

[0030] Further, the specific operation of the feature pooling token mixer FtTokenMixer is as follows, and the calculation process is as follows:

[0031]

[0032] In the formula, T′ ：，i represents the output feature after the pooling token mixing operation at position i, T ：，i represents the original input feature at position i, and M represents the size of the pooling window.

[0033] Further, by imposing an Euclidean distance constraint on the instance pair feature E p of the predicted relationship and the important instance pair feature of the true relationship, the Euclidean distance d t between the instance pair feature of the predicted relationship and the important instance pair feature of the true relationship is obtained, and the calculated Euclidean distances are combined to obtain the distance matrix D t =(d t ) ∈ R (N+1)×(N+1) ;

[0034]

[0035] In the formula, is expressed as the square of the two-norm;

[0036] Calculate the mean v and standard deviation σ of all Euclidean distances in the distance matrix, and obtain the normalized distance matrix D′ based on the mean v and standard deviation σ t =(d′ t )∈R (N+1)×(N+1) ;

[0037]

[0038] In the formula, v is the mean of all Euclidean distances in the distance matrix, and σ is the standard deviation of the Euclidean distances in the distance matrix;

[0039] Sort the values in the normalized distance matrix in descending order and prune the extreme values. Take the average of the absolute values of the pruned Euclidean distances as the instance pair loss Lpair. The calculation process is as follows:

[0040]

[0041] In the formula, N represents the number of instance pair features, η = [δ·N] represents the number of pruned front and back ends of the Euclidean distances after sorting N instance pair features, and δ represents the hyperparameter of the pruning ratio.

[0042] Furthermore, in step 6, add the distance loss L p_mah and the instance pair loss Lpair to obtain the final constraint loss L;

[0043] L = L pair + L p_mah .

[0044] Furthermore, test the trained model in the significant scene graph generation method based on important instance pair features, including the following steps:

[0045] S1-1: Input the test image into the trained model for relationship prediction to obtain the relationship prediction result;

[0046] S1-2. Compare the relationship prediction result with the true predicate label, and evaluate the relationship prediction result using the evaluation index;

[0047] S1-3. Evaluate the prediction ability of the model for important relationships, and evaluate the relationship prediction result using the important relationship recall rate IR@K and the important relationship accuracy rate IP@K;

[0048] S1-4. Based on the relationship importance score, screen the important relationships in the predicted relationships to form a scene graph;

[0049] S1-5. Combine and output the test results based on the results obtained in steps S1-2, S1-3, and S1-4.

[0050] Beneficial effects: A method for generating a significant scene graph based on feature learning of important instance pairs in the present invention designs a plug-and-play feature learning module for important instance pairs. This module first expands the differences between the instance pair features predicted by the model through Manhattan distance constraints, so that the model can better distinguish different instance pair information in relation prediction. Then, it uses the Feature PoolFormer network to fuse the instance pair features and relationship importance scores to enhance the representation of important relationship instance pair features, enabling the model to fully learn the instance pair features of important relationships, thereby improving the model's ability to recognize important relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of a method for generating a significant scene graph based on feature learning of important instance pairs;

[0052] Figure 2 is a network structure diagram of an instance pair feature PoolFormer module based on importance scores;

[0053] Figure 3 is a block diagram of a method for generating a significant scene graph based on feature learning of important instance pairs. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] The present invention will be further described in detail below with reference to the accompanying drawings.

[0055] As Figure 1 shown, a method for generating a significant scene graph based on feature learning of important instance pairs includes the following steps:

[0056] S1. Concatenate the instance labels of the relationships to be predicted and the object instance labels generated by inputting the training images in the dataset into the target detector to form the instance pair labels of the predicted relationships;

[0057] S2. Based on the instance pair labels of the predicted relationships and the instance pair labels of the true relationships, establish corresponding models respectively, and extract the instance pair features of the predicted relationships and the instance pair features of the true relationships;

[0058] S3. Use Manhattan distance constraints to expand the differences between the instance pair features of the predicted relationships to obtain a distance loss;

[0059] S4. Based on the input relationship importance scores, use the instance pair feature PoolFormer module based on importance scores to fuse the instance pair features of the true relationships and the relationship importance scores to generate the important instance pair features of the true relationships;

[0060] S5. Impose an Euclidean distance constraint between the instance pair features of the predicted relationship and the important instance pair features of the true relationship, so that the instance pair features of the predicted relationship are close to the important instance pair features of the true relationship, and generate an instance pair loss;

[0061] S6. Sum the distance loss obtained in step S3 and the instance pair loss obtained in step S5 to obtain the total loss, and train the scene graph generation model based on the obtained total loss;

[0062] S7. Determine whether the set number of training rounds is reached, that is, whether all training pictures have been read. Otherwise, return to step S1. If yes, complete the training and output the trained scene graph generation model, that is, the instance pair feature PoolFormer scene graph generation network model based on the importance score. Similarly, when the set batch size is reached during the training model, return to step S1. The input image is processed by the trained instance pair feature PoolFormer scene graph generation network model based on the importance score and then a significant scene graph is output.

[0063] The instance labels of the relationship to be predicted generated by inputting the training images in the dataset into the target detector. The training images are all from the standard dataset Visual Genome (VG) for scene graph generation. This dataset consists of 108,077 images, including 150 most common instance classes and 50 predicate classes. 70% of the images in the dataset are divided into the training set, and the remaining 30% are the test set in the present invention. The Faster R-CNN is used as the target detector, and the instance labels of the relationship to be predicted are obtained through the target detector. In the working process of most current scene graph generation models, usually a target detector is first deployed. The role of the target detector is to recognize the input image and extract the instance proposals and corresponding features therein.

[0064] In the step S1, the subject instance label of the relationship to be predicted and the object instance label are concatenated to obtain the instance pair label f of the predicted relationship p ; meanwhile, the instance pair label f of the true relationship t is generated by concatenating the true subject instance label and the object instance label ;

[0065]

[0066] In the formula, [.,.] represents column-wise concatenation.

[0067] In the step S2, for the instance pair label f of the predicted relationship pBuild a feature extraction model to extract a number of instance pair features E of the prediction relationship p , and at the same time, for the instance pair label f of the true relationship t Build a feature extraction model to extract a number of instance pair features E of the prediction relationship p ,

[0068] E p = BN(ReLU(FC(f p ))), E t = BN(ReLU(FC(f t )))

[0069] In the formula, FC, BN, and ReLU respectively represent the fully connected layer, batch normalization operation, and ReLU activation function.

[0070] In step S3, in order to expand the difference between any two different instance pair features E pi and E pj , the Manhattan distance constraint method is used to calculate the Manhattan distance d pi between all the instance pair features E pj of the prediction relationship extracted by the model, and the Manhattan distances calculated for each instance pair feature of the prediction relationship are combined to obtain the distance matrix D p =(d p )∈R p ; (N+1)×(N+1) ;

[0071] d p =|E pi -E pj |

[0072] In the formula, i, j ∈ {1, 2, 3,..., N}, N represents the number of instance pair features, and |.| represents the absolute value operation;

[0073] Sort the elements in each row of several rows in the distance matrix D p in ascending order to obtain the sorted distance matrix D' p =(d' p )'∈R (N+1)×(N+1) ; and select the sum of the first k minimum Manhattan distances in each row and take the average value, that is, add up all the first k minimum values in all rows of the matrix and take the average value to obtain the average Manhattan distance

[0074]

[0075] Calculate the distance loss L p_mah through the obtained average Manhattan distance;

[0076]

[0077] where γ is a hyperparameter for adjusting the distance margin, and max is used to find the maximum value. L p_mah The smaller it is, the greater the difference between the features E of any two different instance pairs pi and E pj so that the model can better distinguish different instance pair information in relation prediction.

[0078] In the step S4, the instance pair feature PoolFormer module based on the importance score combines the Pooling operation with the Transformer architecture. The instance pair feature PoolFormer module based on the importance score includes a feature extraction layer, a normalization layer, and a multi-layer perceptron; after the main instance label and the object instance label are concatenated, an instance pair label is obtained, and the instance pair label extracts features through the feature extraction layer to obtain instance pair features; the instance pair features are input into the feature pooling token mixer for operation after batch normalization, and the output of the feature pooling token mixer is added and fused with the instance pair features to obtain instance pair features capturing local features; the instance pair features capturing local features are sequentially subjected to batch normalization and multi-layer perceptron operations, and the output of the multi-layer perceptron is added and fused with the instance pair features capturing local features and then multiplied and fused with the relationship importance score to obtain important instance pair features; the important instance pair features are calculated to obtain the instance pair loss.

[0079] The feature pooling token mixer FtTokenMixer in the instance pair feature PoolFormer module based on the importance score performs one-dimensional pooling on the instance pair features of the true relationship to capture the changes in local features, and retains the information of the original features through residual connection; the calculation process is as follows:

[0080] E′ t = FtTokenMixer(BN(E t )) + E t

[0081] where E′ t is the instance pair feature of the true relationship capturing local features, BN is the batch normalization operation, and FtTokenMixer(.) represents the operation of the feature pooling token mixer;

[0082] The relationship importance score Rim of the true relationship calculated through fuzzy logic in the VG150 dataset is used to weight the instance pair feature E′ t of the true relationship capturing local features to generate important instance pair features to capture important instance pair information, and the calculation process is as follows:

[0083]

[0084] In the formula, GeLU represents the GeLU activation function, and w1 and w2 represent learnable weight parameters.

[0085] The specific operation of the feature pooling token mixer FtTokenMixer is calculated as follows:

[0086]

[0087] In the formula, T ′：，i represents the output feature after the pooling token mixing operation at position i, and T ：，i represents the original input feature at position i, and M represents the size of the pooling window.

[0088] Apply the Euclidean distance constraint to the instance pair features of several predicted relationships and the important instance pair features of several calculated true relationships one by one; by applying the Euclidean distance constraint to the instance pair feature E p of the predicted relationship and the important instance pair feature of the true relationship, obtain the Euclidean distance d t between the instance pair features of several predicted relationships and the important instance pair features of the true relationship, and combine the calculated Euclidean distances to obtain the distance matrix D t =(d t )∈R (N+1)×(N+1) ;

[0089]

[0090] In the formula, represents the square of the two-norm, that is, the square of the Euclidean norm;

[0091] Calculate the mean v and standard deviation σ of all Euclidean distances in the distance matrix, and calculate the normalized distance matrix D' t =(d' t )∈R (N+1)×(N+1) ;

[0092]

[0093] In the formula, v is the mean of all Euclidean distances in the distance matrix, and σ is the standard deviation of the Euclidean distances in the distance matrix;

[0094] To avoid the normalized distance set D tThe influence of extreme values in ′ on the final calculated distance loss. The values in the normalized distance matrix are sorted in descending order and the extreme values are trimmed. The average of the absolute values of the trimmed Euclidean distances is used as the instance pair loss Lpair. The calculation process is as follows:

[0095]

[0096] In the formula, N represents the number of instance pair features, η = [δ·N] represents the number of trimming at the front and back ends of the Euclidean distances after sorting the N instance pair features, δ represents the hyperparameter of the trimming ratio, and d′ tj is the j-th normalized Euclidean distance.

[0097] In step 6, the distance loss L p_mah and the instance pair loss Lpair are added together to obtain the final constraint loss L;

[0098] L = L pair+ L p_mah .

[0099] When testing the scene graph generation model, it is tested separately under three subtasks, namely predicate classification (Predicate Classication, Predcls), whose goal is to understand the relationship categories between instances in the image, rather than deeply processing the recognition of the instances themselves; specifically, the model receives all instance bounding boxes and their labels in the image as input, and the focus is on predicting the semantic relationships between these instances. Since the model can directly obtain the bounding box and category information of the instances, the localization and recognition of the instances are assumed to be completed. Therefore, this task pays more attention to the modeling of interactions between entities and semantic understanding, and less involves the complexity of instance recognition.

[0100] Scene graph classification (Scene Graph Classification, SGcls) is similar to the predicate classification task in terms of input, and also provides the bounding boxes of all instances in the image. However, the difference is that it does not contain the category information of these instances. That is to say, the model first needs to identify the category of each instance according to the content in the bounding box, and then predict the semantic relationship between the instances. This task is more challenging because the accuracy of instance classification directly affects the relationship prediction. It not only requires the model to have high robustness in category recognition and relationship prediction, but also needs to achieve an organic combination of the two, so that the relationship prediction can be completed more accurately based on instance classification.

[0101] Scene Graph Detection (SGdet) is the most challenging of the three tasks. Compared with the previous two tasks, SGdet places more demanding requirements on the model - the input image does not contain any bounding box and class information of instances at all; the model needs to start from scratch and independently complete a series of complex subtasks, including identifying the positions of all instances, determining the class of each instance, and further predicting the semantic relationships between instances. This means that SGdet not only needs to solve the problem of instance detection, but also needs to handle the tasks of instance class recognition and relationship prediction simultaneously. The complexity of this process lies in that there may be biases or errors in instance detection and class recognition, and these biases will be directly transmitted to the relationship prediction stage, thus affecting the accuracy of the overall result.

[0102] Testing the trained model in the significant scene graph generation method based on important instance pairs for feature learning, which is both testing the scene graph generation model and the instance pair feature PoolFormer scene graph generation network model based on importance scores; includes the following steps:

[0103] S1-1: Input the test image into the trained model for relationship prediction to obtain the relationship prediction result;

[0104] S1-2: Compare the relationship prediction result with the true predicate label, and use evaluation metrics to evaluate the relationship prediction result;

[0105] S1-3: Evaluate the prediction ability of the model for important relationships, and use the important relationship recall rate IR@K and the important relationship precision IP@K to evaluate the relationship prediction result;

[0106] S1-4: Based on the relationship importance scores, screen out the important relationships in the predicted relationships to form a scene graph;

[0107] S1-5: Combine and output the test results according to the results obtained in steps S1-2, S1-3, and S1-4.

[0108] In step S1-1, when testing the model, it is tested separately under three sub-tasks: the predicate classification task, the scene graph classification task, and the scene graph detection task. In the predicate classification task, the trained model focuses on predicting the relationship types between instances based on the bounding box and category label information of all instances provided in the test set, given the location and category of each instance. In the scene graph classification task, the trained model cannot directly use the category label information of instances. The trained model needs to independently predict the category of each instance to obtain the instance category results, and then further predict the relationships between instances based on the predicted instance category results. In the scene graph detection task, the trained model cannot access the bounding box and category label information of any instance. The trained model starts from scratch to identify and locate the instances in the image, predict the category of each instance, and then further predict the relationships between instances based on the category of each instance. This task is more challenging because it requires understanding the image content from scratch.

[0109] In step S1-2, the widely recognized evaluation metric Recall@K (R@K) in the current scene graph generation field is adopted. Recall@K refers to the proportion of the number of correctly predicted relationships in the top K predicted results to the total number of relationships. The higher the value of this metric, the better the test results, the higher the relationship prediction accuracy of the model, and the better the performance. In the present invention, the two metrics R@50 and R@100 are specifically selected to evaluate the performance of the model.

[0110] In step S1-3, two key evaluation metrics are adopted - the important relationship recall rate (IR@K) and the important relationship precision rate (IP@K). The selection range of important relationships is also consistent with the previous invention. Among them, the important relationship recall rate (IR@K) measures the proportion of the number of correctly predicted important relationships in the top K predicted results to the total number of important relationships. The role of IR@K is to evaluate how many important relationships the model can retrieve in the top K predicted results, mainly focusing on the coverage ability of the model for important relationships and reflecting its performance in identifying important relationships. The important relationship precision rate (IP@K) represents the proportion of the number of correctly predicted important relationships in the top K predicted results to the total number of correctly predicted relationships. It focuses on the accuracy of the model's importance judgment when correctly predicting relationships and reflects the precision of the model in judging the importance of relationships. These two metrics complement each other and comprehensively evaluate the performance of the model in predicting important relationships from the two perspectives of coverage and precision. In the present invention, IR@5, IR@10, IP@5, and IP@10 are specifically adopted as evaluation metrics to systematically evaluate the effect of the model in predicting important relationships.

[0111] In the steps S1-4, the predicted relationships are sorted from largest to smallest according to the relationship importance scores. The predicted relationships with importance scores greater than 0.5 will be used as important relationships to form a significant scene graph.

[0112] The present invention proposes an instance pair feature PoolFormer scene graph generation network model based on importance scores, which can improve the accuracy of the model in predicting important relationships by modeling important instance pair features. The following three metrics are used for the model proposed by the present invention: R@K, IP@K, and IR@K.

[0113] To verify the beneficial effects of the present invention, the present invention conducts experiments on the VG150 dataset, which contains the 150 most common instance classes and 50 predicate classes. The present invention evaluates the metrics in three subtasks: predicate classification, scene graph classification, and scene graph detection. In the predicate classification task, the evaluation results finally obtained by the method of the present invention are as follows: R@50 is 66.6%, R@100 is 68.7%, IR@5 is 39.08%, IR@10 is 51.40%, IP@5 is 90.79%, IP@10 is 88.57%; in the scene graph classification task, the evaluation results finally obtained by the method of the present invention are as follows: R@50 is 41.3%, R@100 is 42.4%, IR@5 is 25.45%, IR@10 is 32.87%, IP@5 is 89.59%, IP@10 is 87.13%; in the scene graph detection task, the evaluation results finally obtained by the method of the present invention are as follows: R@50 is 32.4%, R@100 is 36.9%, IR@5 is 14.01%, IR@10 is 19.50%, IP@5 is 91.25%, IP@10 is 89.25%.

[0114] Embodiment 1

[0115] In this embodiment, the VG150 dataset will be used to complete the task of generating significant scene graphs based on the importance of fuzzy logic inference relationships. This dataset contains 108,077 images, covering 150 of the most common instance categories and 50 predicate categories. To reasonably partition the dataset, 70% of the images are assigned to the training set, while the remaining 30% are used as the test set. The instance pair feature PoolFormer scene graph generation network model IEPFP-SGG of the present invention based on the importance score is trained by the SGD optimizer for 50K iterations. The initial learning rate and batch size are set to 0.001 and 8 respectively, the parameters k and γ are set to 3 and 7 respectively, and the extreme value pruning ratio δ is set to 0.1. To complete the training and testing, the present invention selects deepin20.4 as the operating system and conducts research in the experimental environment of torch1.7.1+cu110. The main software and hardware environment configurations used for experimental training and testing include an Intel(R) Core(TM) i7-9700k CPU, 32GB of memory, an NVIDIA GeForce RTX 4090 GPU with 24GB of video memory, the Deepin20.4 operating system, and software dependencies such as PyTorch and Numpy.

[0116] The present invention conducts metric evaluations in three subtasks: predicate classification PredCls, scene graph classification SGCls, and scene graph detection SGCls. The present invention uses the commonly used evaluation metric Recall@K (R@K) in current scene graph generation to accurately evaluate the ability of the IEPFP-SGG method in relation prediction. R@K is specifically evaluated using the R@50 and R@100 metrics. The comparison results between the method of the present invention and existing methods on the VG150 dataset are shown in Table 1.

[0117] Table 1 Comparison results between the method of the present invention and existing methods on the VG150 dataset

[0118]

[0119] According to the results in Table 2, the IEPFP-SGG method proposed by the present invention demonstrates excellent performance in relation prediction. In the PredCls task, compared with FLRW-SGG, IEPFP-SGG improves by 0.5% and 0.4% respectively in terms of the R@50 and R@100 metrics. In the SGCls task, compared with FLRW-SGG, IEPFP-SGG increases by 0.6% and 0.5% respectively in R@50 and R@100. In the SGDet task, compared with FLRW-SGG, IEPFP-SGG achieves improvements of 0.8% and 1.0% respectively in R@50 and R@100. The results show that the IEPFP-SGG method proposed by the present invention can reach a relatively high level in the most basic evaluation criterion Recall@K, demonstrating significant capabilities in relation prediction.

[0120] To evaluate the prediction ability of the IEPFP-SGG network model of the present invention in important relations, it is evaluated in terms of two metrics: important relation recall rate (IR@K) and important relation precision rate (IP@K), and compared with the FLRW-SGG method. Under the three subtasks of PredCls, SGCls, and SGDet, the comparison results of the IEPFP-SGG method with FLRW-SGG in IR@5, IR@10, IP@5, and IP@10 are shown in Table 2.

[0121] Table 2 Comparison results of the method of the present invention and the baseline model in terms of IR@K and IP@K metrics

[0122]

[0123] According to the results in Table 3, the IEPFP-SGG method of the present invention can achieve excellent performance in important relation prediction in the three subtasks. In the PredCls task, compared with FLRW-SGG, IEPFP-SGG improves by 0.30% and 0.38% respectively in terms of the IR@5 and IR@10 metrics; and by 0.14% and 0.08% respectively in the IP@5 and IP@10 metrics. In the SGCls task, compared with FLRW-SGG, IEPFP-SGG improves by 0.20% and 0.27% respectively in the IR@5 and IR@10 metrics. In the SGDet task, compared with FLRW-SGG, IEPFP-SGG has the most improvement, with increases of 0.41% and 0.43% respectively in the IR@5 and IR@10 metrics; and by 0.17% and 0.22% respectively in the IP@5 and IP@10 metrics. In summary, it shows that the IEPFP-SGG network model of the present invention can better predict important relations and demonstrates better performance in the most difficult scene graph generation task SGDet.

[0124] Example Two

[0125] This embodiment will introduce an applicable scenario of the present invention. In the field of visual question answering, users often need to understand complex images and accurately answer relevant questions, such as when dealing with landscape pictures in natural scenes or life photos containing diverse objects. Due to the large variety of objects, complex interrelationships, and possible uneven lighting, occlusion, etc. in these images, it is difficult for traditional scene graph generation models to accurately capture the important relationships crucial for answering questions, resulting in the visual question answering system being unable to give accurate and targeted answers. The significant scene graph generation method based on important instance pair feature learning proposed by the present invention can well solve this problem.

[0126] First, the visual question answering system performs preprocessing on the input image, including operations such as removing image noise and adjusting image contrast, so as to improve the clarity and quality of the image. The classic object detection model Faster R-CNN is used to perform instance recognition and segmentation on the preprocessed image to determine the position and category information of each object in the image.

[0127] Use the method of the present invention to generate a significant scene graph. This method constructs a plug-and-play important instance pair feature learning module by means of the true relationship importance scores calculated by fuzzy logic in the VG150 dataset in the previous invention. This module first uses Manhattan distance constraints to increase the difference between the instance pair features predicted by the model, enabling the model to more clearly distinguish different instance pair information during relationship prediction. Then, the Feature PoolFormer network is used to fuse the instance pair features and the relationship importance scores to strengthen the expression of the important relationship instance pair features, helping the model fully learn the instance pair features of important relationships and improving the ability to recognize important relationships. Based on the generated significant scene graph, the visual question answering system can clarify the key relationships among the objects in the image, and then give an accurate and detailed answer to the question raised by the user.

[0128] Finally, the answer given by the system will be displayed to the user together with the original image, providing the user with a clear result of image understanding. The user can also perform secondary queries based on the key information in the answer to quickly filter out images and corresponding answers that meet specific requirements, improving the efficiency of obtaining effective information.

[0129] The above is only a description of the preferred embodiment of the present invention. Those of ordinary skill in the art can make several modifications and optimizations based on the above disclosure without departing from the basic principle content. These improvements and optimizations should be regarded as the protection scope understood by the present invention.

Claims

1. A method for generating a significant scene graph based on important instances for feature learning, characterized in that: Including the following steps: S1. Concatenate the instance labels of the relationships to be predicted and the object instance labels generated by inputting the training images in the dataset into the target detector to form the instance pair labels of the predicted relationships; S2. Based on the instance pair labels of the predicted relationships and the instance pair labels of the true relationships, establish corresponding models respectively, and extract the instance pair features of the predicted relationships and the instance pair features of the true relationships; S3. Use the Manhattan distance constraint to expand the differences between the instance pair features of the predicted relationships to obtain the distance loss; S4. Based on the input relationship importance scores, use the instance pair feature PoolFormer module based on importance scores to fuse the instance pair features of the true relationships and the relationship importance scores to generate the important instance pair features of the true relationships; S5. Apply the Euclidean distance constraint between the instance pair features of the predicted relationships and the important instance pair features of the true relationships to make the instance pair features of the predicted relationships close to the important instance pair features of the true relationships, and generate the instance pair loss; S6. Sum the distance loss obtained in step S3 and the instance pair loss obtained in step S5 to obtain the total loss, and train the scene graph generation model based on the obtained total loss; S7. Judge whether the set number of training rounds is reached. Otherwise, return to step S1. If yes, complete the training and output the trained scene graph generation model.

2. The significant scene graph generation method for feature learning based on important instances according to claim 1, wherein: In the step S1, the subject instance label of the relationship to be predicted and the object instance label are concatenated to obtain the instance pair label f of the predicted relationship p ; meanwhile, the instance pair label f of the true relationship t is generated by using the true subject instance label and the object instance label for concatenation; In the formula, [.,.] represents concatenation by column.

3. A method for generating a significant scene graph for feature learning based on important instances according to claim 1, characterized in that: In the step S2, for the instance pair label f of the prediction relationship p a model is established, and a number of instance pair features E of the prediction relationship are extracted p , and at the same time, for the instance pair label f of the true relationship t a model is established, and a number of instance pair features E of the prediction relationship are extracted p , E p = BN(ReLU(FC(f p )))E t = BN(ReLU(FC(f t ))) In the formula, FC, BN, and ReLU represent the fully connected layer, batch normalization operation, and ReLU activation function respectively.

4. A method for generating a significant scene graph for feature learning based on important instances according to claim 3, wherein: In the step S3, the Manhattan distance constraint method is adopted to calculate the instance pair feature E of all the predicted relationships extracted by the model pi and E pj The Manhattan distance d between them p is calculated, and the Manhattan distances calculated for the instance pair features of each predicted relationship are combined to obtain the distance matrix D p =(d p )∈R (N+1)×(N+1) ; Sort the elements in each of several rows in the distance matrix D p in ascending order to obtain the sorted distance matrix D′ p =(d′ p ) ∈ R (N+1)×(N+1) ; Select the first k minimum values in each row, sum them up and take the average to obtain the average Manhattan distance Calculate the distance loss L based on the obtained average Manhattan distance p_mah ; In the formula, γ is a hyperparameter used to adjust the distance margin.

5. A method for generating a significant scene graph for feature learning based on important instances according to claim 1, characterized in that: The instance pair feature PoolFormer module based on importance scores includes a feature extraction layer, a normalization layer, and a multi-layer perceptron; the instance pair labels are obtained by concatenating the subject instance labels and the object instance labels, and the instance pair features are obtained by extracting features from the instance pair labels through the feature extraction layer; the instance pair features are input into the feature pooling token mixer after batch normalization for operation, and the output of the feature pooling token mixer is added and fused with the instance pair features to obtain the instance pair features capturing local features; the instance pair features capturing local features are sequentially subjected to batch normalization and multi-layer perceptron operations, and the output of the multi-layer perceptron is added and fused with the instance pair features capturing local features and then multiplied and fused with the relationship importance scores to obtain the important instance pair features; the important instance pair features are calculated to obtain the instance pair loss.

6. A method for generating a significant scene graph for feature learning based on important instances according to claim 5, wherein: The feature pooling token mixer FtTokenMixer in the instance pair feature PoolFormer module based on importance scores performs one-dimensional pooling on the instance pair features of the true relationships to capture the changes in local features, and retains the information of the original features through residual connection; the calculation process is as follows: E′ t = FtTokenMixer(BN(E t )) + E t where E′ t is an instance pair feature that captures the true relationship of local features, BN is the batch normalization operation, and FtTokenMixer(.) represents the operation of the feature pooling token mixer; Use the relationship importance score Rim to weight the instance pair of the true relationship that captures the local feature for the feature E’ t to generate In the formula, GeLU represents the GeLU activation function, and w1 and w2 represent learnable weight parameters.

7. A method for generating a significant scene graph for feature learning based on important instances according to claim 6, characterized in that: The specific operation of the feature pooling token mixer FtTokenMixer is calculated as follows: Wherein, T' 1，i represents the output feature after the pooling token mixing operation at position i, T ：，i represents the original input feature at position i, and M represents the size of the pooling window.

8. A method for generating a significant scene graph for feature learning based on important instances according to claim 5, characterized in that: By means of the instance pair of the prediction relationship for feature E p and the important instance pair of the true relationship for the feature applying the Euclidean distance constraint, the Euclidean distance d between the instance pair feature of the prediction relationship and the important instance pair feature of the true relationship is obtained t , and the calculated Euclidean distances are combined to obtain the distance matrix D t =(d t )∈R (N+1)×(N+1) ; wherein, is expressed as the square of the second norm; Calculate the mean \(v\) and standard deviation \(\sigma\) of all Euclidean distances in the distance matrix, and obtain the normalized distance matrix \(D'\) based on the mean \(v\) and standard deviation \(\sigma\). t =(d' t )\(\in\mathbb{R}\) (N+1)×(N+1) ; In the formula, v is the mean of all Euclidean distances in the distance matrix, and σ is the standard deviation of the Euclidean distances in the distance matrix; Sort the values in the normalized distance matrix in descending order and trim the extreme values. Take the average of the absolute values of the trimmed Euclidean distances as the instance pair loss \(L_{pair}\). The calculation process is as follows: In the formula, \(N\) represents the number of instance pair features, \(\eta = [\delta\cdot N]\) represents the number of Euclidean distances at the front and back ends of the sorted instance pair features for trimming, and \(\delta\) represents the hyperparameter of the trimming ratio.

9. A method for generating a significant scene graph for feature learning based on important instances according to claim 8, wherein: In step 6, the distance loss L p_mah and the instance pair loss Lpair are added together to obtain the final constraint loss L; L = L pair + L p_mah 。 10. A method for generating a significant scene graph for feature learning based on important instances according to claim 1, characterized in that: Test the trained scene graph generation model in the significant scene graph generation method based on important instance pair features, including the following steps: S1-1: Input the test image into the trained model for relationship prediction to obtain the relationship prediction result; S1-2: Compare the relationship prediction result with the true predicate label, and evaluate the relationship prediction result using the evaluation metric; S1-3: Evaluate the prediction ability of the model for important relationships, and evaluate the relationship prediction result using the important relationship recall rate \(IR@K\) and the important relationship accuracy \(IP@K\); S1-4: Based on the relationship importance score, screen out the important relationships in the predicted relationships to form a scene graph; S1-5: Combine the results obtained in steps S1-2, S1-3, and S1-4 to output the test result.

Images