Relation-incremental scene graph generation method based on relation independence and feature adapter
By adopting the relationship independent and feature adapter method in scene graph generation, the problem of difficulty in dealing with relationship incremental scenarios in the prior art is solved, and a more efficient and accurate relationship detection effect is achieved.
Patent Information
- Application Number
- CN202210343339.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-31
- Publication Date
- 2025-04-22
- Estimated Expiration
- 2042-03-31
AI Technical Summary
Existing scenario graph generation methods are difficult to effectively deal with relational incremental scenarios, and traditional methods usually regard scene graph generation tasks as ordinary training tasks, and fail to fully consider the long-tail distribution characteristics of relationships.
A relational incremental scene graph generation method based on relationship independence and feature adapter is adopted. By constructing a relationship detector for shared feature extraction parts, it is trained separately on the base sub-data set and incremental data set, and the feature adapter is used to adapt relationship changes.
It realizes the generation of scene graphs more efficiently in relational incremental scenarios, improves the accuracy and adaptability of relationship detection, and can better handle the long-tail distribution characteristics of relationships.
Smart Images
Figure CN114972963B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a relational incremental scene graph generation method based on relation independence and feature adapter. Background Art
[0002] As a technology for extracting structural natural language information from images, scene graph generation is often used as auxiliary information for some high-level visual tasks, such as visual question answering and image captioning. The goal of scene graph generation is to predict the classification label of each object in the image and the relationship between each pair of objects given an image. The key factors of scene graph generation mainly include the uncertainty of the number of relationships between object pairs and the long-tail distribution of the number of relationship annotations. Traditional methods generally regard the scene graph generation task as an ordinary training task, that is, all the relationships that need to be classified are known from the beginning, without considering that the scene graph generation task in actual application scenarios often appears in the form of relationship increment scenarios.
[0003] The existing mainstream scene graph generation method is a two-stage learning framework. It inputs an original three-channel color image, first uses an object detector to detect all objects in the image, then uses a relationship detector to detect the relationship between objects, and finally combines the object detection results and the relationship detection results to generate a scene graph. Summary of the invention
[0004] In view of the above problems, the present invention provides a method for generating a relational incremental scene graph based on relation independence and feature adapter. The specific technical solution adopted by the present invention is as follows:
[0005] A method for generating a relational incremental scene graph based on relation independence and feature adapter comprises the following steps:
[0006] S1. Obtain a training data set for training relational incremental scene graph generation; the training data set includes a base sub-data set and N incremental sub-data sets, and each sub-data set has a data set label;
[0007] S2, build a pre-trained object detector and detect all objects in the images contained in the training dataset; at the same time, build a relationship detector for each sub-dataset that shares the feature extraction part;
[0008] S3, training the relation detector using a relation-independent training method on the base dataset;
[0009] S4, using the relationship detector trained on the base sub-dataset as the basic model, further training a relationship detector suitable for the first incremental sub-dataset using a relationship-independent training method in conjunction with a feature adapter on the first incremental sub-dataset;
[0010] S5, traverse the remaining incremental sub-datasets in turn and train the relationship detectors on the remaining N-1 incremental sub-datasets in turn; during the training process, for any m-th incremental sub-dataset satisfying m≥2, take the m-1-th incremental sub-dataset as a new base sub-dataset, and use the relationship detector trained on the new base sub-dataset as the basic model, and train according to the relationship-independent training method in S4 to obtain a relationship detector suitable for the m-th incremental sub-dataset;
[0011] S6. Use the relationship detector trained on the Nth incremental sub-dataset as the final relationship detector to generate a scene graph for the target image.
[0012] Preferably, the training data set in S1 includes a base data set D (base) and several incremental subsets of data sets D (1) , D (2) ,……,D (N) .
[0013] Preferably, S2 comprises the following sub-steps:
[0014] S21, select a pre-trained object detector ObjDet that is common to the entire scene graph generation task and is used to detect any input training image I train , the corresponding image feature map is obtained through the image feature extraction network And get the initial target detection result through the region candidate network
[0015]
[0016] S22, construct a relation detector {RelDet, C} including a feature extraction part and a classifier, wherein the feature extraction part RelDet is shared for each sub-dataset, and the sub-dataset D (base) , D (1) , D (2) ,……,D (N) The corresponding classifiers C are represented as C (base) , C (1) , C (2) ,……,C (N) .
[0017] Preferably, S3 comprises the following sub-steps:
[0018] S31. Use the pre-trained target detector ObjDet to obtain the base dataset D (base) Each training image in The corresponding image feature map And the initial target detection results
[0019] S32, for each training image in the base dataset D(base) Image feature map Use the relation detector to get relation predictions for all object pairs And the initial target detection results Fine-tune to obtain
[0020]
[0021] Among them C (base) and RelDet are randomly initialized;
[0022] S33, according to D (base) Each training image in Relationship prediction In the object pair (n i , n j The result of object classification is (o i , o j ), then for the object (n i , n j ) Prediction of relation category k The relationship-independent loss function L is calculated by the following formula BCE :
[0023]
[0024] where y i，j，k The object is i , n j ) about the dataset label of relation category k, have:
[0025]
[0026] Where e is the base of natural logarithm; FreqM i，j，k Indicates that in all categories the label is (o i , o j ) of objects (n i , n j ) is the ratio of object pairs with relation k;
[0027] Based on the above loss function L BCE Update RelDet and C (base) , after several rounds of iterative training, we finally get (base) The trained relation detector {RelDet base , C (base)}.
[0028] Preferably, S4 comprises the following sub-steps:
[0029] S41, for the mth incremental sub-dataset D whose initial value is 1 (m) , initialize the feature extraction part of the relation detector RelDet to the feature extraction part of the relation detector obtained by the latest training, and set the incremental sub-dataset D (m) A single training image in Input the target detector ObjDet, and then use the output of the target detector ObjDet as the input of the feature extraction part RelDet of the relationship detector to obtain the target detection results and relationship features in the figure Randomly initialize the classifier C (m) And by classifier C (m) Output Current relationship predictions Then for the incremental sub-dataset D (m) Calculate the loss function L BCE ;
[0030] S42, use a randomly initialized feature adapter A to adapt the features After feature transformation, it is passed through all the currently trained classifier sets As the old classifier C (old) Calculate the classification results:
[0031]
[0032] Where: m = 1 C (base) , when m≥2 Contains C (base) and
[0033] S43, the incremental sub-dataset D (m) A single training image in Through the parameter-fixed relation detector {RelDet, C (old)}, get the relationship classification output on all previous old relationship classes For an object pair (n i , n j The result of object classification is (o i , o j ), then for the object (n i , n j ) about the prediction of relation category k, The corresponding prediction result is The corresponding prediction result is The loss function L is calculated by the following formulaBKD :
[0034]
[0035] S44, combined with the loss function L calculated in S41 BCE and the loss function L calculated in S43 BKD , calculate the total loss function L in a weighted manner:
[0036] L=L BCE +λL BKD ,
[0037] Where λ is the weight value;
[0038] With the goal of minimizing the total loss function L, based on L BCE Update RelDet and C (m) , based on L BKD Update C (old) and A, through several rounds of iterative training, we get (m) Trained relation detector
[0039] Preferably, S5 comprises the following sub-steps:
[0040] S51. For any mth incremental sub-dataset D satisfying m≥2 (m) , taking the m-1th incremental sub-dataset as the new base sub-dataset, and taking the relationship detector trained on the new base sub-dataset as the basic model, iteratively train according to S4 in sequence, and obtain D (m) Trained relation detector
[0041] S53, after completing the training for all N incremental sub-datasets, save the last Nth incremental sub-dataset D (N) The trained relation detector is the final relationship detector.
[0042] Preferably, the S6 comprises the following sub-steps:
[0043] S61, for any target image I * , first use the pre-trained target detector ObjDet to extract the image feature map in the target image And target detection results
[0044] S62, image feature map for target image And target detection results Using the Relationship Detector Get the relationship prediction between objects in the graph And the target detection results after fine-tuning
[0045] Preferably, in S44, the weight value λ=0.5.
[0046] This method is based on a deep neural network, fully utilizes the model of the previous subtask, and independently models each relationship. It models the relationship in the image and can better adapt to the problem of relational incremental scene graph generation. Compared with the traditional scene graph generation method, the present invention has the following beneficial effects:
[0047] First, the present invention proposes a feasible solution for the relational incremental scene graph generation task.
[0048] Secondly, the present invention only improves the classifier in structure and uses a relationship-independent classifier, that is, the method can be used together with most existing relationship detectors and is a plug-in method that is easy to promote.
[0049] Finally, the relational incremental scene graph generation method of the present invention makes full use of the model of the previous subtask to obtain the information of the relationship category contained in the previous subtask, ensuring that the previous subtask information is maintained when the current subtask is trained, thereby improving the final effect of the method. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a schematic diagram of the process of the present invention; DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0052] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.
[0053] refer to Figure 1 , is a method for generating a relational incremental scene graph based on relational independence and feature adapter in the present invention, the method comprising the following steps:
[0054] S1. Obtain a training data set for training relational incremental scene graph generation; the training data set includes a base sub-data set and N incremental sub-data sets, and each sub-data set has a data set label.
[0055] In this embodiment, the training data set in S1 includes a base data set D (base) and several incremental subsets of data sets D (1) , D (2) ,……,D (N) .
[0056] Of course, in order to test the specific technical effects of the present invention, in addition to the above training data set, a corresponding test data set can also be set. In the present invention, the final algorithm goal is to generate a single test image I test Or the scene graph G corresponding to other target images needs to predict the relationship between the base sub-dataset and all incremental sub-datasets.
[0057] S2. Build a pre-trained object detector and detect all objects in the images contained in the training dataset; at the same time, build a relationship detector with a shared feature extraction part for each sub-dataset.
[0058] In this embodiment, S2 includes the following sub-steps:
[0059] S21, select a pre-trained object detector ObjDet that is common to the entire scene graph generation task and is used to detect any input training image I train , the corresponding image feature map is obtained through the image feature extraction network And get the initial target detection result through the region candidate network
[0060]
[0061] S22, construct a relation detector {RelDet, C} including a feature extraction part and a classifier, wherein the feature extraction part RelDet is shared for each sub-dataset, and the sub-dataset D (base) , D (1) , D (2) ,……,D (N) The corresponding classifiers C are represented as C (base) , C (1) , C (2) ,……,C (N) .
[0062] S3. Train the relation detector using a relation-independent training method on the base dataset.
[0063] In this embodiment, S3 includes the following sub-steps:
[0064] S31. Use the pre-trained target detector ObjDet to obtain the base dataset D (base) Each training image in The corresponding image feature map And the initial target detection results
[0065] S32, base sub-dataset D (base) Each training image in Image feature map Use the relation detector to get relation predictions for all object pairs And the initial target detection results Fine-tune to obtain
[0066]
[0067] Among them C (base) and RelDet are randomly initialized;
[0068] S33, according to D (base) Each training image in Relationship prediction In the object pair (n i , n j The result of object classification is (o i , o j ), then for the object (n i , n j ) Prediction of relation category k The relationship-independent loss function L is calculated by the following formula BCE :
[0069]
[0070] where y i，j，k The object is i , n j ) about the dataset label of relation category k, have:
[0071]
[0072] Where e is the base of natural logarithm; FreqM i，j，k Indicates that in all categories the label is (o i , o j ) of objects (n i , n j ) is the ratio of object pairs with relation k;
[0073] Based on the above loss function L BCE Update RelDet and C (base) , after several rounds of iterative training, we finally get (base) The trained relation detector {RelDet base , C (base)}.
[0074] S4. Taking the relationship detector trained on the base sub-dataset as the basic model, further train a relationship detector suitable for the first incremental sub-dataset using a relationship-independent training method in conjunction with a feature adapter on the first incremental sub-dataset.
[0075] In this embodiment, S4 includes the following sub-steps:
[0076] S41, for the mth incremental sub-dataset D whose initial value is 1 (m) , initialize the feature extraction part of the relation detector RelDet to the feature extraction part of the relation detector obtained by the latest training, and set the incremental sub-dataset D (m) A single training image in Input the target detector ObjDet, and then use the output of the target detector ObjDet as the input of the feature extraction part RelDet of the relationship detector to obtain the target detection results and relationship features in the figure Randomly initialize the classifier C (m) And by classifier C (m) Output Current relationship predictions Then for the incremental sub-dataset D (m) Calculate the loss function L BCE (See formula in S33);
[0077] S42, use a randomly initialized feature adapter A to adapt the features After feature transformation, it is passed through all the currently trained classifier sets As the old classifier C (old) Calculate the classification results:
[0078]
[0079] Where: m = 1 C (base) , when m≥2 Contains C (base) and
[0080] S43, the incremental sub-dataset D (m) A single training image in Through the parameter-fixed relation detector {RelDet, C (old)}, get the relationship classification output on all previous old relationship classes For an object pair (n i , n j The result of object classification is (o i , o j ), then for the object (n i , n j ) about the prediction of relation category k, The corresponding prediction result is The corresponding prediction result is The loss function L is calculated by the following formula BKD :
[0081]
[0082] S44, combined with the loss function L calculated in S41 BCE and the loss function L calculated in S43 BKD , calculate the total loss function L in a weighted manner:
[0083] L=L BCE +λL BKD ,
[0084] Where λ is the weight value; the weight value is preferably λ=0.5;
[0085] With the goal of minimizing the total loss function L, based on the loss term L BCE Update RelDet and C (m) , based on the loss term L BKD Update C (old) and A, through several rounds of iterative training, we get (m) Trained relation detector It should be noted that In the incremental sub-dataset D (m) The old classifier C obtained after training (old) .
[0086] S5. Traverse the remaining incremental sub-datasets in turn and train the relationship detectors on the remaining N-1 incremental sub-datasets in turn. During the training process, for any m-th incremental sub-dataset satisfying m≥2, take the m-1-th incremental sub-dataset as the new base sub-dataset, and use the relationship detector trained on the new base sub-dataset as the basic model. According to the relationship-independent training method in S4, train to obtain a relationship detector suitable for the m-th incremental sub-dataset.
[0087] In this embodiment, S5 includes the following sub-steps:
[0088] S51. For any mth incremental sub-dataset D satisfying m≥2 (m) , take the m-1th incremental sub-dataset as the new base sub-dataset, and take the relationship detector trained on the new base sub-dataset as the basic model, take 2, 3, ..., N along m in order, and iteratively train each m according to S4 in turn, and obtain each D (m) Trained relation detector
[0089] S53, after completing the training for all N incremental sub-datasets, save the last Nth incremental sub-dataset D (N) The trained relation detector is the final relationship detector.
[0090] S6. Use the relationship detector trained on the Nth incremental sub-dataset as the final relationship detector to generate a scene graph for the target image.
[0091] In this embodiment, S6 includes the following sub-steps:
[0092] S61, for any target image I * , first use the pre-trained target detector ObjDet to extract the image feature map in the target image And target detection results
[0093] S62, image feature map for target image And target detection results Using the Relationship Detector Get the relationship prediction between objects in the graph And the target detection results after fine-tuning
[0094] The following is an application of the relational incremental scene graph generation method based on relational independence and feature adapter shown in S1 to S6 above to a specific example to demonstrate its specific technical effect.
[0095] Example
[0096] The implementation method of this embodiment is as described in S1 to S6 above, and the specific steps will not be elaborated in detail. The following only shows its effect based on case data. The present invention is implemented on a data set with true value annotations:
[0097] VG dataset: This dataset contains 10,807 images, with 150 object labels and 50 relationship labels, with an average of 38 object labels and 22 relationship labels per image. The 50 relationships are sorted by the number of appearances in the training set. The first 10 categories are the base datasets, and categories 11-20, 21-30, 31-40, and 41-50 are divided into incremental sub-datasets 1, 2, 3, and 4.
[0098] This example is trained on the training set of the VG dataset and tested on all incremental sub-test sets. It is necessary to predict all 50 types of relationships. The specific process (taking incremental sub-dataset 2 as an example) is as follows: Figure 1 shown.
[0099] The detection accuracy of the detection results of the PredCls task (input image and the position and classification of objects in the image, output complete scene graph) of this embodiment is shown in the following table, and the average recall rate (R@K) and the classification average recall rate (mR@K) are mainly used for comparison. The average recall rate (R@K) indicator refers to the proportion of manually labeled (object 1, relationship, object 2) groups covered by each test image when K candidate (object 1, relationship, object 2) groups are given. The classification average recall rate (mR@K) indicator means that when K candidate (object 1, relationship, object 2) groups are given for each test image, the average recall rate of each relationship is first calculated, and then the average recall rate of all relationships is calculated as an arithmetic average.
[0100] Method R@20 R@50 R@100 Motifs-Fixed 21.67 40.82 57.21 Motifs-Finetune 3.76 5.78 9.15 Motifs-LwF 8.80 15.26 22.52 Motifs-RCE+RKD 37.31 57.29 70.13 Motifs-Ours 45.04 62.35 74.05 Transformer-Fixed 21.62 38.54 54.36 Transformer-Finetune 8.27 16.10 26.30 Transformer-LwF 13.16 22.43 31.99 Transformer-RCE+RKD 42.12 60.72 73.06 Transformer-Ours 50.23 66.29 76.89
[0101]
[0102] The detection accuracy of the detection results of the SGCls task (inputting a picture and the position of the object in the picture, and outputting a complete scene graph) in this embodiment is shown in the following table:
[0103] Method R@20 R@50 R@100 Motifs-Fixed 12.45 23.34 32.56 Motifs-Finetune 2.31 3.45 5.25 Motifs-LwF 5.61 9.41 12.95 Motifs-RCE+RKD 21.65 31.36 37.34 Motifs-Ours 22.04 31.67 37.65 Transformer-Fixed 12.71 22.18 30.53 Transformer-Finetune 4.29 8.82 14.71 Transformer-LwF 8348 15.14 21.04 Transformer-RCE+RKD 23.21 32.79 39.66 Transformer-Ours 23.91 33.52 39.92
[0104]
[0105] The detection accuracy of the detection results of the SGDet task (inputting a picture and the position of the object in the picture, and outputting a complete scene graph) in this embodiment is shown in the following table:
[0106] Method R@20 R@50 R@100 Motifs-Fixed 9.39 16.31 24.25 Motifs-Finetune 1.95 3.43 5.57 Motifs-LwF 2.64 5.00 8.05 Motifs-RCE+RKD 12.48 19.75 25.21 Motifs-Ours 16.84 24.32 30.46 Transformer-Fixed 10.43 17.83 25.70 Transformer-Finetune 2.59 4.63 7.83 Transformer-LwF 4.89 8.77 12.79 Transformer-RCE+RKD 15.19 23.10 28.84 Transformer-Ours 16.52 24.01 30.20
[0107]
[0108] As shown in the above six tables, we tested the combination with two different relationship detectors, Motifs and Transformer, respectively. A relationship incremental scene graph generation method based on relationship independence and feature adapter has significant improvements on all indicators of all tasks and all relationship detectors.
[0109] In the above embodiment, the relational incremental scene graph generation method based on relational independence and feature adapter of the present invention first improves the classifier so that the predictions of all relations are independent of each other. On this basis, an additional feature adapter is used and assisted by the model of the previous task to help learn, so as to better complete the relational incremental scene graph generation task.
[0110] Through the above technical solutions, the embodiment of the present invention develops a method for generating incremental scene graphs based on relation independence and feature adapters based on deep learning technology. The present invention improves the classifier so that the predictions of all relations are independent of each other; an additional feature adapter is used and assisted by the model of the previous task to help learn, so as to better adapt to the task of generating incremental scene graphs.
[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A relational incremental scene graph generation method based on relation independence and feature adapter, characterized in that The following steps are involved: S1. Obtain a training data set for training relational incremental scene graph generation; the training data set includes a base sub-data set and N incremental sub-data sets, and each sub-data set has a data set label; S2, build a pre-trained object detector and detect all objects in the images contained in the training dataset; at the same time, build a relationship detector for each sub-dataset that shares the feature extraction part; S3, training the relation detector using a relation-independent training method on the base dataset; S4, using the relationship detector trained on the base sub-dataset as the basic model, further training a relationship detector suitable for the first incremental sub-dataset using a relationship-independent training method in conjunction with a feature adapter on the first incremental sub-dataset; S5, traverse the remaining incremental sub-datasets in turn and train the relationship detector on the remaining N-1 incremental sub-datasets in turn; During the training process, for any m-th incremental sub-dataset satisfying m≥2, the m-1-th incremental sub-dataset is used as a new base sub-dataset, and the relationship detector trained on the new base sub-dataset is used as the basic model. The relationship detector suitable for the m-th incremental sub-dataset is trained according to the relationship-independent training method in S4; S6. Use the relationship detector trained on the Nth incremental sub-dataset as the final relationship detector to generate a scene graph for the target image.
2. The method for generating a relational incremental scene graph based on relation independence and feature adapter according to claim 1, characterized in that The training data set in S1 includes a base data set D (base) and several incremental sub-datasets D (1) , D (2) ,……,D (N) .
3. The method for generating a relational incremental scene graph based on relation independence and feature adapter according to claim 2, characterized in that The S2 comprises the following sub-steps: S21, select a pre-trained object detector ObjDet that is common to the entire scene graph generation task and is used to detect any input training image I train , the corresponding image feature map is obtained through the image feature extraction network And get the initial target detection result through the region candidate network S22, construct a relation detector {RelDet, C} including a feature extraction part and a classifier, wherein the feature extraction part RelDet is shared for each sub-dataset, and the sub-dataset D (base) , D (1) , D (2) ,……,D (N) The corresponding classifiers C are represented as C (base) , C (1) , C (2) ,……,C (N) .
4. The method for generating a relational incremental scene graph based on relation independence and feature adapter according to claim 3 is characterized in that The S3 comprises the following sub-steps: S31. Use the pre-trained target detector ObjDet to obtain the base dataset D (base) Each training image in The corresponding image feature map And the initial target detection results S32, base sub-dataset D (base) Each training image in Image feature map Use the relation detector to get relation predictions for all object pairs And the initial target detection results Fine-tune to obtain Among them C (base) and RelDet are randomly initialized; S33, according to D (base) Each training image in Relationship prediction In the object pair (n i , n j The result of object classification is (o i , o j ), then for the object (n i , n j ) Prediction of relation category k The relationship-independent loss function L is calculated by the following formula BCE : where y i，j，k The object is i , n j ) about the dataset label of relation category k, have: Where e is the base of natural logarithm; FreqM i，j，k Indicates that in all categories the label is (o i , o j ) of objects (n i , n j ) is the ratio of object pairs with relation k; Based on the above loss function L BCE Update RelDet and C (base) , after several rounds of iterative training, we finally get (base) The trained relation detector {RelDet base , C (base) }.
5. The method for generating a relational incremental scene graph based on relation independence and feature adapter according to claim 4 is characterized in that The S4 comprises the following sub-steps: S41, for the mth incremental sub-dataset D whose initial value is 1 (m) , initialize the feature extraction part of the relation detector RelDet to the feature extraction part of the relation detector obtained by the latest training, and set the incremental sub-dataset D (m) A single training image in Input the target detector ObjDet, and then use the output of the target detector ObjDet as the input of the feature extraction part RelDet of the relationship detector to obtain the target detection results and relationship features in the figure Randomly initialize the classifier C (m) And by classifier C (m) Output Current relationship predictions Then for the incremental sub-dataset D (m) Calculate the loss function L BCE ; S42, use a randomly initialized feature adapter A to adapt the features After feature transformation, it is passed through all the currently trained classifier sets As the old classifier C (old) Calculate the classification results: Where: m = 1 C (base) , when m≥2 Contains C (base) and S43, the incremental sub-dataset D (m) A single training image in Through the fixed parameter relationship detector {RelDet, C (old) }, get the relationship classification output on all previous old relationship classes For an object pair (n i , n j The result of object classification is (o i , o j ), then for the object (n i , n j ) about the prediction of relation category k, The corresponding prediction result is The corresponding prediction result is The loss function L is calculated by the following formula BKD : S44, combined with the loss function L calculated in S41 BCE and the loss function L calculated in S43 BKD , calculate the total loss function L in a weighted manner: L=L BCE +λL BKD , Where λ is the weight value; With the goal of minimizing the total loss function L, based on L BCE Update RelDet and C (m) , based on L BKD Update C (old) and A, through several rounds of iterative training, we get (m) Trained relation detector 6. The method for generating a relational incremental scene graph based on relation independence and feature adapter according to claim 5 is characterized in that The S5 comprises the following sub-steps: S51. For any mth incremental sub-dataset D satisfying m≥2 (m) , taking the m-1th incremental sub-dataset as the new base sub-dataset, and taking the relationship detector trained on the new base sub-dataset as the basic model, iteratively train according to S4 in sequence, and obtain D (m) Trained relation detector S53, after completing the training for all N incremental sub-datasets, save the last Nth incremental sub-dataset D (N) The trained relation detector is the final relationship detector.
7. The method for generating a relational incremental scene graph based on relation independence and feature adapter according to claim 6 is characterized in that The S6 comprises the following sub-steps: S61, for any target image I * , first use the pre-trained target detector ObjDet to extract the image feature map in the target image And target detection results S62, image feature map for target image And target detection results Using the Relationship Detector Get the relationship prediction between objects in the graph And the target detection results after fine-tuning 8. The method for generating a multi-label scene graph based on multi-granularity features according to claim 7, characterized in that: In the above S44, the weight value λ=0.5.