A method for generating multi-label scene graphs based on multi-granular features

Abstract

The present invention discloses a method for generating a multi-label scene graph based on multi-granularity features. The method comprises the following steps: obtaining a data set for training scene graph generation; performing coarse and fine granularity classification and statistics on all relationship categories in the data set, obtaining all symbiotic relationships of each relationship category and constructing symbiotic labels of each relationship category. Using a pre-trained target detector, all objects in the data set image are detected; using a multi-granularity module to establish multi-granularity features for each image, and constructing multi-classification labels for each object pair in the image; using multi-granularity features as input, and data set labels, multi-classification labels, and symbiotic labels as supervision, the relationship detector is trained; finally, the prediction results of objects and relationships are obtained from the target image, and converted into a scene graph using a multi-label scene graph generator. The present invention makes full use of the connections between relationship categories and models objects with multi-granularity visual features, which can better adapt to the scene graph generation task.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for generating a multi-label scene graph based on multi-granularity features. Background Art

[0002] As a technology for extracting structural natural language information from images, scene graph generation is often used as auxiliary information for some high-level visual tasks, such as visual question answering and image captioning. The goal of scene graph generation is to predict the classification label of each object in an image and the relationship between each pair of objects given an image. The key factors of scene graph generation mainly include the uncertainty of the number of relationships between object pairs and the incomplete annotation of relationships in the dataset. Traditional methods generally regard the scene graph generation task as a single-label problem for relationships. Although some methods have made some breakthroughs in recall rate, they do not fully consider the diversity of relationships between objects.

[0003] The existing mainstream scene graph generation method is a two-stage learning framework. It inputs an original three-channel color image, first uses an object detector to detect all objects in the image, then uses a relationship detector to detect the relationship between objects, and finally combines the object detection results and the relationship detection results to generate a scene graph. These methods can generate scene graphs, but they do not adequately consider the multi-relationship phenomenon between objects in the scene graph, which limits the effectiveness of the current methods. Summary of the invention

[0004] In view of the above problems, the present invention provides a method for generating a multi-label scene graph based on multi-granularity features. The specific technical solution adopted by the present invention is as follows:

[0005] A method for generating a multi-label scene graph based on multi-granularity features comprises the following steps:

[0006] S1, obtaining a training dataset with dataset labels for training scene graph generation;

[0007] S2. Perform coarse-grained and fine-grained classification and statistics on all relationship categories in all images in the training dataset, obtain all symbiotic relationships of each relationship category, and construct symbiotic labels for each relationship category to obtain the maximum possible number of relationships between object pairs;

[0008] S3. Use the pre-trained object detector to detect all objects in the images of the training dataset and obtain the initial object category prediction;

[0009] S4, using the multi-granularity module to establish multi-granularity features for each training image, and construct multi-classification labels for each object pair in the image;

[0010] S5. Using multi-granularity features as input, dataset labels, multi-classification labels, and co-occurrence labels as supervision, the relationship detector is trained so that the relationship detector can output the relationship prediction between objects in the image, and the object category prediction output by the target detector is fine-tuned to form the final object category prediction result;

[0011] S6. For the target image to be generated with a multi-label scene graph, the pre-trained target detector is used to detect all objects in the target image and obtain an initial object category prediction. Then, the relationship detector is used to output the relationship prediction result between objects in the target image, and the object category prediction output by the target detector for the target image is fine-tuned to form a final object category prediction result. Finally, a multi-label scene graph generator is used to convert the relationship prediction result and the object category prediction result in the target image into a multi-label scene graph corresponding to the target image.

[0012] Preferably, the training data set in S1 includes a plurality of training images I train , each training image I train Each object has an object category label O gt and object position B gt And the relationship category label P of the object pair gt , the total number of relationship categories contained in the entire training data set is Q.

[0013] Furthermore, the S2 comprises the following sub-steps:

[0014] S21. For all the relationship categories in the training data set, the proportion of object pairs belonging to each relationship category in the training data set is counted respectively, and the relationship categories with the largest proportion and the total proportion exceeding the threshold are taken as coarse-grained relationships, and the remaining relationship categories are taken as fine-grained relationships; then, each relationship category in all object pairs in the entire training data set is traversed, and all relationship categories that are annotated in the same object pair as the currently traversed relationship category and have the opposite granularity type to the currently traversed relationship category are taken as the symbiotic relationship categories of the currently traversed relationship category;

[0015] S22. Construct a co-occurrence label p in the form of a Q-dimensional vector for each relationship category q. co , symbiotic label p co Each dimension element in corresponds to a relationship category, where the element value corresponding to the relationship category q is p o , and the element values ​​of all symbiotic relationship categories of relationship category q are evenly divided into the remaining (1-p o ), the element values ​​of other remaining relationship categories are all 0;

[0016] S23. For all object pairs in the training data set, find the maximum value p of the number of relationship category labels marked on the object pairs. max .

[0017] Furthermore, the S3 includes the following sub-steps:

[0018] S31, obtaining a pre-trained object detector ObjDet, wherein the object detector ObjDet includes an image feature extraction network and a region candidate network;

[0019] S32, for each training image I in the training data set train , the corresponding image feature map is obtained through the image feature extraction network And get the target detection result through the region candidate network

[0020]

[0021] in: Contains object position prediction and initial object category prediction

[0022] Furthermore, the S4 comprises the following sub-steps:

[0023] S41. Features given by the target detector for a single training image Use M random erase rates Each generates features at different granularities, and a total of M multi-granularity features are obtained At the same time, the characteristics As the overall feature of the image

[0024] S42, for each training image I in the training data set train Each object pair (n i ,n j ), respectively construct the multi-classification label p of the object pair multi , the multi-classification label p multi is a Q-dimensional vector, in which each element corresponds to a relationship category, and the element values ​​corresponding to all relationship categories marked on the object pair have a probability of equal sum of 1, while the element values ​​of other relationship categories are all 0.

[0025] Furthermore, the S5 comprises the following sub-steps:

[0026] S51, build a relationship detector RelDet, where each training image I train of Each of Use the relation detector RelDet to get the relation predictions for all object pairs And Object category prediction in Fine-tune to obtain

[0027]

[0028] in: Object pair (n i ,n j ) The relationship prediction for relationship category k is

[0029] Then each object pair (n i ,n j ) For each relationship category k, all M+1 relationship prediction outputs Fusion is performed to obtain the relationship detection probability result for:

[0030]

[0031] Where λ is the weight value;

[0032] For each training image I train All Merge, obtain and output the final relationship prediction At the same time, after fine-tuning, the relation detector RelDet outputs the final object category prediction

[0033] S52. Set the loss function of the relationship detector RelDet to:

[0034]

[0035] in is the cross entropy loss function, is the KL loss function with temperature parameter T, P co is the symbiotic label p in the training image co The collection of P multi is the multi-classification label p in the training image multi A collection of;

[0036] S53, use SGD optimization method and back propagation algorithm through loss function To train the relationship detector RelDet until the loss function converges.

[0037] Furthermore, the S6 comprises the following sub-steps:

[0038] S61, for any target image I* , first use the pre-trained target detector ObjDet to extract the image feature map in the target image And target detection results Contains object position prediction and initial object category prediction Then use the relationship detector RelDet to predict the relationship between objects in the target image. And fine-tune Get the final object category prediction

[0039] S62, based on the multi-label scene graph generator MultiGraphGen, for each object pair (n i ,n j ) The first p with the highest confidence max As a result, we use this p max Relation prediction results, combined with fine-tuned object category predictions Object position prediction output by the object detector Generate the scene graph G corresponding to the target image:

[0040]

[0041] Furthermore, in S22, p o =0.5.

[0042] Furthermore, in S41, M=3, r1=0, r1=0.1, r2=0.2, r3=0.3.

[0043] Furthermore, in S5, λ=0.4, α=0.5, β=0.05.

[0044] This method is based on a deep neural network and uses the interconnections between different relationship classes and the multi-granularity features of each object to model the relationships in the image, which can better adapt to scene graph generation in different scenarios. Compared with traditional scene graph generation methods, the present invention has the following beneficial effects:

[0045] Firstly, the scene graph generation method of the present invention points out the problem of multiple relationships between object pairs in scene graph generation, that is, there is more than one predicted relationship between each object pair.

[0046] Secondly, the scene graph generation method of the present invention is based on a deep convolutional neural network, and simultaneously models object features and their multi-granularity features, which can better express visual features and improve the final effect of the method.

[0047] Finally, the scene graph generation method of the present invention is compatible with the relationship detectors in most two-stage scene graph generation models and can be used together. It is a plug-in method that is easy to promote. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the process flow of the method for generating a multi-label scene graph of the present invention;

[0049] Figure 2 A schematic diagram of the model structure. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0052] refer to Figure 1 , which is a flowchart of a method for generating a multi-label scene graph based on multi-granularity features in the present invention, wherein the model structure diagram is as follows Figure 2 The specific implementation steps of the method are described in detail below. The method specifically includes the following steps:

[0053] S1. Obtain a training dataset with dataset labels for training scene graph generation.

[0054] In this embodiment, the training data set in the above step S1 includes a plurality of training images I train , each training image I train Each object has an object category label O gt and object position B gt And the relationship category label P of the object pair gt , the total number of relationship categories contained in the entire training data set is Q.

[0055] Of course, in order to test the specific technical effects of the present invention, in addition to the above training data set, a corresponding test data set can also be set. In the present invention, the final algorithm goal is to generate a single test image I test Or the scene graph G corresponding to other target images.

[0056] S2. Perform coarse-grained and fine-grained classification and statistics on all relationship categories in all images in the training dataset, obtain all symbiotic relationships of each relationship category and construct symbiotic labels for each relationship category, and obtain the maximum possible number of relationships between object pairs.

[0057] In this embodiment, the above step S2 includes the following sub-steps:

[0058] S21. For all the relationship categories in the training data set, the proportion of object pairs belonging to each relationship category in the training data set is counted respectively, and the relationship categories with the largest proportion and the total proportion exceeding the threshold are taken as coarse-grained relations, and the remaining relationship categories are taken as fine-grained relations. In other words, it is necessary to count the relationship categories marked on all object pairs in the entire training data set, and for any relationship category, it is necessary to count the proportion of the number of object pairs marked with this relationship category to all object pairs in the training data set, and sort all the relationship categories according to this proportion. If the sum of the proportions of the top N relationship categories exceeds the threshold and the sum of the proportions of the top N-1 relationship categories is lower than the threshold, then these first N relationship categories are all taken as coarse-grained relations, and the remaining relationship categories are all taken as fine-grained relations; then, each relationship category in all object pairs in the entire training data set is traversed, and each relationship category in the traversal process is taken as the current traversal category in turn, and all relationship categories that are marked in the same object pair as the current traversal relationship category and have the opposite granularity type to the current traversal relationship category are taken as the symbiotic relationship categories of the current traversal relationship category. Moreover, the symbiotic relationship is mutual, that is, if one relationship category is a symbiotic relationship of another relationship category, then the other relationship category is also a symbiotic relationship of this relationship category.

[0059] S22. Construct a co-occurrence label p in the form of a Q-dimensional vector for each relationship category q. co , symbiotic label p co Each dimension element in corresponds to a relationship category, where the element value corresponding to the relationship category q is p o , and the element values ​​of all symbiotic relationship categories of relationship category q are evenly divided into the remaining (1-p o ), and the element values ​​of other remaining relationship categories are all 0. Generally speaking, you can set p o =0.5.

[0060] S23. For all object pairs in the training data set, find the maximum value p of the number of relationship category labels marked on the object pairs. max , the maximum value p max It will be used in the multi-label scene graph generator later.

[0061] S3. Use the pre-trained object detector to detect all objects in the images of the training dataset and obtain the initial object category prediction.

[0062] In this embodiment, the above step S3 includes the following sub-steps:

[0063] S31, obtaining a pre-trained object detector ObjDet, wherein the object detector ObjDet includes an image feature extraction network and a region candidate network. The specific network structure of the object detector ObjDet is not limited, as long as it can accurately detect the target (i.e., the object in the image) after pre-training.

[0064] S32, for each training image I in the training data set train , the corresponding image feature map is obtained through the image feature extraction network And get the target detection result through the region candidate network

[0065]

[0066] in: Contains object position prediction and initial object category prediction

[0067] S4. Use the multi-granularity module to establish multi-granularity features for each training image and construct multi-classification labels for each object pair in the image.

[0068] In this embodiment, the above step S4 includes the following sub-steps:

[0069] S41. Features given by the target detector for a single training image Use M random erase rates Each generates features at different granularities, and a total of M multi-granularity features are obtained At the same time, the characteristics As the overall feature of the image Generally speaking, M=3, r1=0, r1=0.1, r2=0.2, and r3=0.3 can be set.

[0070] S42, for each training image I in the training data set train Each object pair (n i ,n j ), respectively construct the multi-classification label p of the object pair multi , the multi-classification label p multiis a Q-dimensional vector, where each element corresponds to a relationship category, and the element values ​​corresponding to all relationship categories annotated on the object pair have a probability of evenly dividing the sum to 1, while the element values ​​of other relationship categories are all 0. For example, if a certain object pair is manually annotated with 4 relationship categories, then in its multi-classification label, the label (i.e., element) values ​​of these 4 relationship categories are all 0.25, and the label values ​​of the other (Q-4) relationship categories are all 0.

[0071] S5. Use multi-granular features as input, dataset labels, multi-classification labels, and co-occurrence labels as supervision to train the relationship detector so that the relationship detector can output the relationship prediction between objects in the image, and fine-tune the object category prediction output by the target detector to form the final object category prediction result.

[0072] In this embodiment, the above step S5 includes the following sub-steps:

[0073] S51, build a relationship detector RelDet, where each training image I train of Each of Use the relation detector RelDet to get the relation predictions for all object pairs And Object category prediction in Fine-tune to obtain

[0074]

[0075] in: Object pair (n i ,n j ) The relationship prediction for relationship category k is By a series of composition;

[0076] Then each object pair (n i ,n j ) For each relationship category k, all M+1 relationship prediction outputs Fusion is performed to obtain the relationship detection probability result for:

[0077]

[0078] Where λ is the weight value. Generally speaking, λ can be set to 0.4.

[0079] For each training image I train All Merge, obtain and output the final relationship prediction At the same time, after fine-tuning, the relation detector RelDet outputs the final object category prediction

[0080] S52. Set the loss function of the relationship detector RelDet to:

[0081]

[0082] in is the cross entropy loss function, is the KL loss function with temperature parameter T, P co is the symbiotic label p in the training image co The collection of P multi is the multi-classification label p in the training image multi Generally speaking, we can set α=0.5, β=0.05;

[0083] S53, use SGD optimization method and back propagation algorithm through loss function To train the relationship detector RelDet until the loss function converges.

[0084] S6. For the target image to be generated with a multi-label scene graph, the pre-trained target detector is used to detect all objects in the target image and obtain an initial object category prediction. Then, the relationship detector is used to output the relationship prediction result between objects in the target image, and the object category prediction output by the target detector for the target image is fine-tuned to form a final object category prediction result. Finally, a multi-label scene graph generator is used to convert the relationship prediction result and the object category prediction result in the target image into a multi-label scene graph corresponding to the target image.

[0085] In this embodiment, the above step S6 includes the following sub-steps:

[0086] S61, for any target image I * , first use the pre-trained target detector ObjDet to extract the image feature map in the target image And target detection results Contains object position prediction and initial object category prediction Then use the relationship detector RelDet to predict the relationship between objects in the target image. And fine-tune Get the final object category prediction

[0087] It should be noted that the specific input, output and internal processing methods of the pre-trained object detector ObjDet and relationship detector RelDet here are basically the same as the aforementioned steps S3 to S5. The only difference is that the training image is replaced by the target image. In order to distinguish them, the outputs corresponding to the training images are marked with * and recorded as the outputs corresponding to the target image.

[0088] S62, based on the multi-label scene graph generator MultiGraphGen, for each object pair (n i ,n j ) The first p with the highest confidence max As a result, we use this p max Relation prediction results, combined with fine-tuned object category predictions Object position prediction output by the object detector Generate the scene graph G corresponding to the target image, which is expressed by the following formula:

[0089]

[0090] The multi-label scene graph generation method based on multi-granularity features shown in S1 to S6 above is applied to a specific example to demonstrate its specific technical effect.

[0091] Example

[0092] The implementation method of this embodiment is as described in S1 to S6 above, and the specific steps will not be elaborated in detail. The following only shows its effect based on case data. The present invention is implemented on a data set with true value annotations, which are:

[0093] VG dataset: This dataset contains 10,807 images with 150 types of object labels and 50 types of relationship labels, with an average of 38 types of object labels and 22 types of relationship labels per image.

[0094] This example is trained on the training set of the VG dataset and tested on the test set. The specific process is as follows Figure 1 As shown in the figure, MGM is a module for generating multi-granular features, ObjDet is an object detector, RelDet is a relationship detector, and p co is the symbiotic label, p multi It is a multi-classification label.

[0095] The detection accuracy of the detection results of the PredCls task (input image and the position and classification of objects in the image, output complete scene graph) of this embodiment is shown in the following table. The three indicators of average recall rate (R@K), classification average recall rate (mR@K) and average recall rate without training samples (zR@K) are mainly used for comparison. The average recall rate (R@K) indicator refers to the proportion of manually labeled (object 1, relationship, object 2) groups covered by each test image when K candidate (object 1, relationship, object 2) groups are given. The classification average recall rate (mR@K) indicator means that when K candidate (object 1, relationship, object 2) groups are given for each test image, the average recall rate of each relationship is first calculated, and then the average recall rate of all relationships is calculated as an arithmetic average. The average recall rate of untrained samples (zR@K) refers to the evaluation of only the (object 1, relationship, object 2) groups in the test set that have not appeared in the training set when K candidate (object 1, relationship, object 2) groups are given for each test image, to see what proportion of manually annotated (object 1, relationship, object 2) groups that have not appeared in the training set are recalled.

[0096] method R@20 R@50 R@100 Motifs 58.46 65.18 67.01 Motifs+OursMulti 67.19(+8.73) 81.47(+16.29) 88.56(+21.55) IMP 54.34 61.05 63.06 IMP+OursMulti 62.62(+8.28) 77.60(+16.55) 85.13(+22.07) Transformer 59.06 65.55 67.29 Transformer+OursMulti 67.65(+8.59) 82.12(+16.57) 89.05(+21.76)

[0097] method mR@20 mR@50 mR@100 Motifs 11.67 14.79 16.08 Motifs+OursMulti 21.22(+9.55) 34.31(+19.52) 44.47(+28.39) IMP 8.85 10.97 11.77 IMP+OursMulti 16.87(+8.02) 28.42(+17.45) 38.10(+26.33) Transformer 12.77 16.30 17.63 Transformer+OursMulti 23.27(+10.50) 38.22(+21.92) 49.56(+31.93)

[0098] method zR@20 zR@50 zR@100 Motifs 1.08 3.24 5.36 Motifs+OursMulti 4.01(+2.93) 13.12(+9.88) 25.74(+20.38) IMP 12.17 17.66 20.25 IMP+OursMulti 15.39(+3.22) 28.54(+10.88) 40.12(+19.87) Transformer 1.35 3.63 5.64 Transformer+OursMulti 6.09(+4.74) 16.83(+13.20) 30.28(+24.64)

[0099] As shown in the above three tables, we tested the combination with three different relationship detectors, Motifs, IMP and Transformer. The result with +OursMulti is the final result of a multi-label scene graph generation method based on multi-granularity features. It can be seen that a multi-label scene graph generation method based on multi-granularity features has significant improvements in all indicators and all relationship detectors.

[0100] In the above embodiment, the multi-label scene graph generation method based on multi-granularity features of the present invention first models the relationship between all relationship categories. On this basis, the relationship prediction problem originally belonging to single-label classification is transformed into a multi-label classification problem, and can be used in conjunction with a variety of relationship prediction models established based on deep neural networks. Finally, the trained scene graph generation model is used to generate the scene graph corresponding to the test image.

[0101] Through the above technical solutions, the embodiment of the present invention develops a multi-label scene graph generation method based on multi-granularity features based on deep learning technology. The present invention makes full use of the connection between relationship categories and uses multi-granularity visual features to model objects, which can better adapt to real scene graph generation tasks.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a multi-label scene graph based on multi-granularity features, characterized in that The following steps are involved: S1, obtain a training dataset with dataset labels for training scene graph generation; S2. Perform coarse-grained and fine-grained classification and statistics on all relationship categories in all images in the training dataset, obtain all symbiotic relationships of each relationship category, and construct symbiotic labels for each relationship category to obtain the maximum possible number of relationships between object pairs; S3. Use the pre-trained object detector to detect all objects in the images of the training dataset and obtain the initial object category prediction; S4, using the multi-granularity module to establish multi-granularity features for each training image, and construct multi-classification labels for each object pair in the image; S5. Using multi-granularity features as input, dataset labels, multi-classification labels, and co-occurrence labels as supervision, the relationship detector is trained so that the relationship detector can output the relationship prediction between objects in the image, and the object category prediction output by the target detector is fine-tuned to form the final object category prediction result; S6. For the target image to be generated with a multi-label scene graph, use the pre-trained target detector to detect all objects in the target image and obtain an initial object category prediction, then use the relationship detector to output the relationship prediction result between objects in the target image, and fine-tune the object category prediction output by the target detector for the target image to form a final object category prediction result, and finally use the multi-label scene graph generator to convert the relationship prediction result and the object category prediction result in the target image into a multi-label scene graph corresponding to the target image; The S4 comprises the following sub-steps: S41. Features given by the target detector for a single training image Use M random erase rates Each generates features at different granularities, and a total of M multi-granularity features are obtained At the same time, the characteristics As the overall characteristics of the image S42, for each training image I in the training data set train Each object pair (n i ,n j ), respectively construct the multi-classification label p of the object pair multi , the multi-classification label p multi is a Q-dimensional vector, where each dimension element corresponds to a relationship category, and the probability that the element values ​​corresponding to all relationship categories marked on the object pair are evenly divided into 1, while the element values ​​of other relationship categories are all 0; The S6 comprises the following sub-steps: S61, for any target image I * , first use the pre-trained target detector ObjDet to extract the image feature map in the target image And target detection results Contains object position prediction and initial object category prediction Then use the relationship detector RelDet to predict the relationship between objects in the target image. And fine-tune Get the final object category prediction S62, based on the multi-label scene graph generator MultiGraphGen, for each object pair (n i ,n j ) The first p with the highest confidence max As a result, we use this p max Relation prediction results, combined with fine-tuned object category predictions Object position prediction output by the object detector Generate the scene graph G corresponding to the target image:

2. According to claim 1, a method for generating a multi-label scene graph based on multi-granularity features is characterized in that The training data set in S1 includes several training images I train , each training image I train Each object has an object category label O gt and object position B gt And the relationship category label P of the object pair gt , the total number of relationship categories contained in the entire training data set is Q.

3. The method for generating a multi-label scene graph based on multi-granularity features according to claim 2 is characterized in that The S2 comprises the following sub-steps: S21. For all relationship categories in the training data set, count the proportion of object pairs belonging to each relationship category in the training data set, and take the relationship categories with the largest proportion and the total proportion exceeding the threshold as coarse-grained relationships, and the remaining relationship categories as fine-grained relationships; Then, every relationship category in all object pairs in the entire training data set is traversed, and all relationship categories that are annotated in the same object pair as the currently traversed relationship category and whose granularity type is opposite to that of the currently traversed relationship category are taken as the symbiotic relationship categories of the currently traversed relationship category; S22. Construct a co-occurrence label p in the form of a Q-dimensional vector for each relationship category q. co , symbiotic label p co Each dimension element in corresponds to a relationship category, where the element value corresponding to the relationship category q is p o , and the element values ​​of all symbiotic relationship categories of relationship category q are evenly divided into the remaining (1-p o ), the element values ​​of other remaining relationship categories are all 0; S23. For all object pairs in the training data set, find the maximum value p of the number of relationship category labels marked on the object pairs. max .

4. According to the method for generating a multi-label scene graph based on multi-granularity features according to claim 3, it is characterized in that The S3 comprises the following sub-steps: S31, obtaining a pre-trained object detector ObjDet, wherein the object detector ObjDet includes an image feature extraction network and a region candidate network; S32, for each training image I in the training data set train , the corresponding image feature map is obtained through the image feature extraction network And get the target detection result through the region candidate network in: Contains object position prediction and initial object category prediction 5. According to the method for generating a multi-label scene graph based on multi-granularity features according to claim 4, it is characterized in that The S5 comprises the following sub-steps: S51, build a relationship detector RelDet, where each training image I train of Each of Use the relation detector RelDet to get the relation predictions for all object pairs And Object category prediction in Fine-tune to obtain in: Object pair (n i ,n j ) The relationship prediction for relationship category k is Then each object pair (n i ,n j ) For each relationship category k, all M+1 relationship prediction outputs Fusion is performed to obtain the relationship detection probability result for: Where λ is the weight value; For each training image I train All Merge, obtain and output the final relationship prediction At the same time, after fine-tuning, the relation detector RelDet outputs the final object category prediction S52. Set the loss function of the relationship detector RelDet to: in is the cross entropy loss function, is the KL loss function with temperature parameter T, P co is the symbiotic label p in the training image co The collection of P multi is the multi-classification label p in the training image multi A collection of; S53, use SGD optimization method and back propagation algorithm through loss function To train the relationship detector RelDet until the loss function converges.

6. The method for generating a multi-label scene graph based on multi-granularity features according to claim 5, characterized in that: In S22, p o =0.

5.

7. The method for generating a multi-label scene graph based on multi-granularity features according to claim 5, characterized in that: In S41, M=3, r1=0, r1=0.1, r2=0.2, r3=0.

3.

8. The method for generating a multi-label scene graph based on multi-granularity features according to claim 5, characterized in that: In S5, λ=0.4, α=0.5, and β=0.05.

Images