A shape-aware feature based course learning type panoramic scene graph generation method

Abstract

The application discloses a kind of course learning type panoramic scene graph generation method based on shape perception feature, belong to computer vision field.This method includes: using panoramic segmenter to identify object in image, obtain candidate frame and its corresponding mask, further extract boundary box feature, mask feature and boundary feature;Using phased feature fusion strategy, obtain object feature and relationship feature with different complexity;Based on cognitive learning difficulty difference, group predicate, use the predicate of each group to train relationship classifier in stages, and ensure the balance of the number of predicates within the group through predicate sampling strategy;Using course learning strategy gradually improves the complexity of input feature, so that the model learns predicate of different cognitive difficulty in order from easy to difficult, and fuses knowledge distillation strategy to save the knowledge obtained in early stage.Experiments prove that the application can significantly improve the accuracy of the model in relation prediction, and shows excellent performance.

Description

Technical Field

[0001] This invention pertains to the task of panoramic scene image generation in the field of computer vision, and particularly relates to a course-based panoramic scene image generation method based on shape-aware features. Background Technology

[0002] Scene graph generation is a fundamental scene understanding task that goes beyond simply classifying and locating objects by predicting the relationships between them. However, traditional scene graph generation tasks rely on bounding box-based paradigms, which suffer from inaccurate object localization and limited background annotation. Therefore, panoramic scene graph generation utilizes more precise scene mask representation (panoramic segmentation) and defines relationships between background objects, thus providing a more accurate and comprehensive understanding of the scene.

[0003] Since panoramic scene graph generation is built upon scene graph generation, it faces similar challenges: 1) Long-tail effect: A few predicate categories occupy most of the ground truth labels, causing the model's predictions to favor head predicates. 2) Semantic overlap: A subject-object pair may contain multiple valid relations, and these relations may have semantic overlap, while existing training paradigms use one-hot encoding. To overcome these challenges, existing panoramic scene graph generation methods draw inspiration from scene graph generation tasks, employing resampling and reweighting methods to address the long-tail effect, or label smoothing methods to address semantic overlap.

[0004] While existing panoramic scene graph generation methods have made some progress in improving the accuracy of relationship prediction, almost all of them neglect shape-aware features. Specifically, state-of-the-art panoramic scene graph generation methods simply replace object features with better representations from panoramic segmenters; they still utilize bounding box features generated from the minimum bounding boxes of these masks, ignoring crucial shape-aware features. This limitation can hinder a holistic understanding of the scene and potentially lead to semantic confusion in fine-grained visual relationship prediction. Summary of the Invention

[0005] The purpose of this invention is to solve the problem that existing methods rely solely on bounding box features, and to propose a course-based panoramic scene map generation method based on shape-aware features. The aim is to predict objects and their relationships in the scene as accurately as possible, given a panoramic scene map, so as to achieve comprehensive scene understanding.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0007] A method for generating panoramic scene images for curriculum learning based on shape-aware features includes the following steps:

[0008] S1. Pre-acquire several training images, use the trained panoramic segmenter to identify objects in each training image, obtain the mask of each object, use the mask of each object to obtain the candidate box of each object, extract the bounding box features from the candidate box of each object, and extract the mask features and boundary features from the mask of each object.

[0009] S2. Using a phased feature fusion strategy, bounding box features, mask features and boundary features are gradually fused to obtain object features and relationship features with different levels of complexity and corresponding to each stage. The object features are used to train the object classifier, and the relationship features are used to train the relationship classifier.

[0010] S3. Based on the differences in cognitive learning difficulty, the predefined predicate categories contained in the training dataset are divided into several groups. Accordingly, the entire training process of the relation classifier is divided into several training stages. Each training stage uses a dedicated relation classifier to distinguish the predicates of its respective group. The predicate sampling strategy ensures the balance of the number of predicates within the group. The sampled training images are used as the final training images.

[0011] S4. The course learning strategy is used on the final training images to gradually increase the complexity of the input features in each training stage, so that the relation classifier can learn predicates of different cognitive difficulty in order from easy to difficult, and integrate the knowledge distillation strategy to preserve the knowledge obtained in the early stage.

[0012] S5. Input the test image to be processed into the trained panoramic segmenter, identify the objects in the test image, output the panoramic segmentation result of the test image, input the panoramic segmentation result into the relation classifier trained in the last training stage, classify the relationships between the detected objects, and output several triples representing the relationships between pairs of objects in the test image. The elements in the triples represent the subject, predicate, and object, respectively.

[0013] Based on the above scheme, each step can be implemented in the following preferred manner.

[0014] As a preferred embodiment, the specific process of step S1 is as follows:

[0015] S11. For the input training images, the trained panoramic segmenter is used to identify objects in each training image. Each training image is segmented into a set of object masks M. Each object in the training image corresponds to a mask in the mask set M. For the mask m of the i-th object... i ∈{0,1} H×W Calculate each bounding box, and take the smallest bounding box corresponding to the i-th mask as the candidate box b of the i-th object. i , where H and W represent the height and width of the training image, respectively;

[0016] S12, for the candidate box b of the i-th object i The RoIAlign operation is used to extract the bounding box features f of the i-th object. i b ;

[0017] S13, for the mask m of the i-th object i First, a binary erosion operation is used to extract the contour of the i-th object. Then, Zernike moments are used to convert the contour of the i-th object into mask features f. i m ;

[0018] S14. For the subject object mask m of the i-th object... g ∈m i and object mask m j ∈m i Calculate the first intersection region d of the subject object mask and the object object mask. gj The extracted first intersection region is transformed into boundary features using binary erosion and Zernike moments.

[0019] Preferably, the mask feature f i m The function form is:

[0020]

[0021] The first intersection region d gj The function form is:

[0022] d gj =m g ∩m j

[0023] The boundary features The function form is:

[0024]

[0025] Where Zernike denotes Zernike moments; ∩ denotes finding the intersection; Indicates negation; erosion(m i ) represents the mask m for the i-th object. i Perform binary erosion operation; erosion(d gj This indicates that a binary erosion operation is performed on the first intersection region.

[0026] As a preferred embodiment, the specific process of step S2 is as follows:

[0027] S21. The bounding box feature f of the i-th object... i b And the candidate bounding box b of the i-th object i They are input together into the object context encoder trained in the first stage. In the process, the object characteristics F of the first stage are obtained. i (1) ; classify the i-th object into σ i GloVe encoding w i Integrating the object features F from the first stage i (1) and bounding box features f i b In the process, the first-stage fusion features are obtained, and these features are then input into the first-stage trained relation encoder. In the process, the first stage of improved object features are obtained. Improved object features from the first stage Obtain the object features of the subject in the first stage. The characteristics of the object with the verb Obtain subject and object candidate boxes from the object candidate boxes, and calculate the union of the subject and object candidate boxes to obtain the feature u of the joint subject-object candidate box. gj The object characteristics of the subject in the first stage The characteristics of the object with the verb The features are concatenated to obtain the first-stage concatenated features. These features are then combined with the features of the subject-object pair joint candidate boxes. gj Performing a dot product yields the first-stage subject-object relationship characteristics.

[0028] S22, extract the bounding box features f of the i-th object. i b Mask features f i m and GloVe encoding w i and the object characteristics F in the first stage i (1) They are input together into the relation encoder trained in the second stage. In the process, the object features improved in the second stage are obtained. Improved object features from the second stage Obtain the object features of the subject in the second stage. The characteristics of the object with the verb The object characteristics of the subject in the second stage The characteristics of the object with the verb The features are then concatenated to obtain the second-stage concatenated features. These features are then combined with the features of the subject-object pair joint candidate boxes. gjPerforming a dot product yields the relational characteristics of the subject-object pair in the second stage.

[0029] S23. The bounding box feature f of the i-th object... i b Mask features f i m and GloVe encoding w i The object characteristics F in the first and second stages i (2) They are input together into the relation encoder trained in the third stage. In the process, the object features improved in the third stage are obtained. Object features improved from the third stage Obtaining the object features of the subject in the third stage The characteristics of the object with the verb The object characteristics of the subject in the third stage The characteristics of the object with the verb The features are then stitched together to obtain the first feature after the third stage of stitching; the boundary features are then... Features u of the joint candidate box and the subject / object pair gj By concatenating the first and second features from the third stage, we obtain the second feature after concatenation. Then, we perform a dot product on the first and second features from the third stage to obtain the subject-object relationship feature from the third stage.

[0030] As a preferred embodiment, the specific process of step S3 is as follows:

[0031] S31. Sort all predicate categories in descending order of the number of training instances in the training dataset to obtain a sorted predicate set. Then, evenly distribute all predicates in the sorted predicate set into three different groups as the initial grouping.

[0032] S32. Calculate the confusion matrix C based on the pre-obtained baseline model prediction results. Each value c[i][j] in the confusion matrix represents the number of samples that label the i-th relation category as the j-th relation category. For the current predicate in the initial group, determine whether the corresponding confusion matrix value exceeds the preset threshold. If it exceeds the threshold, the current predicate is regarded as a predicate with semantic confusion and adjusted based on semantic similarity. If it does not exceed the threshold, the current predicate pair is kept in the initial group until all predicates in the initial group have been traversed, resulting in three optimized predicate groups based on differences in cognitive difficulty.

[0033] S33. Based on the optimized predicate grouping obtained in S32, the training process of the relation classifier is divided into three training stages, with each training stage using its own independent relation classifier. Except for the relation classifier in the first training stage, the relation classifiers in the remaining training stages all follow a continuously growing classification space.

[0034] S34. Obtain the classification space The frequency of each predicate in the text (freq) r Using frequency of occurrence (freq) r Construct an optimized upsampling frequency; obtain the classification space. The median number of predicates (Med) and the maximum number of predicates (Max) are used to determine the classification space. The k-th predicate r k The number of occurrences, Cnt(r) k Is it greater than the median value (Med)? If it is, then the first downsampling rate will be adjusted. Set to Cnt(r) k ) / Med, and the second downsampling rate Set to Max-Cnt(r) k ) / Max-Med, otherwise use the first downsampling rate Set to 1, and simultaneously set the second downsampling rate Set to [Max-Cnt(r)] k ) / Max-Med]*k 2 , where k 2 The square of the predicate number is used; the weighted sum of the first downsampling rate and the second downsampling rate is used as the optimized downsampling frequency; the training images containing tail predicates in the training dataset are upsampled according to the optimized upsampling frequency; the training images containing head predicates in the training dataset are downsampled according to the optimized downsampling frequency; and finally, the sampled training dataset is used as the final training set.

[0035] Furthermore, the specific rules for adjusting based on semantic similarity are as follows: For the kth initial group If the initial group If a predicate with a confusing occurrence is detected, the predicate with a lower frequency will be moved to the (k+1)th initial group. Alternatively, move the predicates that appear more frequently to the (k-1)th initial group. middle.

[0036] Furthermore, the classification space of the relation classifier is used to identify the predicate categories in the previous and current groups, and its functional form is:

[0037]

[0038] in, ∪ represents the 1st, ..., kth initial groups respectively; ∪ represents finding the union.

[0039] Furthermore, the optimized upsampling frequency rate1 function is in the form of:

[0040]

[0041] The optimized downsampling frequency rate2 function is in the form of:

[0042]

[0043] Where θ and λ both represent hyperparameters; where, This indicates the first downsampling rate biased towards the head predicate; This indicates the second downsampling rate, which is biased towards the tail predicate.

[0044] Preferably, the specific process for each training stage of the relation classifier is as follows: The relation features of the subject-object pairs obtained in S2 for each stage are... The input is fed into the relation classifier for each corresponding training stage. In this way, the predicted value of the relation distribution for each training stage can be obtained. Predicted values ​​based on the relational distribution at each training stage The relationship between the sampled and the true values A cross-entropy loss is constructed, and a KL divergence loss is constructed based on the predicted relation distribution values ​​of two different training stages. The weighted KL divergence loss and the cross-entropy loss are added together as the total loss. The parameters of the relation classifier in the first training stage are updated by minimizing the cross-entropy loss of the first training stage, the parameters of the relation classifier in the second training stage are updated by minimizing the total loss of the second training stage, and the parameters of the relation classifier in the last training stage are updated by minimizing the total loss of the third training stage. The training is iterated until a preset iteration threshold is reached, and the relation classifiers in each training stage converge, finally obtaining the relation classifiers trained in each training stage.

[0045] Preferably, the panoramic segmenter adopts the Panoptic Segformer model or the Panoptic FPN model.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] 1) This invention proposes shape-aware features, which, compared with existing methods based on bounding box features, can better utilize object shape and contour information, thereby better capturing interaction information between objects.

[0048] 2) This invention integrates shape-aware features into the panoramic scene image generation task through a course-based learning approach, enabling the model to learn predicates in an order of increasing difficulty, which aligns with human cognitive processes. Experimental results show that this invention significantly improves model performance and exhibits good generalization ability. Attached Figure Description

[0049] Figure 1 This is an overall flowchart of the present invention;

[0050] Figure 2 This is a schematic diagram of the core steps of the present invention. Detailed Implementation

[0051] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0052] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0053] This invention relates to the task of panoramic scene image generation in the field of computer vision. The task aims to generate a structured description of objects and their relationships within a scene from an input panoramic image. The specific implementation process of this invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] like Figure 1 and Figure 2 As shown, in a preferred embodiment of the present invention, a method for generating panoramic scene images for course learning based on shape-aware features is provided, the basic steps of which are as follows:

[0055] S1. Pre-acquire several training images, use the trained panoramic segmenter to identify objects in each training image, obtain the mask of each object, use the mask of each object to obtain the candidate box of each object, extract the bounding box features from the candidate box of each object, and extract the mask features and boundary features from the mask of each object.

[0056] It should be noted that the specific process of step S1 of the present invention is as follows:

[0057] S11. For the input training images, the trained panoramic segmenter is used to identify objects in each training image. Each training image is segmented into a set of object masks M. Each object in the training image corresponds to a mask in the mask set M. For the mask m of the i-th object... i ∈{0,1} H×W Calculate each bounding box, and take the smallest bounding box corresponding to the i-th mask as the candidate box b of the i-th object. i , where H and W represent the height and width of the training image, respectively.

[0058] S12, for the candidate box b of the i-th object i The RoIAlign operation is used to extract the bounding box features f of the i-th object. i b .

[0059] Furthermore, the mask feature f i m The function form is:

[0060]

[0061] Where Zernike denotes Zernike moments; ∩ denotes finding the intersection; Indicates negation; erosion(m i ) represents the mask m for the i-th object. i Perform binary etch operation.

[0062] S14. For the subject object mask m of the i-th object... g ∈m i and object mask m j ∈m i Calculate the first intersection region d of the subject object mask and the object object mask. gj The extracted first intersection region is transformed into boundary features using binary erosion and Zernike moments.

[0063] Furthermore, the first intersection region d gj The function form is:

[0064] d gj =m g ∩m j

[0065] Furthermore, boundary features The function form is:

[0066]

[0067] Thus, through the steps S11 to S14 above, two shape-aware features, namely mask features and boundary features, are obtained, which can be used for subsequent object features and relationship features.

[0068] It should be noted that in this embodiment, Panoptic FPN is selected as the panoramic segmenter in the visual relationship classification subtask, and Panoptic Segformer is selected as the panoramic segmenter in the scene graph detection subtask. Furthermore, all parameters of the panoramic segmenter need to be frozen during training.

[0069] S2. Using a phased feature fusion strategy, bounding box features, mask features, and boundary features are gradually fused to obtain object features and relationship features with different levels of complexity corresponding to each stage. The object features are used to train the object classifier, and the relationship features are used to train the relationship classifier.

[0070] It should be noted that the specific process of step S2 of the present invention is as follows:

[0071] S21. The bounding box feature f of the i-th object... i b And the candidate bounding box b of the i-th object i They are input together into the object context encoder trained in the first stage. In the process, the object characteristics F of the first stage are obtained. i (1) ; classify the i-th object into σ i GloVe encoding w i Integrating the object features F from the first stage i (1) and bounding box features f i b In the process, the first-stage fusion features are obtained, and these features are then input into the first-stage trained relation encoder. In the process, the first stage of improved object features are obtained. Improved object features from the first stage Obtain the object features of the subject in the first stage. The characteristics of the object with the verb Obtain subject and object candidate boxes from the object candidate boxes, and calculate the union of the subject and object candidate boxes to obtain the feature u of the joint subject-object candidate box. gj The object characteristics of the subject in the first stage The characteristics of the object with the verb The features are concatenated to obtain the first-stage concatenated features. These features are then combined with the features of the subject-object pair joint candidate boxes. gj Performing a dot product yields the first-stage subject-object relationship characteristics.

[0072] Furthermore, the object characteristics in the first stage The function form is:

[0073]

[0074] Furthermore, the object features improved in the first stage The function form is:

[0075]

[0076] Furthermore, the characteristics of the subject-object relationship in the first stage The function form is:

[0077]

[0078] It should be noted that the improved object features obtained in step S21 require the use of a relational encoder. In the context modeling part, the object context encoder used in this embodiment is consistent with the baseline model.

[0079] S22, extract the bounding box features f of the i-th object. i b Mask features f i m and GloVe encoding w i and the object characteristics F in the first stage i (1) They are input together into the relation encoder trained in the second stage. In the process, the object features improved in the second stage are obtained. Improved object features from the second stage Obtain the object features of the subject in the second stage. The characteristics of the object with the verb The object characteristics of the subject in the second stage The characteristics of the object with the verb The features are then concatenated to obtain the second-stage concatenated features. These features are then combined with the features of the subject-object pair joint candidate boxes. gj Performing a dot product yields the relational characteristics of the subject-object pair in the second stage.

[0080] Furthermore, the second phase improved object features The function form is:

[0081]

[0082] Furthermore, the characteristics of the subject-object pair relationship in the second stage The function form is:

[0083]

[0084] S23. The bounding box feature f of the i-th object... i b Mask features f i m and GloVe encoding w i The object characteristics F in the first and second stages i (2) They are input together into the relation encoder trained in the third stage. In the process, the object features improved in the third stage are obtained. Object features improved from the third stage Obtaining the object features of the subject in the third stage The characteristics of the object with the verb The object characteristics of the subject in the third stage The characteristics of the object with the verb The features are then stitched together to obtain the first feature after the third stage of stitching; the boundary features are then... Features u of the joint candidate box and the subject / object pair gj By concatenating the first and second features from the third stage, we obtain the second feature after concatenation. Then, we perform a dot product on the first and second features from the third stage to obtain the subject-object relationship feature from the third stage.

[0085] Furthermore, the third stage improves object features The function form is:

[0086]

[0087] Furthermore, the characteristics of the subject-object relationship in the third stage The function form is:

[0088]

[0089] in, Indicates horizontal splicing; This indicates dot product.

[0090] It should be noted that the subject-object relationship characteristics in the third stage of step S23... In order to capture the interaction between the main and guest objects, boundary features are introduced. Specifically, firstly, the boundary features Features u of the joint candidate boxes with the main and guest pairs ij The pieces are then assembled, and the assembled result is compared with the main and secondary objects to identify their characteristics. Perform a dot product.

[0091] Thus, through the steps S21 to S23 above, the object features and relationship features of each stage are obtained, which are used in the object classifier and relationship classifier of each stage respectively, thereby obtaining the prediction results of the object category and the prediction results of the relationship category.

[0092] S3. Based on the differences in cognitive learning difficulty, the predefined predicate categories contained in the training dataset are divided into several groups. Accordingly, the entire training process of the relation classifier is divided into several training stages. Each training stage uses a dedicated relation classifier to distinguish the predicates of its respective group. The predicate sampling strategy ensures the balance of the number of predicates within the group. The sampled training images are used as the final training images.

[0093] It should be noted that the specific process of step S3 of the present invention is as follows:

[0094] S31. Sort all predicate categories in descending order of the number of training instances in the training dataset to obtain the sorted predicate set. Then sort the predicate set All predicates are evenly distributed into three distinct groups as the initial grouping.

[0095] In this embodiment, considering that the number of training instances corresponding to different predicate categories is different, for example, the number of training instances corresponding to the predicate category "on" is greater than the number of training instances corresponding to the predicate category "standing on", the predicate category "on" is placed before the predicate category "standing on". This ensures that predicate categories with a large number of training instances are easier to learn, while predicate categories with a small number of training instances are more difficult to learn.

[0096] S32. Calculate the confusion matrix C based on the pre-obtained baseline model prediction results. Each value c[i][j] in the confusion matrix represents the number of samples that label the i-th relation category as the j-th relation category. For the current predicate in the initial group, determine whether the corresponding confusion matrix value exceeds a preset threshold: if it does, the current predicate is considered a predicate with semantic confusion and adjusted based on semantic similarity; if it does not exceed the threshold, the current predicate pair is retained in the initial group until all predicates in the initial group have been traversed, resulting in three optimized predicate groups based on differences in cognitive difficulty.

[0097] It should be noted that in this embodiment, the threshold is set to 0.8 in the design of the predicate grouping strategy based on the difference in cognitive difficulty.

[0098] In this embodiment, the specific rules for adjustment based on semantic similarity are as follows: For the kth initial group If the initial group If a predicate with a confusing occurrence is detected, the predicate with a lower frequency will be moved to the (k+1)th initial group. Alternatively, move the predicates that appear more frequently to the (k-1)th initial group. middle.

[0099] S33. Based on the optimized predicate grouping obtained in S32, the training process of the relation classifier is divided into three training stages, with each training stage using its own independent relation classifier. Except for the relation classifier in the first training phase In addition, the relation classifiers in the remaining training phases all follow a continuously growing classification space.

[0100] In this embodiment, the classification space of the relation classifier is used to identify the predicate categories in the previous and current groups, and its functional form is:

[0101]

[0102] in, ∪ denotes the 1st, ..., kth initial group; ∪ denotes finding the union of sets.

[0103] S34. Obtain the classification space The frequency of each predicate in the text (freq) r Using frequency of occurrence (freq) r Construct an optimized upsampling frequency; obtain the classification space. The median number of predicates (Med) and the maximum number of predicates (Max) are used to determine the classification space. The k-th predicate r k The quantity Cnt(r) k Is it greater than the median value (Med)? If it is, then the first downsampling rate will be adjusted. Set to Cnt(r) k ) / Med, and the second downsampling rate Set to Max-Cnt(r) k ) / Max-Med, otherwise use the first downsampling rate Set to 1, and simultaneously set the second downsampling rate Set to [Max-Cnt(r)] k ) / Max-Med]*k 2 The weighted sum of the first downsampling rate and the second downsampling rate is used as the optimized downsampling frequency. The training images containing tail predicates in the training dataset are upsampled according to the optimized upsampling frequency, and the training images containing head predicates in the training dataset are downsampled according to the optimized downsampling frequency. Finally, the sampled training dataset is used as the final training set.

[0104] In step S34 of this embodiment, if the classification space The number of predicates is 10, starting from predicate r1 and continuing to predicate r. 10 The end, then r k Representing r1~r 10 For any predicate in the classification space, when k=5, The number of the fifth predicate r5 is denoted as Cnt(r5), and k in the second downsampling rate is... 2 Set it to 25 (i.e., 5 squared).

[0105] Furthermore, in this embodiment, the optimized upsampling frequency rate1 function is in the form of:

[0106]

[0107]

[0108]

[0109] in, This indicates the first downsampling rate biased towards the head predicate; This indicates the second downsampling rate biased towards the tail predicate; Cnt(r k ) represents the k-th predicate r k The number of occurrences; k 2 Indicates the square of the predicate number.

[0110] Furthermore, in this embodiment, the optimized downsampling frequency rate2 function is in the form of:

[0111]

[0112] Here, θ and λ both represent hyperparameters. In the design of the predicate sampling strategy, the hyperparameters θ and λ are selected as 0.07 and 0.5 respectively.

[0113] In this embodiment, to ensure that each relation classifier is discriminative and unaffected by the long-tail effect's bias towards head predicates, step S34 utilizes a predicate sampling strategy to guarantee a balanced number of predicates within a group. Specifically, for the entire dataset, images containing tail predicate samples are upsampled to enrich the tail predicate samples. The upsampling frequency is related to the frequency of predicate r, freq. r Related. On the other hand, in order to mitigate the significant difference in the number of triplet samples among different predicates within a group, a downsampling frequency rate2 is designed to downsample triples containing head predicate samples.

[0114] Therefore, through steps S31 to S34, three sets of predicates based on differences in cognitive learning difficulty are obtained, and the training of the relation classifier is divided into three stages. Furthermore, the application of a predicate sampling strategy ensures that the number of predicates within the relation classifier remains balanced in each stage.

[0115] S4. The final training images are used with a course learning strategy to gradually increase the complexity of the input features in each training stage, so that the relation classifier can learn predicates of different cognitive difficulty in order from easy to difficult, and a knowledge distillation strategy is integrated to preserve the knowledge obtained in the early stages.

[0116] In this invention, the specific process of each training stage of the relation classifier is as follows: The relation features of the subject-object pairs obtained in S2 for each stage are... The input is fed into the relation classifier for each corresponding training stage. In this way, the predicted value of the relation distribution for each training stage can be obtained. Predicted values ​​based on the relational distribution at each training stage The relationship between the sampled and the true values A cross-entropy loss is constructed, and a KL divergence loss is constructed based on the predicted relation distribution values ​​of two different training stages. The weighted KL divergence loss and the cross-entropy loss are added together as the total loss. The parameters of the relation classifier in the first training stage are updated by minimizing the cross-entropy loss of the first training stage, the parameters of the relation classifier in the second training stage are updated by minimizing the total loss of the second training stage, and the parameters of the relation classifier in the last training stage are updated by minimizing the total loss of the third training stage. The training is iterated until a preset number of iterations is reached and the relation classifier converges, finally obtaining the relation classifier trained in each training stage.

[0117] In this embodiment, the relational features of the subject-object pair obtained in the kth stage from S2 are... The input is fed into the relation classifier at the kth training stage. Thus, the predicted relation distribution value for the k-th training stage is obtained. For the relation classifier in the first training phase (when k=1), the predicted values ​​based on the relation distribution in the first training phase are... and the corresponding true value of the sampled relationship distribution The first training stage involves constructing a cross-entropy loss. The parameters of the first training stage relation classifier are updated based on minimizing this loss. This iterative training continues until a preset threshold of iterations is reached, at which point the first training stage relation classifier converges, resulting in the trained relation classifier for the first training stage. For the second training stage relation classifier (when k=2), the predicted relation distribution values ​​from the second training stage are used... and the corresponding true value of the sampled relationship distribution Construct the cross-entropy loss for the second training phase, based on the predicted values ​​of the relation distribution from the first training phase. The predicted distribution of the relationship between the slices in the second training phase and the predicted values A KL divergence loss is constructed for the second training stage. This KL divergence loss is weighted and then added to the cross-entropy loss of the first training stage to obtain the total loss for the second training stage. The parameters of the relation classifier in the second training stage are updated based on minimizing this total loss. This process is iterated until a preset threshold of iterations is reached, at which point the relation classifier in the second training stage converges, resulting in the final relation classifier trained in the second training stage. For the relation classifier in the third training stage (when k=3), the predicted relation distribution values ​​from the third training stage are used... and the corresponding true value of the sampled relationship distribution Construct a cross-entropy loss for the third training phase, based on the predicted values ​​of the relation distribution from the first training phase. The predicted distribution of the relationship between the slices in the second training phase and the predicted values The predicted value of the relation distribution in the second training phase The predicted distribution of the relationship between the slices in the third training phase and the predicted values Predicted relation distribution values ​​in the first training phase The predicted distribution of the relationship between the slices in the third training phase and the predicted values Construct the KL divergence loss for the third training stage. After weighting each KL divergence loss in the third training stage, add it to the cross-entropy loss of the training stage to obtain the total loss of the third training stage. Update the parameters of the relation classifier in the third training stage based on minimizing the total loss of the third training stage. Iterate the training until the preset number of iterations threshold is reached, and the relation classifier in the third training stage converges. Finally, the relation classifier trained in the third training stage is obtained.

[0118] Furthermore, the distribution of predicted relations in the k-th training stage The function is in the following form:

[0119]

[0120] Furthermore, the cross-entropy loss function for the k-th training stage is in the form of:

[0121]

[0122] Furthermore, the KL divergence loss function takes the form of:

[0123]

[0124] Furthermore, the total loss function takes the form:

[0125] L total =L ce +αL kl

[0126] Wherein, α represents the hyperparameter. In this embodiment, the hyperparameter α is set to 1 in the design of the training objective function. This indicates the relation classifier from the k1th training stage. Relation classifier at the k2th training stage A set of paired knowledge matching; This represents the true value of the relation distribution after sampling in the k1th training phase; This represents the predicted relation distribution value at the k-th training stage; This represents the predicted relation distribution value of the k2-th training stage slice, which is not included in the previous classification space. To ensure that the dimensions are equal, subsets that do not exist in the set are excluded. This represents the predicted relation distribution value at the k1th training stage.

[0127] It should be noted that, since the relation classifier at each stage focuses on distinguishing predicates within a group, an additional knowledge distillation strategy is employed to preserve knowledge acquired in earlier stages. During the relation classifier training process, KL divergence loss is used to measure the similarity between two relation distributions, SGD is used as the optimizer, the initial learning rate is set to 0.01, and the relation classifier is trained for 12 epochs.

[0128] S5. Input the test image to be processed into the trained panoramic segmenter to identify objects in the test image and output the panoramic segmentation result of the test image. This panoramic segmentation result includes the object mask and candidate boxes. Input the panoramic segmentation result into the relation classifier trained in the last training stage. In the test image, the relationships between detected objects are classified, and several triples representing the relationships between pairs of objects are output. The elements in the triples represent the subject, predicate, and object, respectively.

[0129] The steps S1 to S5 described above constitute a framework for a course-based panoramic scene graph generation method based on shape-aware features. Therefore, this invention, based on steps S1 to S5, can fully utilize shape-aware features to accurately capture interaction information between objects, achieving a better understanding of complex visual scenes. This invention reveals a key problem in existing panoramic scene graph generation tasks: over-reliance on bounding box features while neglecting shape-aware features. The core of the above method is to introduce shape-aware features into the panoramic scene graph generation task. Furthermore, inspired by human cognitive processes, this invention allows the relation classifier to learn predicates in a progressively more complex manner, thereby improving the accuracy of relation detection and achieving accurate understanding of complex scenes. This invention is a model-independent method that can be flexibly integrated into any two-stage panoramic scene graph generation model. Compared to other panoramic scene graph generation methods, this invention effectively overcomes the limitation of relying solely on bounding box features by integrating shape-aware features into the panoramic scene graph generation task.

[0130] The present invention will now demonstrate the application effect of the shape-aware feature-based course learning panoramic scene image generation method described in S1 to S5 of the above embodiments on a specific dataset through a specific example, so as to facilitate understanding of the essence of the present invention.

[0131] Example

[0132] This embodiment conducted experiments on the publicly available panoramic scene graph dataset PSG. This dataset contains 48,749 images, covering 133 object categories (80 entity categories and 53 background categories) and 56 relation categories. Each image is labeled with panoramic segmentation and scene graph. Furthermore, this embodiment conducted experiments based on three baseline models: Motifs, VCTree, and Transformer.

[0133] This embodiment focuses on experiments in two sub-tasks of panoramic scene graph generation: 1) Visual Relationship Classification (PredCls): Given an image and masks of all objects in the image, as well as object categories, the model only needs to predict the visual relationship categories between objects. 2) Scene Graph Detection (SGDet): Given only an image, the model needs to detect all objects and predict the categories of all objects and the visual relationship categories between objects.

[0134] To objectively evaluate the performance of this invention, this embodiment selects three main evaluation metrics: 1) Recall@K (R@K): This metric calculates the proportion of times the visual triples are correctly predicted among the top K visual triple prediction results with the highest confidence. 2) Mean Recall@K (mR@K): This metric first calculates R@K for each predicate category separately, and then averages the R@K for all predicate categories. 3) Mean: This metric is obtained by averaging R@K and mR@K. Since the former tends to favor head predicate categories and the latter tends to favor tail predicate categories, the Mean metric can better evaluate the performance of all predicate categories.

[0135] Table 1. Experimental results of the method of the present invention on the visual relationship classification subtask.

[0136]

[0137] Table 2. Experimental results of the method of the present invention on the scene graph detection subtask.

[0138]

[0139]

[0140] ※ The unlabeled values ​​indicate that the model was trained for 60 epochs, while the unlabeled values ​​indicate that the model was trained for 12 epochs.

[0141] This embodiment compared several existing panoramic scene image generation models, and the experimental results are shown in Tables 1 and 2. This embodiment used three baseline models: Motifs, VCTree, and Transformer. The results show that the present invention significantly outperforms the baseline models in terms of Mean metrics and exhibits good accuracy in relation detection. Furthermore, the present invention achieves consistently good results across multiple sub-tasks, and its model is independent, improving the performance of any baseline model and demonstrating good generalization ability. The panoramic scene graph generation models used for comparison are as follows: IMP is an image scene graph generation model, derived from existing technical literature: Xu D, Zhu Y, Choy CB, et al. Scene graph generation by iterative message passing[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2017:5410-5419; GPSNet is an image scene graph generation model, derived from existing technical literature: Lin X, Ding C, Zeng J, et al. Gps-net: Graph property sensing network for scene graph generation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2020:3746-3753; C-SGG is an unbiased image scene graph generation model, derived from existing technical literature: Jin T, Guo F, Meng Q, et al. Fast contextual scene graph generation with unbiased context augmentation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023:6302-6311; Motifs are image scene graph generation models, which are derived from existing technical literature: Zellers R, Yatskar M, Thomson S, et al.Neural motifs: Scene graph parsing with global context[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2018:5831-5840; BGNN is a two-stage unbiased scene graph generation model, which is derived from existing technical literature: Li R, Zhang S, Wan B, et al. Bipartite graph network with adaptive message passing for unbiased scene graph generation[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2021:11109-11119; GCL is a two-stage unbiased scene graph generation model, which is derived from existing technical literature: Dong X, Gan T, Song X, et al. Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 19427-19436; PSGTR is a single-stage panoramic scene graph generation model, which is derived from existing technical literature: Yang J, Ang YZ, Guo Z, et al. Panoptic scene graph generation [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 178-196; PSGFormer is a single-stage panoramic scene graph generation model, which is derived from existing technical literature: Yang J, Ang YZ, Guo Z, et al. Panoptic scene graph generation [C] / / European Conference on Computer Vision.Cham: Springer Nature Switzerland, 2022: 178-196; IETrans is an image scene graph generation model, derived from existing technical literature: Zhang A, Yao Y, Chen Q, et al. Fine-grained scene graph generation with data transfer[C] / / Europeanconference on computer vision. Cham: Springer Nature Switzerland, 2022: 409-424; ADTrans is a panoramic scene graph generation model, derived from existing technical literature: Li L, Ji W, Wu Y, et al. Panoptic scene graph generation with semantics-prototype learning[J]. arXiv preprint arXiv:2307.15567,2023.

[0142] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for generating panoramic scene images for curriculum learning based on shape-aware features, characterized in that, Includes the following steps: S1. Pre-acquire several training images, use the trained panoramic segmenter to identify objects in each training image, obtain the mask of each object, use the mask of each object to obtain the candidate box of each object, extract the bounding box features from the candidate box of each object, and extract the mask features and boundary features from the mask of each object. S2. Using a phased feature fusion strategy, bounding box features, mask features and boundary features are gradually fused to obtain object features and relationship features with different levels of complexity and corresponding to each stage. The object features are used to train the object classifier, and the relationship features are used to train the relationship classifier. S3. Based on the differences in cognitive learning difficulty, the predefined predicate categories contained in the training dataset are divided into several groups. Accordingly, the entire training process of the relation classifier is divided into several training stages. Each training stage uses a dedicated relation classifier to distinguish the predicates of its respective group. The predicate sampling strategy ensures the balance of the number of predicates within the group. The sampled training images are used as the final training images. S4. The course learning strategy is used on the final training images to gradually increase the complexity of the input features in each training stage, so that the relation classifier can learn predicates of different cognitive difficulty in order from easy to difficult, and integrate the knowledge distillation strategy to preserve the knowledge obtained in the early stage. S5. Input the test image to be processed into the trained panoramic segmenter, identify the objects in the test image, output the panoramic segmentation result of the test image, input the panoramic segmentation result into the relation classifier trained in the last training stage, classify the relationships between the detected objects, and output several triples representing the relationships between pairs of objects in the test image. The elements in the triples represent the subject, predicate, and object, respectively. The specific process of step S3 is as follows: S31. Sort all predicate categories in descending order of the number of training instances in the training dataset to obtain a sorted predicate set. Then, evenly distribute all predicates in the sorted predicate set into three different groups as the initial grouping. S32. Calculate the confusion matrix based on the pre-obtained baseline model prediction results. Each value in the confusion matrix Indicates the first The relationship category is labeled as the first... The number of samples for each relation category; For the current predicate in the initial group, determine whether the corresponding confusion matrix value exceeds the preset threshold: if it exceeds, the current predicate is regarded as a predicate with semantic confusion and adjusted based on semantic similarity; if it does not exceed, the current predicate pair is kept in the initial group until all predicates in the initial group have been traversed, resulting in three optimized predicate groups based on differences in cognitive difficulty. S33. Based on the optimized predicate grouping obtained in S32, the training process of the relation classifier is divided into three training stages, with each training stage using its own independent relation classifier. Except for the relation classifier in the first training stage, the relation classifiers in the remaining training stages all follow a continuously growing classification space. ; S34. Obtain the classification space Frequency of each predicate Using frequency of occurrence Construct an optimized upsampling frequency; obtain the classification space. Median value of the number of predicates and the maximum number of all predicates. Determine the classification space The k-th predicate Number of occurrences Is it greater than the median value? If it is greater than 0, then the first downsampling rate will be adjusted. Set as And the second downsampling rate Set as Otherwise, the first downsampling rate will be... Set to 1, and simultaneously set the second downsampling rate Set as ,in The square of the predicate number is used; the weighted sum of the first downsampling rate and the second downsampling rate is used as the optimized downsampling frequency; the training images containing tail predicates in the training dataset are upsampled according to the optimized upsampling frequency; the training images containing head predicates in the training dataset are downsampled according to the optimized downsampling frequency; and finally, the sampled training dataset is used as the final training set.

2. The method for generating panoramic scene images for course learning based on shape-aware features as described in claim 1, characterized in that, The specific process of step S1 is as follows: S11. For the input training images, the trained panoramic segmenter is used to identify objects in each training image. Each training image is segmented into a set of object masks. Each object in the training image corresponds to a set of masks. For each mask in the image, the mask for the i-th object is... Calculate each bounding box, and take the smallest bounding box corresponding to the i-th mask as the candidate box for the i-th object. ,in These represent the height and width of the training image, respectively; S12, For the candidate bounding box of the i-th object The RoIAlign operation is used to extract the bounding box features of the i-th object. ; S13, the mask for the i-th object First, a binary erosion operation is used to extract the contour of the i-th object. Then, Zernike moments are used to convert the contour of the i-th object into mask features. ; S14. For the subject object mask of the i-th object and object mask Calculate the first intersection region of the subject object mask and the object object mask. The extracted first intersection region is transformed into boundary features using binary erosion and Zernike moments. .

3. The method for generating panoramic scene images for course learning based on shape-aware features as described in claim 2, characterized in that, The mask features The function form is: ； The first intersection region The function form is: ； The boundary features The function form is: ； in, Represents Zernike moments; This indicates finding the intersection; Indicates negation; Represents the mask for the i-th object. Perform binary etch operation; This indicates that a binary erosion operation is performed on the first intersection region.

4. The method for generating panoramic scene images for course learning based on shape-aware features as described in claim 2, characterized in that, The specific process of step S2 is as follows: S21. The bounding box features of the i-th object... Candidate bounding boxes of the i-th object They are input together into the object context encoder trained in the first stage. In the process, the first stage of object characteristics is obtained. ; Classify the i-th object GloVe encoding Integrating object features from the first stage and bounding box features In the process, the first-stage fusion features are obtained, and these features are then input into the first-stage trained relation encoder. In the process, the first stage of improved object features are obtained. ; Improved object features from the first stage Obtain the object features of the subject in the first stage. The characteristics of the object with the verb Obtain subject and object candidate boxes from the object's candidate boxes, and calculate the union of the subject and object candidate boxes to obtain the features of the joint subject-object candidate box. ; The object characteristics of the subject in the first stage The characteristics of the object with the verb The features are concatenated to obtain the first-stage concatenated features. These features are then combined with the features of the subject-object pairs in the joint candidate boxes. Performing a dot product yields the first-stage subject-object relationship characteristics. ; S22. The bounding box features of the i-th object... Mask features and GloVe encoding and the characteristics of objects in the first stage They are input together into the relation encoder trained in the second stage. In the process, the object features improved in the second stage are obtained. ; Improved object features from the second stage Obtain the object features of the subject in the second stage. The characteristics of the object with the verb The object characteristics of the subject in the second stage The characteristics of the object with the verb The features are then concatenated to obtain the second-stage concatenated features. These features are then combined with the features of the subject-object pairs in the joint candidate boxes. Performing a dot product yields the relational characteristics of the subject-object pair in the second stage. ; S23. The bounding box features of the i-th object... Mask features and GloVe encoding Object characteristics in the first and second stages They are input together into the relation encoder trained in the third stage. In the process, the object features improved in the third stage are obtained. ; Improved object features from the third stage Obtaining the object features of the subject in the third stage The characteristics of the object with the verb The object characteristics of the subject in the third stage The characteristics of the object with the verb The parts are spliced ​​together to obtain the first feature after the third stage of splicing; Boundary features Features of the joint candidate box and its counterpart. By concatenating the first and second features from the third stage, we obtain the second feature after concatenation. Then, we perform a dot product on the first and second features from the third stage to obtain the subject-object relationship feature from the third stage. .

5. The method for generating panoramic scene images for course learning based on shape-aware features as described in claim 1, characterized in that, The specific rules for adjusting based on semantic similarity are as follows: For the first... Initial groups If the initial group If a predicate with confusing characteristics is detected, the predicate with lower frequency will be moved to the next position. Initial groups Alternatively, move the most frequently occurring predicates to the first position. Initial groups middle.

6. The method for generating panoramic scene images for course learning based on shape-aware features as described in claim 1, characterized in that, The classification space of the relation classifier is used to identify the predicate categories in the previous and current groups, and its functional form is: ； in, They represent the first one, , No. One initial group; This indicates finding the union of sets.

7. The method for generating panoramic scene maps for course learning based on shape-aware features as described in claim 1, characterized in that, The optimized upsampling frequency The function is in the following form: ； The optimized downsampling frequency The function is in the following form: ； in, , All represent hyperparameters; among them, This indicates the first downsampling rate biased towards the head predicate; This indicates the second downsampling rate, which is biased towards the tail predicate.

8. The method for generating panoramic scene maps for course learning based on shape-aware features as described in claim 1, characterized in that, The specific process for each training stage of the relation classifier is as follows: The relation features of the subject-object pairs obtained in S2 for each stage are... The input is fed into the relation classifier for each corresponding training stage. In this way, the predicted value of the relation distribution for each training stage can be obtained. Predicted values ​​based on the relational distribution at each training stage The relationship between the sampled and the true values A cross-entropy loss is constructed, and a KL divergence loss is constructed based on the predicted relation distribution values ​​of two different training stages. The weighted KL divergence loss and the cross-entropy loss are added together as the total loss. The parameters of the relation classifier in the first training stage are updated by minimizing the cross-entropy loss of the first training stage, the parameters of the relation classifier in the second training stage are updated by minimizing the total loss of the second training stage, and the parameters of the relation classifier in the last training stage are updated by minimizing the total loss of the third training stage. The training is iterated until a preset iteration threshold is reached, and the relation classifiers in each training stage converge, finally obtaining the relation classifiers trained in each training stage.

9. The method for generating panoramic scene maps for course learning based on shape-aware features as described in claim 1, characterized in that, The panoramic segmenter uses either the Panoptic Segformer model or the Panoptic FPN model.

Images