Scene graph generation method based on context modeling and fractal dimension calibration

Abstract

The invention belongs to the technical field of artificial intelligence and computer vision, and particularly relates to a scene graph generation method based on context modeling and fractal dimension calibration. According to the method, by introducing the triple space modeling and fractal dimension calibration mechanism, the recognition and recall capability of the tail class relation is improved, and therefore the problem that in the prior art, the long tail relation is difficult to recognize is solved. According to the method, local structure reasoning of spatial context modeling and triple feature fusion is introduced, then confidence coefficient calibration of fractal dimensions is carried out, and finally a scene graph structure is output, so that the recognition rate of the long-tail relationship can be effectively improved. Compared with a traditional scene graph generation method depending on convolutional neural network feature extraction, the method not only can better capture the dependency relationship between entities, but also can refine semantic reasoning by using geometric and local features, so that the recognition capability of the tail class relationship is obviously improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, specifically relating to a method for generating scene graphs based on context modeling and fractal dimension calibration. Background Technology

[0002] Scene Graph Generation (SGG) is an increasingly popular field in computer vision. Its goal is to depict objects and their semantic relationships in an image using structured graphs. The most significant highlight of this technique is its integration of object detection results with relational reasoning, extracting features and relationships, and fusing these inferences. This provides semantic-level technical support for subsequent tasks such as image captioning, visual question answering, cross-modal retrieval, and even autonomous driving perception.

[0003] In existing research, traditional methods mainly rely on Convolutional Neural Networks (CNNs) and Long Short-Term Memory Networks (LSTMs) to construct sequence models. A typical example is the MOTIFS model proposed by Zellers et al. (CVPR 2018), which uses bidirectional LSTMs to obtain contextual information and tightly integrates this information, making relationship predictions more coherent and the contextual logic sufficient. However, since these models are based on sequence structures, they do not adequately depict long-range dependencies between objects in complex scenes, and are prone to inference errors and overfitting in the presence of noise. Subsequently, the VTree model proposed by Tang et al. (CVPR 2019) introduced a tree structure to represent hierarchical dependencies, strengthening the dependency relationships. Although this alleviated the problems of noise errors and overfitting to some extent, inference computation still faces serious problems such as high computational cost and insufficient model stability when dealing with large amounts of data.

[0004] Existing methods still have significant limitations in modeling complex spatial relationships, identifying long-tail relationships, and computational efficiency. Summary of the Invention

[0005] To achieve the above objectives, this invention provides a method for generating scene graphs based on context modeling and fractal dimension calibration:

[0006] (1) This invention abandons the serialization path and instead introduces the Triplet3DModeling module, which displays the geometric features such as the relative position, size and angle between encoded entities, and combines them with local visual aggregation features. It uses the structure-aware attention mechanism to capture spatial context dependencies, thereby enhancing the model’s ability to represent complex spatial relationships.

[0007] (2) To further integrate multi-source features, this invention designs a triplet fusion modeling module to integrate entity semantic features, scene-level common features and geometric coding features, and constructs a triplet representation with structure awareness, providing a more discriminative feature foundation for relation prediction.

[0008] (3) This invention innovatively introduces a fractal dimension estimator and a fractal confidence calibration module to optimize long-tail distributions. At the same time, by estimating the fractal dimension of the relationship analogy, long-tail relationships are identified and fractal confidence calibration is performed based on the original output of the relationship prediction module, thereby improving the recognition recall rate of long-tail relationships.

[0009] (4) The entire process of this invention achieves efficient feature extraction and information transmission through various designed modules, which maintains efficient reasoning while avoiding excessive costs.

[0010] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0011] A scene graph generation method based on context modeling and fractal dimension calibration includes the following steps:

[0012] Step 1: Preprocess the input image to generate a normalized image tensor, then proceed to Step 2;

[0013] Step 2: Input the training images from the dataset into the candidate box generator to generate entity visual features and corresponding candidate box information, then proceed to Step 3;

[0014] Step 3: Use the backbone network to extract multi-scale deep feature maps of the image, further model them to generate refined instance features, and proceed to Step 4;

[0015] Step 4: Using the scene-level common feature extraction module, generate corresponding joint region features based on candidate box pairs, and proceed to Step 5; Step 5: Use triplet Relation3D modeling to model the spatial structural relationship between entity pairs, generate structure-enhanced triplet vectors, and proceed to Step 6;

[0016] Step 6: Use a geometric encoder to calculate the geometric feature deviation of the entity pair, then proceed to Step 7;

[0017] Step 7: Integrate features through the triplet fusion module to form a unified triplet feature representation, then proceed to Step 8;

[0018] Step 8: Input the triplet features into the relation prediction module to obtain the corresponding predicate probability distribution, and proceed to Step 9;

[0019] Step 9: Calculate the fractal dimension values ​​of various predicates in the dataset using the fractal dimension estimation module, and proceed to Step 10;

[0020] Step 10: Using the fractal dimension confidence calibration module, calibrate the predicate prediction probability distribution based on the fractal dimension value, and proceed to Step 11;

[0021] Step 11: Generate a visualization chart of the fractal dimension confidence calibration score from Step 10 using the confidence calibration visualization module, and proceed to Step 12;

[0022] Step 12: Output the calibrated scene graph structure, and the process ends.

[0023] Furthermore, as a preferred embodiment of the present invention, step 5, which involves modeling the spatial structural relationship between entity pairs using triplet Relation3D modeling, is as follows:

[0024] Step 5-1: Display the relative spatial positional relationship between encoded entity pairs by constructing a spatial geometric relationship vector, as shown in formula (1):

[0025]

[0026] Where Δg is a geometric relation quantity, Δx represents the coordinate difference between the center points of the subject and object bounding boxes on the horizontal (x) axis of the image, Δy represents the coordinate difference between the center points of the subject and object bounding boxes on the vertical (y) axis of the image, and ω o h o ω s h s These represent the width and height of the bounding boxes for the object and subject instances, respectively.

[0027] Step 5-2: Generate geometric attention bias by linearly transforming the geometric relation vector, as shown in formula (2):

[0028] φ(Δg)=W g ·Δg+b g (2)

[0029] Where φ(Δg) is the geometric attention bias obtained after linear transformation of the geometric relation vector, W g Let b be the learnable weight matrix for linear transformation. g This is a learnable bias term for the linear transformation.

[0030] As a further preferred embodiment of the present invention, step 7, in which the features are integrated through the triplet fusion module, is as follows:

[0031] Step 7-1: Transfer the main instance features F sObject instance features F0, joint region features F joint and geometric coding features F geo Then, the parts are assembled.

[0032] Step 7-2: Generate a unified triplet feature representation using a gating fusion mechanism, as shown in formulas (3) and (4):

[0033] g=σ(W g ·[F s ;F o ;F joint ;F geo ]+b g (3)

[0034] F triplet =g☉ReLU(W f [F s ;F o ;F joint ;F geo ]+b f (4)

[0035] Among them W g ,b g W f ,b f Here are the learnable parameters, σ represents the Sigmoid activation function, ⊙ represents element-wise multiplication, and F... s F o F represents the visual features of the subject and object instances, respectively. joint For scene-level common features, F geo These are geometric coding features.

[0036] Furthermore, as a preferred embodiment of the present invention, in step 8, the predicate classification step of the relationship prediction module is as follows:

[0037] Step 8-1: Let the triplet features f triplet Through a nonlinear transformation layer, the hidden feature representation h is obtained, as shown in formula (5):

[0038] h = ReLU(W) f ·f triplet +b1) (5)

[0039] Among them W f b1 and b1 are learnable parameters;

[0040] Step 8-2: Pass the hidden feature representation h through a linear transformation layer to a Softax classifier to obtain the predicted probability distribution of the predicate, as shown in formula (6):

[0041] P(r|f triplet=Softmax(W2·h+b2) (6)

[0042] Where P(r|f) triplet ) represents the conditional probability distribution, and W2 and b2 are learnable parameters.

[0043] Furthermore, as a preferred embodiment of the present invention, step 9, which involves using the fractal dimension estimation module to calculate the fractal dimension values ​​of various predicates in the VG dataset, is as follows:

[0044] Step 9-1: Based on the requirement of evaluating the complexity of the predicate space distribution, the predicate space distribution features in the VG dataset are used as input parameters for calculating the fractal dimension. The processing procedure is as follows:

[0045] For each predicate category, firstly, the spatial coordinates of its entity pairs in all training images are collected to form a two-dimensional spatial distribution point set. Secondly, the spatial point set is covered by grids of different sizes, and the minimum number of grids under each scale coverage is counted. At the same time, a set of decreasing grid scales needs to be selected to ensure that the scale range can capture the distribution features, thereby improving the accuracy of the fractal dimension value.

[0046] Step 9-2: Define the fractal dimension calculation system for the input parameters in Step 9-1. The analysis of the spatial point set is divided into three key elements, mainly including grid scale, grid number, and fractal dimension. Among them, the grid scale is divided into large scale, medium scale, and small scale; the changing trend of the grid number is divided into rapid decrease, medium decrease, and slow decrease; the fractal dimension value is divided into three levels: high dimension, medium dimension, and low dimension; the measured fractal dimension value is mainly distributed in the range of 1.1 to 1.9, and the numerical range of the quantitative index of fractal dimension is defined within this range.

[0047] Step 9-3: Calculate the fractal dimension using box counting. Simultaneously, for the spatially distributed point set, estimate the fractal dimension using linear regression, as shown in formula (7):

[0048]

[0049] Where ∈ i For the i-th grid scale, N c (∈ i ) represents the minimum number of grid cells required to cover the point set, and m represents the number of scales. and This represents the average of the corresponding values;

[0050] Step 9-4: Perform statistical analysis on the spatial distribution of all predicate categories in the VisualGenome dataset, and define a fractal dimension threshold to distinguish predicate complexity. First, use box counting to calculate the fractal dimension value D for each predicate category.c Secondly, the overall distribution of the values ​​was analyzed; the analysis results show that D c The distribution of values ​​exhibits a clear trimodal characteristic; therefore, the following rule is defined:

[0051] If the fractal dimension value is greater than 1.7, the predicate category is marked as high complexity;

[0052] If the fractal dimension value is between 1.3 and 1.7, the predicate category is labeled as medium complexity;

[0053] If the fractal dimension value is less than 1.3, the predicate category is marked as low complexity;

[0054] Based on the size of the fractal dimension value, the predicate category is divided into three spatial distribution patterns: Rule 1 indicates an extremely complex distribution, belonging to the difficult samples in the long-tail distribution; Rule 2 indicates a moderately complex distribution; and Rule 3 indicates a simple distribution, belonging to the high-frequency category.

[0055] Step 9-5: The fractal dimension estimation module outputs the fractal dimension value for each predicate category, which serves as the basis for subsequent confidence calibration. The specific calculation process of the fractal dimension value is shown in formula (8):

[0056]

[0057] Where D c N represents the fractal dimension value of the predicate. c (∈) represents the minimum number of grids with side length ∈ required to cover all spatial points of this type of predicate.

[0058] Furthermore, as a preferred embodiment of the present invention, in step 10, the step of calibrating the predicted probability distribution through the fractal dimension confidence calibration module is as follows:

[0059] Step 10-1: Based on the fractal dimension value D of the predicate category c Calculate its importance weight w c As shown in formula (9):

[0060] w c =1+γ·(D c -μ) (9)

[0061] Where γ is the scaling factor and μ is the mean of the fractal dimension or the target offset, both of which are adjustable hyperparameters;

[0062] Step 10-2: Use the weights to directly scale the original logic values ​​to obtain the calibrated probability distribution. As in formula (10):

[0063]

[0064] Where w c z represents the importance weight of predicate category c. c w represents the raw logical value (score before Softmax) output by the relation prediction module for predicate category c. j z is the original logical value. j represents the importance weights for the fractal dimension.

[0065] A testing method for testing the output of a trained model in a scene graph generation method based on context modeling and fractal dimension calibration includes the following steps:

[0066] Step a: Input the test image into the trained model above to predict the relationship, then proceed to step b;

[0067] Step b: Compare the relation prediction results with the true predicate labels, evaluate the overall performance using standard evaluation metrics, and proceed to step c;

[0068] Step c: Filter the long-tail relationships in the prediction results based on the fractal dimension value, generate a salient scene map, and proceed to step d;

[0069] Step d: Integrate overall and long-tail performance metrics and visualization results to form a complete test report.

[0070] Furthermore, as a preferred technical solution of the present invention, the test images in step a are selected from the official test set partitioning widely used in the Visual Genome dataset. This partitioning method is consistent with the traditional research partitioning to ensure the comparability of experimental results.

[0071] As a preferred embodiment of the present invention, in step b, the relation prediction results are compared with the true predicate labels. This invention adopts widely accepted evaluation criteria in the current field of scene graph generation, namely Recall@K (R@K) and mean Recall@K (mR@K), to accurately evaluate the model's test performance. The larger the values ​​of these two evaluation metrics, the better the test results, indicating higher robustness of the model in the face of diverse data and complex scenarios, and better model performance. The Recall@K metric measures the model's comprehensive ability to correctly identify predicate categories in its first K predicted relations. Specifically, this invention uses R@20, R@50, and R@100 metrics for evaluation; mean Recall@K represents the model's average recall performance for each predicate category. This metric effectively reflects the model's recognition effect and generalization ability for long-tail relations. A higher mR@K value indicates a more balanced discrimination of different relation categories and a stronger recall ability for tail-type relations. Specifically, the present invention uses mR@20, mR@50, and mR@100 as evaluation indicators.

[0072] As a further preferred technical solution of the present invention, in step c, the predicate categories corresponding to all predicted relations are first sorted according to the calculated fractal dimension value, and then long-tail relations with fractal dimension values ​​greater than 1.5 are selected. These relations and their associated entity nodes are combined to form the final output salient scene graph.

[0073] The scene graph generation method based on context modeling and fractal dimension calibration described in this invention has the following technical advantages compared with existing technologies:

[0074] This invention improves the recognition rate of long-tail relationships by introducing local structural reasoning through spatial context modeling and triplet feature fusion, followed by fractal dimension confidence calibration, and finally outputting the scene graph structure. Compared to traditional scene graph generation methods that rely on convolutional neural network feature extraction, this invention not only better captures the dependencies between entities but also refines semantic reasoning using geometric and local features, thus significantly improving the recognition ability of tail-class relationships. Experimental results show that, on the same dataset, this method significantly improves the recall rate of tail-class relationships and also improves the overall relationship prediction accuracy. By introducing fractal dimension calibration, the model's prediction confidence for tail-class relationships is significantly enhanced, effectively alleviating the problem of excessive proportion of head-class relationships in the prediction results, making the generated scene graph structure more complete and reliable. Attached Figure Description

[0075] Figure 1 Generate a comparative illustration for the scene graph;

[0076] Figure 2 A framework diagram for generating scene graphs by modeling triple predicates in existing methods;

[0077] Figure 3 This is a network framework diagram for scene graph generation based on the fusion of context modeling and fractal dimension calibration in this invention.

[0078] Figure 4 This is a flowchart illustrating the training process of the scene graph generation method based on context modeling and fractal dimension calibration of the present invention. Detailed Implementation

[0079] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0080] To achieve the aforementioned objectives, this invention proposes a scene graph generation method based on context modeling and fractal dimension calibration. The overall process includes twelve steps: input image preprocessing, candidate box generation, backbone network feature extraction, scene-level common feature extraction, triplet structure modeling, context encoding, geometric encoding, triplet feature fusion, relation prediction, fractal dimension estimation, fractal calibration processing, confidence visualization, and final scene graph generation. These steps work collaboratively to alleviate the difficulty of relation recognition under long-tailed distributions in existing methods.

[0081] like Figure 1 As shown, scene graphs are generated from the same image. The upper right side shows the scene graph generated by the traditional scene graph generation method, where the predicted relationships are usually head predicates such as "on" and "has". The lower right side shows the scene graph generated by the scene graph generation method based on context modeling and fractal dimension calibration, which can accurately identify tail predicates such as "wearing" and "riding".

[0082] like Figure 2 The diagram shows the framework for generating scene graphs using triple predicate modeling proposed by existing methods.

[0083] like Figure 3 As shown, a scene graph generation method based on context modeling and fractal dimension calibration includes the following steps:

[0084] Step 1: Preprocess the input image to generate a normalized image tensor, then proceed to Step 2;

[0085] Step 2: Input the training images from the dataset into the candidate box generator to generate entity visual features and corresponding candidate box information, then proceed to Step 3;

[0086] Step 3: Use the backbone network to extract multi-scale deep feature maps of the image, further model them to generate refined instance features, and proceed to Step 4;

[0087] Step 4: Using the scene-level common feature extraction module, generate corresponding joint region features based on candidate box pairs, and proceed to Step 5; Step 5: Use triplet Relation3D modeling to model the spatial structural relationship between entity pairs, generate structure-enhanced triplet vectors, and proceed to Step 6;

[0088] Step 6: Use a geometric encoder to calculate the geometric feature deviation of the entity pair, then proceed to Step 7;

[0089] Step 7: Integrate features through the triplet fusion module to form a unified triplet feature representation, then proceed to Step 8;

[0090] Step 8: Input the triplet features into the relation prediction module to obtain the corresponding predicate probability distribution, and proceed to Step 9;

[0091] Step 9: Calculate the fractal dimension values ​​of various predicates in the dataset using the fractal dimension estimation module, and proceed to Step 10;

[0092] Step 10: Using the fractal dimension confidence calibration module, calibrate the predicate prediction probability distribution based on the fractal dimension value, and proceed to Step 11;

[0093] Step 11: Generate a visualization chart of the fractal dimension confidence calibration score from Step 10 using the confidence calibration visualization module, and proceed to Step 12;

[0094] Step 12: Output the calibrated scene graph structure, and the process ends.

[0095] The steps for modeling the spatial structural relationship between entity pairs using triplet Relation3D modeling in step 5 are as follows:

[0096] Step 5-1: Construct a spatial geometric relationship vector to show the relative spatial relationships between coded entity pairs, as shown in Formula 1:

[0097]

[0098] Where Δg is a geometric relation quantity, Δx represents the coordinate difference between the center points of the subject and object bounding boxes on the horizontal (x) axis of the image, Δy represents the coordinate difference between the center points of the subject and object bounding boxes on the vertical (y) axis of the image, and ω o h o ω s h s These represent the width and height of the bounding boxes for the object and subject instances, respectively.

[0099] Step 5-2: Generate geometric attention bias by linearly transforming the geometric relation vector, as shown in Equation 2:

[0100] φ(Δg)=W g ·Δg+b g (2)

[0101] Where φ(Δg) is the geometric attention bias obtained after linear transformation of the geometric relation vector, W g Let b be the learnable weight matrix for linear transformation. g This is a learnable bias term for the linear transformation.

[0102] The steps in step 7 for integrating features using the triplet fusion module are as follows:

[0103] Step 7-1: Transfer the main instance features F sObject instance features F0, joint region features F joint and geometric coding features F geo Then, the parts are assembled.

[0104] Step 7-2: Generate a unified triplet feature representation using a gating fusion mechanism, as shown in formulas 3 and 4:

[0105] g=σ(W g ·[F s ;F o ;F joint ;F geo ]+b g (3)

[0106] F triplet =g☉ReLU(W f ·[F s ;F o ;F joint ;F geo ]+b f (4)

[0107] Among them W g b g W f ,b f Here are the learnable parameters, σ represents the Sigmoid activation function, ⊙ represents element-wise multiplication, and F... s F o F represents the visual features of the subject and object instances, respectively. joint For scene-level common features, F geo These are geometric coding features.

[0108] The predicate classification steps in step 8 using the relation prediction module are as follows:

[0109] Step 8-1: Extract the triplet features f triplet The hidden feature representation h is obtained through a nonlinear transformation layer, as shown in Equation 5:

[0110] h = ReLU(W) f ·f triplet +b1) (5)

[0111] Among them W f b1 and b1 are learnable parameters;

[0112] Step 8-2: Pass the hidden feature representation h through a linear transformation layer and then connect it to a Softmax classifier to obtain the predicted probability distribution of the predicate as shown in Formula 6:

[0113] P(r|f triplet =Softmax(W2·h+b2) (6)

[0114] Where P(r|f) triplet ) represents the conditional probability distribution, and W2 and b2 are learnable parameters.

[0115] The steps in step 9 to calculate the fractal dimension values ​​of various predicates in the VG dataset using the fractal dimension estimation module are as follows:

[0116] Step 9-1: Based on the requirement of evaluating the complexity of the predicate space distribution, the predicate space distribution features in the VG dataset are used as input parameters for calculating the fractal dimension. The processing procedure is as follows:

[0117] For each predicate category, firstly, the spatial coordinates of its entity pairs in all training images are collected to form a two-dimensional spatial distribution point set. Secondly, the spatial point set is covered by grids of different sizes, and the minimum number of grids under each scale coverage is counted. At the same time, a set of decreasing grid scales needs to be selected to ensure that the scale range can capture the distribution features, thereby improving the accuracy of the fractal dimension value.

[0118] Step 9-2: Define the fractal dimension calculation system for the input parameters in Step 9-1. The analysis of the spatial point set consists of three key elements, mainly including grid scale, grid number, and fractal dimension. The grid scale can be divided into large-scale, medium-scale, and small-scale; the trend of the number of grids can be divided into rapid decrease, medium decrease, and slow decrease; the fractal dimension value can be divided into three levels: high-dimensional, medium-dimensional, and low-dimensional. The invention found that the measured fractal dimension values ​​are mainly distributed in the range of 1.1 to 1.9. Therefore, the numerical range of the quantitative index of the fractal dimension is defined within this range.

[0119] Step 9-3: Calculate the fractal dimension using box counting. Simultaneously, for the spatially distributed point set, estimate the fractal dimension using linear regression, as shown in Formula 7:

[0120]

[0121] Where ∈ i For the i-th grid scale, N c (∈ i ) represents the minimum number of grids required to cover the point set, m is the number of scales, and log ∈ and This represents the average of the corresponding values.

[0122] Step 9-4: Perform statistical analysis on the spatial distribution of all predicate categories in the VisualGenome dataset. This invention defines a fractal dimension threshold for distinguishing predicate complexity. Specifically, firstly, the fractal dimension value D of each predicate category is calculated using box counting. c Secondly, the overall distribution of the values ​​was analyzed. The analysis results show that D cThe distribution of values ​​exhibits a clear trimodal characteristic; therefore, the following rule is defined:

[0123] (1) If the fractal dimension value is greater than 1.7, the predicate category is marked as "high complexity";

[0124] (2) If the fractal dimension value is between 1.3 and 1.7, the predicate category is marked as “medium complexity”;

[0125] (3) If the fractal dimension value is less than 1.3, the predicate category is marked as "low complexity".

[0126] Based on the fractal dimension value, predicate categories are divided into three spatial distribution patterns: Rule 1 indicates an extremely complex distribution, belonging to the difficult samples in the long-tail distribution; Rule 2 indicates a moderately complex distribution; and Rule 3 indicates a simple distribution, belonging to the high-frequency category. The fractal dimension estimation module in steps 9-5 calculates the specific value of each predicate category (Formula 8), which will be used for subsequent confidence calibration.

[0127] Step 9-5: The fractal dimension estimation module outputs the fractal dimension value for each predicate category, which serves as the basis for subsequent confidence calibration. The specific calculation process for the fractal dimension value is shown in Formula 8:

[0128]

[0129] Where D c N represents the fractal dimension value of the predicate. c (∈) represents the minimum number of grids with side length ∈ required to cover all spatial points of this type of predicate.

[0130] The steps in step 10 to calibrate the predicted probability distribution using the fractal dimension confidence calibration module are as follows:

[0131] Step 10-1: Based on the fractal dimension value D of the predicate category c Calculate its importance weight ω c For example, in formula 9:

[0132] w c =1+γ·(D c -μ) (9)

[0133] Where γ is the scaling factor and μ is the mean of the fractal dimension or the target offset, both of which are adjustable hyperparameters.

[0134] Step 10-2: Use the weights to directly scale the original logic values ​​to obtain the calibrated probability distribution. As in formula 10:

[0135]

[0136] Where w c z represents the importance weight of predicate category c. c w represents the raw logical value (score before Softmax) output by the relation prediction module for predicate category c. j z is the original logical value. j represents the importance weights for the fractal dimension.

[0137] The training flowchart of the scene graph generation method based on context modeling and fractal dimension calibration proposed in this invention is as follows: Figure 4 As shown, the batch size can be adjusted according to the computer's performance during the training process. The training process of this invention is described below: A testing method for testing the output trained model in a scene graph generation method based on context modeling and fractal dimension calibration includes the following steps:

[0138] Step a: Input the test image into the trained model above to predict the relationship, then proceed to step b;

[0139] Step b: Compare the relation prediction results with the true predicate labels, evaluate the overall performance using standard evaluation metrics, and proceed to step c;

[0140] Step c: Filter the long-tail relationships in the prediction results based on the fractal dimension value, generate a salient scene map, and proceed to step d;

[0141] Step d: Integrate overall and long-tail performance metrics and visualization results to form a complete test report.

[0142] The test images in step a are selected from the official test set partitioning widely used in the Visual Genome dataset. This partitioning method is consistent with the traditional research partitioning to ensure the comparability of experimental results.

[0143] In step b, this invention adopts widely accepted evaluation criteria in the current field of scene graph generation, namely Recall@K (R@K) and mean Recall@K (mR@K), to accurately evaluate the model's test performance. The larger the values ​​of these two evaluation metrics, the better the test results, indicating higher robustness of the model in the face of diverse data and complex scenarios, and better model performance. The Recall@K metric measures the model's comprehensive ability to correctly identify predicate categories in its first K predicted relations. Specifically, this invention uses R@20, R@50, and R@100 metrics for evaluation; mean Recall@K represents the model's average recall performance for each predicate category, effectively reflecting the model's recognition effect and generalization ability for long-tail relations. A higher mR@K value indicates a more balanced discrimination of different relation categories and stronger recall ability for tail-class relations. Specifically, this invention uses mR@20, mR@50, and mR@100 metrics for evaluation.

[0144] In step c, the predicate categories corresponding to all predicted relations are first sorted according to the fractal dimension values ​​calculated according to claim 2. Then, long-tail relations with fractal dimension values ​​greater than 1.5 are selected, and these relations and their associated entity nodes are combined to form the final output salient scene graph.

[0145] In practice, this invention uses the Ubuntu 20.04 operating system and a PyTorch 1.12 + CUDA 11.6 experimental environment. The model training batch size is set to 8, and the alternation update frequencies of the Transformer stream and the GNN stream are set to every 3 and 7 iterations, respectively. The model is trained using the SGD optimizer for 50K iterations, with an initial learning rate of 10. -3 All experiments were conducted on servers equipped with NVIDIA RTX 4090 GPUs. Training and testing data were derived from the VG (Visual Genome) dataset, which contains 108,077 images covering 150 instance categories and 50 predicate categories. In this embodiment, 60% of the images were used as the training set, 20% as the validation set, and 20% as the test set to ensure the objectivity of the model evaluation.

[0146] This invention employs Faster R-CNN to accurately detect instances and their location information in images. This architecture has been widely applied and matured in the field of object detection, accurately locating and classifying objects, while providing valuable experience and a solid foundation for numerous subsequent object detection models. Furthermore, it has been widely used to solve various practical problems, such as visual processing in autonomous driving systems and dynamic target tracking in intelligent security, demonstrating its broad applicability in modern technology.

[0147] Example 1:

[0148] This invention underwent comprehensive performance evaluation on the Visual Genome standard dataset, validating the model's effectiveness through three progressive tasks: predicate classification, scene graph classification, and scene graph detection. The evaluation employed a dual metric system: Recall@K (R@K) and meanRecall@K (mR@K). R@K measures the proportion of correct predicates among the top K predictions, meanRecall@K is the average Recall@K for each relation predicate class, and mR@K calculates the average Recall@K for each class to eliminate long-tail distribution bias. Specifically, this invention used R@20, R@50, R@100, and mR@20, mR@50, mR@100 metrics for evaluation. The comparison results between this invention's method and existing solutions on the predicate classification task are shown in Table 1. The comparison results on the scene graph detection task are shown in Table 2. The comparison results on the scene graph classification task are shown in Table 3. All three sub-tasks tested in this invention were conducted on the VG dataset.

[0149] Table 1. Comparison of the method of the present invention and existing solutions on the predicate classification task.

[0150]

[0151] Table 2 Comparison of the method of the present invention and existing solutions in scene graph detection tasks.

[0152]

[0153] Table 3 Comparison of the method of the present invention and existing solutions in scene graph classification tasks.

[0154]

[0155] Based on the results in the table above, the experimental results show that the proposed fusion-based scene graph generation method exhibits superior performance in all three sub-tasks: predicate classification, scene graph classification, and scene graph detection. In the predicate classification task, the R@20, R@50, and R@100 metrics are improved by approximately 1.8%–2.3% compared to the existing PE-Net method, while the mR@20, mR@5, and mR@100 metrics are also improved by approximately 0.9%–1.2%, indicating that this method can more comprehensively improve the prediction ability for different relation categories. In the scene graph classification task, the proposed method improves the Recall@20, Recall@50, and Recall@100 metrics by approximately 0.6%–1.0%, while the mean Recall@20, mean Recall@50, and mean Recall@100 metrics are also improved by approximately 0.3%–0.6% compared to the baseline model, verifying the model's advantages in capturing global semantic consistency and local contextual information. In scene graph detection tasks, the method of this invention improves both Recall@K and mean Recall@K metrics. Specifically, the improvement in Recall@20, Recall@50, and Recall@100 is approximately 0.5% to 0.9%, while the improvement in mean Recall@20, mean Recall@50, and mean Recall@100 is approximately 0.2% to 0.4%.

[0156] In summary, the method of this invention outperforms the most advanced comparative methods in all three core sub-tasks, demonstrating its superior performance in improving long-tail relationship recall and spatial semantic understanding.

[0157] Example 2:

[0158] This embodiment will introduce an applicable scenario of the present invention in the field of autonomous driving.

[0159] In the environmental perception system of autonomous vehicles, the user vehicle needs to identify and understand various targets and their spatial relationships in traffic scenes in real time (such as "vehicle-entering-current lane", "pedestrian-crossing-road"), and make safe and reasonable decisions quickly. Traditional perception algorithms are not accurate enough in semantic representation of complex spatial relationships such as "entering" and "crossing", and their recognition performance is poor for rare but high-risk long-tail scenarios (such as "child-chasing-ball-arriving-roadside"). The method proposed in this invention can solve this problem.

[0160] First, the system collects images and spatial information of the surrounding environment through multi-source sensors such as vehicle-mounted cameras and LiDAR, and uses a Faster R-CNN detector to identify the main target objects in the scene (such as "vehicles", "pedestrians", "traffic lights", "cones" etc.), while extracting their appearance features and geometric position information.

[0161] Next, the core module of this invention begins to operate. The triplet Relation3D modeling module models the spatial structural relationships between different entities and encodes their geometric deviations (including relative distance, direction angle, and scale ratio), thereby accurately distinguishing scenarios that are semantically similar but have significantly different real-world relationships, such as "parallel" and "driving in," or "stationary at" and "passing through." Simultaneously, the fractal dimension estimation module quantifies the complexity of predicate relationships in the training data, enabling the identification of long-tail relationship categories with high spatial complexity, such as "chasing," "running," and "intersecting."

[0162] Then, the fractal confidence calibration module adjusts the prediction score of high-risk relationships based on the fractal dimension estimation results. If the corresponding fractal dimension is high, the module will automatically increase the confidence of the "child-chasing-ball" relationship, enabling the vehicle system to identify potential risks in advance and issue a deceleration command.

[0163] Ultimately, the system generates a structured scene graph, clearly marking key semantic relationships such as "ball - rolling towards - center of the road" and "child - chasing - ball." The planning and control module makes real-time decisions based on this scene graph, thereby improving the vehicle's safety and ability to cope in complex traffic environments. Through the technical solution proposed in this invention, the autonomous driving system can achieve more accurate and intelligent semantic understanding and safety decisions when facing multi-objective, dynamic, and long-tailed scenarios.

[0164] Example 3:

[0165] This embodiment will introduce an application scenario of the present invention in the field of intelligent security.

[0166] In large transportation hubs such as airports and train stations, surveillance footage often shows multiple types of targets and their complex interactions, such as "passengers - left-behind - packages" and "crowds - gathering - entrances / exits." Traditional security algorithms, when dealing with such relationships, often only focus on target detection results, lacking fine-grained modeling of the spatial semantic relationships between targets. This is especially true for low-frequency, long-tail relationships like "left-behind" and "gathering," where accuracy is low, easily leading to missed detections or false alarms. The method proposed in this invention can solve this problem.

[0167] First, the system acquires real-time video streams from security surveillance cameras and extracts keyframes for preprocessing. Preprocessing includes image resizing, brightness correction, and noise removal. Then, Faster R-CNN is used to detect entity targets (such as "person," "suitcase," and "trolley") in the keyframes, outputting corresponding visual features and bounding box coordinates, providing an input basis for subsequent relationship modeling.

[0168] Next, the triplet spatial modeling module of this invention performs feature association on the detected targets and explicitly encodes the geometric information between targets, including relative position, size ratio, and direction of movement, through the Relation3D structure. At the same time, the fractal dimension estimation module calculates the distribution complexity values ​​of various relation predicates based on historical monitoring data, and uses this to distinguish between common relations and high-complexity long-tail relations.

[0169] Then, the fractal confidence calibration module performs a secondary calibration on the relationship prediction results based on the aforementioned dimensional indicators. For relationships belonging to the long-tail category (such as "legacy" or "clustering"), the system dynamically increases their confidence weight during the output stage, thereby enhancing the detection sensitivity for rare but high-risk events. Simultaneously, the confidence visualization module generates confidence distribution maps and statistical reports for relationship identification, helping security personnel quickly assess the reliability and priority of the identification results.

[0170] Finally, the system outputs a calibrated scene map, clearly showing key risk relationships such as "unclaimed suitcase - left in - waiting area". This scene map will simultaneously generate structured alarm records for real-time early warning and subsequent event tracing by the security system. Through the method of this invention, the intelligent security system can accurately capture long-tail and abnormal behavior relationships in complex environments, achieving more efficient monitoring analysis and public safety prevention and control.

[0171] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for scene graph generation based on context modeling and fractal dimension calibration, characterized in that, The method comprises the following steps: Step 1: preprocessing the input image to generate a normalized image tensor, and entering step 2; Step 2: inputting the training image in the data set into the candidate box generator to generate entity visual features and corresponding candidate box information, and entering step 3; Step 3: using the backbone network to extract multi-scale deep feature maps of the image, and further modeling to generate refined instance features, and entering step 4; Step 4: using a scene-level public feature extraction module to generate corresponding joint region features based on the candidate box, and entering step 5; Step 5: using a triple Relation3D modeling to model the spatial structure relationship between the entity pairs, and generating structure-enhanced triple vector, and entering step 6; Step 6: using a geometric encoder to calculate the geometric feature deviation of the entity pairs, and entering step 7; Step 7: integrating the features through a triple fusion module to form a unified triple feature representation, and entering step 8; Step 8: inputting the triple feature into a relationship prediction module to obtain the corresponding predicate probability distribution, and entering step 9; Step 9: using a fractal dimension estimation module to calculate the fractal dimension values of each predicate in the data set, and entering step 10; Step 10: using a fractal dimension confidence calibration module to calibrate the predicate prediction probability distribution based on the fractal dimension values, and entering step 11; Step 11: using a confidence calibration visualization module to generate a visualization chart for the fractal dimension confidence calibration score in step 10, and entering step 12; Step 12: outputting the calibrated scene graph structure, and ending.

2. The method of claim 1, wherein, In step 5, the step of modeling the spatial structure relationship between the entity pairs using triple Relation3D modeling is as follows: Step 5-1: display the relative spatial position relationship between the encoded entity pairs by constructing a spatial geometric relationship vector, as shown in formula (1): wherein Ag is a geometric relationship quantity, Ax represents a coordinate difference value of the center points of the subject and object bounding boxes in the horizontal direction x axis of the image, Ay represents a coordinate difference value of the center points of the subject and object bounding boxes in the vertical direction y axis of the image, and o , h o , ω s , h s respectively represent the width and height of the object and subject instance bounding boxes. Step 5-2: generate a geometric attention bias by linearly transforming the geometric relationship vector, as shown in formula (2): φ(Δg) = W g • Δg + b g (2) where φ(Δg) is the geometric attention bias obtained by linearly transforming the geometric relationship vector, W g is a learnable weight matrix for the linear transformation, and b g is a learnable bias term for the linear transformation.

3. The method of claim 2, wherein, In step 7, the step of integrating the features through the triple fusion module is as follows: Step 7-1: The body instance feature F s , the object instance feature F0, the joint region feature F joint , and the geometry encoding feature F geo are stitched together. Step 7-2: generate a unified triple feature representation using a gated fusion mechanism, as shown in formulas (3) and (4): g = σ(W g • [F s ; F o ; F joint ; F geo ] + b g ) (3) F triplet = g ReLU(W f [F s ; F o ; F joint ; F geo + b f (4) where W g , b g , W f , b f are learnable parameters, σ denotes a sigmoid activation function, ⊙ denotes element-wise multiplication, F s , F o denote visual features of the subject and object instances, respectively, F joint denotes a scene-level common feature, and F geo denotes a geometry encoding feature.

4. The method of claim 3, wherein, In step 8, the step of predicate classification by the relationship prediction module is as follows: Step 8-1: Let the triple feature f triplet The hidden feature representation h is obtained by a nonlinear transformation layer as in equation (5): h = ReLU(W f .f triplet + b1) (5) where W f and b1are learnable parameters; Step 8-2: pass the hidden feature representation h through a linear transformation layer and a Softmax classifier to obtain the prediction probability distribution of the predicate, as shown in formula (6): P(r|f triplet ) = Softmax(W2 · h + b2) (6) where P(r|f triplet ) is the conditional probability distribution, and W2 and b2 are learnable parameters.

5. The method of claim 4, wherein, In step 9, the step of calculating the fractal dimension values of each predicate in the VG data set using the fractal dimension estimation module is as follows: Step 9-1: according to the evaluation requirements of the predicate spatial distribution complexity, the spatial distribution features of the predicates in the VG data set are used as input parameters for fractal dimension calculation, and the processing process is as follows: For each predicate category, first, collect the spatial position coordinates of the entity pairs in all training images to form a two-dimensional spatial distribution point set, and second, use different sizes of grids to cover the spatial point set, and count the minimum grid number under each scale coverage, and at the same time, a set of decreasing grid scales need to be selected to ensure that the scale range can capture the distribution features, thereby improving the accuracy of the fractal dimension value; Step 9-2: Define the fractal dimension calculation system for the input parameters in step 9-1. The analysis of the spatial point set is divided into three key elements, mainly including grid scale, grid number and fractal dimension; Among them, the grid scale is divided into large scale, medium scale and small scale; the trend of grid number change is divided into fast reduction, medium reduction and slow reduction; the fractal dimension value is divided into high dimension, medium dimension and low dimension; the fractal dimension value is mainly distributed in the interval of 1.1 to 1.9, and the quantization index value range of fractal dimension is limited in this interval; Step 9-3: Calculate the fractal dimension value by box counting method, and estimate the fractal dimension of the spatial distribution point set by linear regression method, as formula (7): where ∈ i is the ith grid scale, N c (∈ i ) is the minimum number of grids required to cover the point set, m is the number of scales, and are the mean values of the respective numbers. Step 9-4: Statistical analysis of the spatial distribution of all predicate categories in the Visual Genome dataset, defining a fractal dimension threshold for distinguishing predicate complexity, first using box counting method to calculate the fractal dimension value D of each predicate category c Secondly, the overall distribution of values is analyzed; the analysis results show that the distribution of D c values presents obvious three-peak characteristics, so the following rules are defined: If the fractal dimension value is greater than 1.7, the predicate category is marked as high complexity; If the fractal dimension value is between 1.3 and 1.7, the predicate category is marked as medium complexity; If the fractal dimension value is less than 1.3, the predicate category is marked as low complexity; According to the size of fractal dimension value, the predicate category is divided into three spatial distribution modes: rule 1 represents extremely complex distribution, which belongs to difficult samples in long tail distribution; rule 2 represents moderate complexity of distribution; rule 3 indicates simple distribution, which belongs to high frequency category; Step 9-5: The fractal dimension estimation module outputs the fractal dimension value of each predicate category as the basis for subsequent confidence calibration, and the specific calculation process of fractal dimension value is as formula (8): where D c represents the fractal dimension value of the predicate, N c (∈) represents the minimum number of grids with side length ∈ that covers all spatial points of this class of predicates.

6. The method of claim 5, wherein, In step 10, the fractal dimension confidence calibration module calibrates the prediction probability distribution as follows: Step 10-1: Calculate the fractal dimension value D according to the predicate category c , and calculate its importance weight w c as formula (9): w c = 1 + γ · (D c - μ) (9) Where γ is the scaling factor, μ is the fractal dimension mean or target offset, both of which are adjustable hyperparameters; Step 10-2: Use the weights to directly scale the original logical values to get the calibrated probability distribution As equation (10): where w c is the importance weight of the predicate class c, z c is the raw logistic value before Softmax output by the relation prediction module for the predicate class c, w j is the raw logistic value, z j is the importance weight of the fractal dimension.

7. A test method for testing the trained model output by the context modeling and fractal dimension calibration based scene graph generation method of any one of claims 1-6, comprising the following steps: Step a: input the test image into the trained model to predict the relationship, and enter step b; Step b: compare the relationship prediction result with the real predicate label, evaluate the overall performance using standard evaluation indicators, and enter step c; Step c: filter the long tail relationship in the prediction result based on the fractal dimension value, generate a significant scene graph, and enter step d; Step d: integrate the overall and long tail performance indicators and the visualization results to form a complete test report.

8. The test method of claim 7, wherein, The test image in step a is selected from the official test set widely used in VisualGenome dataset, which is consistent with the traditional research division to ensure the comparability of the experimental results.

9. The test method of claim 7, wherein, In step b, the relationship prediction result is compared with the real predicate label, and the evaluation standards widely recognized in the current scene graph generation field, namely Recall@K(R@K) and meanRecall@K(mR@K), are adopted to accurately evaluate the test effectiveness of the model; the larger the values of the two evaluation indicators, the better the test result, which represents the higher robustness of the model in the face of diversified data and complex scenes, and the better the model performance; Wherein the Recall@K index measures the comprehensive ability of the model to correctly identify the predicate category in its first K predicted relationships, specifically, the R@20, R@50, R@100 indexes are used for evaluation; Wherein the mean Recall@K represents the average recall performance of the model for each predicate category, and the index can effectively reflect the identification effect and generalization ability of the model for long-tail relationships. The higher the mR@K value, the more balanced the model discriminates different relationship categories, and the stronger the recall ability for tail relationship. Specifically, the mR@20, mR@50, mR@100 indexes are used for evaluation.

10. The test method of claim 7, wherein, In step c, first, the predicate categories corresponding to all predicted relationships are sorted according to the calculated fractal dimension value, and then long-tail relationships with a fractal dimension value greater than 1.5 are screened out, and these relationships and the associated entity nodes are combined to form the final output significant scene graph.

Images