A method, system and apparatus for constructing an unbiased scene graph based on generative templates

Abstract

The application discloses a kind of methods, systems and equipment based on the construction method of unbiased scene graph of generative template, obtain the different images under rich scene of to be detected;Different images of rich scene are extracted using deep learning model context features;According to the context features of different images, the semantic information existing in different images is inferred through the Transformer block in deep learning model, the semantic information existing in different images includes the classification of target object and the confidence score between targets;Image triplet prediction results are output according to the confidence score height-low;The deep learning model is the visual context feature of image extracted by Transformer encoder-decoder network architecture, and is obtained after data enhancement, training is carried out to deep neural network.The application carries out a large number of experiments on public scene graph dataset VG, and can effectively construct scene graph in line with image content.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, image processing and natural language processing, and specifically relates to a method, system and device for constructing unbiased scene graphs based on generative templates. Background Technology

[0002] In recent years, image content understanding, as a core technology for bridging images and text, has attracted significant attention in the field of computer vision research. Reconstructing digitized images into textual knowledge graphs is the primary goal and crucial function of this visual task. Scene graph models have wide applications in various fields of intelligent systems, such as assisting intelligent transportation systems in analyzing road scenes and assisting intelligent medical systems in achieving accurate judgments. Furthermore, they can be combined with word embedding models to handle advanced upstream tasks, including visual question answering and image retrieval. Common scene graph generation methods are typically based on two-stage contextual feature encoding and decoding. Candidate object sequences are obtained through object detection, and then the most closely related semantic object triples are inferred using Transformer self-attention and cross-attention networks.

[0003] Early methods employed graph convolution-based message passing, updating a node by combining the receptive fields of neighboring nodes (see Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation[C]. in Proceedings of the Europeanconference on computer vision (ECCV), 2018: 670–685.). Subsequent methods mostly employed recurrent neural networks, such as LSTM, to extract visual context and achieve two-stage reasoning of objects and relationships. To extract deeper information, more powerful and flexible Bi-LSTM and Bi-TreeLSTM were proposed (see Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic treestructures for visual contexts[C]. in Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition, 2019: 6619–6628), which can encode bidirectionally and adapt to possible one-to-many object combinations under the current scene semantics. With improvements in logic theory, more complex models have emerged, such as hierarchical graphs, probabilistic graphs, and perceptual graphs. However, these models still have limitations in utilizing contextual information. The popularity of Transformer network architectures, with their powerful visual feature extraction capabilities, has enabled end-to-end models. Transformer-dominated frameworks aim to bypass object detection and inference through entity and predicate decoders, significantly improving speed and accuracy (see Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer[C]. in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022: 19486–19496). However, these methods still face challenges in adapting to imbalanced data distributions.

[0004] Existing technologies suffer from the following problems: For complex traffic and road scene images captured by cameras, the image content cannot be accurately processed, resulting in inaccurate reflections of real-time road conditions. For some challenging samples in medical imaging, such as those with intractable diseases, there are issues such as complex image content and a scarcity of usable samples, making it impossible to obtain accurate information from a limited number of samples and establish a reasonable diagnosis. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention aims to provide a more accurate method, system, and device for constructing unbiased scene graphs based on deep learning and data augmentation. This method can obtain high-quality scene graphs, thereby accurately representing image content information and enhancing the performance of intelligent systems and automated devices.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for constructing an unbiased scene graph based on generative templates includes the following steps:

[0008] Acquire different images from various scenes to be detected;

[0009] Deep learning models are used to extract contextual features from different images in rich scenes;

[0010] Based on the contextual features of different images, semantic information existing in different images is inferred through the Transformer block in the deep learning model. The semantic information existing in different images includes the classification of target objects and the confidence scores between targets.

[0011] Output the image triplet prediction results according to the confidence score;

[0012] The deep learning model is obtained by extracting visual context features of an image through a Transformer encoder-decoder network architecture, performing data augmentation on the classification of visual context features of the image, and then training a deep neural network.

[0013] Furthermore, the visual context features of the image are extracted through the Transformer encoder-decoder network architecture, including the following process: using ResNet-101 and FPN networks as backbone networks and Faster RCNN as detector, the training set is trained to obtain candidate object boxes, and the visual features of the candidate objects are obtained based on the candidate object boxes.

[0014] The visual features of the candidate targets are input into an encoder-decoder network based on Transformer blocks to obtain visual context features.

[0015] Furthermore, the visual context features of the image are calculated using the following formula:

[0016] (1)

[0017] in, Visual features representing candidate targets. This indicates the relative geometric position information between targets. This represents the word embedding features corresponding to the target. ，i This represents the category corresponding to the i-th sample. Represents the visual context features of an image.

[0018] Furthermore, deep learning models are obtained through the following process:

[0019] Based on the classification difficulty of the training set samples, all samples are divided into easy samples and hard samples, and the corresponding visual context feature matrices of easy samples and hard samples are obtained.

[0020] Based on the visual context feature matrices of simple and hard samples, distillation meta-learning is performed to obtain feature templates for identifying categories;

[0021] After supplementing the feature templates of the identifier categories with data, a deep neural network is trained to obtain a deep learning model.

[0022] Furthermore, through distillation meta-learning, after data augmentation of the feature templates for identifying categories, a deep neural network is trained to obtain a deep learning model, including the following steps:

[0023] The feature template for identifying categories includes simple and hard samples corresponding to the feature matrix;

[0024] A template indicator is constructed using the visual context feature matrix corresponding to simple samples. The template indicator consists of a multi-head self-attention layer and a multilayer perceptron, and the indicator parameters are updated through incremental learning.

[0025] Then, the indicator is fine-tuned using the corresponding difficult samples of the feature matrix;

[0026] The feature matrix is ​​updated using the fine-tuned indicator to obtain the feature template that identifies the category.

[0027] Furthermore, the feature templates for identifying categories are supplemented with data through the following process:

[0028] Reconstructed supplementary samples are obtained through MCMC sampling;

[0029] Compare the feature similarity between background class samples and feature templates of the label class in the training set, and take samples with feature similarity higher than the threshold as high-confidence samples;

[0030] The reconstructed supplementary samples and high-confidence samples are combined to form out-of-distribution sample data, which is a balanced dataset. The combined data of the training set and the balanced dataset is used to train the deep neural network to obtain a deep learning model.

[0031] Furthermore, out-of-distribution sample data as follows:

[0032] (4)

[0033] In the formula, As the category center, This is a deviation.

[0034] An unbiased scene graph construction system based on generative templates, comprising:

[0035] The image acquisition module is used to acquire different images from various scenes to be detected.

[0036] The feature extraction module is used to extract contextual features of different images in rich scenes using a deep learning model;

[0037] The semantic information inference module is used to infer the semantic information present in different images based on the contextual features of different images through the Transformer block in the deep learning model. The semantic information present in different images includes the classification of target objects and the confidence scores between targets.

[0038] The output module is used to output the image triplet prediction results according to the confidence score.

[0039] The deep learning model is obtained by extracting visual context features of an image through a Transformer encoder-decoder network architecture, performing data augmentation on the classification of visual context features of the image, and then training a deep neural network.

[0040] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the generative template-based unbiased scene graph construction method.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] This invention employs a novel Transformer encoder-decoder network architecture to extract visual contextual features from images, while effectively balancing data distribution and enabling secondary training of the classifier. When building the deep learning model, self-driven distillation meta-learning is first used for transfer learning, ensuring that difficult and easy samples have the same learning capacity, and extracting a unique feature template for each class. Furthermore, considering that uneven data distribution can impair the training effect of the neural network, this invention supplements the feature templates that identify the class with additional data.

[0043] Extensive experiments have confirmed that this invention is easy to train. The improved network achieves end-to-end functionality, shortening training time, improving model efficiency, and solving the challenge of imbalanced data. This invention can generate high-quality scene graphs in most scenarios and convert digital images captured by hardware devices into text information, thereby accurately conveying image content. With the assistance of scene graphs, analysis of road traffic conditions and the causes of medical images has significantly improved the accuracy of final judgments.

[0044] Furthermore, by applying MCMC sampling, reconstructed supplementary samples are obtained. That is, a central radiation strategy is adopted to compare the feature similarity between the background class samples and the feature templates of the label class in the training set, and samples with feature similarity higher than a threshold are regarded as high-confidence samples. That is, a threshold screening strategy is adopted. The reconstructed supplementary samples and the high-confidence samples are combined to form out-of-distribution sample data, thereby providing a rich source of supplementary data. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0046] Figure 2 This is a schematic diagram of the network structure of the method of the present invention;

[0047] Figure 3 This is a schematic diagram showing the distribution of samples from some categories after data balancing.

[0048] Figure 4 This invention generates visualization results of the scene graph model on the large public dataset Visual Genome test set. Figures (a) and (b) are training set images from random testing, and Figures (c) and (d) are the scene graphs obtained from Figures (a) and (b) through the deep learning model, respectively. These graphs are composed of multiple sets of <subject, predicate, object> triples, which accurately describe the image content information.

[0049] Figure 5 This is a schematic diagram of the unbiased scene graph construction system based on generative templates of the present invention. Detailed Implementation

[0050] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0051] See Figure 1 The process involves obtaining the scene images to be retrieved, labeling and preprocessing them, and obtaining the labeled dataset of the scene images to be retrieved; the labeled dataset of the scene images to be retrieved is then divided into a training set and a test set.

[0052] Further, a Transformer-based encoder-decoder network is used to extract visual context features from the images, aiming to build a template generation network and summarize category templates from these features. This method employs distillation-based learning for transfer learning, ensuring that fine-grained samples (difficult samples) and coarse-grained samples (easy samples) have the same learning ability. Simultaneously, a template application network is designed, using center radiation and threshold filtering to process category templates, constructing a large number of high-quality out-of-distribution supplementary samples for secondary training. These data augmentation techniques balance the sample distribution of the dataset, ensuring the accuracy of scene graph generation. The model is trained using stochastic gradient descent. By setting two loss functions, the comparative loss ensures the inter-class differences and intra-class similarities of the category templates, while the cross-entropy loss calculates the accuracy of the classification prediction results. Finally, the scene is output according to the confidence score. Figure 3 Tuple prediction results. The specific method is as follows:

[0053] Step 1: Obtain the scene images to be retrieved, perform annotation and preprocessing, and obtain the annotated dataset of the scene images to be retrieved; divide the annotated dataset of the scene images to be retrieved into training set and test set;

[0054] The specific process of annotation and preprocessing is as follows: Data cleaning is performed on a rich collection of various scene images to be retrieved, selecting samples with clear semantic relationships. The targets present in the images and the relationships between them are then labeled. Finally, an irregular digitized image is converted into a structured semantic graph with <subject-verb-object> as the basic unit, resulting in an annotated dataset of the scene images to be retrieved.

[0055] Step 2: First, input the training set images and obtain candidate bounding boxes using the object detection model. Second, design a Transformer-based encoder-decoder network to extract visual contextual features between object combinations, enhancing the visual contextual feature representation ability of all categories in the training set, thereby improving classification performance. Further, distillation-meta-learning is employed to inductively generate feature templates for different categories. Transfer learning is performed through distillation-meta-learning to ensure that difficult and easy samples have the same learning ability; the specific process is as follows:

[0056] Two-stage training is performed on the training set:

[0057] The first stage employs a highly mature object detection model. This invention uses ResNet-101 (a fast-training residual network) and FPN (Feature Pyramid Networks) as the backbone networks, and a pre-trained Faster RCNN as the detector. The model is trained on the training set to obtain a series of candidate bounding boxes. Based on these candidate bounding boxes, the visual features of the object detector are obtained.

[0058] The second stage involves detecting the correlation between candidate targets. For example... Figure 2 As shown, a template generation network was first designed. Essentially, it obtains the feature matrix of class i based on the visual features of the object detector through a Transformer block-based encoder-decoder network. The Transformer block-based encoder-decoder network includes a multi-head self-attention layer, a normalization layer, and a fully connected layer. The formula is as follows:

[0059] (1)

[0060] in Represents the visual features of the target detector. This indicates the relative geometric position information between targets. This represents the word embedding features corresponding to the target. ，i This represents the category corresponding to the i-th sample. This represents the feature matrix of category i, i.e., the visual context features of the image.

[0061] However, a key challenge remains: the data contains coarse-grained samples (simple samples) and fine-grained samples (difficult samples), and the model's classification performance differs significantly for these. Therefore, this invention is based on the feature matrix of category i. Based on the classification difficulty of the training set samples, all samples are divided into simple samples and difficult samples. Then, according to formula (1), the visual context feature matrices of the simple samples and difficult samples are obtained.

[0062] A self-driven distillation meta-learning strategy is further designed to achieve an average performance improvement across all categories. Specifically, a template indicator is constructed using coarse-grained samples corresponding to the feature matrix of category i. The template indicator consists of a multi-head self-attention layer and a multilayer perceptron, and the indicator parameters are updated through incremental learning. The learnable indicator can accurately distinguish most coarse-grained samples, thus gaining strong discriminative power from simple samples.

[0063] Then, the indicator is fine-tuned using the corresponding fine-grained samples of the feature matrix of category i, while simultaneously inheriting the category discrimination capability, thereby achieving an average performance improvement for all categories.

[0064] Update the feature matrix of category i using the fine-tuned indicator. The improved category center was obtained. and deviation Feature templates that uniquely identify categories The following formula is known to be used for calculation:

[0065] (2)

[0066] (3)

[0067] in These are the weighting coefficients. Let the class center be the value in the t-th iteration. Let be the deviation in the t-th iteration, where t is the number of iterations.

[0068] After the iteration is completed, the category center is obtained. and deviation .

[0069] Step 3: Based on the feature template of the unique identifier category The class of a sample can be determined by calculating the cosine similarity between a specific sample and its class center. However, this invention observes that the negative impact of significant data distribution disparities still exists. Therefore, it proposes a template application network based on template-constructed pseudo-labels to achieve data augmentation and secondary training. This network incorporates two strategies: center radiation calculates the improved class center. and deviation Through MCMC sampling, reconstructed supplementary samples are obtained. On the other hand, considering the large number of background class samples in the data, there may be instances with potential semantic information. Therefore, a threshold filtering method is designed to compare background class samples with foreground class templates (i.e., feature templates that uniquely identify the category) in the training set. Based on feature similarity, samples with feature similarity exceeding a threshold are designated as high-confidence samples. These high-confidence samples are then used as supplementary samples. The reconstructed supplementary samples, along with the high-confidence samples, form the out-of-distribution sample data. Out-of-distribution sample data To balance the dataset The training set and the balanced dataset The combined data is then used for secondary training of the classification head.

[0070] Out-of-distribution sample data as follows:

[0071] (4)

[0072] In the formula, As the category center, This is a deviation.

[0073] See Figure 3 The constructed balanced dataset The data distribution is more even, eliminating the negative impact of the long-tailed distribution of the original dataset.

[0074] Step 4: To evaluate the effectiveness of the model's predictions, this invention designs two effective losses to calculate the distance between the predicted and true values. Among them, a contrastive loss is designed to verify the quality of the templates obtained by the template generation network. Calculate the distances between positive and negative samples belonging to different categories, and between positive samples belonging to the same category. Contrast loss. This ensures inter-class diversity and intra-class similarity in template generation, improving the quality of template generation. Simultaneously, a cross-entropy loss function is designed. The final multi-class prediction result is calculated by taking the negative logarithm of the probability of the correct class component.

[0075] The augmented data samples generated in step 3 are used for secondary training, and the cross-entropy loss is calculated. The cross-entropy loss is added to the contrastive loss to obtain the multi-task loss function, which serves as a direct constraint on the model. The multi-task loss function is then applied to the balanced dataset. A deep neural network is trained on a training set consisting of the training set D and the training set D. The parameters of the deep neural network are optimized until the loss function of backpropagation tends to converge. Training is then stopped to obtain a deep learning model. The parameters for training the deep neural network are set as follows: the base learning rate is 0.004, the weight decay is 0.1, and the batch size is set to 12.

[0076] Step 5: Use a deep learning model to extract contextual features from different images in rich scenes in the test set. Then, use the transformer block in the deep learning model to infer the semantic information in the images, including the classification of target objects and the relationships between them. Finally, output the scene according to the confidence score. Figure 3 Tuple prediction results.

[0077] Figure 4 The visualization of the model's detection results shows that the predicted results are basically consistent with the semantics of the actual images.

[0078] The deep neural network is trained using backpropagation and stochastic gradient descent, with continuous updates and iterations. The model evaluation metric used is mean recall (mR).

[0079]

[0080] The following is a specific embodiment.

[0081] The neural network of this invention is implemented based on the PyTorch framework, and the workstation used for training is equipped with a 3090 GPU for computational acceleration. Experiments were conducted on the Visual Genome public scene image dataset published by Stanford University, which provides image datasets for various scenes, consisting of a total of 108K images and 2.3M relation annotations. The experiments used the most commonly used VG150 subset, which contains the 150 most common object classes and 50 predicate classes. Data cleaning was performed on some lower-quality classes with fewer samples to ensure the effectiveness of training. The training and test sets were further divided in a 7:3 ratio. Features of the relationships between image elements were extracted using the trained network and stored in a feature file. The trained classification head was used to classify the features, outputting the top-k triples with the highest confidence scores. Experimental data are shown in Table 1. Comparison with existing methods verifies the significant improvement in the generation accuracy of deep learning models by this invention.

[0082] Table 1. Recall Evaluation of Scene Graph Generation Model

[0083]

[0084]

[0085] Furthermore, there are differences in the difficulty of classification among different categories, so the average recall rate is a more objective evaluation indicator. The measured data is shown in Table 2.

[0086] Table 2 Recall Evaluation of Scene Graph Generation Model

[0087]

[0088]

[0089] In addition, the visualization results on the VG test set are as follows: Figure 4The first column represents the image to be tested, and the second column represents the scene image corresponding to the first column. Horizontal lines below represent incorrect predictions. The visualization and quantitative evaluation results demonstrate the effectiveness and robustness of the algorithm of this invention. The generated scene image clearly expresses the image content, assisting intelligent transportation, autonomous driving, and other systems in more accurately grasping real-time road conditions.

[0090] See Figure 5 The unbiased scene graph construction system based on generative templates of the present invention includes:

[0091] The image acquisition module is used to acquire different images from various scenes to be detected.

[0092] The feature extraction module is used to extract contextual features of different images in rich scenes using a deep learning model;

[0093] The semantic information inference module is used to infer the semantic information present in different images based on the contextual features of different images through the Transformer block in the deep learning model. The semantic information present in different images includes the classification of target objects and the confidence scores between targets.

[0094] The output module is used to output the image triplet prediction results according to the confidence score.

[0095] The deep learning model is obtained by extracting visual context features of an image through a Transformer encoder-decoder network architecture, performing data augmentation on the classification of visual context features of the image, and then training a deep neural network.

[0096] This invention provides a novel scene graph construction scheme that avoids the degradation of scene graph quality caused by data imbalance. The generated high-quality scene graph provides more comprehensive image content analysis, which can assist in processing image data captured by cameras in intelligent transportation systems, analyzing real-time traffic conditions on monitored roads, and the spatial location and behavioral relationships between various traffic elements (vehicles, pedestrians). Simultaneously, this invention focuses on solving the data imbalance problem. Therefore, it can effectively perform image content analysis on some difficult samples in medical images, such as those with complex and rare diseases, thereby accurately pinpointing the cause of lesions and assisting doctors in making judgments. Thus, scene graph technology has broad research significance and application prospects in image understanding.

[0097] This invention describes the semantic content of images by constructing scene graphs based on generative templates. The method is simple and effective, and the experimental results are accurate. It can effectively realize the task of image content parsing and can be applied to intelligent transportation systems, medical image analysis, etc.

[0098] The above description is only of the preferred embodiment of the present invention and should not be construed as limiting the scope of the claims. The present invention is not limited to the above embodiments, and variations in its specific structure are permitted. All variations made within the scope of the independent claims of the present invention are also within the scope of protection of the present invention.

[0099] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

Claims

1. A method for constructing an unbiased scene graph based on generative templates, characterized in that, Includes the following steps: Acquire different images from various scenes to be detected; Deep learning models are used to extract contextual features from different images in rich scenes; Based on the contextual features of different images, semantic information existing in different images is inferred through the Transformer block in the deep learning model. The semantic information existing in different images includes the classification of target objects and the confidence scores between targets. Output the image triplet prediction results according to the confidence score; The deep learning model is obtained by extracting the visual context features of the image through the Transformer encoder-decoder network architecture, performing data augmentation on the classification of the visual context features of the image, and then training the deep neural network. Deep learning models are obtained through the following process: Based on the classification difficulty of the training set samples, all samples are divided into easy samples and hard samples, and the corresponding visual context feature matrices of easy samples and hard samples are obtained. Based on the visual context feature matrices of simple and hard samples, distillation meta-learning is performed to obtain feature templates for identifying categories; After supplementing the feature templates of the identifier categories with data, a deep neural network is trained to obtain a deep learning model. The feature templates for identifying categories are supplemented with data through the following process: Reconstructed supplementary samples are obtained through MCMC sampling; Compare the feature similarity between background class samples and feature templates of the label class in the training set, and take samples with feature similarity higher than the threshold as high-confidence samples; The reconstructed supplementary samples and high-confidence samples are combined to form out-of-distribution sample data, which is a balanced dataset. The combined data of the training set and the balanced dataset is used to train the deep neural network to obtain a deep learning model.

2. The method for constructing an unbiased scene graph based on generative templates according to claim 1, characterized in that, Visual context features of images are extracted using a Transformer encoder-decoder network architecture. The process includes the following steps: using ResNet-101 and FPN networks as backbone networks, Faster RCNN as the detector, training on the training set to obtain candidate object boxes, and obtaining the visual features of the candidate objects based on the candidate object boxes; The visual features of the candidate targets are input into an encoder-decoder network based on Transformer blocks to obtain visual context features.

3. The method for constructing an unbiased scene graph based on generative templates according to claim 2, characterized in that, The visual context features of an image are calculated using the following formula: (1) in, Visual features representing candidate targets. This indicates the relative geometric position information between targets. This represents the word embedding features corresponding to the target. ，i This represents the category corresponding to the i-th sample. Represents the visual context features of an image.

4. The method for constructing an unbiased scene graph based on generative templates according to claim 1, characterized in that, By using distillation meta-learning, after data augmentation of the feature templates for identifying categories, a deep neural network is trained to obtain a deep learning model, including the following steps: The feature template for identifying categories includes simple and hard samples corresponding to the feature matrix; A template indicator is constructed using the visual context feature matrix corresponding to simple samples. The template indicator consists of a multi-head self-attention layer and a multilayer perceptron, and the indicator parameters are updated through incremental learning. Then, the indicator is fine-tuned using the corresponding difficult samples of the feature matrix; The feature matrix is ​​updated using the fine-tuned indicator to obtain the feature template that identifies the category.

5. The method for constructing an unbiased scene graph based on generative templates according to claim 1, characterized in that, Out-of-distribution sample data as follows: (4) In the formula, As the category center, This is a deviation.

6. An unbiased scene graph construction system based on generative templates, characterized in that, include: The image acquisition module is used to acquire different images from various scenes to be detected. The feature extraction module is used to extract contextual features of different images in rich scenes using a deep learning model; The semantic information inference module is used to infer the semantic information present in different images based on the contextual features of different images through the Transformer block in the deep learning model. The semantic information present in different images includes the classification of target objects and the confidence scores between targets. The output module is used to output the image triplet prediction results according to the confidence score. The deep learning model is obtained by extracting the visual context features of the image through the Transformer encoder-decoder network architecture, performing data augmentation on the classification of the visual context features of the image, and then training the deep neural network. Deep learning models are obtained through the following process: Based on the classification difficulty of the training set samples, all samples are divided into easy samples and hard samples, and the corresponding visual context feature matrices of easy samples and hard samples are obtained. Based on the visual context feature matrices of simple and hard samples, distillation meta-learning is performed to obtain feature templates for identifying categories; After supplementing the feature templates of the identifier categories with data, a deep neural network is trained to obtain a deep learning model. The feature templates for identifying categories are supplemented with data through the following process: Reconstructed supplementary samples are obtained through MCMC sampling; Compare the feature similarity between background class samples and feature templates of the label class in the training set, and take samples with feature similarity higher than the threshold as high-confidence samples; The reconstructed supplementary samples and high-confidence samples are combined to form out-of-distribution sample data, which is a balanced dataset. The combined data of the training set and the balanced dataset is used to train the deep neural network to obtain a deep learning model.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the unbiased scene graph construction method based on generative templates as described in any one of claims 1 to 5.

Images