Method for generating a salient scene graph based on importance of fuzzy logic inference relations

Abstract

The application relates to a significant scene graph generation method and device based on fuzzy logic reasoning relationship importance, a scene graph generation model test method and device, and a computer device. The method comprises the following steps: S100, calculating the importance score of the real relationship of each image sample in a target data set; S200, constructing a target detector; S300, constructing a semantic extractor and an instance feature refinement module; S400, constructing a bounding box modeling module and a feature splicing module; S500, constructing a feature fusion module and a relationship representation modeling module; S600, constructing a relationship loss weighting module; S700, under the condition that the number of training image samples reaches the batch processing size, returning to step S200; under the condition that all image samples in the training set are read, entering step S800 to output a scene graph generation model. The method can flexibly and comprehensively evaluate the importance of the relationship without reducing the number of relationship samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and computer vision, in particular to a significant scene graph generation method and device based on fuzzy logic reasoning relationship importance, a scene graph generation model test method and device, and a computer device. BACKGROUND

[0002] Scene graph generation is a frontier topic in the field of artificial intelligence and is one of the key tasks of computer vision. As an intermediate task, scene graph helps to connect the upstream target detection task and the downstream advanced visual understanding task, and its effect largely depends on the prediction of important relationships by the model.

[0003] Although most current scene graph generation models have made progress in reducing relationship prediction bias by balancing predicate distribution, using resampling techniques, and enhancing semantic information, they often output too many unnecessary relationships that do not necessarily reflect the main content of the image. This approach can generate redundancy for downstream tasks such as image caption generation that use scene graphs, affecting the focus on the key content of the image and thus reducing the processing efficiency of the downstream task. Moreover, traditional significant scene graph generation methods rely on training with fixed datasets and fail to adapt to the actual needs of diversified downstream tasks, limiting the application range of the model. SUMMARY

[0004] Therefore, it is necessary to provide a significant scene graph generation method and device based on fuzzy logic reasoning relationship importance, a scene graph generation model test method and device, and a computer device that can accurately predict the importance of relationships and improve the ability of scene graph generation models to predict important relationships.

[0005] In a first aspect, the present application provides a significant scene graph generation method based on fuzzy logic reasoning relationship importance, comprising:

[0006] Step S100, calculating the importance score of each image sample in the target dataset using fuzzy logic, and dividing the target dataset into a training set and a test set, wherein each image sample includes a subject object and an object object;

[0007] Step S200, constructing a target detector that takes an image sample as input and outputs the label of each object in the image sample and the corresponding bounding box information of the predicted relationship instance between the objects in the image sample;

[0008] Step S300, constructing a semantic extractor taking the labels of the relationship instances to be predicted between the objects in the image sample as input and the instance features of the subject object and the instance features of the object object as output, and constructing an instance feature refinement module to refine the instance features of the subject object and the instance features of the object object;

[0009] Step S400, constructing a bounding box modeling module taking the bounding box information of the relationship instances to be predicted between the objects in the image sample as input and taking the bounding box features of the subject object and the bounding box features of the object object as output, and constructing a feature splicing module to splice the bounding box features of the subject object and the instance features of the subject object to output the instance representation of the subject object and to splice the bounding box features of the object object and the instance features of the object object to output the instance representation of the object object;

[0010] Step S500, constructing a feature fusion module taking the instance representation of the subject object and the instance representation of the object object as input and taking the relationship features of the relationship to be predicted as output, and constructing a relationship representation modeling module taking the relationship features of the relationship to be predicted as input and taking the relationship representation of the relationship to be predicted as output;

[0011] Step S600, constructing a relationship loss weighting module for generating the label distribution of the relationship representation of the relationship to be predicted, combining the labels of the real relationships for loss calculation, and weighting and summing the losses of the relationships of each image sample with the importance scores of the real relationships as weights for scene graph generation training;

[0012] Step S700, from the input to the output direction, the target detector, the semantic extractor, the instance feature refinement module, the bounding box modeling module, the feature splicing module, the feature fusion module, the relationship representation modeling module, and the relationship loss weighting module constitute a to-be-trained network structure; based on the target data set, taking the image sample as input and the scene graph as output, the to-be-trained network structure is trained to obtain a scene graph generation model; in the case that the number of training image samples reaches the batch processing size, return to step S200; in the case that all image samples in the training set are read, enter step S800;

[0013] Step S800, outputting the scene graph generation model.

[0014] In one of the embodiments, the step S100 uses fuzzy logic to calculate the importance score of the real relationship of each image sample in the target data set, which includes:

[0015] According to the influence factors of the relationship importance, the relationship area, the predicate frequency, and the instance class, predicate class, and attribute class data specified by the user in the target data set are preprocessed as input parameters of the fuzzy reasoning system;

[0016] Dividing a fuzzy set for an input parameter of the fuzzy inference system;

[0017] Formulating a membership function, and converting the input parameter of the fuzzy inference system into a fuzzy value through the membership function;

[0018] Defining a fuzzy rule according to a factor influencing the importance of the relationship, and evaluating the fuzzy value according to the fuzzy rule to generate a fuzzy output of each fuzzy rule;

[0019] Aggregating the fuzzy output of the fuzzy rule through the fuzzy inference system to generate a fuzzy output parameter, and converting the fuzzy set of the fuzzy output parameter into an importance score through a defuzzification process by using a centroid method.

[0020] In one of the embodiments, the step S400 comprises:

[0021] Concatenating the coordinates of the upper left corner, the coordinates of the lower right corner of the bounding box of the instance, and the area of the bounding box by column, and generating the bounding box feature through modeling;

[0022] Concatenating the instance feature and the bounding box feature through modeling to form the instance representation.

[0023] In one of the embodiments, the step S600 of weighting the loss of the relationship of each image sample by using the importance score of the real relationship as the weight comprises:

[0024] Extracting the predicate feature in the target data set by using the semantic extractor;

[0025] Generating the refined predicate feature through modeling;

[0026] Performing standardization processing on the relationship representation and the predicate feature;

[0027] Calculating the loss of the label distribution of the relationship to be predicted and the real relationship label through a weighted cross-entropy loss function.

[0028] In a second aspect, the present application further provides a scene graph generation model testing method, which is used for testing the scene graph generation model provided in any one of the embodiments of the first aspect, and comprises:

[0029] In step S100, the image sample in the test set is input into the scene graph generation model taking the image sample as the input and the scene graph as the output to perform relationship prediction;

[0030] In step S200, the relationship prediction result is compared with the real predicate label, and the comparison result is evaluated by using the target evaluation index;

[0031] In step S300, the target prediction index is introduced to evaluate the result of the relationship prediction;

[0032] Step S400: Select important relationships from the relationship prediction results of the scene graph generation model and generate scene graphs;

[0033] Step S500: Output the generated scene graph.

[0034] In one embodiment, step S300 includes:

[0035] We introduce important relation recall and important relation precision to evaluate the results of relation prediction.

[0036] In one embodiment, the method further includes the following steps prior to step S400:

[0037] Calculate the importance score of the relationship for each image sample in the test set, and sort the predicted relationships from largest to smallest based on the importance score, with relationships having an importance score greater than 0.5 being considered important relationships.

[0038] Thirdly, this application also provides a salient scene graph generation device based on the importance of fuzzy logic reasoning relationships, including:

[0039] The importance score calculation module is used to calculate the importance score of the true relationship of each image sample in the target dataset using fuzzy logic, and to divide the target dataset into a training set and a test set, where each image sample includes a subject object and an object object;

[0040] The object detector building module is used to build an object detector that takes image samples as input and outputs the labels of the predicted relationship instances between objects in the image samples and the corresponding bounding box information of the predicted relationship instances.

[0041] The instance feature extraction and refinement module is used to construct a semantic extractor that takes the labels of the predicted relationship instances between objects in the image sample as input and the instance features of the subject object and the object object as output. It also constructs an instance feature refinement module to refine the instance features of the subject object and the object object.

[0042] The instance representation generation module is used to construct a bounding box modeling module that takes the bounding box information of the predicted relationship instances between objects in the image sample as input and the bounding box features of the subject object and the object object as output. It also constructs a feature concatenation module that concatenates the bounding box features and instance features of the subject object to output the instance representation of the subject object, and concatenates the bounding box features and instance features of the object object to output the instance representation of the object object.

[0043] a relationship representation generation module, configured to construct a feature fusion module taking an instance representation of a subject object and an instance representation of an object object as input and a relationship feature of a to-be-predicted relationship as output, and construct a relationship representation modeling module taking the relationship feature of the to-be-predicted relationship as input and a relationship representation of the to-be-predicted relationship as output;

[0044] a relationship loss weighting module, configured to construct a relationship loss weighting module for generating a label distribution of a relationship representation of a to-be-predicted relationship, performing loss calculation in combination with a label of a real relationship, weighting a loss of a relationship of each image sample with an importance score of the real relationship as a weight, and performing scene graph generation training;

[0045] a scene graph generation model training module, configured to, from an input direction to an output direction, constitute a to-be-trained network structure with a target detector, a semantic extractor, an instance feature refinement module, a bounding box modeling module, a feature splicing module, a feature fusion module, a relationship representation modeling module, and a relationship loss weighting module; based on a target data set, take an image sample as input and a scene graph as output, train the to-be-trained network structure, and obtain a scene graph generation model; in a case where a quantity of training image samples reaches a batch processing size, return the target detector construction module; in a case where all training image samples are read in the training set, enter a scene graph generation model output module;

[0046] a scene graph generation model output module, configured to output the scene graph generation model.

[0047] In a fourth aspect, the present application further provides a scene graph generation model testing device for testing the scene graph generation model provided in the first aspect, comprising:

[0048] a relationship prediction module, configured to input an image sample in a test set into the scene graph generation model taking the image sample as input and the scene graph as output to perform relationship prediction;

[0049] a comparison result evaluation module, configured to compare a relationship prediction result with a real predicate label, and evaluate a comparison result using a target evaluation index;

[0050] a prediction result evaluation module, configured to introduce a target prediction index to evaluate a result of relationship prediction;

[0051] a scene graph generation module, configured to screen out an important relationship in a relationship prediction result of the scene graph generation model to generate a scene graph;

[0052] a scene graph output module, configured to output the generated scene graph.

[0053] In a fifth aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for significant scene graph generation based on the importance of fuzzy logic inference relationship and the method for testing the scene graph generation model when executing the computer program.

[0054] The scene graph generation method and device, the scene graph generation model test method and device, and the computer equipment can calculate the importance score of the real relationship of each image sample in the target data set according to the factors affecting the importance of the relationship, without reducing the number of relationship samples, and can flexibly and comprehensively evaluate the importance of the relationship. The target data set is divided into a training set and a test set, each image sample in the target data set includes a subject object and an object object, the training set is used to train the scene graph generation model, and the test set is used to test the scene graph generation model. A target detector is constructed, which takes an image sample as input and outputs the label of a to-be-predicted relationship instance between objects in the image sample and the corresponding bounding box information of the to-be-predicted relationship instance. A semantic extractor is constructed, which takes the label of the to-be-predicted relationship instance between objects in the image sample as input and outputs the instance feature of the subject object and the instance feature of the object object. An instance feature refinement module is constructed to refine the instance feature of the subject object and the instance feature of the object object. A bounding box modeling module is constructed, which takes the corresponding bounding box information of the to-be-predicted relationship instance between objects in the image sample as input and outputs the bounding box feature of the subject object and the bounding box feature of the object object. A feature splicing module is constructed to splice the bounding box feature of the subject object and the instance feature of the subject object to output the instance representation of the subject object, and to splice the bounding box feature of the object object and the instance feature of the object object to output the instance representation of the object object. A feature fusion module is constructed, which takes the instance representation of the subject object and the instance representation of the object object as input and outputs the relationship feature of the to-be-predicted relationship. A relationship representation modeling module is constructed, which takes the relationship feature of the to-be-predicted relationship as input and outputs the relationship representation of the to-be-predicted relationship. A relationship loss weighting module is constructed, which is used for generating the label distribution of the relationship representation of the to-be-predicted relationship, calculating the loss by combining the label of the real relationship, weighting the loss of the relationship of each image sample by using the importance score of the real relationship as a weight, and performing scene graph generation training. The relationship loss weighting module is plug-and-play, and the model pays more attention to important relationships by weighting the loss of each relationship sample by using the relationship importance score as a weight.From the input to the output direction, the target detector, the semantic extractor, the instance feature refinement module, the bounding box modeling module, the feature splicing module, the feature fusion module, the relationship representation modeling module, and the relationship loss weighting module constitute a network structure to be trained; based on the target data set, taking an image sample as input and a scene graph as output, the network structure to be trained is trained, and a scene graph generation model is obtained; in the case that the number of training image samples reaches the batch processing size, the step of constructing a target detector taking an image sample as input and the label of each object in the image sample and the corresponding bounding box information of the relationship instance to be predicted as output is repeated, and the above iteration training step is repeated; in the case that all image samples in the training set are read, the scene graph generation model is output, the ability of the scene graph generation model to predict important relationships is improved, and key information can be more effectively extracted and analyzed from a large number of social interaction images. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A schematic diagram of a target data set annotating relationships for a given image in an embodiment;

[0056] Figure 2 A flowchart of a significant scene graph generation method based on fuzzy logic reasoning relationship importance in an embodiment;

[0057] Figure 3 A general framework diagram of a relationship loss weighting scene graph generation network FLRW-SGG based on fuzzy logic in an embodiment;

[0058] Figure 4 A flowchart of an embodiment using fuzzy logic to infer relationship importance scores;

[0059] Figure 5 A module diagram of a relationship loss weighting module in a significant scene graph generation method based on fuzzy logic reasoning relationship importance in an embodiment;

[0060] Figure 6 A flowchart of a scene graph generation model testing method in an embodiment;

[0061] Figure 7 A training flowchart of FLRW-SGG in an embodiment;

[0062] Figure 8 A module diagram of a significant scene graph generation device based on fuzzy logic reasoning relationship importance in an embodiment;

[0063] Figure 9 A module diagram of a scene graph generation model testing device in an embodiment. DETAILED DESCRIPTION

[0064] Scene graphs integrate image recognition and natural language processing techniques to recognize and classify various instances from an image and predict the visual relationships between these instances. These instances and relationships are usually described by nouns and predicates and output in the form of triples. As shown in Figure 1 , “man-riding-bike” is an example of a triple, where “man” is the subject instance, “bike” is the object instance, and “riding” is the relationship between them. By combining all the triples in an image, a scene graph of the image can be constructed, as shown in Figure 1 Scene graph generation serves as an intermediate task that helps to connect the upstream task of object detection and the downstream task of advanced visual understanding, such as image caption generation, visual navigation, image retrieval, human pose estimation, and visual question answering.

[0065] Traditional two-stage scene graph generation methods first perform instance recognition and feature extraction on an input image using a target detector, and then in the second stage, the model combines the features of the subject and object to predict the relationship between them. Although most current scene graph generation models have made progress in reducing relationship prediction bias by balancing predicate distribution, using resampling techniques, and enhancing semantic information, they often output too many unnecessary relationships that do not necessarily reflect the main content of the image. For example, most models use the most common Visual Genome 150 (VG150) dataset for scene graph generation, and some instances and relationships annotated in this dataset are not important for understanding the main content of the entire image scene. The model cannot determine important relationships during training and instead treats all relationships equally. As shown in Figure 1 , for the given image on the left, the diagram on the right shows the relationships annotated by the VG150 dataset for the given image, where <man-riding-bike> 、 <man-near-train>For understanding the main content of the image, the scene graph in the lower right corner treats all relationships equally and outputs them all. This approach can generate redundancy for downstream tasks such as image caption generation using scene graphs, affecting the focus on the key content of the image, and thus reducing the processing efficiency of the downstream task.

[0066] In order to make the model predict important relationships, and let the scene graph better serve the downstream task, researchers began to study the significant scene graph generation method, which can focus on identifying significant relationships (also known as important relationships or key relationships) in the image. As the Significance-guided Message Passing (SMP) method, using an object interaction encoder to enhance the representation of object relationships by jointly utilizing the appearance, semantic and spatial relationships between objects, and further estimating the significance of each visual relationship through ordinal regression. With object and relationship features as conditions, SMP enhances scene graph generation by performing message passing on objects and significant relationships. Training and testing are carried out on the dataset VG-KR (Visual Genome Keyword Recognition) with significant relationship annotations, but the dataset is too absolute in dividing key relationships, which divides relationships into key relationships and unimportant relationships according to the caption content of the image, visual attention to color contrast, center position, instance size, etc. And only the annotation of key relationships is retained in the dataset for training, which may not be sufficient for the model using the dataset to learn fully due to the reduction of samples. In fact, as a bridge between object detection and downstream tasks, different downstream tasks in different fields have different levels of attention to instances and relationships in images. For example, in urban traffic research, the main focus is on traffic-related elements such as vehicles and traffic lights; while in social behavior research, more attention is paid to human interaction. Therefore, training the scene graph generation model only on VG-KR reduces the accuracy of the model in predicting relationships, and also fails to meet the application needs of different research fields.

[0067] Scene graph generation helps downstream advanced visual understanding tasks, and its effectiveness depends largely on the model's prediction of important relationships. If the importance of relationships can be accurately predicted, scene graphs will play a greater role in downstream tasks. Existing significant scene graph generation methods rely on training on fixed datasets and fail to adapt to the actual needs of diversified downstream tasks, limiting the application range of the model. In order to solve this problem, the present invention proposes a significant scene graph generation method based on fuzzy logic reasoning relationship importance.

[0068] In an exemplary embodiment, as Figure 2 As shown, a significant scene graph generation method based on the importance of fuzzy logic reasoning relationship is provided, and the method comprises the following steps:

[0069] In step S100, the importance score of the real relationship of each image sample in the target data set is calculated using fuzzy logic, and the target data set is divided into a training set and a test set, wherein each image sample comprises a subject object and an object object.

[0070] Specifically, the target data set comprises a VG (Visual Genome) 150 data set.

[0071] Exemplarily, the importance score of the real relationship in the VG150 data set is calculated using fuzzy logic, and step S200 is entered.

[0072] In step S200, a target detector is constructed, which takes an image sample as input, and the label of the relationship instance to be predicted between the objects in the image sample and the boundary box information corresponding to the relationship instance to be predicted as output.

[0073] Preferably, the training images in step S200 are all from the standard data set VG150 of scene graph generation. The data set consists of 108077 images, containing the most common 150 instance classes and 50 predicate classes. In this embodiment, 70% of the images in the data set are divided into a training set, and the remaining 30% of the images are a test set. The target detector adopts Faster R-CNN, and the instance label and the corresponding boundary box information are obtained through the target detector.

[0074] Exemplarily, the training images in the VG150 data set are input into the target detector to generate the instance label of the relationship to be predicted and the corresponding boundary box information, and step S300 is entered.

[0075] In step S300, a semantic extractor is constructed, which takes the label of the relationship instance to be predicted between the objects in the image sample as input, and the instance features of the subject object and the instance features of the object object as output, and an instance feature refinement module is constructed to refine the instance features of the subject object and the instance features of the object object.

[0076] Exemplarily, the semantic extractor is used to extract the instance features of the subject and the object, and the further modeling generates refined instance features of the subject and the object, and step S400 is entered.

[0077] Step S400, a bounding box modeling module is constructed, which takes the boundary box information corresponding to the to-be-predicted relationship instance between each object in the image sample as input, and outputs the boundary box feature of the subject object and the boundary box feature of the object object.

[0078] Specifically, the boundary box information includes boundary box coordinates and area.

[0079] Exemplarily, the boundary box coordinates and area of the subject and the object are modeled to generate the boundary box features of the subject and the object, and the boundary box features are spliced with the instance features to form the instance representation of the subject and the object, and step S500 is entered.

[0080] Step S500, a feature fusion module is constructed, which takes the instance representation of the subject object and the instance representation of the object object as input, and outputs the relationship feature of the to-be-predicted relationship, and a relationship representation modeling module is constructed, which takes the relationship feature of the to-be-predicted relationship as input, and outputs the relationship representation of the to-be-predicted relationship.

[0081] Exemplarily, the instance representations of the subject and the object are fused to obtain the relationship feature, and the relationship representation is generated by further modeling the relationship feature, and step S600 is entered.

[0082] Step S600, a relationship loss weighting module is constructed for generating the label distribution of the relationship representation of the to-be-predicted relationship, loss calculation is performed in combination with the label of the real relationship, the importance score of the real relationship is used as a weight to weight the loss of each relationship sample of the relationship, and scene graph generation training is performed.

[0083] Exemplarily, the relationship loss weighting is performed. The label distribution of the relationship representation of step S500 is generated, loss calculation is performed with the real relationship label, the relationship importance score of step S100 is used as a weight to weight the loss of each relationship sample, and then scene graph generation training is performed, and step S700 is entered.

[0084] Step S700, from the input to the output direction, the target detector, the semantic extractor, the instance feature refinement module, the bounding box modeling module, the feature splicing module, the feature fusion module, the relationship representation modeling module, and the relationship loss weighting module constitute a to-be-trained network structure; based on the target data set, taking the image sample as input and the scene graph as output, the to-be-trained network structure is trained, and a scene graph generation model is obtained; in the case that the number of training image samples reaches the batch processing size, return to step S200; in the case that all image samples in the training set are read, enter step S800.

[0085] Exemplarily, if the set batch-size size is reached, return to step S200; if all the training pictures are read, go to step S800.

[0086] In step S800, output the scene graph generation model.

[0087] Exemplarily, output the trained model, and end.

[0088] The above significant scene graph generation method based on the importance of fuzzy logic inference relationship can calculate the importance score of the real relationship in the VG150 dataset according to the factors affecting the importance of the relationship, such as the relationship area, the predicate frequency, and the user-specified data, such as the specified instance class, the predicate class, and the attribute class. The importance of the relationship can be flexibly and comprehensively evaluated without reducing the number of relationship samples. The training set is input into the target detector, and the importance score of the real relationship is used as a weight to weight the loss of each image sample in the training set, and the scene graph generation model is iteratively trained through multiple steps of generating instance labels and corresponding bounding box information, instance features, instance representations, and relationship representations of the relationship to be predicted. By designing the plug-and-play relationship loss weighting module, the loss of each relationship sample is weighted by using the importance score of the relationship as a weight, so that the scene graph generation model pays more attention to important relationships. The relationship loss weighting module can be applied to the scene graph generation baseline model with excellent performance, and the ability of the scene graph generation model to predict important relationships can be improved.

[0089] In one embodiment, the present application proposes a fuzzy logic relation loss weighted scene graph generation network FLRW-SGG (Fuzzy Logic Relation Loss Weighted Scene Graph Generation Network). The overall framework diagram of the network is as follows: Figure 3 As shown, the importance score of the real relationship in the VG150 dataset is calculated using fuzzy logic, the training image is input into the Faster R-CNN to obtain the predicted instance class label and the predicted instance bounding box; the semantic extraction and feature refinement are performed according to the predicted instance class label to obtain the instance feature of the subject and the instance feature of the object, the bounding box modeling is performed based on the predicted instance bounding box to further obtain the bounding box feature of the subject and the bounding box feature of the object; the bounding box feature and the instance feature are spliced to form the instance representation of the subject and the object, the instance representation is modeled and the feature is fused to obtain the relationship feature, the relationship feature is modeled to generate the relationship representation, and the relationship representation is input into the relationship loss weighting module. The VG150 dataset is preprocessed and input into the fuzzy logic reasoning module to obtain the relationship importance score, and the significant scene graph is obtained based on the relationship importance score and the relationship loss weighting module, including "woman", "laptop", "pant", "chair", "shelf", "holding", "wearing", "sitting on", and "behind".

[0090] Further, the process of obtaining the relationship importance score using fuzzy logic reasoning is as shown in Figure 4 The relationship area, predicate frequency and specified attribute class are used as input values of the Gaussian function to obtain the fuzzy set; the specified instance class and the specified predicate class are used as input values of the S-shaped function to obtain the fuzzy set, the fuzzy rules are defined and the fuzzy rule reasoning is performed to generate the fuzzy output parameter; the de-fuzzification process is performed by the centroid method to obtain the output value, i.e. the relationship importance score.

[0091] Further, the relationship loss weighting module is as shown in Figure 5 The semantic extractor is used to perform semantic extraction on the predicate label to obtain the predicate feature; the matrix multiplication is performed on the refined predicate feature and the relationship representation to obtain the label distribution of the predicted relationship, and the loss of the label distribution of the predicted relationship and the real relationship label is calculated by the weighted cross-entropy loss function combined with the relationship importance score. Wherein, represents the multiplication of the matrix.

[0092] In an exemplary embodiment, the step S100 of using fuzzy logic to calculate the importance score of the real relationship of each image sample in the target data set comprises: according to the influence factors of the importance of the relationship, pre-processing the area of the relationship region, the predicate frequency and the user-specified instance class, predicate class and attribute class data in the target data set as input parameters of the fuzzy reasoning system; dividing the fuzzy set of the input parameters of the fuzzy reasoning system; formulating the membership function, and converting the input parameters of the fuzzy reasoning system into fuzzy values through the membership function; defining fuzzy rules according to the factors affecting the importance of the relationship, and evaluating the fuzzy values according to the fuzzy rules to generate the fuzzy output of each fuzzy rule; aggregating the fuzzy output of the fuzzy rules through the fuzzy reasoning system to generate the fuzzy output parameter, and converting the fuzzy set of the fuzzy output parameter into the importance score through the centroid method.

[0093] Exemplarily, the step of using fuzzy logic to calculate the importance score of the real relationship in the VG150 data set comprises:

[0094] In step S101, according to the influence factors of the importance of the relationship, the area of the relationship region, the predicate frequency and the user-specified instance class, predicate class and attribute class data in the VG150 data set are pre-processed as input parameters of the fuzzy reasoning system, and the processing process comprises:

[0095] For the area of the relationship region, the area of each instance is calculated according to the instance bounding box information in the VG150 data set, and normalized processing is performed to eliminate the influence of image size; the area of the relationship is obtained by weighted average of the area of the subject and object instances; the predicate frequency needs to be normalized to the value range of 0-1 due to the large difference in the number of occurrences in the data set, so as to reduce the deviation of high-frequency predicates on the analysis result; for specific instance and predicate categories, this embodiment adopts binary classification labeling: if the subject or object in the relationship belongs to the preset category, the relationship is labeled as 1, otherwise as 0; for the attribute class, considering that each instance may have multiple attributes, the attribute richness of the relationship is determined by calculating the total number of specific attribute categories contained in the subject and object instances in each relationship.

[0096] Step S102, the fuzzy sets of the input parameters in step S101 are divided. The fuzzy set of the relationship area (RA) is divided into three subsets: {large region (LR), medium region (MR), small region (SR)}; the fuzzy set of the predicate frequency (PF) is {high frequency (HF), medium frequency (MF), low frequency (LF)}; the fuzzy set of the specified instance class (SO) is divided into two subsets: {include (IN), not include (NI)}; the fuzzy set of the specified predicate class (SP) is {belong to (BE), not belong to (NB)}; the fuzzy set of the specified attribute class (SA) is divided into three subsets: {more attributes (MA), medium attributes (MD), few attributes (FA)}.

[0097] In addition, the relationship importance (RI) is taken as the output parameter of the fuzzy inference system, and the fuzzy set is divided into three subsets: {important (HI), medium important (MI), not important (LI)}.

[0098] Step S103, the input parameters are converted into fuzzy values by using appropriate membership functions.

[0099] Specifically, for the parameters of the relationship area, the predicate frequency and the specified attribute class, their fuzzy sets obviously present different levels of division, and the Gauss function (Gaussmf) is used as the membership function μ gaussmf (X1; a, β) as shown in formula (1), wherein X1 ∈ {RA, PF, SA} is the parameter of the relationship area, the predicate frequency and the specified attribute class, a is a hyperparameter reflecting the peak value of the Gauss function, and β is a hyperparameter used to represent the decay speed of the Gauss function from the peak value. The values of a and β are determined according to the numerical range of each parameter.

[0100]

[0101] Further, for the two input parameters of the specified instance class and the specified predicate class, their fuzzy sets represent two extreme states of yes and no respectively, and the S-shaped function (Sigmf) has the characteristics of a continuous curve, which can provide a smooth transition between the two states, and is used as the membership function μ sigmf (X2; h, k) as shown in formula (2), wherein X2 ∈ {SO, SP} is the parameter of the specified instance class and the predicate class. ε and h are hyperparameters reflecting the middle position and steepness of the S-shaped curve respectively, and the greater h is, the more rapid the transition of the function near ε. The values of ε and h are determined according to the numerical range of each parameter.

[0102]

[0103] Step S104, according to the factors affecting the importance of the relationship, the following fuzzy rules are defined, which will evaluate the fuzzy value of step S103 to generate the fuzzy output of each rule.

[0104] (1) if (SO = IN) U (SP = BE) U (SA = MA), Then RI = HI

[0105] (2) if (SO = NI) ∩ (SP = NB) ∩ (SA = FA), Then RI = LI

[0106] (3) if (RA = LA) ∩ (SA = MD), Then RI = HI

[0107] (4) if (PF = HF) ∩ (SA = MD), Then RI = HI

[0108] (5) if (PF = LF) U (PF = MF) U (PF = HF), Then RI = MI

[0109] Among them, rule (1) means that if the subject or object in the relationship belongs to the specified instance category, or the relationship belongs to the specified predicate category, or the number of attributes in the relationship is large, then the importance of the relationship is "important"; rule (2) means that when there is no specified instance predicate category in the relationship, and the number of attributes is small, then the importance of the relationship is "not important"; rule (3) means that if the relationship area is large and the number of attributes is moderate, then the importance of the relationship is "important"; rule (4) means that if the predicate frequency in the relationship is low and the number of attributes is moderate, then the importance of the relationship is also "important"; rule (5) is the default rule, which ensures that when other factors do not provide enough decisive information, that is, the above rules are not met, the system can assign the relationship importance to "moderate importance".

[0110] Step S105, the fuzzy inference system aggregates the outputs of all rules to generate a comprehensive fuzzy output parameter RI to reflect the importance of the relationship, and finally through the centroid method to defuzzify, the fuzzy set of RI is converted into a specific numerical result as the relationship importance score R im of the experiment. The process of using the centroid method to defuzzify the relationship importance score R im is shown in formula (3). Wherein, α ∈ {HI, MI, LI} RI represents a subset in the fuzzy set of the fuzzy output parameter RI, μ RI is a Gaussian function used for RI, μ RI (α) represents the membership degree of each subset in the fuzzy set of RI. R im ∈ [0, 1] indicates the importance score of each relationship in the data set.

[0111]

[0112] In this embodiment, the importance score of the real relationship of each image sample in the target data set is calculated using fuzzy logic, which helps to calculate the relationship loss of the image sample as a weight in the process of training the scene graph generation model, thereby improving the accuracy of the scene graph generation model.

[0113] In an exemplary embodiment, the step S300 of generating refined subject and object instance features is as follows:

[0114] Step S301, a semantic extractor is used to extract the instance features f s 、f o , as shown in equations (4) and (5). Where w s 、w o are learnable weight parameters of the subject and object instances, T represents matrix transpose, c s 、c o represent the word embeddings of the subject and object labels, respectively.

[0115] f s =w s T c s (4)

[0116] f o =w o T c o (5)

[0117] Step S302, the instance features of the subject and object are further modeled to generate refined subject and object instance features Refined subject instance features As shown in equation (6). Where FC(·), LN(·), and ReLU(·) represent full connection layer, layer normalization operation, and ReLU activation function, respectively, and are connected through a residual connection to improve the expression ability of the instance features. Further, the refined object instance features are derived in the same way according to equation (6)

[0118]

[0119] In an exemplary embodiment, the significant scene graph generation method based on fuzzy logic reasoning relationship importance includes step S400:

[0120] The coordinates of the upper left corner and the lower right corner of the instance bounding box and the area of the bounding box are spliced by column, and the bounding box features are generated by modeling; the instance features and the bounding box features are spliced to form the instance representation by modeling.

[0121] Specifically, in step S401, the coordinates b l , b r of the upper left corner and the lower right corner of the bounding box of the instance, and the area b a of the bounding box are concatenated to model the bounding box feature of the instance, and the bounding box feature of the subject is generated as shown in formula (7). Wherein, [·, ·] represents column concatenation, and LReLU(·) is a LeakyReLU activation function. Further, the bounding box feature of the object is derived in the same way.

[0122]

[0123] In step S402, the instance features and the bounding box features are further modeled to generate the instance representations of the subject and the object according to formula (8). Wherein, s is the subject instance representation, which represents the concatenation of the subject instance features and the subject bounding box features. Further, the object instance representation o is derived in the same way.

[0124]

[0125] In this embodiment, the method of generating bounding box features based on bounding box information and concatenating the bounding box features with instance features to form instance representations is further described, which helps to train the scene graph generation model and improve the result accuracy of the scene graph generation model.

[0126] In one exemplary embodiment, the significant scene graph generation method based on fuzzy logic reasoning relationship importance comprises the following steps:

[0127] In step S501, the instance representations s and o of the subject and the object are fused to obtain the relationship feature F(s, o), as shown in formula (9):

[0128] F(s, o) = ReLU(s + o) - (s - o) 2 (9)

[0129] In step S502, the relationship feature is further modeled to enhance the expression ability of the feature, and the specific operation is shown in formula (10). Wherein, MLP represents a multi-layer perceptron, which effectively captures important features in input data through multiple nonlinear transformations, and provides rich feature representation for the model.

[0130]

[0131] In an exemplary embodiment, the step of weighting the loss of the relation of each image sample with the importance score of the real relation as the weight in the significant scene graph generation method based on fuzzy logic inference relation importance includes:

[0132] The predicate features in the target data set are extracted by a semantic extractor; refined predicate features are generated by modeling; the relation representation and the predicate features are standardized; and the loss of the label distribution of the predicted relation and the real relation label is calculated by a weighted cross-entropy loss function.

[0133] Exemplarily, the step of weighting the relation loss in step S600 is as follows:

[0134] In step S601, a semantic extractor is used to extract predicate features f p , as shown in formula (11). Wherein, w p is a learnable weight parameter of the predicate, T represents matrix transposition, c p is a word embedding representing the predicate label.

[0135] f p =w p T c p (11)

[0136] In step S602, the predicate features are further modeled to generate refined predicate features , as shown in formula (12).

[0137]

[0138] In step S603, the relation representation and the predicate features are standardized, and the label distribution r p of the predicted relation is calculated by formula (13). Wherein, ||·|| represents L2 norm, ⊙ is dot product operation, exp is exponential operation, and γ is a learnable scaling factor, used to adjust the range of the relation label distribution.

[0139]

[0140] In step S604, the loss of the label distribution of the predicted relation and the real relation label is calculated by a weighted cross-entropy loss function. Specifically, the importance score R im of each relation sample calculated in step S100 is taken as the weight to multiply the standard cross-entropy loss, so that the loss of the important relation sample contributes more to the total loss, as shown in formula (14). Wherein, and are the real label and the predicted distribution of the jth relation sample, respectively, is the importance score of the jth relationship sample, and N is the total number of relationship samples on each image.

[0141]

[0142] As preferred, the present application is tested under three sub-tasks, namely predicate classification (Predcls), scene graph classification (SGcls) and scene graph detection (SGdet) respectively.

[0143] In the predicate classification task, the model takes all instance bounding box information and labels of the image as input, and the goal is to predict the relationship class between instances, focusing on understanding the interaction between entities, without involving the challenge of instance recognition; the scene graph classification task is similar to predicate classification, and the input also includes the bounding box information of all instances, but does not contain instance class information. Therefore, the model first needs to identify the class of each instance, and then predict the relationship between instances, and this task is more challenging because the accuracy of instance classification directly affects relationship prediction; the scene graph detection is the most challenging task, and the input image does not contain any instance bounding box and class information. The model needs to independently identify the location and class of all instances and predict the relationship between them. The instance recognition from scratch may cause deviation from the manually annotated data, increase the difficulty of relationship prediction, and thus may affect the overall accuracy.

[0144] In this embodiment, the method of relationship loss weighting is further described, which continuously iteratively corrects the loss between the true relationship and the relationship to be predicted, thereby improving the relationship prediction ability of the scene graph generation model.

[0145] In an exemplary embodiment, in order to test the output of the trained model in the significant scene graph generation method based on fuzzy logic reasoning relationship importance, as shown in Figure 6 A scene graph generation model testing method is provided for testing the scene graph generation model provided in the above embodiments, comprising:

[0146] Step S100, inputting the image samples in the test set into the scene graph generation model taking the image samples as input and the scene graph as output for relationship prediction.

[0147] Among them, the test image is selected from the test set divided in the VG150 data set.

[0148] Exemplarily, the test image is input into the trained scene graph generation model for relationship prediction, and step S200 is entered.

[0149] Step S200, compare the relationship prediction result with the real predicate label, and evaluate the comparison result using a target evaluation index.

[0150] Illustratively, the relationship prediction result is compared with the real predicate label, the result is evaluated using R@50, R@100 evaluation indexes, and step S300 is entered.

[0151] Step S300, introduce a target prediction index to evaluate the result of relationship prediction.

[0152] Illustratively, in order to evaluate the prediction ability of the scene graph generation model on important relationships, two new indexes are introduced to evaluate the result, and step S400 is entered.

[0153] Specifically, the IR@5, IR@10, IP@5 and IP@10 indexes are used for evaluation in this embodiment.

[0154] Step S400, screen out important relationships in the relationship prediction result of the scene graph generation model to generate a scene graph.

[0155] Illustratively, the important relationships in the predicted relationships are screened to form a scene graph, and step S500 is entered.

[0156] Step S500, output the generated scene graph.

[0157] Illustratively, the test result is output, and the process ends.

[0158] The scene graph generation model test method described above tests the scene graph generation model provided in the above embodiment to ensure the effect of the scene graph output by the scene graph generation model. In order to compare the relationship prediction result with the real predicate label, the evaluation standard Recall@K (R@K) widely recognized in the field of scene graph generation is adopted. Recall@K refers to the proportion of the number of correctly predicted relationships in the first K prediction results to the total number of relationships. The larger the value of this index, the better the test result, which means that the higher the accuracy of the model relationship prediction, and the better the model performance. In order to estimate the prediction ability of the model on important relationships, two evaluation indexes, important relationship recall rate (IR@K) and important relationship accuracy rate (IP@K), are introduced. The important relationship recall rate (IR@K) refers to the proportion of the number of correctly predicted important relationships in the first K prediction results to the total number of important relationships. The important relationship accuracy rate (IP@K) refers to the proportion of the number of correctly predicted important relationships in the first K prediction results to the number of correctly predicted relationships. The two indexes complement each other and can comprehensively evaluate the performance of the model in important relationship prediction.

[0159] In an exemplary embodiment, the image samples in the test set are input into a scene graph generation model taking image samples as input and scene graphs as output for relationship prediction, wherein the test images are selected from the test set divided in the VG150 dataset, and the specific steps include:

[0160] Step S101, performing a predicate classification task. The scene graph generation model uses the position and class label information of all instances provided by the test set, and under the premise that the exact position and class of each instance are known, the scene graph generation model focuses on predicting the relationship between instances.

[0161] Step S102, performing a scene graph classification task. The scene graph generation model does not access the class label information of instances in the image, but needs to independently predict the class of each instance. These prediction results are subsequently used for the next step of relationship prediction, and the prediction of the relationship between instances is continued based on this step.

[0162] Step S103, performing a scene graph detection task. The scene graph generation model cannot access the position and class label information of instances in the image. It first needs to identify and locate each instance in the image, and then predict the class of these instances. After completing these steps, the scene graph generation model further predicts the relationship between instances. Since it is necessary to understand the image content from scratch, this process has higher complexity and challenges compared to the previous two tasks.

[0163] In an exemplary embodiment, step S300 in the scene graph generation model test method includes:

[0164] Important relationship recall and important relationship accuracy are introduced to evaluate the results of relationship prediction.

[0165] Specifically, important relationship recall (IR@K) refers to the proportion of the number of correctly predicted important relationships in the top K prediction results to the total number of important relationships; and important relationship accuracy (IP@K) refers to the proportion of the number of correctly predicted important relationships in the top K prediction results to the number of correctly predicted relationships.

[0166] Specifically, the present application uses IR@5, IR@10, IP@5 and IP@10 indicators for evaluation.

[0167] In this embodiment, the important relationship recall rate (IR@K) is used to measure how many important relationships can be recalled in the first K prediction results. The index focuses on the coverage of important relationships by the model and can reflect the effect of the model in identifying important relationships. The important relationship accuracy (IP@K) refers to the proportion of the number of correctly predicted important relationships in the first K prediction results to the number of correctly predicted relationships. The index is used to measure how many correctly predicted relationships are important relationships in the first K prediction results. The index focuses on the importance judgment ability of the model when correctly predicting relationships and can reflect the accuracy of the model in predicting the importance of relationships. In combination with the relationship importance score, the two indexes complement each other and can comprehensively evaluate the performance of the model in predicting important relationships.

[0168] In an exemplary embodiment, the step S400 in the scene graph generation model testing method further comprises:

[0169] The importance score of each image sample relationship in the test set is calculated, and the predicted relationships are sorted in descending order according to the importance score. The relationships with an importance score greater than 0.5 are regarded as important relationships.

[0170] Specifically, the predicted relationships are sorted in descending order according to the relationship importance score calculated by the significant scene graph generation method based on fuzzy logic reasoning, and the relationships with an importance score greater than 0.5 are regarded as important relationships to form a significant scene graph.

[0171] In this embodiment, the importance score of each image sample relationship is used to determine important relationships, and based on the important relationships, a more accurate significant scene graph can be generated.

[0172] In an exemplary embodiment, the present application proposes a relationship loss weighting scene graph generation network model based on fuzzy logic. The model uses fuzzy logic to calculate the importance score of each relationship according to the actual needs of users and uses the importance score as a weight to improve the ability of the model to predict important relationships, so that the model can meet the application needs of different research fields.

[0173] Further, the following three indexes R@K, IP@K and IR@K are used for the model proposed in the present application.

[0174] Further, experiments are performed on the VG150 dataset, which contains the most common 150 instance classes and 50 predicate classes. The present application respectively evaluates the indicators in the predicate classification, scene graph classification and scene graph detection three sub-tasks. In the predicate classification task, the evaluation results finally obtained by the method of the present application are as follows: R@50 is 66.14%, R@100 is 68.36%, IR@5 is 38.78%, IR@10 is 51.02%, IP@5 is 90.65%, and IP@10 is 88.49%; in the scene graph classification task, the evaluation results finally obtained by the method of the present application are as follows: R@50 is 40.66%, R@100 is 41.85%, IR@5 is 24.46%, IR@10 is 31.82%, IP@5 is 89.52%, and IP@10 is 87.08%; in the scene graph detection task, the evaluation results finally obtained by the method of the present application are as follows: R@50 is 32.38%, R@100 is 36.87%, IR@5 is 13.60%, IR@10 is 19.07%, IP@5 is 91.08%, and IP@10 is 89.03%.

[0175] In an exemplary embodiment, the training flowchart of the FLRW-SGG proposed by the present application is as shown in Figure 7 The batch-size size can be adjusted according to the performance of the computer in the training flowchart, and the training flowchart of the present application includes:

[0176] Step S1, input the image and annotation information in VG150, and enter step S2.

[0177] Step S2, calculate the importance score of the real relationship in the VG150 dataset using fuzzy logic, i.e., use fuzzy logic to infer the relationship importance score, and enter step S3.

[0178] Step S3, input the training image in the dataset into the target detector to generate the instance label and the corresponding bounding box information of the predicted relationship, i.e., generate the predicted instance label and the bounding box information through the target detector, and enter step S4.

[0179] Step S4, use the semantic extractor to extract the instance features of the subject and object, and further model to generate refined subject and object instance features, i.e., model the instance features of the subject and object, and enter step S5.

[0180] Step S5, model the bounding box coordinates and area of the subject and object to generate the bounding box features of the subject and object, i.e., model the bounding box features of the subject and object, and enter step S6.

[0181] Step S6, concatenate the bounding box features and the instance features to generate the instance representation of the subject and object, and enter step S7.

[0182] Step S7, the instance representations of the subject and the object are fused to obtain a relation feature, and the relation feature is further modeled to generate a relation representation, i.e., the instance representations of the subject and the object are fused, the relation representation is modeled, and step S8 is entered.

[0183] Step S8, the fused relation feature is matched with the predicate prototype, and step S9 is entered.

[0184] Step S9, relation loss weighting is performed. A label distribution of the relation representation of step S6 is generated, loss calculation is performed with a real relation label, the relation importance score of step S2 is taken as a weight to weight the loss of each relation sample, and then training of scene graph generation is performed, and step S10 is entered.

[0185] Step S10, the scene graph generation model performs relation prediction, and step S11 is entered.

[0186] Step S11, if a set batch-size size is reached, i.e., a specified number of model training rounds is reached, step S3 is returned to; if all training pictures are read, step S12 is entered.

[0187] Step S12, a trained model is output, and the process ends.

[0188] It should be understood that these steps are not necessarily executed in the order indicated by the arrows. As described above, at least a part of the steps in the flowchart involved in each of the embodiments can include multiple steps or multiple stages, which can be executed at different times, or alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0189] In an exemplary embodiment, the present embodiment will utilize the VG150 dataset to complete the significant scene graph generation task based on the importance of fuzzy logic reasoning relationship. The dataset contains 108,077 images, covering 150 most common instance categories and 50 predicate categories. In order to reasonably divide the dataset, 70% of the images are assigned to the training set, and the remaining 30% are used as the test set. In order to complete the training and testing, the present embodiment selects deepin 20.4 as the operating system and conducts research in the experimental environment of torch1.7.1+cu110. The FLRW-SGG in the present embodiment is trained by the SGD optimizer for 50,000 iterations, and the initial learning rate and batch size are set to 0.001 and 8, respectively. In the process of fuzzification using the Gaussian function as the membership function, since the fuzzy set of the input parameter is divided into three and uniformly distributed in the range of 0 to 1, a is set as the center of the three fuzzy sets, respectively 0.25, 0.5 and 0.75, and β is set to 1 / 6 to ensure smooth transition between fuzzy sets; in the process of fuzzification using the S-shaped function as the membership function, since the fuzzy set of the input parameter is divided into two extreme states of yes and no, the S-shaped curve should ensure smooth transition between 0 and 1, then ε representing the intermediate position is set to 0.5, and h representing the steepness is set to 20 and -20. All experiments are performed on hardware equipped with NVIDIA GeForce RTX 4090 GPU to ensure the efficiency of the training and testing process.

[0190] The present application accurately detects instances and their location information in images by using Faster R-CNN. Due to its excellent performance and flexibility, Faster R-CNN has been widely used in various computer vision tasks, including but not limited to object detection, image segmentation, and face detection. In addition, it is also widely used to solve various practical problems, such as visual processing of autonomous driving systems, dynamic target tracking in security monitoring, and key areas such as medical image analysis, demonstrating its extensive influence in modern technology applications.

[0191] The present application evaluates the indicators in the predicate classification, scene graph classification and scene graph detection sub-tasks respectively. The present application uses the evaluation indicator Recall@K (R@K) to accurately evaluate the ability of FLRW-SGG in relationship prediction. R@K refers to the probability of measuring the correct predicate class label in the top K relationship prediction results. Specifically, the present application uses R@50 and R@100 indicators for evaluation. The comparison results of the present method and the existing method on the VG150 dataset are shown in Table 1.

[0192] Table 1 Comparison results of the present method and the traditional method on the VG150 dataset

[0193]

[0194] According to the results in Table 1, the method provided in the present application can achieve superior performance in relation prediction on three sub-tasks. Compared with Transformer under PredCls task, FLRW-SGG improves by 0.5% and 0.9% in R@50 and R@100, respectively; compared with PE-Net under SGCls task, FLRW-SGG improves by 1.3% and 1.2% in R@50 and R@100, respectively. Under SGDet task, compared with Motifs, FLRW-SGG improves by 0.3% and 0.1% in R@50 and R@100, respectively. These results show that the method FLRIW-Net of the present application can achieve a high level in the most basic evaluation standard Recall@K, and has the ability of relation prediction.

[0195] In order to evaluate the prediction ability of FLRW-SGG in important relations, the present application uses two indicators of important relation recall rate (IR@K) and important relation accuracy rate (IP@K) to evaluate, and selects three baseline models of scene graph generation of Motifs, Transformer and PE-Net to verify the effectiveness of the method FLRW-SGG of the present application and the relation loss weighting module in the method. Specifically, the present application reproduces the baseline model, and then inserts the relation loss weighting module of the present application into the baseline model, and penet+RW, motif+RW and transformer+RW are the relation loss weighting module on these baseline models. This way can verify whether the relation loss weighting module of the present application can improve the ability of the baseline model in predicting important relations. All models are trained and tested on the VG150 dataset using the same batch size (batchsize) and iteration number, and the evaluation results of IR@5, IR@10, IP@5 and IP@10 are recorded. Under the three sub-tasks of PredCls, SGCls and SGDet, the test results of the baseline model of the relation loss weighting module and the model after adding the relation loss weighting module are compared, and the results are shown in Table 2.

[0196] Table 2 Comparison results of the method of the present application and the baseline model in IR@K and IP@K indicators

[0197]

[0198]

[0199] According to the results in Table 2, the FLRW-SGG of the application and the relationship loss weighting module can achieve excellent performance in important relationship prediction on three sub-tasks. In the PredCls task, the FLRW-SGG achieves the best test results in IR@K and IP@K indicators; for the relationship loss weighting module, penet+RW improves the IR@5, IR@10, IP@5 and IP@10 of penet by 4.71%, 5.04%, 2.67% and 1.98% respectively, penet is a better unbiased model at present, it pursues to predict more tail predicates, so it cannot have better performance in the hit rate of the first K relationships, while the relationship loss weighting module of the application can greatly improve penet. On the motif and transformer, which are the baseline models with better relationship recall rate, the insertion of the relationship loss weighting module can further improve their ability to predict important relationships. In the SGCls task, the FLRW-SGG achieves the best test results in IR@K indicators; the models with the relationship loss weighting module can further improve the test results of IR@K and IP@K compared with the original model, and motif+RW achieves the best IP@5 and IP@10 results, which are improved by 1.15% and 1.05% respectively compared with motif. In the SGDet task, the FLRW-SGG achieves the best test results in IR@K, and the insertion of the relationship loss weighting module can further improve the ability of the baseline model to predict important relationships.

[0200] In an exemplary embodiment, a suitable scenario of the application is provided. In social behavior research, users often need to analyze a large number of images of social interactions, such as crowd behavior captured in public places or specific activities. These images often have complex shooting environments (such as cluttered backgrounds) that make it difficult for traditional scene graph generation models to accurately predict important relationships related to social behavior in the images, and users cannot generate accurate image captions. The significant scene graph generation method based on fuzzy logic reasoning of relationship importance proposed in the application can effectively cope with this challenge.

[0201] Firstly, the research platform pre-processes the uploaded images, such as image size adjustment and color correction, to optimize the image quality. Then, the Faster R-CNN model is used to extract features from the pre-processed images to identify and locate each instance in the image. Next, the method proposed in this application is used to generate a significant scene graph related to social behavior. This method scores the importance of relationships through fuzzy logic and uses a relationship loss weighting module to prioritize and highlight those relationships that are important in social behavior research, such as gaze, conversation, handshake, etc. Even in complex situations, users can generate relevant and accurate captions for the pictures based on the scene graph. Finally, these generated captions are displayed together with the images on the research platform, providing users with an in-depth perspective and detailed description of social interactions. In addition, users can search for keywords through the caption content to quickly locate images containing specific social behaviors, thereby speeding up the research analysis process.

[0202] Through the application of this application, users can more effectively extract and analyze key information from a large number of social interaction images, further promoting the depth and accuracy of social behavior research.

[0203] Based on the same inventive concept, the embodiments of the present application also provide a significant scene graph generation device based on fuzzy logic reasoning relationship importance for implementing the significant scene graph generation method based on fuzzy logic reasoning relationship importance as described above, and a scene graph generation model testing device for implementing the scene graph generation model testing method as described above. The implementation scheme for solving the problem provided by the above device is similar to the implementation scheme described in the above method, so the specific limitations in one or more device embodiments provided below can refer to the limitations of the corresponding method in the above text, which will not be repeated here.

[0204] In one exemplary embodiment, as shown in Figure 8 a significant scene graph generation device based on fuzzy logic reasoning relationship importance is provided, comprising:

[0205] an importance score calculation module 801 for calculating the importance score of each image sample in the target data set using fuzzy logic, and dividing the target data set into a training set and a test set, wherein each image sample includes a subject object and an object object;

[0206] a target detector construction module 802 for constructing a target detector taking an image sample as input and the label of each object relationship instance in the image sample and the corresponding bounding box information of the object relationship instance as output;

[0207] The instance feature extraction refinement module 803 is configured to construct a semantic extractor taking the label of the instance of the relationship to be predicted between the objects in the image sample as input and taking the instance feature of the subject object and the instance feature of the object object as output, and construct an instance feature refinement module to refine the instance feature of the subject object and the instance feature of the object object.

[0208] The instance representation generation module 804 is configured to construct a bounding box modeling module taking the bounding box information of the instance of the relationship to be predicted between the objects in the image sample as input and taking the bounding box feature of the subject object and the bounding box feature of the object object as output, and construct a feature splicing module to splice the bounding box feature of the subject object and the instance feature of the subject object to output the instance representation of the subject object, and splice the bounding box feature of the object object and the instance feature of the object object to output the instance representation of the object object.

[0209] The relationship representation generation module 805 is configured to construct a feature fusion module taking the instance representation of the subject object and the instance representation of the object object as input and taking the relationship feature of the relationship to be predicted as output, and construct a relationship representation modeling module taking the relationship feature of the relationship to be predicted as input and taking the relationship representation of the relationship to be predicted as output.

[0210] The relationship loss weighting module 806 is configured to construct a relationship loss weighting module for generating a label distribution of the relationship representation of the relationship to be predicted, performing loss calculation in combination with the label of the real relationship, weighting the loss of the relationship of each image sample with the importance score of the real relationship, and performing scene graph generation training.

[0211] The scene graph generation model training module 807 is configured to form a to-be-trained network structure from the input to the output direction, the target detector, the semantic extractor, the instance feature refinement module, the bounding box modeling module, the feature splicing module, the feature fusion module, the relationship representation modeling module, and the relationship loss weighting module; based on a target data set, taking an image sample as input and a scene graph as output, the to-be-trained network structure is trained to obtain a scene graph generation model; in the case that the number of training image samples reaches the batch processing size, the target detector construction module is returned; in the case that all image samples in the training set are read, the scene graph generation model output module is entered.

[0212] The scene graph generation model output module 808 is configured to output the scene graph generation model.

[0213] In an exemplary embodiment, as shown in Figure 9 a scene graph generation model testing device is provided, comprising:

[0214] The relationship prediction module 901 is configured to input the image sample in the test set into the scene graph generation model taking the image sample as input and the scene graph as output for relationship prediction.

[0215] The comparative result evaluation module 902 is configured to compare the relationship prediction result with the real predicate label, and evaluate the comparison result using a target evaluation index;

[0216] The prediction result evaluation module 903 is configured to introduce a target prediction index to evaluate the result of relationship prediction;

[0217] The scene graph generation module 904 is configured to screen out important relationships in the scene graph generation model relationship prediction result to generate a scene graph;

[0218] The scene graph output module 905 is configured to output the generated scene graph.

[0219] The above-mentioned significant scene graph generation device based on the relationship importance of fuzzy logic reasoning and the scene graph generation model test device can be realized by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned modules.

[0220] For those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims. < / man-riding-bike>

Claims

1. A significant scene graph generation method based on the importance of fuzzy logic inference relationship, comprising: Step S100, using fuzzy logic to calculate the importance score of the real relationship of each image sample in the target data set, and dividing the target data set into a training set and a test set, wherein each image sample comprises a subject object and an object object; Wherein the step S100 using fuzzy logic to calculate the importance score of the real relationship of each image sample in the target data set comprises: According to the influence factors of the importance of the relationship, the relationship area in the target data set, the predicate frequency and the user-specified instance class, predicate class and attribute class data are preprocessed as input parameters of the fuzzy inference system; Divide the fuzzy set of the input parameters of the fuzzy inference system; Formulate the membership function, and convert the input parameters of the fuzzy inference system into fuzzy values through the membership function; Define the fuzzy rules according to the factors affecting the importance of the relationship, and evaluate the fuzzy values according to the fuzzy rules to generate the fuzzy output of each fuzzy rule; The fuzzy output of the fuzzy rule is aggregated by the fuzzy inference system to generate the fuzzy output parameter, and the fuzzy set of the fuzzy output parameter is converted into the importance score through the centroid method defuzzification process; Step S200, constructing a target detector taking an image sample as input and taking the label of the relationship instance to be predicted between the objects in the image sample and the boundary box information corresponding to the relationship instance to be predicted as output; Step S300, constructing a semantic extractor taking the label of the relationship instance to be predicted between the objects in the image sample as input, and taking the instance features of the subject object and the instance features of the object object as output, and constructing an instance feature refinement module to refine the instance features of the subject object and the instance features of the object object; Step S400, constructing a boundary box modeling module taking the boundary box information corresponding to the relationship instance to be predicted between the objects in the image sample as input, and taking the boundary box features of the subject object and the boundary box features of the object object as output, and constructing a feature splicing module that splices the boundary box features of the subject object and the instance features of the subject object to output the instance representation of the subject object, and splices the boundary box features of the object object and the instance features of the object object to output the instance representation of the object object; Step S500, constructing a feature fusion module taking the instance representation of the subject object and the instance representation of the object object as input, and taking the relationship features of the relationship to be predicted as output, and constructing a relationship representation modeling module taking the relationship features of the relationship to be predicted as input, and taking the relationship representation of the relationship to be predicted as output; Step S600, constructing a relationship loss weighting module for generating the label distribution of the relationship representation of the relationship to be predicted, combining the label of the real relationship for loss calculation, taking the importance score of the real relationship as a weight to weight the loss of the relationship of each image sample, and performing scene graph generation training; Step S700, from input to output direction, the target detector, the semantic extractor, the instance feature refinement module, the bounding box modeling module, the feature splicing module, the feature fusion module, the relationship representation modeling module, and the relationship loss weighting module constitute a to-be-trained network structure; based on the target data set, taking an image sample as input and a scene graph as output, the to-be-trained network structure is trained to obtain a scene graph generation model; in the case that the number of training image samples reaches the batch processing size, return to step S200; in the case that all image samples in the training set are read, enter step S800; Step S800, output the scene graph generation model.

2. The method of claim 1, wherein, The step S400 comprises: The coordinates of the upper left corner of the bounding box of the instance, the coordinates of the lower right corner of the bounding box, and the area of the bounding box are spliced by column, and the bounding box feature is generated by modeling; The instance feature and the bounding box feature are spliced to form an instance representation by modeling.

3. The method of claim 1, wherein, In the step S600, the importance score of the real relationship is used as a weight to weight the loss of the relationship of each image sample, comprising: The semantic extractor is used to extract the predicate feature in the target data set; The refined predicate feature is generated by modeling; The relationship representation and the predicate feature are standardized; The label distribution of the to-be-predicted relationship and the loss of the real relationship label are calculated by a weighted cross-entropy loss function.

4. A scene graph generation model testing method for testing the scene graph generation model according to any one of claims 1 to 3, comprising: Step S100, input the image sample in the test set into the scene graph generation model taking the image sample as input and the scene graph as output for relationship prediction; Step S200, compare the relationship prediction result with the real predicate label, and evaluate the comparison result using a target evaluation index; Step S300, introduce a target prediction index to evaluate the result of relationship prediction; Step S400, screen out important relationships in the scene graph generation model relationship prediction result to generate a scene graph; Step S500, output the generated scene graph.

5. The method of claim 4, wherein, The step S300 comprises: Introducing the important relationship recall rate and the important relationship accuracy rate to evaluate the result of relationship prediction.

6. The method of claim 4, wherein, The step S400 further comprises: Calculate the importance score of each image sample relationship in the test set, and sort the predicted relationships from large to small according to the importance score, wherein the relationship with an importance score greater than 0.5 is an important relationship.

7. A significant scene graph generation device based on fuzzy logic reasoning relationship importance, for implementing the method according to any one of claims 1-3, comprising: An importance score calculation module for calculating the importance score of each image sample real relationship in the target data set using fuzzy logic, and dividing the target data set into a training set and a test set, wherein each image sample includes a subject object and an object object; A target detector construction module for constructing a target detector taking an image sample as input and outputting the label of each object relationship instance to be predicted in the image sample and the bounding box information of the relationship instance to be predicted; An instance feature extraction refinement module is configured to construct a semantic extractor taking labels of relationship instances to be predicted between objects in an image sample as input and taking instance features of a subject object and instance features of an object object as output, and to construct an instance feature refinement module to refine the instance features of the subject object and the instance features of the object object; An instance representation generation module is configured to construct a bounding box modeling module taking bounding box information of relationship instances to be predicted between objects in an image sample as input and taking bounding box features of a subject object and bounding box features of an object object as output, and to construct a feature splicing module to splice the bounding box features of the subject object and the instance features of the subject object to output an instance representation of the subject object and to splice the bounding box features of the object object and the instance features of the object object to output an instance representation of the object object; A relationship representation generation module is configured to construct a feature fusion module taking the instance representation of the subject object and the instance representation of the object object as input and taking relationship features of a relationship to be predicted as output, and to construct a relationship representation modeling module taking the relationship features of the relationship to be predicted as input and taking a relationship representation of the relationship to be predicted as output; A relationship loss weighting module is configured to construct a relationship loss weighting module for generating a label distribution of a relationship representation of a relationship to be predicted, performing loss calculation in combination with a label of a real relationship, weighting and summing the loss of the relationship of each image sample with an importance score of the real relationship as a weight, and performing scene graph generation training; A scene graph generation model training module is configured to form a network structure to be trained from input to output by the target detector, the semantic extractor, the instance feature refinement module, the bounding box modeling module, the feature splicing module, the feature fusion module, the relationship representation modeling module, and the relationship loss weighting module; based on a target data set, taking an image sample as input and a scene graph as output, training the network structure to be trained to obtain a scene graph generation model; and returning the target detector construction module when the number of training image samples reaches a batch processing size; When all image samples in the training set are read, a scene graph generation model output module is entered; The scene graph generation model output module is configured to output the scene graph generation model.

8. A scene graph generation model testing device for testing the scene graph generation model according to any one of claims 1 to 3, comprising: A relationship prediction module is configured to input an image sample in a test set into a scene graph generation model taking an image sample as input and a scene graph as output to perform relationship prediction; A comparison result evaluation module is configured to compare the relationship prediction result with a real predicate label and evaluate the comparison result using a target evaluation index; A prediction result evaluation module is configured to introduce a target prediction index to evaluate the result of relationship prediction; A scene graph generation module is configured to screen out important relationships in the relationship prediction result of the scene graph generation model to generate a scene graph; A scene graph output module is configured to output the generated scene graph. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

Images