Salient scene graph generation method based on importance of fuzzy logic reasoning relationship

Abstract

The invention relates to a salient scene graph generation method and device based on fuzzy logic reasoning relation importance, a scene graph generation model test method and device and computer equipment. The method comprises the following steps: step S100, calculating an importance score of a real relationship of each image sample in a target data set; step S200, constructing a target detector; step S300, constructing a semantic extractor and an instance feature refining module; step S400, constructing a bounding box modeling module and a feature splicing module; step S500, constructing a feature fusion module and a relation representation modeling module; step S600, constructing a relation loss weighting module; step S700, returning to the step S200 when the number of the training image samples reaches the batch processing size; and under the condition that all the image samples in the training set are read, entering the step S800 to output the scene graph generation model. According to the method, the importance of the relationship can be flexibly and comprehensively evaluated on the basis of not reducing the number of the relationship samples.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and computer vision technology, and in particular to a method and device for generating a significant scene graph based on the importance of fuzzy logic reasoning relationships, a method and device for testing a scene graph generation model, and a computer device. Background Art

[0002] Scene graph generation is a cutting-edge topic in the field of artificial intelligence and one of the key tasks in computer vision. As an intermediate task, scene graph helps connect the upstream object detection task and the downstream high-level visual understanding task, and its effect depends largely on the model's prediction of important relationships.

[0003] Although most current scene graph generation models have made progress in reducing relation prediction bias by balancing predicate distribution, using resampling techniques, and enhancing semantic information, they often output too many unnecessary relations that do not necessarily reflect the main content of the image. This approach may cause redundancy for downstream tasks that use scene graphs, such as image caption generation, affecting the focus on the key content of the image, thereby reducing the processing efficiency of downstream tasks. In addition, traditional salient scene graph generation methods rely on the training of fixed data sets and fail to adapt to the actual needs of diverse downstream tasks, limiting the scope of application of the model. Summary of the invention

[0004] Based on this, it is necessary to provide a significant scene graph generation method and device based on fuzzy logic reasoning of the importance of relationships, a scene graph generation model testing method and device, and computer equipment to address the above technical problems, which can accurately predict the importance of relationships and improve the ability of scene graph generation models to predict important relationships.

[0005] In a first aspect, the present application provides a method for generating a salient scene graph based on fuzzy logic reasoning relationship importance, comprising:

[0006] Step S100, using fuzzy logic to calculate the importance score of the true relationship of each image sample in the target data set, and dividing the target data set into a training set and a test set, wherein each image sample includes a subject object and an object object;

[0007] Step S200, constructing an object detector that takes an image sample as input and takes labels of the to-be-predicted relationship instances between objects in the image sample and bounding box information corresponding to the to-be-predicted relationship instances as output;

[0008] Step S300, constructing a semantic extractor that takes the labels of the relationship instances to be predicted between the objects in the image sample as input and the instance features of the subject object and the instance features of the object object as output, and constructing an instance feature refinement module to refine the instance features of the subject object and the instance features of the object object;

[0009] Step S400, constructing a bounding box modeling module that takes the bounding box information corresponding to the to-be-predicted relationship instance between each object in the image sample as input and takes the bounding box features of the subject object and the bounding box features of the object object as output, and constructing a feature splicing module that splices the bounding box features of the subject object and the instance features of the subject object to output the instance representation of the subject object, and splices the bounding box features of the object object and the instance features of the object object to output the instance representation of the object object;

[0010] Step S500, constructing a feature fusion module that takes the instance representation of the subject object and the instance representation of the object object as input and the relational features of the to-be-predicted relation as output, and constructing a relational representation modeling module that takes the relational features of the to-be-predicted relation as input and the relational representation of the to-be-predicted relation as output;

[0011] Step S600, constructing a relationship loss weighting module for generating a label distribution of a relationship representation of a relationship to be predicted, performing loss calculation in combination with the label of the real relationship, weighting the loss of the relationship of each image sample by taking the importance score of the real relationship as a weight, and performing scene graph generation training;

[0012] Step S700, from input to output direction, the target detector, semantic extractor, instance feature refinement module, bounding box modeling module, feature splicing module, feature fusion module, relationship representation modeling module, and relationship loss weighting module constitute the network structure to be trained; based on the target data set, with image samples as input and scene graph as output, the network structure to be trained is trained to obtain a scene graph generation model; when the number of training image samples reaches the batch processing size, return to step S200; when all image samples in the training set are read, enter step S800;

[0013] Step S800, outputting a scene graph generation model.

[0014] In one embodiment, the step S100 of using fuzzy logic to calculate the importance score of the true relationship of each image sample in the target data set includes:

[0015] According to the factors affecting the importance of the relationship, the relationship area, predicate frequency, and the instance class, predicate class, and attribute class data specified by the user in the target data set are preprocessed as input parameters of the fuzzy reasoning system;

[0016] Partition fuzzy sets for input parameters of fuzzy inference system;

[0017] Formulate a membership function, and convert the input parameters of the fuzzy inference system into fuzzy values ​​through the membership function;

[0018] Define fuzzy rules according to the factors that affect the importance of the relationship, evaluate the fuzzy values ​​according to the fuzzy rules, and generate the fuzzy output of each fuzzy rule;

[0019] The fuzzy outputs of fuzzy rules are aggregated through a fuzzy inference system to generate fuzzy output parameters, and the fuzzy set of fuzzy output parameters is converted into importance scores through the centroid method defuzzification process.

[0020] In one embodiment, step S400 includes:

[0021] The coordinates of the upper left corner, the coordinates of the lower right corner, and the area of ​​the bounding box of the instance are concatenated in columns, and the bounding box features are generated through modeling;

[0022] The instance features and bounding box features are concatenated through modeling to form instance representation.

[0023] In one embodiment, in step S600, using the importance score of the true relationship as a weight to weight the loss of the relationship of each image sample includes:

[0024] A semantic extractor is used to extract predicate features in the target dataset;

[0025] Generate refined predicate features through modeling;

[0026] Standardize the relation representation and predicate features;

[0027] The weighted cross entropy loss function is used to calculate the loss of the label distribution of the relationship to be predicted and the true relationship label.

[0028] In a second aspect, the present application further provides a scene graph generation model testing method, which is used to test the scene graph generation model provided by any embodiment of the first aspect, including:

[0029] Step S100, inputting image samples in the test set into a scene graph generation model that takes image samples as input and scene graphs as output to perform relationship prediction;

[0030] Step S200, comparing the relationship prediction result with the true predicate label, and evaluating the comparison result using the target evaluation index;

[0031] Step S300, introducing a target prediction index to evaluate the result of relationship prediction;

[0032] Step S400, filtering out important relationships in the scene graph generation model relationship prediction results to generate a scene graph;

[0033] Step S500: output the generated scene graph.

[0034] In one embodiment, step S300 includes:

[0035] The important relationship recall rate and important relationship accuracy are introduced to evaluate the results of relationship prediction.

[0036] In one embodiment, before step S400, the method further includes:

[0037] The importance score of each image sample relationship in the test set is calculated, and the predicted relationships are sorted from large to small according to the importance score, where the relationships with importance scores greater than 0.5 are regarded as important relationships.

[0038] In a third aspect, the present application also provides a device for generating a significant scene graph based on the importance of fuzzy logic reasoning relationships, comprising:

[0039] An importance score calculation module is used to calculate the importance score of the true relationship of each image sample in the target data set using fuzzy logic, and divide the target data set into a training set and a test set, wherein each image sample includes a subject object and an object object;

[0040] An object detector construction module is used to construct an object detector that takes an image sample as input and takes labels of the relationship instances to be predicted between objects in the image sample and bounding box information corresponding to the relationship instances to be predicted as output;

[0041] An instance feature extraction and refinement module is used to construct a semantic extractor that takes the labels of the relationship instances to be predicted between the objects in the image sample as input and the instance features of the subject object and the instance features of the object object as output, and to construct an instance feature refinement module to refine the instance features of the subject object and the instance features of the object object;

[0042] An instance representation generation module is used to construct a bounding box modeling module that takes the bounding box information corresponding to the to-be-predicted relationship instance between each object in the image sample as input and takes the bounding box features of the subject object and the bounding box features of the object object as output, and to construct a feature splicing module that splices the bounding box features of the subject object and the instance features of the subject object to output the instance representation of the subject object, and splices the bounding box features of the object object and the instance features of the object object to output the instance representation of the object object;

[0043] A relation representation generation module is used to construct a feature fusion module that takes the instance representation of the subject object and the instance representation of the object object as input and the relational features of the to-be-predicted relation as output, and to construct a relation representation modeling module that takes the relational features of the to-be-predicted relation as input and the relational representation of the to-be-predicted relation as output;

[0044] A relation loss weighting module, which is used to construct a label distribution for generating a relation representation of a relation to be predicted, calculate the loss in combination with the label of the true relation, use the importance score of the true relation as a weight to weight the loss of the relation of each image sample, and perform scene graph generation training;

[0045] The scene graph generation model training module is used for the direction from input to output. The target detector, semantic extractor, instance feature refinement module, bounding box modeling module, feature splicing module, feature fusion module, relationship representation modeling module, and relationship loss weighting module constitute the network structure to be trained; based on the target data set, with image samples as input and scene graphs as output, the network structure to be trained is trained to obtain the scene graph generation model; when the number of training image samples reaches the batch processing size, the target detector construction module is returned; when all image samples in the training set are read, the scene graph generation model output module is entered;

[0046] The scene graph generation model output module is used to output the scene graph generation model.

[0047] In a fourth aspect, the present application further provides a scene graph generation model testing device, which is used to test the scene graph generation model provided in the first aspect, comprising:

[0048] A relationship prediction module, used to input the image samples in the test set into a scene graph generation model that takes the image samples as input and the scene graph as output to perform relationship prediction;

[0049] A comparison result evaluation module is used to compare the relationship prediction results with the true predicate labels and evaluate the comparison results using the target evaluation index;

[0050] A prediction result evaluation module is used to introduce target prediction indicators to evaluate the results of relationship prediction;

[0051] A scene graph generation module is used to filter out important relationships in the relationship prediction results of the scene graph generation model to generate a scene graph;

[0052] The scene graph output module is used to output the generated scene graph.

[0053] In a fifth aspect, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of a significant scene graph generation method based on the importance of fuzzy logic reasoning relationships and a scene graph generation model testing method.

[0054] The above-mentioned significant scene graph generation method and device, scene graph generation model testing method and device, and computer equipment can flexibly and comprehensively evaluate the importance of relationships without reducing the number of relationship samples by using fuzzy logic to calculate the importance score of the true relationship of each image sample in the target data set based on the factors affecting the importance of the relationship. The target data set is divided into a training set and a test set, where each image sample includes a subject object and an object object. The training set is used to train the scene graph generation model, and the test set is used to test the scene graph generation model. Construct an object detector that takes image samples as input and takes the labels of the relationship instances to be predicted between each object in the image sample and the bounding box information corresponding to the relationship instances to be predicted as output; construct a semantic extractor that takes the labels of the relationship instances to be predicted between each object in the image sample as input and the instance features of the subject object and the instance features of the object object as output, and construct an instance feature refinement module to refine the instance features of the subject object and the instance features of the object object; construct a bounding box modeling module that takes the bounding box information corresponding to the relationship instances to be predicted between each object in the image sample as input and takes the bounding box features of the subject object and the bounding box features of the object object as output, construct a bounding box modeling module that concatenates the bounding box features of the subject object and the instance features of the subject object to output the instance representation of the subject object, and concatenates the bounding box features of the object object and the bounding box features of the object object to output the instance representation of the subject object. A feature splicing module that splices instance features of objects and outputs instance representations of object objects; a feature fusion module that takes instance representations of subject objects and instance representations of object objects as inputs and relational features of the relationship to be predicted as outputs, and a relational representation modeling module that takes relational features of the relationship to be predicted as inputs and relational representations of the relationship to be predicted as outputs; a relational loss weighting module that constructs a label distribution for generating relational representations of the relationship to be predicted, calculates losses in combination with the labels of the true relationships, uses the importance scores of the true relationships as weights to weight the loss of the relationship of each image sample, and performs scene graph generation training; a plug-and-play relational loss weighting module is used to weight the loss of each relation sample by using the relational importance score as a weight, so that the model pays more attention to important relationships.From input to output, the target detector, semantic extractor, instance feature refinement module, bounding box modeling module, feature splicing module, feature fusion module, relationship representation modeling module, and relationship loss weighting module constitute the network structure to be trained; based on the target data set, with image samples as input and scene graphs as output, the network structure to be trained is trained to obtain a scene graph generation model, and when the number of training image samples reaches the batch processing size, the step of constructing a target detector with image samples as input and labels of the relationship instances to be predicted between objects in the image samples and the bounding box information corresponding to the relationship instances to be predicted as output is returned and the above iterative training steps are repeated; when all image samples in the training set have been read, the scene graph generation model is output to improve the ability of the scene graph generation model to predict important relationships, so that key information can be more effectively extracted and analyzed from a large number of social interaction images. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of the annotation relationship between a target data set and a given image in one embodiment;

[0056] Figure 2 is a flow chart of a method for generating a salient scene graph based on fuzzy logic reasoning relationship importance in one embodiment;

[0057] Figure 3 The overall framework diagram of a fuzzy logic-based relational loss weighted scene graph generation network FLRW-SGG in one embodiment is shown;

[0058] Figure 4 A schematic diagram of a process for using fuzzy logic to infer relationship importance scores in one embodiment;

[0059] Figure 5 A schematic diagram of a module of a relation loss weighting module in a method for generating a salient scene graph based on fuzzy logic reasoning relation importance in one embodiment;

[0060] Figure 6 A schematic diagram of a flow chart of a scene graph generation model testing method in one embodiment;

[0061] Figure 7 is a training flow chart of FLRW-SGG in one embodiment;

[0062] Figure 8 A schematic diagram of modules of a device for generating a salient scene graph based on fuzzy logic reasoning relationship importance in one embodiment;

[0063] Fig. 9 A schematic diagram of modules of a scene graph generation model testing device in one embodiment. DETAILED DESCRIPTION

[0064] Scene graphs combine image recognition and natural language processing techniques to identify and classify various instances from images and predict the visual relationships between these instances. These instances and relationships are usually described by nouns and predicates and output in the form of triples. Figure 1 As shown in the figure, “man-riding-bike” is an example of a triple, where “man” is the subject instance, “bike” is the object instance, and “riding” is the relationship between them. By combining all the triples in the image, we can construct a scene graph for the image. The scene graph is shown in Figure 1 As shown in the lower right corner, scene graph generation, as an intermediate task, helps connect the upstream object detection task and the downstream high-level visual understanding tasks, such as image caption generation, visual navigation, image retrieval, human pose estimation, and visual question answering.

[0065] The traditional two-stage scene graph generation method first uses an object detector to perform instance recognition and feature extraction on the input image. Then, in the second stage, the model combines the features of the subject and the object to predict the relationship between them. Although most current scene graph generation models have made progress in reducing relationship prediction bias by balancing predicate distribution, using resampling techniques, and enhancing semantic information, they often output too many unnecessary relationships that do not necessarily reflect the main content of the image. For example, the vast majority of models use the most popular Visual Genome 150 (VG150) dataset for scene graph generation. Some of the instances and relationships annotated in this dataset are not important for understanding the main content of the entire image scene. The model cannot judge important relationships during training, but instead treats all relationships equally. Figure 1 As shown in the figure, for the given picture on the left, the upper right corner shows the relationship between the VG150 dataset and the given image annotation, where <man-riding-bike> 、 <man-near-train>The most important thing for understanding the main content of the image is the scene graph in the lower right corner, but all relationships are treated equally and all are output. This approach may cause redundancy for downstream tasks such as image caption generation that use scene graphs, affecting the focus on the key content of the image, thereby reducing the processing efficiency of downstream tasks.

[0066] In order to enable the model to predict important relationships and make the scene graph better serve downstream tasks, researchers began to study salient scene graph generation methods, which can focus on identifying salient relationships (also known as important relationships or key relationships) in images. For example, the Significance-guided Message Passing (SMP) method uses an object interaction encoder to enhance the object relationship representation by jointly utilizing the appearance, semantics, and spatial relationships between objects, and further estimates the saliency of each visual relationship through ordinal regression. Conditioned on object and relationship features, SMP enhances scene graph generation by performing message passing on objects and salient relationships. It was trained and tested on the VG-KR (Visual Genome Keyword Recognition) dataset with salient relationship annotations, but the dataset is too absolute in dividing key relationships. It divides relationships into key relationships and unimportant relationships based on factors such as the image's caption content, visually focused color contrast, center position, and instance size, and only retains annotations of key relationships in the dataset for training. For models using this dataset, the reduction in samples may make it unable to fully learn. In fact, scene graphs serve as a bridge between object detection and downstream tasks. Downstream tasks in different fields pay different attention to instances and relationships in images. For example, in urban traffic research, the focus is on traffic-related elements such as vehicles and traffic lights; while in social behavior research, more attention is paid to the interaction between people. Therefore, if the scene graph generation model is only trained on VG-KR, the reduction of samples will affect the accuracy of the model's relationship prediction, and it is not enough to meet the application needs of different research fields.

[0067] Scene graph generation is helpful for downstream high-level visual understanding tasks, and its effectiveness depends largely on the model's prediction of important relationships. If the importance of relationships can be accurately predicted, scene graphs will play a greater role in downstream tasks. Existing salient scene graph generation methods rely on training with fixed data sets, fail to adapt to the actual needs of diverse downstream tasks, and limit the scope of application of the model. In order to solve this problem, the present invention proposes a salient scene graph generation method based on fuzzy logic reasoning of relationship importance.

[0068] In an exemplary embodiment, Figure 2 As shown, a method for generating a salient scene graph based on fuzzy logic reasoning relationship importance is provided, and the method includes:

[0069] Step S100, using fuzzy logic to calculate the importance score of the true relationship of each image sample in the target data set, and dividing the target data set into a training set and a test set, wherein each image sample includes a subject object and an object object.

[0070] Specifically, the target dataset includes the VG (Visual Genome) 150 dataset.

[0071] Exemplarily, fuzzy logic is used to calculate the importance score of the real relationship in the VG150 data set, and the process proceeds to step S200.

[0072] Step S200 , constructing an object detector that takes an image sample as input and takes labels of the to-be-predicted relationship instances between objects in the image sample and bounding box information corresponding to the to-be-predicted relationship instances as output.

[0073] Preferably, the training images in step S200 are all from the standard dataset VG150 generated by scene graphs. The dataset consists of 108,077 images, including the 150 most common instance classes and 50 predicate classes. In this embodiment, 70% of the images in the dataset are divided into a training set and the remaining 30% of the images are divided into a test set. The target detector uses Faster R-CNN, and the instance label and the corresponding bounding box information are obtained through the target detector.

[0074] Exemplarily, the training images in the VG150 dataset are input into the object detector to generate instance labels of the relationship to be predicted and corresponding bounding box information, and then proceed to step S300.

[0075] Step S300, construct a semantic extractor that takes the labels of the relationship instances to be predicted between the objects in the image sample as input and the instance features of the subject object and the instance features of the object object as output, and construct an instance feature refinement module to refine the instance features of the subject object and the instance features of the object object.

[0076] Exemplarily, a semantic extractor is used to extract instance features of the subject and the object, which are further modeled to generate refined subject and object instance features, and then the process proceeds to step S400.

[0077] Step S400, construct a bounding box modeling module that takes the bounding box information corresponding to the relationship instance to be predicted between each object in the image sample as input and takes the bounding box features of the subject object and the bounding box features of the object object as output, and construct a feature splicing module that splices the bounding box features of the subject object and the instance features of the subject object to output the instance representation of the subject object, and splices the bounding box features of the object object and the instance features of the object object to output the instance representation of the object object.

[0078] Specifically, the bounding box information includes bounding box coordinates and area.

[0079] Exemplarily, the bounding box coordinates and areas of the subject and the object are modeled respectively to generate bounding box features of the subject and the object, and the bounding box features are concatenated with the instance features to form instance representations of the subject and the object, and then the process proceeds to step S500.

[0080] Step S500, construct a feature fusion module that takes the instance representation of the subject object and the instance representation of the object object as input and the relational features of the relationship to be predicted as output, and construct a relational representation modeling module that takes the relational features of the relationship to be predicted as input and the relational representation of the relationship to be predicted as output.

[0081] Exemplarily, the instance representations of the subject and the object are fused to obtain the relationship features, and the relationship features are further modeled to generate the relationship representation, and then the process proceeds to step S600.

[0082] Step S600, construct a relationship loss weighting module for generating a label distribution of a relationship representation of a relationship to be predicted, combining the label of the real relationship to calculate the loss, using the importance score of the real relationship as a weight to weight the loss of the relationship of each image sample, and performing scene graph generation training.

[0083] Exemplarily, the relationship loss is weighted. The label distribution of the relationship representation in step S500 is generated, and the loss is calculated with the real relationship label, and the relationship importance score in step S100 is used as a weight to weight the loss of each relationship sample, and then the scene graph generation training is performed, and the process proceeds to step S700.

[0084] Step S700, from input to output direction, the target detector, semantic extractor, instance feature refinement module, bounding box modeling module, feature splicing module, feature fusion module, relationship representation modeling module, and relationship loss weighting module constitute the network structure to be trained; based on the target data set, with image samples as input and scene graph as output, the network structure to be trained is trained to obtain a scene graph generation model; when the number of training image samples reaches the batch processing size, return to step S200; when all image samples in the training set are read, enter step S800;

[0085] Exemplarily, if the set batch processing (batch-size) size is reached, return to step S200; if all training images are read, go to step S800.

[0086] Step S800, outputting a scene graph generation model.

[0087] Exemplarily, the trained model is output and the process ends.

[0088] The above-mentioned significant scene graph generation method based on fuzzy logic reasoning relationship importance uses fuzzy logic to calculate the importance score of the real relationship in the VG150 dataset according to factors affecting the importance of the relationship, such as the area of ​​the relationship region, the frequency of the predicate, and the data flexibly specified by the user, such as the specified instance class, predicate class, and attribute class. It can flexibly and comprehensively evaluate the importance of the relationship without reducing the number of relationship samples. The training set is input into the object detector, and the importance score of the real relationship is used as a weight to weight the loss of the relationship of each image sample in the training set through multiple steps such as generating instance labels of the relationship to be predicted, corresponding bounding box information, instance features, instance representation, and relationship representation, and the scene graph generation model is iteratively trained. By designing a plug-and-play relationship loss weighting module, the relationship importance score is used as a weight to weight the loss of each relationship sample, so that the scene graph generation model pays more attention to important relationships. This relationship loss weighting module can be applied to the scene graph generation baseline model with excellent performance, which can improve the ability of the scene graph generation model to predict important relationships.

[0089] In one embodiment, the present application proposes a fuzzy logic-based relation loss weighted scene graph generation network FLRW-SGG (Fuzzy Logic Relation Loss Weighted Scene Graph Generation Network), the overall framework of which is shown in FIG. Figure 3 As shown in the figure, fuzzy logic is used to calculate the importance score of the real relationship in the VG150 dataset, and the training image is input into Faster R-CNN to obtain the predicted instance class label and the predicted instance bounding box; semantic extraction and feature refinement are performed based on the predicted instance class label to obtain the instance feature of the subject and the instance feature of the object, and bounding box modeling is performed based on the predicted instance bounding box to further obtain the bounding box feature of the subject and the bounding box feature of the object; the bounding box feature and the instance feature are spliced ​​to form the instance representation of the subject and the object, and the instance representation is modeled and feature fused to obtain the relationship feature, and the relationship feature is modeled to generate the relationship representation, and the relationship representation is input into the relationship loss weighting module. The VG150 dataset is preprocessed and input into the fuzzy logic reasoning module to obtain the relationship importance score, and the salient scene graph is obtained based on the relationship importance score and the relationship loss weighting module, including "woman", "laptop", "pant", "chair", "shelf", "holding", "wearing", "sitting on", "behind".

[0090] Furthermore, the process of using fuzzy logic to infer the relationship importance score is as follows Figure 4 As shown in the figure, the parameters of the relationship region area, predicate frequency and specified attribute class are used as the input values ​​of the Gaussian function to obtain the fuzzy set; the two parameters of the specified instance class and the specified predicate class are used as the input values ​​of the S-type function to obtain the fuzzy set, define the fuzzy rules and perform fuzzy rule reasoning to generate fuzzy output parameters; the output value is obtained by the centroid method to defuzzify the process, which is the relationship importance score.

[0091] Furthermore, the relation loss weighting module is as follows Figure 5 As shown. The semantic extractor is used to extract semantics from the predicate label to obtain the predicate features; the predicate features are refined and matrix multiplied with the relationship representation to obtain the label distribution of the predicted relationship. The weighted cross entropy loss function is combined with the relationship importance score to calculate the loss of the predicted relationship label distribution and the true relationship label. Among them, Represents matrix multiplication.

[0092] In an exemplary embodiment, the use of fuzzy logic in step S100 to calculate the importance score of the true relationship of each image sample in the target data set includes: preprocessing the relationship region area, predicate frequency, and user-specified instance class, predicate class, and attribute class data in the target data set as input parameters of the fuzzy inference system according to factors affecting the relationship importance; dividing the input parameters of the fuzzy inference system into fuzzy sets; formulating a membership function, and converting the input parameters of the fuzzy inference system into fuzzy values ​​through the membership function; defining fuzzy rules according to factors affecting the relationship importance, and evaluating the fuzzy values ​​according to the fuzzy rules to generate fuzzy outputs for each fuzzy rule; aggregating the fuzzy outputs of the fuzzy rules through the fuzzy inference system to generate fuzzy output parameters, and converting the fuzzy sets of the fuzzy output parameters into importance scores through a centroid method defuzzification process.

[0093] Exemplarily, the steps of using fuzzy logic to calculate the importance score of the real relationship in the VG150 dataset include:

[0094] Step S101, based on the factors affecting the importance of the relationship, the relationship area, predicate frequency, and the instance class, predicate class, and attribute class data specified by the user in the VG150 data set are preprocessed as input parameters of the fuzzy inference system. The processing process includes:

[0095] For the relationship region area, the area of ​​each instance is calculated based on the instance bounding box information in the VG150 dataset, and normalized to eliminate the influence of image size; the relationship region area is obtained by the weighted average of the subject and object instance areas; the predicate frequency needs to be normalized to a numerical range of 0-1 due to the large difference in the number of times it appears in the dataset, so as to reduce the deviation of high-frequency predicates on the analysis results; for specific instances and predicate categories, this embodiment adopts binary classification labeling: if the subject or object in the relationship belongs to a preset category, the relationship is marked as 1, otherwise it is 0; for the attribute class, considering that each instance may have multiple attributes, the attribute richness of the relationship is determined by calculating the total number of specific attribute categories contained in the subject and object instances in each relationship.

[0096] Step S102, divide the fuzzy sets for the input parameters of step S101. The fuzzy set of the relationship region area (RA) is divided into three subsets: {larger area (LR), medium area (MR), smaller area (SR)}; the fuzzy set of the predicate frequency (PF) is {higher (HF), medium frequency (MF), lower frequency (LF)}; the fuzzy set of the specified instance class (SO) is divided into two subsets: {include (IN), not include (NI)}; the fuzzy set of the specified predicate class (SP) is {belongs to (BE), does not belong to (NB)}; the fuzzy set of the specified attribute class (SA) is divided into three subsets: {more attributes (MA), medium attributes (MD), fewer attributes (FA)}.

[0097] In addition, relation importance (RI) is used as the output parameter of the fuzzy inference system, and the fuzzy set is divided into three subsets: {important (HI), moderately important (MI), and not important (LI)}.

[0098] Step S103, formulate a suitable membership function to convert the input parameters into fuzzy values.

[0099] Specifically, for the parameters of relational area, predicate frequency and specified attribute class, their fuzzy sets obviously present different levels of division, and Gaussian function (Gaussmf) is used as the membership function μ of these fuzzy sets. gaussmf (X 1 ; a, β), as shown in formula (1), where X 1 ∈{RA,PF,SA} is the area of ​​the relation region, the frequency of the predicate and the specified attribute class parameters, a is a hyperparameter that reflects the peak value of the Gaussian function; β is a hyperparameter used to represent the decay speed of the Gaussian function from the peak value to the outside. The values ​​of a and β are determined according to the numerical range of each parameter.

[0100]

[0101] Furthermore, for the two input parameters of the specified instance class and the specified predicate class, their fuzzy sets represent two extreme states of yes and no, respectively, and the S-type function (Sigmf) has the characteristics of a continuous curve and can provide a smooth transition between the two states. It is used as the membership function μ of the two input parameters. sigmf (X 2 ; h, k), as shown in formula (2), where X 2 ∈{SO,SP} is a parameter that specifies the instance class and predicate class. ε and h are hyperparameters that reflect the middle position and steepness of the S-curve, respectively. The larger h is, the faster the function transitions near ε. The values ​​of ε and h are determined according to the numerical range of each parameter.

[0102]

[0103] Step S104, based on the factors that affect the importance of the relationship, define the fuzzy rules shown below. The fuzzy rules will evaluate the fuzzy value of step S103 and generate a fuzzy output for each rule.

[0104] (1)if(SO=IN)∪(SP=BE)∪(SA=MA),ThenRI=HI

[0105] (2)if(SO=NI)∩(SP=NB)∩(SA=FA),ThenRI=LI

[0106] (3)if(RA=LA)∩(SA=MD),ThenRI=HI

[0107] (4)if(PF=HF)∩(SA=MD),ThenRI=HI

[0108] (5)if(PF=LF)∪(PF=MF)∪(PF=HF),ThenRI=MI

[0109] Among them, rule (1) indicates that if the subject or object in the relationship belongs to the specified instance category, or the relationship belongs to the specified predicate category, or the number of attributes in the relationship is large, then the importance of the relationship is "important"; rule (2) indicates that when there is no specified instance predicate category in the relationship and the number of attributes is small, then the importance of the relationship is "unimportant"; rule (3) indicates that if the area of ​​the relationship is large and the number of attributes is moderate, then the importance of the relationship is "important"; rule (4) indicates that if the predicate frequency in the relationship is low and the number of attributes is moderate, then the importance of the relationship is also "important"; rule (5) is the default rule, which ensures that when other factors do not provide sufficient decisive information, that is, when the above rules are not met, the system can assign the relationship importance as "medium important".

[0110] Step S105, the fuzzy inference system aggregates the outputs of all rules to generate a comprehensive fuzzy output parameter RI to reflect the importance of the relationship. Finally, the fuzzy set of RI is converted into a specific numerical result as the relationship importance score R of this experiment through the centroid method defuzzification process. im . Use the centroid method to defuzzify the relationship importance score R im The process is shown in formula (3). Where α∈{HI,MI,LI} RI The subset of the fuzzy set representing the fuzzy output parameter RI, μ RI is the Gaussian function used for RI, μ RI (α) represents the membership degree of each subset in the fuzzy set for calculating RI. im ∈[0,1] indicates the importance score of each relationship in the dataset.

[0111]

[0112] In this embodiment, fuzzy logic is used to calculate the importance score of the true relationship of each image sample in the target data set, which helps to calculate the relationship loss of image samples as a weight during the training of the scene graph generation model, thereby improving the accuracy of the scene graph generation model.

[0113] In an exemplary embodiment, the steps of generating refined subject-object instance features in step S300 are as follows:

[0114] Step S301, using a semantic extractor to extract instance features of the subject and object s 、f o , as shown in formula (4) and formula (5). Where, w s 、w o are the learnable weight parameters of the subject and object instances, T represents the matrix transpose, c s 、c o Represent the word embeddings of subject and object labels respectively.

[0115] f s =w s T c s (4)

[0116] f o =w o T c o (5)

[0117] Step S302: further modeling the subject and object instance features to generate refined subject and object instance features Refined subject instance features As shown in formula (6). FC(·), LN(·), and ReLU(·) represent the fully connected layer, layer normalization operation, and ReLU activation function, respectively, and the residual connection is used to improve the expression ability of instance features. Further, according to formula (6), we derive

[0118]

[0119] In an exemplary embodiment, step S400 in the method for generating a salient scene graph based on fuzzy logic reasoning relationship importance includes:

[0120] The coordinates of the upper left corner, the coordinates of the lower right corner and the area of ​​the bounding box of the instance are spliced ​​in columns, and the bounding box features are generated through modeling; the instance features and the bounding box features are spliced ​​through modeling to form an instance representation.

[0121] Specifically, step S401, the coordinate b of the upper left corner of the bounding box of the instance is l , the coordinates of the lower right corner b r and the area of ​​the bounding box b a Splice by column, model it to generate instance bounding box features, subject bounding box features The generation of is shown in formula (7). Where [·,·] represents column concatenation, and LReLU(·) is the LeakyReLU activation function. Furthermore, the bounding box feature of the object Derived in the same way.

[0122]

[0123] Step S402: further model the instance features and bounding box features to generate instance representations of the subject and object according to the steps shown in formula (8). Where s is the subject instance representation, represents the concatenation of the subject instance feature and the subject bounding box feature. Furthermore, the object instance representation o is derived in the same way.

[0124]

[0125] In this embodiment, a method of generating bounding box features based on bounding box information and concatenating bounding box features with instance features to form instance representation is further described, which helps to train the scene graph generation model and improve the accuracy of the results of the scene graph generation model.

[0126] In an exemplary embodiment, step S500 in the method for generating a salient scene graph based on fuzzy logic reasoning relationship importance includes:

[0127] Step S501, the instance representations s and o of the subject and object are fused to obtain the relation feature F(s,o), as shown in formula (9):

[0128] F(s,o)=ReLU(s+o)-(so) 2 (9)

[0129] Step S502, further modeling the relationship features to enhance the expressiveness of the features, the specific operation is shown in formula (10). Wherein, MLP represents a multi-layer perceptron, which effectively captures important features in the input data through multi-layer nonlinear transformations, providing the model with rich feature representation.

[0130]

[0131] In an exemplary embodiment, in the method for generating a salient scene graph based on fuzzy logic reasoning relationship importance, using the importance score of the real relationship as a weight to weight the loss of the relationship of each image sample in step S600 includes:

[0132] A semantic extractor is used to extract predicate features in the target dataset; refined predicate features are generated through modeling; the relationship representation and predicate features are standardized; and the label distribution of the relationship to be predicted and the loss of the true relationship label are calculated through the weighted cross entropy loss function.

[0133] Exemplarily, the steps of performing relationship loss weighting in step S600 are as follows:

[0134] Step S601, using a semantic extractor to extract predicate features f p , as shown in formula (11). Where w p is the learnable weight parameter of the predicate, T represents the matrix transpose, c p is the word embedding representing the predicate label.

[0135] f p =w p T c p (11)

[0136] Step S602: further modeling the predicate features to generate refined predicate features As shown in formula (12).

[0137]

[0138] Step S603: Relationship representation and predicate features Perform standardization and predict the label distribution r of the relationship p It is calculated by formula (13). In it, ||·|| represents the L2 norm, ⊙ is the dot product operation, exp is the exponential operation, and γ is a learnable scaling factor used to adjust the range of the relationship label distribution.

[0139]

[0140] Step S604, calculate the loss of the predicted relationship label distribution and the true relationship label by using the weighted cross entropy loss function. Specifically, the importance score R of each relationship sample calculated in step S100 is im As a weight multiplied by the standard cross entropy loss, the loss of important relationship samples contributes more to the total loss, as shown in formula (14). and are the true label and predicted distribution of the j-th relation sample, respectively. is the importance score of the jth relation sample, and N is the total number of relation samples on each image.

[0141]

[0142] Preferably, the present application is tested separately under three subtasks, namely, predicate classification (PredicateClassification, Predcls), scene graph classification (Scene Graph Classification, SGcls) and scene graph detection (Scene Graph Detection, SGdet).

[0143] In the predicate classification task, the model takes the bounding box information and labels of all instances in the image as input. The goal is to predict the relationship category between instances, focusing on understanding the interaction between entities without involving the challenge of instance recognition. The scene graph classification task is similar to predicate classification. The input also includes the bounding box information of all instances, but does not contain instance category information. Therefore, the model first needs to identify the category of each instance and then predict the relationship between instances. This task is more challenging because the accuracy of instance classification directly affects relationship prediction. Scene graph detection is the most challenging task. The input image does not contain the bounding box and category information of any instance. The model needs to independently identify the location and category of all instances and predict the relationship between them. Instance recognition from scratch may lead to deviations from manually annotated data, increase the difficulty of relationship prediction, and may affect the overall accuracy.

[0144] In this embodiment, a method for weighting the relationship loss is further described, and the relationship prediction capability of the scene graph generation model is improved by continuously and iteratively correcting the loss between the real relationship and the relationship to be predicted.

[0145] In an exemplary embodiment, in order to test the output of the trained model in the salient scene graph generation method based on the importance of fuzzy logic reasoning relations, Figure 6 As shown, a scene graph generation model testing method is provided, which is used to test the scene graph generation model provided in the above embodiment, including:

[0146] Step S100: input the image samples in the test set into a scene graph generation model that takes the image samples as input and the scene graph as output to perform relationship prediction.

[0147] Among them, the test images are selected from the test set divided in the VG150 dataset.

[0148] Exemplarily, the test image is input into the above-mentioned trained scene graph generation model for relationship prediction, and the process proceeds to step S200.

[0149] Step S200, comparing the relationship prediction result with the true predicate label, and evaluating the comparison result using the target evaluation index.

[0150] Exemplarily, the relationship prediction result is compared with the true predicate label, and the result is evaluated using the R@50 and R@100 evaluation indicators, and then the process proceeds to step S300.

[0151] Step S300: introducing a target prediction index to evaluate the result of relationship prediction.

[0152] Exemplarily, in order to evaluate the prediction ability of the scene graph generation model in important relationships, two new indicators are introduced to evaluate the results, and the process proceeds to step S400.

[0153] Specifically, the IR@5, IR@10, IP@5 and IP@10 indicators are used for evaluation in this embodiment.

[0154] Step S400, filtering out important relationships in the scene graph generation model relationship prediction results to generate a scene graph.

[0155] Exemplarily, important relationships among the predicted relationships are screened to form a scene graph, and the process proceeds to step S500.

[0156] Step S500: output the generated scene graph.

[0157] Exemplarily, the test results are output and the process ends.

[0158] The above-mentioned scene graph generation model testing method tests the scene graph generation model provided in the above-mentioned embodiment to ensure the effect of the scene graph output by the scene graph generation model. Among them, in order to compare the relationship prediction results with the true predicate labels, the widely recognized evaluation standard Recall@K (R@K) in the field of scene graph generation is adopted. Recall@K refers to the proportion of the number of correctly predicted relationships in the first K prediction results to the total number of relationships. The larger the value of this indicator, the better the test result, which represents the higher the accuracy of the model relationship prediction and the better the model performance. In order to evaluate the prediction ability of the model in important relationships, two evaluation indicators are introduced, namely, the important relationship recall rate (IR@K) and the important relationship accuracy rate (IP@K). The important relationship recall rate (IR@K) refers to the proportion of the number of correctly predicted important relationships in the first K prediction results to the total number of important relationships; the important relationship accuracy rate (IP@K) refers to the proportion of the number of correctly predicted important relationships in the first K prediction results to the number of correctly predicted relationships. The two indicators complement each other and can comprehensively evaluate the performance of the model in important relationship prediction.

[0159] In an exemplary embodiment, image samples in the test set are input into a scene graph generation model that takes image samples as input and scene graphs as output to perform relationship prediction, wherein the test images are selected from the test set divided in the VG150 data set. The specific steps include:

[0160] Step S101, performing the predicate classification task. The scene graph generation model uses the location and category label information of all instances provided by the test set. Under the premise of knowing the exact location and category of each instance, the scene graph generation model focuses on predicting the relationship between instances.

[0161] Step S102, perform the scene graph classification task. The scene graph generation model does not have access to the category label information of the instances in the image, but needs to independently predict the category of each instance. These prediction results are then used in the next step of relationship prediction, and the prediction of the relationship between instances is continued based on this step.

[0162] Step S103, performing the scene graph detection task. The scene graph generation model cannot access the location and category label information of the instances in the image. It first needs to identify and locate each instance in the image, and then predict the category of these instances. After completing these steps, the scene graph generation model further predicts the relationship between instances. Since it is necessary to understand the image content from scratch, this process is more complex and challenging than the previous two tasks.

[0163] In an exemplary embodiment, step S300 in the scene graph generation model testing method includes:

[0164] The important relationship recall rate and important relationship accuracy are introduced to evaluate the results of relationship prediction.

[0165] Specifically, the important relation recall rate (IR@K) refers to the ratio of the number of important relations predicted correctly in the first K prediction results to the total number of important relations; the important relation precision rate (IP@K) refers to the ratio of the number of important relations predicted correctly in the first K prediction results to the total number of important relations predicted correctly.

[0166] Specifically, this application uses IR@5, IR@10, IP@5 and IP@10 indicators for evaluation.

[0167] In this embodiment, the important relationship recall rate (IR@K) is used to measure how many important relationships can be retrieved in the first K prediction results. This indicator focuses on the model's coverage of important relationships and can reflect the model's effectiveness in identifying important relationships. The important relationship accuracy rate (IP@K) refers to the ratio of the number of correctly predicted important relationships in the first K prediction results to the number of correctly predicted relationships. It is used to measure how many of the correctly predicted relationships in the first K prediction results are important relationships. This indicator focuses on the model's ability to judge the importance of correctly predicting relationships and can reflect the model's accuracy in predicting the importance of relationships. Combined with the relationship importance score, the two indicators complement each other and can comprehensively evaluate the performance of the model in predicting important relationships.

[0168] In an exemplary embodiment, before step S400, the scene graph generation model testing method further includes:

[0169] The importance score of each image sample relationship in the test set is calculated, and the predicted relationships are sorted from large to small according to the importance score, where the relationships with importance scores greater than 0.5 are regarded as important relationships.

[0170] Specifically, the predicted relationships are sorted from large to small according to the relationship importance scores calculated in the salient scene graph generation method based on fuzzy logic reasoning relationship importance, and the relationships with importance scores greater than 0.5 will be regarded as important relationships to form a salient scene graph.

[0171] In this embodiment, the important relationship is determined by the importance score of each image sample relationship, and a more accurate salient scene graph can be generated based on the important relationship.

[0172] In an exemplary embodiment, the present application proposes a relationship loss weighted scene graph generation network model based on fuzzy logic. The model uses fuzzy logic to calculate the importance score of each relationship according to the actual needs of the user, and uses it as a weight to enhance the model's ability to predict important relationships, thereby enabling the model to meet the application needs of different research fields.

[0173] Furthermore, the following three indicators are used for the model proposed in this application: R@K, IP@K and IR@K.

[0174] Furthermore, experiments were conducted on the VG150 dataset, which contains the most common 150 instance classes and 50 predicate classes. This application evaluates the indicators in three subtasks: predicate classification, scene graph classification, and scene graph detection. In the predicate classification task, the final evaluation results of the method of the present application are as follows: R@50 is 66.14%, R@100 is 68.36%, IR@5 is 38.78%, IR@10 is 51.02%, IP@5 is 90.65%, and IP@10 is 88.49%; in the scene graph classification task, the final evaluation results of the method of the present application are as follows: R@50 is 40.66%, R@100 is 41.85%, IR@5 is 24.46%, IR@10 is 31.82%, IP@5 is 89.52%, and IP@10 is 87.08%; in the scene graph detection task, the final evaluation results of the method of the present application are as follows: R@50 is 32.38%, R@100 is 36.87%, IR@5 is 13.60%, IR@10 is 19.07%, IP@5 is 91.08%, and IP@10 is 89.03%.

[0175] In an exemplary embodiment, the training flow chart of the FLRW-SGG proposed in this application is as follows: Figure 7 As shown, in the training process, the batch size can be adjusted according to the performance of the computer. The training process of this application includes:

[0176] Step S1, input the image and annotation information in VG150, and proceed to step S2.

[0177] Step S2, use fuzzy logic to calculate the importance score of the real relationship in the VG150 data set, that is, use fuzzy logic to infer the relationship importance score, and then enter step S3.

[0178] Step S3, input the training image in the data set into the target detector to generate the instance label of the relationship to be predicted and the corresponding bounding box information, that is, generate the predicted instance label and bounding box information through the target detector, and enter step S4.

[0179] Step S4, using a semantic extractor to extract instance features of the subject and the object, further modeling them to generate refined subject and object instance features, that is, modeling the instance features of the subject and the object, and then entering step S5.

[0180] Step S5, modeling the bounding box coordinates and areas of the subject and the object respectively to generate the bounding box features of the subject and the object, that is, modeling the bounding box features of the subject and the object, and then proceeding to step S6.

[0181] Step S6, concatenate the bounding box features with the instance features to generate instance representations of the subject and the object, and proceed to step S7.

[0182] Step S7, the instance representations of the subject and the object are fused to obtain the relational features, and the relational features are further modeled to generate the relational representation, that is, the instance representations of the subject and the object are fused, and the relational representation is modeled to generate the relational representation, and then step S8 is entered.

[0183] Step S8, matching the fused relational features with the predicate prototype, and proceeding to step S9.

[0184] Step S9: weighting the relationship loss. Generate the label distribution of the relationship representation in step S6, calculate the loss with the real relationship label, use the relationship importance score in step S2 as the weight to weight the loss of each relationship sample, and then perform scene graph generation training, and enter step S10.

[0185] Step S10, the scene graph generation model performs relationship prediction, and then proceeds to step S11.

[0186] Step S11, if the set batch processing (batch-size) size is reached, that is, the number of model training rounds reaches the specified number of rounds, return to step S3; if all training images are read, enter step S12.

[0187] Step S12, output the trained model and end.

[0188] It should be understood that these steps are not necessarily performed in the order indicated by the arrows. At least a portion of the steps in the flowcharts of the above-mentioned embodiments may include multiple steps or multiple stages, which may be performed at different times, or may be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0189] In an exemplary embodiment, this embodiment will use the VG150 dataset to complete the task of generating a significant scene graph based on the importance of fuzzy logic reasoning relationships. The dataset contains 108,077 images, covering the 150 most common instance categories and 50 predicate categories. In order to reasonably divide the dataset, 70% of the images are assigned to the training set, and the remaining 30% are used as the test set. In order to complete the training and testing, this embodiment selects deepin 20.4 as the operating system and conducts research in the experimental environment of torch1.7.1+cu110. The FLRW-SGG in this embodiment is trained by the SGD optimizer for 50,000 iterations, and the initial learning rate and batch size are set to 0.001 and 8, respectively. In the process of fuzzification using Gaussian function as membership function, since the fuzzy sets of input parameters are divided into three and uniformly distributed in the range of 0 to 1, a is set to 0.25, 0.5 and 0.75 as the center of the three fuzzy sets, and β is set to 1 / 6 to ensure smooth transition between fuzzy sets; in the process of fuzzification using S-type function as membership function, since the fuzzy sets of input parameters are divided into two extreme states of yes and no, the S-type curve should ensure smooth transition between 0 and 1, so ε representing the middle position is set to 0.5, and h representing the steepness is set to 20 and -20. All experiments are performed on hardware equipped with NVIDIA GeForce RTX 4090 GPU to ensure the efficiency of training and testing process.

[0190] This application uses FasterR-CNN to accurately detect instances and their location information in images. Thanks to its excellent performance and flexibility, FasterR-CNN has been widely used in multiple computer vision tasks, including but not limited to object detection, image segmentation, and face detection. In addition, it is also widely used to solve various practical problems, such as visual processing of autonomous driving systems, dynamic target tracking in security monitoring, and key areas such as medical image analysis, demonstrating its wide influence in modern technology applications.

[0191] This application performs indicator evaluation in three subtasks: predicate classification, scene graph classification, and scene graph detection. This application uses the evaluation indicator Recall@K (R@K) to accurately evaluate the ability of FLRW-SGG in relationship prediction. R@K refers to the probability of the correct predicate class label appearing in the first K relationship prediction results. Specifically, this application uses R@50 and R@100 indicators for evaluation. The comparison results of the method of this application and the existing methods on the VG150 dataset are shown in Table 1.

[0192] Table 1 Comparison results between the proposed method and the traditional method on the VG150 dataset

[0193]

[0194] According to the results in Table 1, the methods provided in this application can achieve superior performance in relation prediction on the three subtasks. Compared with Transformer in the PredCls task, FLRW-SGG improves R@50 and R@100 by 0.5% and 0.9% respectively; compared with PE-Net in the SGCls task, FLRW-SGG improves R@50 and R@100 by 1.3% and 1.2% respectively. In the SGDet task, compared with Motifs, FLRW-SGG improves R@50 and R@100 by 0.3% and 0.1% respectively. These results show that the method FLRIW-Net of this application can achieve a high level in the most basic evaluation standard Recall@K, and has the ability to predict relations.

[0195] In order to evaluate the predictive ability of FLRW-SGG in important relations, this application uses two indicators, the important relation recall rate (IR@K) and the important relation precision rate (IP@K), for evaluation. The baseline models generated by the three scene graphs, Motifs, Transformer, and PE-Net, are selected to verify the effectiveness of the FLRW-SGG method of this application and the relation loss weighting module in this method. Specifically, this application reproduces the baseline model, and then inserts the relation loss weighting module of this application into the baseline model. Penet+RW, motif+RW, and transformer+RW are relation loss weighting modules on these baseline models. This method can verify whether the relation loss weighting module of this application can improve the ability of the baseline model to predict important relations. All models are trained and tested on the VG150 dataset using the same batch size and number of iterations, and the evaluation results of IR@5, IR@10, IP@5, and IP@10 are recorded. Under the three subtasks of PredCls, SGCls and SGDet, the test results of the baseline model with the relational loss weighted module and the model with the relational loss weighted module are compared, and the results are shown in Table 2.

[0196] Table 2 Comparison results between the proposed method and the baseline model on IR@K and IP@K indicators

[0197]

[0198]

[0199] According to the results in Table 2, the FLRW-SGG and the relational loss weighting module of the present invention can achieve excellent performance in the prediction of important relations in the three subtasks. In the PredCls task, FLRW-SGG achieved the best test results in both IR@K and IP@K indicators; for the relational loss weighting module, penet+RW improved penet's IR@5, IR@10, IP@5 and IP@10 by 4.71%, 5.04%, 2.67% and 1.98% respectively. As the best unbiased model currently, penet seeks to predict more tail predicates, so it cannot have good performance in the hit rate of the first K relations, and the relational loss weighting module of the present application can give penet a significant improvement. On the baseline models motif and transformer with better relationship recall performance, the insertion of the relational loss weighting module can also further improve their ability to predict important relations. In the SGCls task, FLRW-SGG achieved the best test results in the IR@K index; the models with the relational loss weighting module can further improve the test results of IR@K and IP@K compared with the original model, and motif+RW achieved the best IP@5 and IP@10 results, which were 1.15% and 1.05% higher than motif, respectively. In the SGDet task, FLRW-SGG achieved the best test results in IR@K, and the insertion of the relational loss weighting module can further improve the ability of the baseline model to predict important relations.

[0200] In an exemplary embodiment, an applicable scenario of the present application is provided. In social behavior research, users often need to analyze a large number of images of social interactions, such as crowd behavior captured in public places or specific activities. These images are often difficult to accurately predict important relationships related to social behavior in the image due to the complex shooting environment (such as cluttered background, etc.), and users are unable to generate accurate image captions. The salient scene graph generation method based on fuzzy logic reasoning relationship importance proposed in this application can effectively meet this challenge.

[0201] First, the research platform preprocesses the uploaded images, such as basic image processing steps such as image resizing and color correction, to optimize image quality. Then, the Faster R-CNN model is used to extract features from these preprocessed images in order to identify and locate individual instances in the image. Next, the method proposed in this application is used to generate a significant scene graph related to social behavior. This method scores the importance of relationships through fuzzy logic, and uses a relationship loss weighting module to prioritize and highlight those relationships that are important in social behavior research, such as: gaze, conversation, handshake, etc. Even in complex scenes, users can generate relevant and accurate captions for pictures based on the scene graph. Finally, these generated captions are displayed on the research platform together with the images, providing users with an in-depth perspective and detailed description of social interactions. In addition, users can search for keywords through the caption content to quickly locate images containing specific social behaviors, thereby accelerating the research and analysis process.

[0202] Through the application of this application, users can more effectively extract and analyze key information from a large number of social interaction images, further promoting the in-depth and precise research on social behavior.

[0203] Based on the same inventive concept, the embodiments of the present application also provide a salient scene graph generation device based on the importance of fuzzy logic reasoning relationships for implementing the above-mentioned salient scene graph generation method based on the importance of fuzzy logic reasoning relationships, and a scene graph generation model testing device for implementing the above-mentioned scene graph generation model testing method. The implementation solutions to the problems provided by the above devices are similar to the implementation solutions recorded in the above methods, so the specific limitations in one or more device embodiments provided below can refer to the limitations of the corresponding methods above, and will not be repeated here.

[0204] In an exemplary embodiment, Figure 8 As shown, a salient scene graph generation device based on fuzzy logic reasoning relationship importance is provided, comprising:

[0205] The importance score calculation module 801 is used to calculate the importance score of the true relationship of each image sample in the target data set using fuzzy logic, and divide the target data set into a training set and a test set, wherein each image sample includes a subject object and an object object;

[0206] The object detector construction module 802 is used to construct an object detector that takes an image sample as input and takes labels of the to-be-predicted relationship instances between objects in the image sample and bounding box information corresponding to the to-be-predicted relationship instances as output;

[0207] The instance feature extraction and refinement module 803 is used to construct a semantic extractor that takes the labels of the relationship instances to be predicted between the objects in the image sample as input and the instance features of the subject object and the instance features of the object object as output, and construct an instance feature refinement module to refine the instance features of the subject object and the instance features of the object object;

[0208] The instance representation generation module 804 is used to construct a bounding box modeling module that takes the bounding box information corresponding to the to-be-predicted relationship instance between each object in the image sample as input and takes the bounding box features of the subject object and the bounding box features of the object object as output, and construct a feature splicing module that splices the bounding box features of the subject object and the instance features of the subject object to output the instance representation of the subject object, and splices the bounding box features of the object object and the instance features of the object object to output the instance representation of the object object;

[0209] The relation representation generation module 805 is used to construct a feature fusion module that takes the instance representation of the subject object and the instance representation of the object object as input and the relation feature of the relationship to be predicted as output, and to construct a relation representation modeling module that takes the relation feature of the relationship to be predicted as input and the relation representation of the relationship to be predicted as output;

[0210] A relation loss weighting module 806, which is used to construct a label distribution for generating a relation representation of a relation to be predicted, perform loss calculation in combination with the label of the real relation, use the importance score of the real relation as a weight to weight the loss of the relation of each image sample, and perform scene graph generation training;

[0211] The scene graph generation model training module 807 is used to form a network structure to be trained from input to output direction, including a target detector, a semantic extractor, an instance feature refinement module, a bounding box modeling module, a feature splicing module, a feature fusion module, a relationship representation modeling module, and a relationship loss weighting module; based on the target data set, with image samples as input and scene graphs as output, the network structure to be trained is trained to obtain a scene graph generation model; when the number of training image samples reaches the batch processing size, the target detector construction module is returned; when all image samples in the training set are read, the scene graph generation model output module is entered;

[0212] The scene graph generation model output module 808 is used to output the scene graph generation model.

[0213] In an exemplary embodiment, Fig. 9 As shown, a scene graph generation model testing device is provided, comprising:

[0214] A relationship prediction module 901 is used to input the image samples in the test set into a scene graph generation model that takes the image samples as input and the scene graph as output to perform relationship prediction;

[0215] A comparison result evaluation module 902 is used to compare the relationship prediction result with the true predicate label and evaluate the comparison result using a target evaluation index;

[0216] A prediction result evaluation module 903 is used to introduce a target prediction index to evaluate the result of relationship prediction;

[0217] A scene graph generation module 904 is used to filter out important relationships in the relationship prediction results of the scene graph generation model to generate a scene graph;

[0218] The scene graph output module 905 is used to output the generated scene graph.

[0219] Each module in the above-mentioned salient scene graph generation device based on the importance of fuzzy logic reasoning relationship and scene graph generation model testing device can be fully or partially implemented by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0220] For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these modifications and improvements are all within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims. < / man-riding-bike>

Claims

1. A method for generating a salient scene graph based on fuzzy logic reasoning relationship importance, comprising: Step S100, using fuzzy logic to calculate the importance score of the true relationship of each image sample in the target data set, and dividing the target data set into a training set and a test set, wherein each image sample includes a subject object and an object object; Step S200, constructing an object detector that takes an image sample as input and takes labels of the to-be-predicted relationship instances between objects in the image sample and bounding box information corresponding to the to-be-predicted relationship instances as output; Step S300, constructing a semantic extractor that takes the labels of the relationship instances to be predicted between the objects in the image sample as input and the instance features of the subject object and the instance features of the object object as output, and constructing an instance feature refinement module to refine the instance features of the subject object and the instance features of the object object; Step S400, constructing a bounding box modeling module that takes the bounding box information corresponding to the to-be-predicted relationship instance between each object in the image sample as input and takes the bounding box features of the subject object and the bounding box features of the object object as output, and constructing a feature splicing module that splices the bounding box features of the subject object and the instance features of the subject object to output the instance representation of the subject object, and splices the bounding box features of the object object and the instance features of the object object to output the instance representation of the object object; Step S500, constructing a feature fusion module that takes the instance representation of the subject object and the instance representation of the object object as input and the relational features of the to-be-predicted relation as output, and constructing a relational representation modeling module that takes the relational features of the to-be-predicted relation as input and the relational representation of the to-be-predicted relation as output; Step S600, constructing a relationship loss weighting module for generating a label distribution of a relationship representation of a relationship to be predicted, performing loss calculation in combination with the label of the real relationship, weighting the loss of the relationship of each image sample by taking the importance score of the real relationship as a weight, and performing scene graph generation training; Step S700, from input to output direction, the target detector, semantic extractor, instance feature refinement module, bounding box modeling module, feature splicing module, feature fusion module, relationship representation modeling module, and relationship loss weighting module constitute the network structure to be trained; based on the target data set, with image samples as input and scene graph as output, the network structure to be trained is trained to obtain a scene graph generation model; when the number of training image samples reaches the batch processing size, return to step S200; when all image samples in the training set are read, enter step S800; Step S800, outputting a scene graph generation model.

2. The method according to claim 1, characterized in that The step S100 uses fuzzy logic to calculate the importance score of the true relationship of each image sample in the target data set, including: According to the factors affecting the importance of the relationship, the relationship area, predicate frequency, and the instance class, predicate class, and attribute class data specified by the user in the target data set are preprocessed as input parameters of the fuzzy reasoning system; Partition fuzzy sets for input parameters of fuzzy inference system; Formulate a membership function, and convert the input parameters of the fuzzy inference system into fuzzy values ​​through the membership function; Define fuzzy rules according to the factors that affect the importance of the relationship, evaluate the fuzzy values ​​according to the fuzzy rules, and generate the fuzzy output of each fuzzy rule; The fuzzy outputs of fuzzy rules are aggregated through a fuzzy inference system to generate fuzzy output parameters, and the fuzzy set of fuzzy output parameters is converted into importance scores through the centroid method defuzzification process.

3. The method according to claim 1, characterized in that The step S400 includes: The coordinates of the upper left corner, the coordinates of the lower right corner, and the area of ​​the bounding box of the instance are concatenated in columns, and the bounding box features are generated through modeling; The instance features and bounding box features are concatenated through modeling to form instance representation.

4. The method according to claim 2, characterized in that: In step S600, the importance score of the true relationship is used as a weight to weight the loss of the relationship of each image sample, including: A semantic extractor is used to extract predicate features in the target dataset; Generate refined predicate features through modeling; Standardize the relation representation and predicate features; The weighted cross entropy loss function is used to calculate the loss of the label distribution of the relationship to be predicted and the true relationship label.

5. A scene graph generation model testing method, used to test the scene graph generation model as claimed in any one of claims 1 to 4, comprising: Step S100, inputting image samples in the test set into a scene graph generation model that takes image samples as input and scene graphs as output to perform relationship prediction; Step S200, comparing the relationship prediction result with the true predicate label, and evaluating the comparison result using the target evaluation index; Step S300, introducing a target prediction index to evaluate the result of relationship prediction; Step S400, filtering out important relationships in the scene graph generation model relationship prediction results to generate a scene graph; Step S500: output the generated scene graph.

6. The method according to claim 5, characterized in that The step S300 includes: The important relationship recall rate and important relationship accuracy are introduced to evaluate the results of relationship prediction.

7. The method according to claim 5, characterized in that Before step S400, the following steps are also included: The importance score of each image sample relationship in the test set is calculated, and the predicted relationships are sorted from large to small according to the importance score, where the relationships with importance scores greater than 0.5 are regarded as important relationships.

8. A salient scene graph generation device based on fuzzy logic reasoning relationship importance, comprising: An importance score calculation module is used to calculate the importance score of the true relationship of each image sample in the target data set using fuzzy logic, and divide the target data set into a training set and a test set, wherein each image sample includes a subject object and an object object; An object detector construction module is used to construct an object detector that takes an image sample as input and takes labels of the relationship instances to be predicted between objects in the image sample and bounding box information corresponding to the relationship instances to be predicted as output; An instance feature extraction and refinement module is used to construct a semantic extractor that takes the labels of the relationship instances to be predicted between the objects in the image sample as input and the instance features of the subject object and the instance features of the object object as output, and to construct an instance feature refinement module to refine the instance features of the subject object and the instance features of the object object; An instance representation generation module is used to construct a bounding box modeling module that takes the bounding box information corresponding to the to-be-predicted relationship instance between each object in the image sample as input and takes the bounding box features of the subject object and the bounding box features of the object object as output, and to construct a feature splicing module that splices the bounding box features of the subject object and the instance features of the subject object to output the instance representation of the subject object, and splices the bounding box features of the object object and the instance features of the object object to output the instance representation of the object object; A relation representation generation module is used to construct a feature fusion module that takes the instance representation of the subject object and the instance representation of the object object as input and the relational features of the to-be-predicted relation as output, and to construct a relation representation modeling module that takes the relational features of the to-be-predicted relation as input and the relational representation of the to-be-predicted relation as output; A relation loss weighting module, which is used to construct a label distribution for generating a relation representation of a relation to be predicted, calculate the loss in combination with the label of the true relation, use the importance score of the true relation as a weight to weight the loss of the relation of each image sample, and perform scene graph generation training; The scene graph generation model training module is used for the direction from input to output. The target detector, semantic extractor, instance feature refinement module, bounding box modeling module, feature splicing module, feature fusion module, relationship representation modeling module, and relationship loss weighting module constitute the network structure to be trained. Based on the target data set, with image samples as input and scene graphs as output, the network structure to be trained is trained to obtain the scene graph generation model. When the number of training image samples reaches the batch processing size, the target detector construction module is returned. When all the image samples in the training set have been read, the scene graph generation model output module is entered; The scene graph generation model output module is used to output the scene graph generation model.

9. A scene graph generation model testing device, used to test the scene graph generation model according to any one of claims 1 to 4, comprising: A relationship prediction module, used to input the image samples in the test set into a scene graph generation model that takes the image samples as input and the scene graph as output to perform relationship prediction; A comparison result evaluation module is used to compare the relationship prediction results with the true predicate labels and evaluate the comparison results using the target evaluation index; A prediction result evaluation module is used to introduce target prediction indicators to evaluate the results of relationship prediction; A scene graph generation module is used to filter out important relationships in the relationship prediction results of the scene graph generation model to generate a scene graph; The scene graph output module is used to output the generated scene graph.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images