A visual scene graph generation method based on differentiable fuzzy logic reasoning
By constructing a visual scene graph generation method based on differentiable fuzzy logic reasoning, and combining deep learning and fuzzy logic, the semantic ambiguity and lack of logical common sense in the long-tail relation processing of visual scene graph generation methods are solved, and the accurate identification of rare relations and black-box interpretability are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAN INST OF OPTICS & PRECISION MECHANICS CHINESE ACAD OF SCI
- Filing Date
- 2026-02-02
- Publication Date
- 2026-05-15
AI Technical Summary
Existing visual scene graph generation methods suffer from semantic ambiguity and a lack of logical common sense when dealing with long-tail relationships, making it difficult for the models to explain and generate predictions that violate physical common sense in practical applications.
A visual scene graph generation method based on differentiable fuzzy logic reasoning is constructed. By combining deep learning and fuzzy logic, a scene graph triple is generated using an object detection module, a fuzzy mapping layer, a logic tensor reasoning module, and a gated residual fusion module. Logical reasoning and visual classification are performed by combining common sense rules from an external knowledge base, and end-to-end training is achieved.
It improves the accuracy of rare relation categories, achieves black-box interpretability, solves the semantic ambiguity and lack of logical common sense in the processing of long-tail relations by traditional methods, and enhances the interpretability and logical reasoning ability of the model.
Smart Images
Figure CN121616903B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and cognitive intelligence technology, specifically relating to a method for generating visual scene graphs that combines deep convolutional neural networks and differentiable fuzzy logic. Background Technology
[0002] With the rapid development of computer vision technology, image understanding tasks have evolved from simple "object detection" (i.e., identifying what objects are in an image, such as people, cars, and trees) to the higher-level "visual semantic reasoning" stage. Scene graph generation, as a core technology in this field, aims to transform unstructured image data into structured "subject-verb-object" triples (such as <pedestrian, holding, mobile phone>), which is of decisive significance for realizing robot autonomous cognition, visual question answering (VQA), and image-text retrieval.
[0003] Looking at the development history of scene graph generation technology, it has mainly gone through three stages: (1) the early statistical stage: using the co-occurrence frequency of objects as prior knowledge to predict scene graphs. Although this method is simple, it ignores the specific visual content of the image and is prone to bias; (2) the stage based on deep visual features: with the popularization of convolutional neural networks and recurrent neural networks, researchers began to use deep networks to extract the visual and spatial features of objects and classify them; (3) the stage based on message passing and deep learning networks: this is the current mainstream direction. It uses graph convolutional networks to pass contextual information between object nodes. This method can significantly improve the accuracy of relationship prediction.
[0004] However, although existing deep learning-based methods (such as MotifNet, VCTree, and Transformer-based models) have achieved high accuracy on standard datasets, they encounter insurmountable bottlenecks in "interpretability" and "logical reasoning ability" in practical applications, specifically manifested as follows:
[0005] 1. Black box uninterpretability: Deep neural networks are end-to-end black boxes. When the model misidentifies "a person leaning against a tree" as "a person standing on a tree", it is impossible to know whether it is a visual feature extraction error or the model lacks understanding of the "leaning" posture.
[0006] 2. Long-tailed distribution and semantic ambiguity: Relationships in the real world are distributed very unevenly, and many relationships are linguistically ambiguous. For example, "Near" and "Next to" are very similar in pixel space. Traditional hard classification forcibly classifies them into mutually exclusive categories, which not only violates language habits but also leads to extremely poor robustness of the model when dealing with boundary samples.
[0007] 3. Lack of logical common sense: Purely data-driven models lack symbolic logical constraints and often generate predictions that violate physical common sense (such as <car, swallows, person>), which is unacceptable in high-reliability scenarios such as medical or security.
[0008] In conclusion, there is an urgent need to develop a visual scene graph generation method that can solve the above problems. Summary of the Invention
[0009] The purpose of this invention is to provide a visual scene graph generation method based on differentiable fuzzy logic reasoning, which solves the technical problems of semantic ambiguity and lack of logical common sense in existing scene graph generation methods when dealing with long-tail relationships.
[0010] To achieve the above objectives, the technical solution provided by this invention is as follows:
[0011] A visual scene graph generation method based on differentiable fuzzy logic reasoning, characterized by the following steps:
[0012] Step 1: Prepare training images and test images; the relationships between objects in the training images and test images are different;
[0013] Step 2: Construct a network model; the network model includes a target detection module, a fuzzy mapping layer, a logical tensor inference module, and a gated residual fusion module connected in sequence, as well as a relation classifier; the target detection module is a pre-trained target detection network; the input of the relation classifier is connected to the target detection module, and the output is connected to the gated residual fusion module;
[0014] Step 3: Input the training image into the object detection module. The object detection module extracts the multidimensional visual information of all objects in the training image. After performing high-frequency sinusoidal position encoding enhancement processing on the visual information, the corresponding high-dimensional geometric embedding vectors are output to the relation classifier and the fuzzy mapping layer respectively.
[0015] Step 4: Perform visual classification prediction on the received high-dimensional geometric embedding vector through the relation classifier, and output the visual prediction branch to the gated residual fusion module;
[0016] Step 5: Learn the fuzzy set parameters of the received high-dimensional geometric embedding vector through the fuzzy mapping layer, and output the fuzzy membership vector of the predefined linguistic variables to the logical tensor inference module;
[0017] Step 6: Combining common sense rules from the external knowledge base, the logic tensor reasoning module uses differentiable logic operators to simulate the logical reasoning process and outputs the logical reasoning branch of each pair of object relationships in the training image that satisfies the preset logical rules to the gated residual fusion module.
[0018] Step 7: The gated residual fusion module uses an attention mechanism to perform weighted fusion of the visual prediction branch output from Step 4 and the logical reasoning branch output from Step 6, thereby generating the scene of the training image. Figure 3 tuple;
[0019] Step 8: Construct a loss function and iteratively train the network model until the loss function converges to obtain the scene graph generation model;
[0020] Step 9: Input the test image into the scene graph generation model to generate a visual scene graph based on differentiable fuzzy logic reasoning.
[0021] Further, in step 3, the multidimensional visual information of all objects in the training image is extracted by the object detection module. Specifically, the object candidate boxes in the training image are extracted by the pre-trained object detection network, and then the multidimensional visual information of all objects in the object candidate boxes is extracted by the RoIAlign extraction method. The multidimensional visual information includes appearance feature vectors and spatial geometric feature vectors.
[0022] Furthermore, in step 3, the object detection module selects the Faster R-CNN model based on ResNet-101.
[0023] Furthermore, in step 4, the visual prediction branch is determined by the corresponding relation category probability distribution. To reflect, The expression is as follows:
[0024]
[0025] In the formula, It is the weight matrix of the relation classifier; These represent the appearance feature vectors of object i and object j, respectively; Represents object pairs A high-dimensional geometric embedding vector.
[0026] Furthermore, in step 5, the fuzzy mapping layer is a parameterized Gaussian fuzzy layer, which contains a set of learnable semantic centers and variance parameters, used to map high-dimensional geometric embedding vectors into a set of fuzzy membership vectors of predefined linguistic variables.
[0027] The expression for the fuzzy membership vector is:
[0028] ;
[0029] In the formula, Represents object pairs fuzzy membership vector; Represents the fuzzy center vector; represents the fuzzy precision matrix, activated using the Softplus function; k represents a predefined language variable.
[0030] Furthermore, step 6 specifically includes:
[0031] 6.1 Transform common sense rules from external knowledge bases into initial logical adjacency tensors;
[0032] 6.2 The logic tensor reasoning module uses differentiable logic operators, combined with the initial logic adjacency tensor, to perform matrix-based logic deduction on the fuzzy membership vector in the feature space to obtain the deduced logic adjacency tensor; the object relationships contained in the deduced logic adjacency tensor and the initial logic adjacency tensor are in one-to-one correspondence.
[0033] 6.3 Calculate the truth confidence of each pair of object relationships satisfying the preset logical rules in the inference logic adjacency tensor, and output the logical reasoning branch of each pair of object relationships satisfying the preset logical rules in the training image to the gated residual fusion module.
[0034] Furthermore, in step 6.2, the differentiable logic operator is a differentiable logic operator based on the T-norm and the smooth maximum value;
[0035] In step 6.3, the logical reasoning branch for each pair of object relationships in the training image that satisfies the preset logical rules is obtained by calculating the weighted sum of the trigger strength of each rule in the common sense rules and the learnable confidence weight of that rule to obtain the relationship category probability distribution. To reflect, the aforementioned The expression is:
[0036]
[0037] in, These are the learnable confidence weights of the common-sense rule r; This indicates the specific relationship between object i and object j. For indicator functions; Represents object pairs The trigger strength under common sense rule r, and:
[0038]
[0039] In the formula, Represents a set of linguistic variables.
[0040] Furthermore, step 7 specifically includes:
[0041] 7.1 The gated residual fusion module uses the attention mechanism to calculate the weights of the visual prediction branch output in step 4 and the logical reasoning branch output in step 6, and then fuses them to obtain the logical reasoning result.
[0042] 7.2 The logical reasoning result is used as a priori correction term and injected into the visual prediction branch output from step 4 to generate the scene of the training image. Figure 3 tuple.
[0043] Further, in step 7.1, the expression for the logical reasoning result is as follows:
[0044]
[0045] In the formula, The predicted score is the result of logical reasoning. Let be the fusion coefficient, and ,in, For the Sigmoid function, It is the weight matrix of the gated residual fusion module.
[0046] Furthermore, in step 8, the loss function is a joint loss function, the expression of which is as follows:
[0047]
[0048] In the formula, For the joint loss function; and These are the first and second term balancing hyperparameters of the joint loss function, respectively. Loss to the main task; Losses due to logical violations; This represents the loss due to fuzzy entropy regularization.
[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0050] 1. This invention provides a visual scene graph generation method based on differentiable fuzzy logic reasoning. Its core lies in constructing a "neural-symbolic" hybrid network model that can be trained end-to-end. For rare relationships with very few samples, the relationship classifier is often difficult to train. However, this invention, through the logic tensor reasoning module, enables the network model to reason out rare relationships by combining common geometric concepts, achieving generalization with zero or few samples. This effectively solves the problems of semantic ambiguity and lack of logical common sense in traditional scene graph generation models when dealing with long-tail relationships.
[0051] 2. The network model constructed in this invention can output intermediate variables (fuzzy membership vectors) and the triggering intensity of common sense rules, so that users can know for sure that the network model has detected the distance between two objects and triggered the corresponding common sense rules, thereby making a correct judgment on the relationship between the two objects, achieving a breakthrough in black box interpretability.
[0052] 3. This invention is based on differentiable fuzzy logic reasoning. All fuzzy operations can be implemented using Gaussian functions and matrix multiplication, and are fully compatible with existing deep learning training frameworks (PyTorch / TensorFlow). No step-by-step training is required, thereby improving training efficiency.
[0053] 4. This invention achieves the unity of perception and cognition through end-to-end gradient optimization, significantly improving the recognition accuracy of rare relationship categories. Attached Figure Description
[0054] Figure 1 This is a flowchart illustrating an embodiment of the present invention.
[0055] Figure 2 This is a schematic diagram of the mapping process of the fuzzy mapping layer in step 5 of an embodiment of the present invention.
[0056] Figure 3 This is a diagram illustrating the reasoning process of the logic tensor reasoning module in step 6 of this embodiment of the invention. Detailed Implementation
[0057] How to organically combine the powerful perceptual capabilities (feature extraction) of deep learning with the inherent reasoning and interpretability (handling uncertainty and rules) of fuzzy logic to construct a hybrid "neural-symbolic" system that can both utilize backpropagation for training and output explicit reasoning rules is a pressing problem in the field of visual cognition. Against this backdrop, this invention proposes a visual scene graph generation method based on differentiable fuzzy logic reasoning. This method is applicable to scenarios requiring high-level semantic understanding, such as indoor navigation for service robots, assistive visual glasses for the blind, and large-scale image retrieval systems.
[0058] To make the objectives, advantages, and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.
[0059] like Figure 1 As shown, this embodiment provides a method for generating visual scene graphs based on differentiable fuzzy logic reasoning, including the following steps:
[0060] Step 1: Prepare training images and test images; the relationships between objects in the training images and test images are different.
[0061] Step 2: Construct a network model; the network model includes a target detection module, a fuzzy mapping layer, a logical tensor inference module, and a gated residual fusion module connected in sequence, as well as a relation classifier; the target detection module is a pre-trained target detection network; the input end of the relation classifier is connected to the target detection module, and the output end is connected to the gated residual fusion module.
[0062] Step 3: Input the training images into the object detection module. The object detection module selects the Faster R-CNN model based on ResNet-101 and pre-trains it. The object detection module extracts the multi-dimensional visual information of all objects in the training images and performs high-frequency sinusoidal position encoding enhancement processing on this visual information. This high-frequency sinusoidal position encoding enhancement processing is used to capture subtle spatial position changes. Then, the corresponding high-dimensional geometric embedding vectors are output to the relation classifier and the fuzzy mapping layer, respectively.
[0063] In this embodiment, the multidimensional visual information of all objects in the training image is extracted through the object detection module. Specifically, the object candidate boxes in the training image are extracted through a pre-trained object detection network, and then the multidimensional visual information of all objects in the object candidate boxes is extracted using the RoI Align (Region of Interest Align) extraction method. The multidimensional visual information includes appearance feature vectors. and spatial geometric eigenvectors RoI Align is a technique used in object detection and instance segmentation to accurately extract candidate region features, thereby improving localization and segmentation accuracy.
[0064] For appearance feature vectors For the detected i-th and j-th objects, their features are extracted using RoI Align and mapped to... 3D appearance feature vector , ,in, , Let represent the appearance feature vectors of the i-th object and the j-th object, respectively. yes , The vector dimension.
[0065] For spatial geometric feature vectors First, extract the object candidate bounding boxes, then calculate the object pairs within the candidate bounding boxes. The relative geometric parameters, including the relative distance between the two. Relative angle Intersection and Union and area ratio ;in, These are the areas of the bounding boxes of objects i and j, respectively.
[0066] For high-dimensional geometric embedding vectors To enable the network model to better distinguish fine spatial relationships, instead of directly using the above values, they are projected into a high-dimensional embedding space.
[0067]
[0068] After mapping using this formula, the object pair is obtained. The corresponding high-dimensional geometric embedding vector ,Right now ,in, It is a high-frequency sinusoidal position coding function, where x represents the relative geometric parameters and L is the number of frequency bands for position coding.
[0069] Step 4: Using a relation classifier, the traditional SGG (Scene Graph Generation) model is used to perform visual classification prediction on the received high-dimensional geometric embedding vectors, and the visual prediction branch is output to the gated residual fusion module.
[0070] The visual prediction branch described in this embodiment is based on the probability distribution of the corresponding relation categories. To reflect, The expression is as follows:
[0071]
[0072] In the formula, It is the weight matrix of the relation classifier; These represent the appearance feature vectors of object i and object j, respectively; Represents object pairs High-dimensional geometric embedding vectors; yes function.
[0073] Step 5: To intuitively understand the principle of fuzzy mapping, Figure 2 The diagram illustrates the fuzzy membership mapping in the relative distance dimension (actual calculations are performed in the high-dimensional embedding space). The fuzzy mapping layer learns the fuzzy set parameters of the received high-dimensional geometric embedding vector and outputs the fuzzy membership vector of the predefined linguistic variables to the logical tensor inference module. Figure 2 middle, This represents the fuzzy center value of the linguistic variable Near. The bandwidth of the Gaussian distribution of the linguistic variable Near is represented. This represents the relative distance between the currently input object pairs.
[0074] The fuzzy mapping layer described in this embodiment is a parametric Gaussian fuzzy layer, which contains a set of learnable semantic centers and variance parameters. It is used to map high-dimensional geometric embedding vectors into a set of fuzzy membership vectors of predefined linguistic variables (such as Near, Covering, Tall). This process realizes a soft transformation from pixel space to semantic space.
[0075] Predefined linguistic variables are predefined sets of spatial fuzzy concepts. ,and For each concept in the set (k represents a predefined language variable, i.e.) (and other concepts), define a learnable Gaussian kernel, where: the input information is a high-dimensional geometric embedding vector. The parameter is the fuzzy center vector. and fuzzy precision matrix (All are used as network parameters in gradient descent); the output information is a fuzzy membership vector, expressed as membership scores. To indicate:
[0076] ;
[0077] In the formula, Represents object pairs Membership score, i.e., the number of object pairs The fuzzy membership vector; k represents a predefined linguistic variable; This represents the corresponding high-dimensional geometric embedding vector; Represents the fuzzy center vector; This represents the fuzzy precision matrix, activated using the Softplus function to ensure positive variance. Through this step, the object... The high-dimensional geometric embedding vector is transformed into a set of semantic membership vectors. ,in, This represents the membership score for near and far.
[0078] Step 6: As Figure 3 As shown, combining common sense rules from an external knowledge base, the logic tensor reasoning module uses differentiable logic operators to simulate the logical reasoning process. The logic tensor reasoning module simulates the human reasoning process through matrix operations (i.e., the learnable rule matrix), and finally outputs the logical reasoning branch of each pair of object relationships in the training image that satisfies the preset logical rules to the gated residual fusion module.
[0079] 6.1 Transform common sense rules from external knowledge bases into initial logical adjacency tensors.
[0080] Suppose there are R common sense rules in an external knowledge base, and one of these common sense rules, r, is: "IF i is Leftof j AND i is Near j, THEN relation is 'Next to'" Figure 3 (As shown).
[0081] Based on this, common-sense rules can be encoded as sparse tensors. ,in It is the number of common-sense rules. It is the number of input fuzzy concepts. It outputs the number of relation categories. It is a sparse tensor representation of logical rules, namely the initial logical adjacency tensor.
[0082] 6.2 The logical tensor inference module uses a differentiable logical operator based on the T-norm and smooth maximum value, combined with the initial logical adjacency tensor, to perform matrix-based logical deduction on the fuzzy membership vector in the feature space to obtain the deduced logical adjacency tensor; the object relationships contained in the deduced logical adjacency tensor and the initial logical adjacency tensor are in one-to-one correspondence.
[0083] To support backpropagation, this embodiment uses smooth logic operators instead of Boolean operations:
[0084] Logical AND: using the product T-norm, i.e.: .
[0085] Logical OR: using probability sum, i.e.: .
[0086] Logical implication: using Reichenbach implication or simple weighted activation, where These are the symbols representing the operands.
[0087] 6.3 Calculate the truth confidence of each pair of object relationships satisfying the preset logical rules in the inference logic adjacency tensor, and output the logical reasoning branch of each pair of object relationships satisfying the preset logical rules in the training image to the gated residual fusion module.
[0088] In this embodiment, the logical reasoning branch for each pair of object relationships in the training image that satisfies the preset logical rules is obtained by calculating the probability distribution of relationship categories by weighting the trigger strength of each common sense rule and the learnable confidence weight of that common sense rule. This is what is reflected in.
[0089] For each common-sense rule r, first calculate its trigger strength. :
[0090]
[0091] In the formula, A set representing linguistic variables.
[0092] For example, for the rule "Left AND Near", object pairs The trigger strength under this rule is .
[0093] Then, the corresponding probability distribution of the relationship categories is deduced. :
[0094]
[0095] in It is the learnable confidence weight of this common-sense rule r. This indicates the specific relationship between i and j. As an indicator function, this process is equivalent to letting the network model itself filter out which common sense rules are reliable.
[0096] Step 7: The gated residual fusion module uses an attention mechanism to perform weighted fusion of the visual prediction branch output from Step 4 and the logical reasoning branch output from Step 6, thereby generating the scene of the training image. Figure 3 tuple.
[0097] 7.1 The gated residual fusion module utilizes an attention mechanism to calculate the weights of the visual prediction branch output from step 4 and the logical reasoning branch output from step 6, respectively, and then fuses them to obtain the logical reasoning result. Its expression is as follows:
[0098] ;
[0099] In the formula, The predicted score is the result of logical reasoning. Let be the fusion coefficient, and ,in, For the Sigmoid function, It is the weight matrix of the gated residual fusion module; Let i and j represent the appearance feature vectors of object i and object j, respectively. When the visual features of the training image are blurred (e.g., severely occluded), When the image size increases, the network model automatically relies more on the logical reasoning branch; when the visual features of the training image are clear, it relies more on visual perception (i.e., the visual prediction branch).
[0100] 7.2 The logical reasoning result is used as a priori correction term and injected into the visual prediction branch output from step 4 to generate the scene of the training image. Figure 3 tuple.
[0101] Step 8: Combine the loss function to iteratively train the network model until the loss function converges, and obtain the scene graph generation model.
[0102] The loss function described in this embodiment is a joint loss function, and its expression is as follows:
[0103]
[0104] In the formula, For the joint loss function; and These are the first and second term balancing hyperparameters of the joint loss function, respectively. The main task loss, i.e., the standard cross-entropy loss, is used to supervise whether the logical reasoning result obtained in step 7 is consistent with the ground truth. The loss for logical violation is used to calculate the probability distribution of relation categories based on the output of step 6. If a mutual exclusion rule is defined (e.g., "Left" and "Right" cannot both be true), then the penalty for a logical violation is:
[0105]
[0106] This will force the semantic concepts learned by the fuzzy mapping layer to conform to mutual exclusion logic.
[0107] The fuzzy entropy regularization loss is used to calculate the fuzzy membership vector based on the output of step 5. To prevent all fuzzy membership degrees from collapsing to around 0.5, an entropy penalty is introduced, and fuzzy membership degrees are encouraged to tend towards 0 or 1, making the reasoning more decisive.
[0108] This step constructs a joint loss function that includes "predicate classification loss", "logical consistency regularization term" and "fuzzy set mutual exclusion constraint". Through the backpropagation algorithm, the parameters of the network model and the semantic parameters of the fuzzy logic layer are updated simultaneously. This forces the network model to satisfy the preset logical common sense while fitting the data, thereby ensuring the authenticity of the final prediction result.
[0109] Step 9: Input the test image into the scene graph generation model to generate a visual scene graph based on differentiable fuzzy logic reasoning.
[0110] This invention achieves the unity of perception and cognition through end-to-end gradient optimization, significantly improving the recognition accuracy of rare relationship categories and effectively solving the problems of semantic ambiguity and lack of logical common sense in the processing of long-tail relationships in existing technologies.
[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein, and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.
Claims
1. A method for generating visual scene graphs based on differentiable fuzzy logic reasoning, characterized in that, Includes the following steps: Step 1: Prepare training images and test images; the relationships between objects in the training images and test images are different; Step 2: Construct a network model; the network model includes a target detection module, a fuzzy mapping layer, a logical tensor inference module, and a gated residual fusion module connected in sequence, as well as a relation classifier; the target detection module is a pre-trained target detection network; the input of the relation classifier is connected to the target detection module, and the output is connected to the gated residual fusion module; Step 3: Input the training image into the object detection module. The object detection module extracts the multidimensional visual information of all objects in the training image. After performing high-frequency sinusoidal position encoding enhancement processing on the visual information, the corresponding high-dimensional geometric embedding vectors are output to the relation classifier and the fuzzy mapping layer respectively. Step 4: Perform visual classification prediction on the received high-dimensional geometric embedding vector through the relation classifier, and output the visual prediction branch to the gated residual fusion module; Step 5: Learn the fuzzy set parameters of the received high-dimensional geometric embedding vector through the fuzzy mapping layer, and output the fuzzy membership vector of the predefined linguistic variables to the logical tensor inference module; Step 6: Combining common sense rules from the external knowledge base, the logic tensor reasoning module uses differentiable logic operators to simulate the logical reasoning process and outputs the logical reasoning branch of each pair of object relationships in the training image that satisfies the preset logical rules to the gated residual fusion module. Step 7: The gated residual fusion module uses an attention mechanism to perform weighted fusion of the visual prediction branch output from Step 4 and the logical reasoning branch output from Step 6, thereby generating scene graph triples of the training images. Step 8: Construct a loss function and iteratively train the network model until the loss function converges to obtain the scene graph generation model; Step 9: Input the test image into the scene graph generation model to generate a visual scene graph based on differentiable fuzzy logic reasoning.
2. The visual scene graph generation method based on differentiable fuzzy logic reasoning according to claim 1, characterized in that: In step 3, the multidimensional visual information of all objects in the training image is extracted through the object detection module. Specifically, the object candidate boxes in the training image are extracted through the pre-trained object detection network, and then the multidimensional visual information of all objects in the object candidate boxes is extracted through the RoI Align extraction method. The multidimensional visual information includes appearance feature vectors and spatial geometric feature vectors.
3. The visual scene graph generation method based on differentiable fuzzy logic reasoning according to claim 2, characterized in that: In step 3, the object detection module selects the Faster R-CNN model based on ResNet-101.
4. The visual scene graph generation method based on differentiable fuzzy logic reasoning according to claim 1, 2, or 3, characterized in that: In step 4, the visual prediction branch is based on the corresponding relation category probability distribution. To reflect, The expression is as follows: ; In the formula, It is the weight matrix of the relation classifier; These represent the appearance feature vectors of object i and object j, respectively; Represents object pairs A high-dimensional geometric embedding vector.
5. The visual scene graph generation method based on differentiable fuzzy logic reasoning according to claim 4, characterized in that: In step 5, the fuzzy mapping layer is a parameterized Gaussian fuzzy layer, which contains a set of learnable semantic centers and variance parameters, used to map high-dimensional geometric embedding vectors into a set of fuzzy membership vectors of predefined linguistic variables. The expression for the fuzzy membership vector is: ； In the formula, Represents object pairs fuzzy membership vector; Represents the fuzzy center vector; represents the fuzzy precision matrix, activated using the Softplus function; k represents a predefined language variable.
6. The visual scene graph generation method based on differentiable fuzzy logic reasoning according to claim 5, characterized in that, Step 6 specifically involves: Step 6.1: Convert the common sense rules in the external knowledge base into the initial logical adjacency tensor; Step 6.2: The logic tensor reasoning module uses differentiable logic operators, combined with the initial logic adjacency tensor, to perform matrix-based logic deduction on the fuzzy membership vector in the feature space to obtain the deduced logic adjacency tensor; the object relationships contained in the deduced logic adjacency tensor and the initial logic adjacency tensor are in one-to-one correspondence. Step 6.3: Calculate the truth confidence of each pair of object relationships in the inference logic adjacency tensor that satisfies the preset logic rules, and output the logical reasoning branch of each pair of object relationships in the training image that satisfies the preset logic rules to the gated residual fusion module.
7. The visual scene graph generation method based on differentiable fuzzy logic reasoning according to claim 6, characterized in that: In step 6.2, the differentiable logic operator is a differentiable logic operator based on the T-norm and the smooth maximum value; In step 6.3, the logical reasoning branch for each pair of object relationships in the training image that satisfies the preset logical rules is obtained by calculating the weighted sum of the trigger strength of each rule in the common sense rules and the learnable confidence weight of that rule to obtain the relationship category probability distribution. To reflect, the aforementioned The expression is: ; in, These are the learnable confidence weights of the common-sense rule r; This indicates the specific relationship between object i and object j. For indicator functions; Represents object pairs The trigger strength under common sense rule r, and: ; In the formula, A set representing linguistic variables.
8. The visual scene graph generation method based on differentiable fuzzy logic reasoning according to claim 7, characterized in that, Step 7 specifically includes: Step 7.1: The gated residual fusion module uses the attention mechanism to calculate the weights of the visual prediction branch output in Step 4 and the logical reasoning branch output in Step 6, and then fuses them to obtain the logical reasoning result. Step 7.2: Use the logical reasoning result as a priori correction term and inject it into the visual prediction branch output in Step 4 to generate scene graph triples of the training images.
9. The visual scene graph generation method based on differentiable fuzzy logic reasoning according to claim 8, characterized in that: In step 7.1, the expression for the logical reasoning result is as follows: ; In the formula, The predicted score is the result of logical reasoning. Let be the fusion coefficient, and ,in, For the Sigmoid function, It is the weight matrix of the gated residual fusion module.
10. The visual scene graph generation method based on differentiable fuzzy logic reasoning according to claim 9, characterized in that: In step 8, the loss function is a joint loss function, and its expression is as follows: ; In the formula, For the joint loss function; and These are the first and second term balancing hyperparameters of the joint loss function, respectively. Loss to the main task; Losses due to logical violations; This represents the loss due to fuzzy entropy regularization.