Predicate similarity discrimination method integrating sensory clues and perceptual clues

Abstract

The invention discloses a predicate similarity discrimination method integrating sensory and perceptual clues, and the method comprises the following specific steps: extracting appearance features, spatial features and entity tags of an input image in a target detection process, and combining target pairs to construct joint features; and then the sensory sensitive module is responsible for capturing intuitive visual clues in the image as sensory clues, aggregating entity-level visual information, and realizing coarse-grained screening and feature alignment of candidate entity pairs. And the perception enhancement module further models the deep relationship between the entity pairs on the basis of the output of the perception module, and realizes semantic differentiation and optimization of similar predicates through semantic mapping and discriminative learning so as to conclude perception clues. And finally, performing deep association on visual scenes with a unified relationship in the same region by the distinguished triple category in a cognitive discrimination layer in a perception enhancement module, and ensuring that predicates embedded between entity pairs can capture discovered fine-grained semantics on the premise of not influencing semantics.

Description

TECHNICAL FIELD

[0001] The application relates to a predicate similarity discrimination method integrating sensory and perceptual clues and belongs to the technical field of deep learning. BACKGROUND

[0002] With the increasing demand of researchers for multi-modal semantic information understanding, how to fully mine and reason the visual semantic information in the image has become a work of great interest today. The scene graph generation task at the present stage has become the main means to realize the reasoning of visual scene semantic information. However, due to the high similarity of the semantics between some predicates, the semantics of the tail predicate is often covered by the head predicate, resulting in the confusion of the relationship semantics. For example, a group of entity pairs composed of elephants and streets, the relationship predicate of which may exist to describe detailed sitting on, lying on or walking in, but is simply classified as "elephant on street". These prediction biases mainly manifest that the tail predicate (such as "lying on") is easily generalized to the high-frequency head predicate (such as "on") due to its semantic similarity, and the essence lies in the misplacement of language representation and visual clues. To distinguish the similarity between the relationship semantics, in addition to combining the actual scene for discrimination, the difference of the specific context of the predicate should also be handled well.

[0003] The sensory and perceptual dichotomy mentioned in the Gestalt school of cognitive psychology shows that the joint processing of external stimuli and internal experience helps to distinguish the similarity between similar targets. The research emphasizes that the sense can respond to the external stimulus, while the perception is a more complex conscious experience. The scene graph generation task can be regarded as a reasoning process dominated by two kinds of information, that is, the sensory cognitive process dominated by visual information and the perceptual cognitive process dominated by deep experience discrimination. In real life, when light stimuli reach the receptors of human eyes, humans passively receive the entities in the image. The sensory ability stimulated by color, edge and other information is the primary processing stage of the sensory system, which can infer simple relationships by using entity information to obtain a primary sensory cognitive result. Subsequently, when humans generate conscious experience from sensory stimuli, they begin to process the entire image and deeply explain the internal association. At this time, the brain enters the deep thinking stage, which can mobilize the deep knowledge stored in the human brain. This concentrated thinking ability can help the model to further distinguish the semantic information carried by similar relationships, achieving the ability of advanced perceptual cognition. The ability of perceptual reasoning can help the model to better identify the small differences between similar scenes.

[0004] Previous scene graph generation methods focus on exploring shallow visual information such as location, color and integrating them into the relationship, which is the primary information of external stimuli for humans. Some researchers have also found the importance of visual context, which helps the model explore how to think about the problem to improve the cognitive ability of the world. However, existing research mainly improves the fine-grained predicate recognition ability through attention mechanism or hierarchical structure. Therefore, the current SGG task should also try to solve the semantic similarity dilemma between the head and tail predicates while improving the entity detection ability. It is necessary to explore a new mode to help the model distinguish similar semantic relationships to alleviate the long-tail distribution. SUMMARY

[0005] In view of the problems in the prior art, the present application proposes a predicate similarity discrimination method integrating sensory and perceptual cues, so that the model can fully learn the entity information and interaction area associated with each predicate, and the model can separate and deeply understand the scene graph knowledge layer by layer to distinguish the different information expressed by different predicates in different scenes.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is a predicate similarity discrimination method integrating sensory and perceptual cues, comprising the following steps:

[0007] S1) Use a target extractor to extract the appearance features, spatial features and entity labels of entity suggestions, and capture their joint regions as joint features;

[0008] S2) Low-level visual information is sent to the sensory sensitive module for similar entity pair information integration, and the visual features of the entity context in the data are perceived through an adaptive perception mechanism for further differentiation of similar predicate instances;

[0009] S3) The perceptual enhancement module further distinguishes the differences between similar predicates;

[0010] Model the context information through the global attention of the Transformer, associate the isolated object features with the overall semantic of the scene, and perform the global attention of the Transformer.

[0011] The context semantic information based on the predicate is evaluated and classified according to the similarity, and then the output result is compared with the triple information of the same predicate to re-rank the confidence;

[0012] S4) The model projects the obtained multi-label relationship instances through two fully connected layers into the embedding representation r of the predicate and the triple, and fuses the semantic perception cues extracted in the perceptual discrimination process into the new relationship representation.

[0013] The optimized predicate similarity discrimination method that integrates sensory and perceptual cues, in step S1), inputs the dataset into the target extraction model to extract initial information and detects the appearance features a of the object. i Spatial features s i and object category labels i These appearance and spatial features are initialized into corresponding entity representations. Then, the relative spatial representation between entity i and entity j is encoded with the joint ROI feature into a joint feature u. ij ;

[0014] In step S1), Faster RCNN is used as the object detector to extract initial information, and ResNeXt-101-FPN is used as the backbone network to generate convolutional feature representations for subsequent modules;

[0015] The target extraction model is trained on the Visual Genome dataset using sensory and perceptual cues. The model learns the entity information and interaction regions associated with each predicate in the corresponding dataset.

[0016] In the optimized predicate similarity discrimination method that integrates sensory and perceptual cues, in step S1), the initial information extracted from the original image is combined to obtain the interaction information between pairs of entities; the extracted combined features and the initial features are sent together to the sensory sensitive module and the perceptual sensitive module for sensory cue extraction and perceptual cue discrimination.

[0017] In the optimized predicate similarity discrimination method that integrates sensory and perceptual cues, step S2) follows the Hybrid Attention (HA) modeling approach, where each layer contains two self-attention units and two cross-attention units to construct the scene context.

[0018] Self-attention is responsible for capturing information within a modality, while cross-attention is responsible for processing information between modalities.

[0019] The hybrid attention layer can be calculated as follows:

[0020]

[0021] Where X h and Y h It is the input X of the h-th layer h-1 Y h-1 Obtained through encoding;

[0022] Through self-attention and cross-modal attention, the model dynamically focuses on key regions, suppresses irrelevant background noise, and optimizes the utilization of sensory information in sensory-sensitive modules;

[0023] Visual information from such low-level entity embeddings and predicate embeddings is fed into the sensory-sensitive module for similar entity pair information integration.

[0024] In the optimized predicate similarity discrimination method that integrates sensory and perceptual cues, in step S2), for the i-th and j-th entities, their appearance features a i ,a j Spatial features s i ,s j and category features l i ,l j All are stitched together by a multilayer perceptron (MLP);

[0025] Appearance feature a i ,a j Spatial features s i ,s j and category features l i ,l j After further concatenation, the initial logic is obtained to construct the logical semantic features f of the sensory cues. ij ;

[0026] Visual information is first divided into foreground pair information and overall information; spatial features are extracted by a multilayer perceptron (MLP) and a position-aware matrix (M), respectively, to obtain the encodings s of paired entities in spatial features. ij ;

[0027] Encodings ij Represented as

[0028] s ij =M(MLP([s i s j ]))

[0029] Where M is the location-aware matrix, taking values ​​of 0 or 1, used to distinguish between meaningful spatial locations and invalid entity pairs, respectively, representing the [·] link operation;

[0030] The entire sensory cue layer is defined as:

[0031] f ij =MLP([[a i ,aj],s ij [l] i , l j ]]);

[0032] Processed sensory cues f ij Combined entity features e refined by the target encoder i ,e j The information is processed in the memory processing layer to extract the perceived information.

[0033] Optimized, the above-mentioned predicate similarity discrimination method of integrating sensory and perceptual cues, after extracting the perceptual information in step S2), the entity information is further refined and integrated, and the visual logical features obtained by intuition are processed into new sensory cues:

[0034]

[0035] Where v i is the context-sensitive information of entity i obtained after intuitive perception of visual information, which is obtained by weighting the original refined features and logical features with the suggestion of the same entity i;

[0036] Then align the feature dimension d v of the visual information to the concept space dimension d; initialize the concept prototype matrix by using external knowledge GloVe, where each row represents a learnable concept embedding;

[0037] The cosine similarity matrix between visual information and concept prototypes calculated by the memory processing layer is represented as:

[0038]

[0039] Further, the soft alignment distribution of the sensory-sensitive module is:

[0040]

[0041] At this time, the model output loss is:

[0042]

[0043] Where y ij ∈{0,1} is the true label;

[0044] Then the perceptual sensitive information, location perceptual information and global information are fed into the perceptual enhancement module for deep fusion and extraction; the sensory-sensitive module comprehensively captures the local details and global structure of the image by using sensory cues, and improves the representation ability of sensory information.

[0045] Optimized, the above-mentioned predicate similarity discrimination method of integrating sensory and perceptual cues, in step S3), the initial entity embedding predicate embedding and the real entity category C e and the real predicate category C p are input into the Transformer layer, and the new distribution X e ,X p is output through the trainable parameter matrix W e and W p .

[0046] X e ,X p =Transformer(W e E e ,W p E p );

[0047] For the predicate branch p i , the sensitive information extracted in the sensory sensitive module and the entity distribution X e obtained by the Transformer layer,

[0048]

[0049] wherein, is the output after unifying the dimensions of different modal features through linear projection;

[0050] Subsequently, the model calculates the prediction confidence distribution using the Softmax function and indexes the maximum value to obtain the preliminary coarse-grained predicate label;

[0051] The perceptual enhancement module further distinguishes the differences between similar predicates;

[0052] The global attention of the Transformer models the context information, associating isolated object features with overall scene semantics;

[0053] The predicate-based context semantic information is newly evaluated and classified according to the similarity, and then the output result is compared with the triple information of the same predicate to re-rank the confidence.

[0054] The optimized predicate similarity discrimination method integrating sensory and perceptual cues described above, in step S3),

[0055] The head predicate set is set to the predicates involved in the first 30% of samples, and the remaining 70% of samples are all focused on in the form of the tail predicate set to enhance the model's learning of a large number of low-frequency and semantically rich relationships;

[0056] When the prediction result is consistent with the true label, and the label does not belong to the head predicate set, the model directly outputs the prediction result;

[0057] When the prediction result is inconsistent with the true label, the model will activate the perception enhancement module to further dig deep semantic features, and consider that other prediction labels are still reasonable in semantics; in this stage, the model first queries the k nearest neighbor predicate set {p1,...,p k} and its confidence distribution associated with the current entity; then, by comparing the consistency measure between these candidate labels and the average one-hot representation of other query predicates, the final optimal label is determined;

[0058] During training, the model updates at most two candidate predicate labels per round; when the prediction results of the two candidate labels are different, and the consistency distance is lower than the threshold θ, by comparing the confidence distribution, the label with higher confidence is selected as the new prediction label, and it is re-labeled as the pseudo label of the current instance; when the consistency distance is higher than θ, the original prediction label is kept unchanged, and it is still fed back as a pseudo label for subsequent training.

[0059] The above-mentioned predicate similarity discrimination method integrating sensory and perceptual cues optimizes the training target in step S4) by minimizing the distance between the relationship embedding r and the corresponding predicate and triple class embedding, and performs collaborative optimization of semantic consistency constraint and discrimination ability,

[0060]

[0061] wherein, is the final predicted predicate label;

[0062] The loss of the perception refinement layer after perceptual discrimination is:

[0063]

[0064] wherein, p + and t + represent positive samples under the same predicate category and the same triple category respectively, ρ represents a learnable temperature, N is the number of predicate categories, and α, β are learnable hyperparameters;

[0065] The final loss of the whole sensory cue and perceptual cue joint training can be defined as:

[0066]

[0067] wherein, represents the cross-entropy loss of the entity category, represents the cross-entropy loss of the predicate category, and λ is a learnable hyperparameter.

[0068] The predicate similarity discrimination method integrating sensory and perceptual cues is optimized, the target extractor in step S1) is pre-trained on an ImageNet dataset based on a backbone network of ResNeXt-101-FPN, and the training dataset is an image dataset.

[0069] The predicate similarity discrimination method integrating sensory and perceptual cues is optimized, in step S1), the model is trained on a Visual Genome dataset to integrate sensory cues and perceptual cues, so that the model can sufficiently learn the entity information and interaction regions associated with each predicate in the corresponding dataset, and the model can distinguish different information expressed by different predicates in different scenes.

[0070] The predicate similarity discrimination method integrating sensory and perceptual cues is optimized, the sensory cues are based on extraction and processing of global and local information of visual information, and the perceptual cues are the ability to integrate such visual stimuli for comparison to intensify the differences between them to improve relationship confusion.

[0071] The predicate similarity discrimination method integrating sensory and perceptual cues is optimized, in step S2), the initial information extracted from the original image is represented in a joint manner to obtain interaction information between pairs of entities; the extracted joint features and initial features are sent to a sensory sensitive module and a perceptual sensitive module for extraction of sensory cues and discrimination of perceptual cues.

[0072] The predicate similarity discrimination method integrating sensory and perceptual cues is optimized, the sensory reasoning module improves the ability to extract pair-level object information from visual information and the ability to preliminarily integrate similar triplets, and improves the understanding of similar semantics of entity context; the perceptual reasoning module deeply divides the relationship instances between similar semantics, and ensures that the predicates embedded between entity pairs can capture the discovered fine-grained semantics without affecting the semantics.

[0073] The predicate similarity discrimination method integrating sensory and perceptual cues is optimized, in step S4), the predicate perception factor α and the triple knowledge guidance factor β are reasonably balanced to improve the long-tail relationship recognition ability of the model in the perceptual sensitive module. At the same time, the perception sensitive factor λ shows the influence of the existence of different sensory sensitive modules on the model accuracy and long-tail distribution. In Figure 3 and Figure 4 the influence of different influence factors on the performance of the model is verified, especially in Figure 3 the results of the joint influence of the predicate perception factor α and the triple knowledge guidance factor β are given.

[0074] The beneficial effects of the present application are:

[0075] In the technical solution of the application, the data is processed and decided from a more essential angle in the way that human beings recognize the world, with the sensory-perception dichotomy as a theoretical guidance model, the primary information of visual stimulation and the experience information of deep processing are integrated to form a deep knowledge expression algorithm for identifying and distinguishing predicate similar relationships, and the semantic similarity dilemma between the head and tail predicates is solved. The process of model processing task is explicitly modeled to more clearly show the logical reasoning process of the scene graph generation task. Through the sensory capture of coarse-grained information and the perceptual reasoning of fine-grained focus, a new modeling idea of scene graph generation model design is proposed to alleviate the long-tail distribution problem of scene graph related data sets.

[0076] In the technical solution of the application, a sensory reasoning module is proposed to improve the ability to extract paired level object information from visual information and the ability to preliminarily integrate similar triples, and to improve the grasp of similar semantics of entity context. In order to further distinguish the features between semantically similar predicates, the perceptual reasoning module is designed to deeply divide the relationship instances between similar semantics, and to ensure that the predicates embedded between entity pairs can capture the discovered fine-grained semantics without affecting the semantics. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 The principle schematic diagram of the integrated sensory and perceptual clue process provided by the application;

[0078] Figure 2 The frame schematic diagram of the sensory clue extraction used by the embodiment of the application;

[0079] Figure 3 The influence schematic diagram of the predicate perception factor and the triple knowledge guide factor in the perceptual sensitive module used by the embodiment of the application on the performance of the model on the Visual Genome data set;

[0080] Figure 4 The value of the perceptual sensitive factor λ used by the embodiment of the application on the Visual Genome data set and the experimental data comparison diagram of different models;

[0081] Figure 5 The experimental result comparison diagram of the relationship prediction of different benchmark models used by the embodiment of the application on the Visual Genome data set;

[0082] Figure 6 The relationship prediction distribution diagram of the embodiment of the application compared with the Motifs method on the Visual Genome data set. DETAILED DESCRIPTION

[0083] The technical features of the application will be further described below in combination with specific embodiments.

[0084] The predicate similarity discrimination method integrating sensory and perceptual cues provided by the application includes the following steps:

[0085] The original image is input into a pre-trained object detection model to extract edge, position, space and other information of the image, multi-granularity entity features are extracted through a sensory sensitive module, and sensory cues are formed based on the extraction and processing of global and local information of visual information.

[0086] Then, the sensory cues are combined with a visual-concept alignment strategy to realize the relationship semantics of the coarse object pair, and a cross-attention mechanism of "subject-environment-relation" is constructed in the feature space.

[0087] Subsequently, the perceptual enhancement module fuses the paired context enhanced by the sensory sensitive module with global features to re-integrate visual and language information as relationship features, and distinguishes the similarity of entity pair features to divide into triples containing various semantics.

[0088] Subsequently, the classified triple categories are spatially mapped in the cognitive discrimination layer of the perceptual enhancement module to make abstract discrimination and form new perceptual cues to assist the model in predicate similarity discrimination, and the model further associates the visual scene of the same region uniformly.

[0089] As shown in Figure 1 The application provides a predicate similarity discrimination method integrating sensory and perceptual cues.

[0090] In one embodiment of the application, first, a target extractor is constructed by using Faster-RCNN to extract the appearance, space and other features of entity proposals, and capture their joint regions as joint features.

[0091] The sensory sensitive module captures intuitive visual cues in the image, aggregates entity-level visual information, and realizes coarse-grained filtering and feature alignment of candidate entity pairs.

[0092] The perceptual enhancement module further models the deep relationship between entity pairs, and realizes semantic discrimination and optimization of similar predicates through semantic mapping and discriminative learning.

[0093] Finally, the integration of sensory and perceptual cues realizes the collaborative optimization of semantic consistency constraints and discrimination ability.

[0094] Specifically, the steps of the scene graph generation method based on core knowledge reasoning are as follows:

[0095] 1) Faster R-CNN is used as the object detector to extract initial information. The readily available ResNeXt-101-FPN is used as its backbone network to generate convolutional feature representations for subsequent modules. For each object detected in the input image I, its corresponding appearance features a are predicted. i Spatial features s i and object category labels i These appearance and spatial features are initialized into corresponding entity representations. Then, the relative spatial representation between entity i and entity j is encoded with the joint ROI feature into a joint feature u. ij .

[0096] 2) The sensory sensitivity module incorporates visual sensitivity and detail perception into the task reasoning process. The appearance, spatial, and category features of the input objects directly influence the perceived range of the extracted entity information. Generally, entities that are closer together are more likely to form a relationship, while entities with high category relevance should form a reasonable relationship even if they are extremely far apart. Based on this, the sensory sensitivity module integrates input features according to different ways of visual stimulation to rationally plan visual information and enhance the model's understanding of the scene.

[0097] For the i-th and j-th entities, their appearance features a i ,a j Spatial features s i ,s j and category features l i ,l j All features are concatenated using a multilayer perceptron (MLP), and then the three types of features are concatenated to obtain the initial logic, which is used to construct the logical semantic features f of the sensory cues. ij Here, visual information is first divided into foreground pair information and overall information to improve the model's ability to process entity information in different spatial contexts. Spatial features are extracted using a multilayer perceptron (MLP) and a position-aware matrix (M), respectively, to obtain the encodings s of paired entities on the spatial features. ij .

[0098] s ij =M(MLP([s i s j ]))

[0099] Where M is the location-aware matrix, taking values ​​of 0 or 1, used to distinguish meaningful spatial locations from invalid entity pairs, and representing the [·] linking operation. Therefore, the entire sensory cue layer is defined as follows:

[0100]

[0101] Then processed sensory cues f ijCombined entity features e refined by the target encoder i ,e j The information is processed in the memory processing layer to extract the perceived information.

[0102] 3) Entity information is further refined and integrated here, and visual logical features obtained through intuition are processed into entirely new sensory cues:

[0103]

[0104] Among them, v i It is the context-sensitive information of entity i obtained after intuitively perceiving visual information, and it is also a major part of our sensory cues. It is obtained by weighting the suggestions of the original refined features and logical features for the same entity i. Then, the feature dimension d of the visual information is... v Align to the concept space dimension d.

[0105] The concept prototype matrix is ​​initialized using external knowledge GloVe, where each row... This represents a learnable concept embedding. The memory processing layer calculates the cosine similarity matrix between visual information and the concept prototype:

[0106]

[0107] Therefore, the soft alignment assignment of the sensory-sensitive module is obtained as follows:

[0108]

[0109] At this point, the model output loss is:

[0110]

[0111] Among them, y ij ∈{0,1} represents the true label.

[0112] Subsequently, sensory information, location-aware information, and global information are fed into the perception enhancement module for deep fusion and extraction. The sensory sensitivity module enhances the ability to represent sensory information by utilizing sensory cues to comprehensively capture local details (such as object edges) and global structure (such as object layout) of the image.

[0113] 4) In order to make reasonable use of the visual stimulus information extracted from the sensory-sensitive module, for a given instance, the predicate branch p i The sensory-sensitive module extracts rich sensitive information. The entity distribution X obtained from the Transformer layer e Similarity is measured using KL divergence to maximize the difference between them.

[0114]

[0115] in, This is the output after unifying the dimensions of features from different modalities through linear projection. Subsequently, the model uses the Softmax function to calculate the prediction confidence distribution and indexes its maximum value to obtain preliminary coarse-grained predicate labels. Specifically, the model first queries the set of k nearest-neighbor predicates {p1,...,p...} associated with the current entity. k The candidate labels and their confidence distributions were then analyzed. Subsequently, the final optimal label was determined by comparing the consistency measure between these candidate labels and the average one-hot representations of other query predicates.

[0116] During training, the model updates a maximum of two candidate predicate labels per round. When the prediction results of the two candidate labels are different and the consistency distance is less than the threshold θ, the model compares their confidence distributions, selects the label with higher confidence as the new predicted label, and relabels it as the pseudo label of the current instance; when the consistency distance is greater than θ, the original predicted label remains unchanged and is still used as a pseudo label for subsequent training.

[0117] 5) Finally, the model projects the obtained multi-label relation instances into predicate and triple embedding representations r through two fully connected layers, and integrates the semantic perception cues extracted during the perceptual discrimination process into the new relation representation. The training objective is to achieve synergistic optimization of semantic consistency constraints and discriminative ability by minimizing the distance between the relation embedding r and the corresponding predicate and triple category embeddings.

[0118]

[0119] in, This represents the final predicted predicate label. The subsequent perception refinement layer obtains the corresponding loss after perception discrimination:

[0120]

[0121] Where, p + With t + Let represent positive samples under the same predicate category and the same triple category, respectively. ρ represents the learnable temperature, N is the number of predicate categories, and α and β are learnable hyperparameters.

[0122] The final loss of the entire sensory and perceptual cue joint training It can be defined as:

[0123]

[0124] in, The cross-entropy loss represents the entity category. Let λ represent the cross-entropy loss of the predicate category, where λ is a learnable hyperparameter.

[0125] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should be protected by the present invention.

Claims

1. A method for predicate similarity discrimination that integrates sensory and perceptual cues, characterized in that: Includes the following steps: S1) Use the target extractor to extract the appearance features, spatial features, and entity labels of the proposed entities, and capture their joint region as a joint feature; S2) Low-level visual information is sent to the sensory-sensitive module for similar entity pair information integration. The visual features of the entity context in the data are perceived through an adaptive perception mechanism, which is used for further differentiation of similar predicate instances. S3) The perception enhancement module further distinguishes the differences between similar predicates; By modeling contextual information through global attention of Transformer, isolated object features are associated with the overall semantics of the scene; The contextual semantic information based on predicates is evaluated and classified in a new way according to similarity. Then, the confidence ranking is re-ranked based on the output results compared with the triple information of the same predicate. The S4 model projects the obtained multi-label relation instances into an embedded representation r of predicates and triples through two fully connected layers, and integrates the semantic perception cues extracted during the perception discrimination process into the new relation representation.

2. The predicate similarity discrimination method integrating sensory and perceptual cues according to claim 1, characterized in that: In step S1), the dataset is input into the target extraction model to extract initial information and detect the appearance features a of the object. i Spatial features s i and object category labels i These appearance and spatial features are initialized into corresponding entity representations. Then, the relative spatial representation between entity i and entity j is encoded with the joint ROI feature into a joint feature u. ij ; In step S1), Faster RCNN is used as the object detector to extract initial information, and ResNeXt-101-FPN is used as the backbone network to generate convolutional feature representations for subsequent modules; The target extraction model is trained on the Visual Genome dataset using sensory and perceptual cues. The model learns the entity information and interaction regions associated with each predicate in the corresponding dataset.

3. The predicate similarity discrimination method integrating sensory and perceptual cues according to claim 2, characterized in that: In step S1), the initial information extracted from the original image is combined to obtain the interaction information between pairs of entities; the extracted combined features and the initial features are sent together to the sensory sensitive module and the perceptual sensitive module for sensory cue extraction and perceptual cue discrimination.

4. The predicate similarity discrimination method integrating sensory and perceptual cues according to claim 1, characterized in that: In step S2), following the hybrid attention modeling approach, each layer contains two self-attention units and two cross-attention units to construct the scene context; Self-attention is responsible for capturing information within a modality, while cross-attention is responsible for processing information between modalities. The hybrid attention layer can be calculated as follows: Where X h and Y h It is the input X of the h-th layer h-1 Y h-1 Obtained through encoding; Through self-attention and cross-modal attention, the model dynamically focuses on key regions, suppresses irrelevant background noise, and optimizes the utilization of sensory information in sensory-sensitive modules; Visual information from such low-level entity embeddings and predicate embeddings is fed into the sensory-sensitive module for similar entity pair information integration.

5. The predicate similarity discrimination method integrating sensory and perceptual cues according to claim 1, characterized in that: In step S2), for the i-th and j-th entities, their appearance features a i ,a j Spatial features s i ,s j and category features l i ,l j All are stitched together by a multilayer perceptron (MLP); Appearance feature a i ,a j Spatial features s i ,s j and category features l i ,l j After further concatenation, the initial logic is obtained to construct the logical semantic features f of the sensory cues. ij ; Visual information is first divided into foreground information and overall information; Spatial features are extracted using a multilayer perceptron (MLP) and a position-aware matrix (M), respectively, to obtain the paired entity codes s on the spatial features. ij ; Encodings ij Represented as s ij =M(MLP([s i ,s j ])) Where M is the location-aware matrix, taking values ​​of 0 or 1, used to distinguish between meaningful spatial locations and invalid entity pairs, respectively, representing the [·] link operation; The entire sensory cue layer is defined as: Processed sensory cues f ij Combined entity features e refined by the target encoder i ,e j The information is processed in the memory processing layer to extract the perceived information.

6. The predicate similarity discrimination method integrating sensory and perceptual cues according to claim 5, characterized in that: In step S2), after extracting the perceptual information, the entity information is further refined and integrated, and the visual logical features obtained through intuition are processed into entirely new sensory cues: Among them, v i It is the context-sensitive information of entity i obtained after intuitively perceiving visual information, which is obtained by weighting the suggestions of the original refined features and logical features of the same entity i; Then, the feature dimension d of the visual information is... v Align to the concept space dimension d; initialize the concept prototype matrix using external knowledge GloVe, where each row... Represents a learnable concept embedding; The cosine similarity matrix between visual information and conceptual prototypes calculated by the memory processing layer is represented as: Therefore, the soft alignment assignment of the sensory-sensitive module is obtained as follows: At this point, the model output loss is: Where y ij ∈{0,1} represents the true label; Subsequently, the sensory information, location information, and global information are fed into the perception enhancement module for deep fusion and extraction; the sensory sensitivity module enhances the ability to represent sensory information by utilizing sensory cues to comprehensively capture the local details and global structure of the image.

7. The predicate similarity discrimination method integrating sensory and perceptual cues according to claim 1, characterized in that: In step S3), the initial entity is embedded into E. e : Predicate embedding E p : and real entity category C e and real predicate category C p The input is fed into the Transformer layer through a trainable parameter matrix W. e and W p Output the new distribution X e ,X p ; X e ,X p =Transformer(W e E e ,W p E p ); For a given instance, the predicate branch p i The sensitive information E extracted from the sensory sensitive module v : The entity distribution X obtained from the Transformer layer e Similarity is measured using KL divergence. in, This is the output after unifying the dimensions of different modal features through linear projection; Subsequently, the model uses the Softmax function to calculate the predicted confidence distribution and indexes its maximum value to obtain preliminary coarse-grained predicate labels; The perception enhancement module further distinguishes the differences between similar predicates; By modeling contextual information through global attention of Transformer, isolated object features are associated with the overall semantics of the scene; The contextual semantic information based on predicates is evaluated and classified in a new way according to similarity. Then, the confidence ranking is re-ranked based on the output results compared with the triple information of the same predicate.

8. The predicate similarity discrimination method integrating sensory and perceptual cues according to claim 7, characterized in that: In step S3), after the model uses the Softmax function to calculate the prediction confidence distribution, The head predicate set is set to include the predicates involved in the first 30% of the samples, while the remaining 70% of the samples are focused on using the tail predicate set. When the predicted result matches the true label and the label does not belong to the head predicate set, the model directly outputs the predicted result; When the predicted result is inconsistent with the true label, the model will activate the perception enhancement module to further mine deep semantic features and assume that other predicted labels are still semantically reasonable. At this stage, the model first queries the set of k nearest-neighbor predicates {p1,...,p...} associated with the current entity. k The candidate labels and their confidence distributions are then analyzed; subsequently, the final optimal label is determined by comparing the consistency measure between these candidate labels and the average one-hot representations of other query predicates. During training, the model updates at most two candidate predicate labels per round. When the prediction results of the two candidate labels are different and the consistency distance is lower than the threshold θ, the label with higher confidence is selected as the new prediction label by comparing the confidence distribution and it is relabeled as the pseudo label of the current instance. When the consistency distance is higher than θ, the original prediction label remains unchanged and is still used as a pseudo label for subsequent training.

9. The predicate similarity discrimination method integrating sensory and perceptual cues according to claim 7, characterized in that: In step S4), the training objective optimizes semantic consistency constraints and discriminative ability by minimizing the distance between the relation embedding r and the corresponding predicate and triplet category embeddings. in, The predicate labels for the final prediction; The loss obtained by the perceptual refinement layer after perceptual discrimination is: Where, p + With t + Let represent positive samples under the same predicate category and the same triplet category, respectively. ρ represents the learnable temperature, N is the number of predicate categories, and α and β are learnable hyperparameters. The final loss of the entire sensory and perceptual cue joint training It can be defined as: in, The cross-entropy loss represents the entity category. Let λ represent the cross-entropy loss of the predicate category, where λ is a learnable hyperparameter.

Images