An image description method fusing knowledge graph and sentiment recognition
By combining ResNet-50 and Faster R-CNN for image feature extraction, using graph attention networks to complete the knowledge graph, and introducing the MCAM module into the Transformer decoder, deep fusion of image, knowledge, and sentiment features is achieved, solving the problems of insufficient semantic understanding and lack of sentiment expression in existing technologies, and generating high-quality image descriptions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-02-27
- Publication Date
- 2026-06-02
AI Technical Summary
Existing image description technologies suffer from insufficient semantic understanding, lack of emotional expression, and weak external knowledge support, resulting in descriptions that lack logical coherence and emotional color.
Image feature extraction is performed using the Faster R-CNN object detection model combined with the ResNet-50 residual network. The knowledge graph is completed using a graph attention network. The deep fusion of image, knowledge and sentiment features is achieved through the Transformer decoder and the MCAM module to generate descriptive text.
It generates semantically accurate, content-rich, and emotionally resonant image descriptions, enhancing the logical coherence and emotional expressiveness of the descriptions.
Smart Images
Figure CN122135159A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to an image description method that integrates knowledge graphs and emotion recognition. Background Technology
[0002] Existing image captioning techniques are primarily based on an Encoder-Decoder architecture, using convolutional neural networks to extract image features and recurrent neural networks or Transformers to generate descriptive text. However, these methods generally suffer from the following prominent problems:
[0003] Insufficient semantic understanding: It is difficult to capture the complex semantic relationships between objects in an image, resulting in a lack of logical coherence in the description.
[0004] Lack of emotional expression: The generated descriptions are mostly objective statements and fail to convey the emotional nuances of the images.
[0005] Weak external knowledge support and crude integration: Although some studies have attempted to introduce knowledge graphs and sentiment recognition, they usually only perform simple feature splicing or shallow fusion, failing to achieve deep and dynamic interaction between visual content, structured knowledge and emotional information, resulting in stiff descriptions and insufficient information utilization.
[0006] In summary, there is an urgent need for a new image description method to achieve deep and adaptive fusion of image visual features, external structured knowledge, and image emotional features, thereby generating semantically accurate, content-rich, and emotionally relevant image descriptions. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention proposes an image description method that integrates knowledge graphs and emotion recognition. This method includes:
[0008] S1: Acquire the image to be processed and perform feature extraction and object detection to obtain object information, image features, and sentiment features of the image;
[0009] S2: Complete the external general knowledge graph using the graph attention network to obtain a dense knowledge graph; input the object information into the dense knowledge graph to query entity relationships and obtain relation knowledge vectors;
[0010] S3: The Transformer decoder is used to process the relational knowledge vector, image features, and sentiment features to obtain the image description text.
[0011] Preferably, the process of feature extraction and object detection of an image includes:
[0012] The Faster R-CNN object detection model, which combines ResNet-50 residual network, is used to process the image to obtain object information;
[0013] The regression head is used to accurately correct the position of the target bounding box, and the high-dimensional visual feature vectors corresponding to each target are extracted from the RoI Pooling layer and defined as image features.
[0014] A convolutional neural network with an integrated channel attention module is used to process images to obtain emotional features.
[0015] Preferably, in step S2, the process of obtaining the relational knowledge vector includes:
[0016] Using object information as semantic anchors, we retrieve associated entities and initial relationships in an external general knowledge graph and construct a sparse semantic subgraph.
[0017] A dense knowledge graph is obtained by using a graph attention network to perform link prediction and aggregation on sparse subgraphs.
[0018] Object information is input as entity information into a dense knowledge graph, and multiple candidate entity relationships are extracted through semantic retrieval.
[0019] Information gain filtering is performed on multiple candidate entity relationships to obtain the optimal entity relationship;
[0020] The selected optimal entity relationships are vectorized and mapped to the same dimensional space as the image features to generate a relationship knowledge vector K.
[0021] Furthermore, the process of information gain filtering for multiple candidate entity relationships includes:
[0022] Define a set of multiple candidate entity relations as a candidate relation set R;
[0023] Using image features V as visual evidence, we establish a mapping logic between candidate relation sets and image content, calculate the conditional probability distribution P(R|V) of each candidate relation in the current image context, and calculate the prior entropy H(R) of the candidate relation set R to characterize the prior uncertainty of the relation in the general knowledge graph.
[0024] Calculate the conditional entropy H(R|V) of the candidate relation set R under the condition of known image features V;
[0025] The information gain value of each candidate relation is calculated using the formula IG(R) = H(R) - H(R|V); candidate entity relations whose information gain value does not reach the threshold are eliminated.
[0026] The remaining candidate entity relationships are sorted from high to low according to their information gain values, and one or more relationship paths with the highest gain values are selected as the optimal entity relationships.
[0027] Preferably, in step S3, the process of using a Transformer decoder to process the relational knowledge vector, image features, and sentiment features includes:
[0028] The word sequence generated at the current time step t is converted into a word embedding vector through an embedding layer, and then represented by positional encoding to obtain the word embedding vector;
[0029] By inputting the word embedding vectors into the masked multi-head self-attention layer of the Transformer decoder, the hidden state vector of the current step is obtained by capturing the internal temporal dependencies of the generated text sequence. ;
[0030] Using learnable linear transformation matrices For the hidden state vector Perform projection mapping to generate query vectors for interaction with multimodal features;
[0031] Based on the query vector, the MCAM module is used to process the relational knowledge vector, image features, and sentiment features to obtain a fused context vector;
[0032] A feedforward neural network is used to process the fused context vector to obtain the image description text.
[0033] Furthermore, the MCAM module's processing of relational knowledge vectors, image features, and sentiment features includes:
[0034] The feature interaction layer processes the relational knowledge vector, image features, and sentiment features based on the query vector to generate intermediate knowledge vector, intermediate image vector, and intermediate sentiment vector.
[0035] A dynamic weight generation layer is used to process the intermediate knowledge vector, intermediate image vector, and intermediate sentiment vector respectively to obtain knowledge weight, image weight, and sentiment weight.
[0036] A weighted fusion layer is used to perform weighted summation of relational knowledge vectors, image features, and sentiment features based on knowledge weights, image weights, and sentiment weights to obtain a fused context vector.
[0037] Furthermore, the process by which the feature interaction layer processes the relational knowledge vector, image features, and sentiment features based on the query vector includes:
[0038] The feature interaction layer utilizes the query vector to perform parallel cross-attention calculations with the three feature paths respectively, achieving semantic alignment between heterogeneous modalities. The processing procedure is as follows:
[0039] intermediate image vector :
[0040]
[0041] Intermediate sentiment vector :
[0042]
[0043] intermediate knowledge vector :
[0044]
[0045] Where Q is the query vector; V, E, and K are the image feature, sentiment feature, and relational knowledge vectors, respectively; and For the first and second learnable linear transformation matrices, Scaling factor express Activation function.
[0046] Furthermore, the process of using a dynamic weight generation layer to process the intermediate knowledge vector, intermediate image vector, and intermediate sentiment vector separately includes:
[0047] The dynamic weight generation layer dynamically calculates the contribution weight of each intermediate vector in the current generation step through a gating mechanism. The processing procedure is expressed as follows:
[0048] For each intermediate vector, an unnormalized energy score is generated using linear projection mapping. :
[0049]
[0050] in, Let represent the unnormalized energy score of the i-th intermediate vector. These are the first, second, and third weight matrices, respectively. Represents the query vector. Represents the intermediate vector of the i-th path. Let V, E, and K represent the bias term, and V, E, and K be the image feature, sentiment feature, and relational knowledge vectors, respectively.
[0051] The energy scores are normalized using the Softmax function to obtain the final weights.
[0052] The beneficial effects of this invention are as follows:
[0053] Significantly enhanced fusion depth: Through the cross-attention mechanism of the MCAM module, deep interaction and dynamic weighted fusion of image, knowledge, and sentiment features in the vector space are achieved, rather than simple splicing, making the generated description highly consistent in semantics and sentiment.
[0054] Systematic Solution: The SDAM module provides a complete and optimized technical path from multi-source information extraction and knowledge enhancement to deep fusion generation, systematically ensuring the quality of the final description.
[0055] The generated descriptions are closer to human expression: Due to the deep integration of external knowledge and internal emotions, the generated descriptions are closer to human cognition and expression habits in terms of accuracy, completeness and emotional richness. Attached Figure Description
[0056] Figure 1 This is a flowchart of the image description method that integrates knowledge graph and emotion recognition in this invention;
[0057] Figure 2 This is a schematic diagram of knowledge graph completion and relationship query in this invention;
[0058] Figure 3 This is a schematic diagram of the MCAM module structure in this invention. Detailed Implementation
[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0060] This invention proposes an image description method that integrates knowledge graphs and emotion recognition, such as... Figure 1 As shown, the method includes the following:
[0061] S1: Acquire the image to be processed and perform feature extraction and object detection to obtain object information, image features, and emotional features of the image.
[0062] The process of feature extraction and object detection in an image includes:
[0063] The system uses a Faster R-CNN object detection model, improved based on the ResNet-50 residual network, to process images; specifically:
[0064] Candidate region extraction:
[0065] The image to be processed is input into the ResNet-50 backbone network for feature mapping to generate a shared feature map. The Region Proposal Network (RPN) is used to perform a sliding window scan on the feature map and extract candidate regions that may contain the target by using anchor boxes of various preset scales and proportions.
[0066] Object information acquisition:
[0067] The feature of candidate regions of different sizes is aligned to a fixed dimension by using the Region of Interest Pooling (RoI Pooling) layer. The input is then fed into the Classification Head to determine the category and obtain the object category label. After removing the labels with confidence scores below a preset threshold, the object information of the image (i.e., the object label list L) is formed.
[0068] The process of feature extraction from an image includes:
[0069] The regression head is used to accurately correct the target bounding box position, and the high-dimensional visual feature vectors corresponding to each target are extracted from the RoI Pooling layer and defined as image features V, which are used to characterize the local semantic information of salient objects in the image.
[0070] The system employs a convolutional neural network with an integrated channel attention module (SE module) to perform global perceptual analysis of images. Specifically:
[0071] Global space compression (Squeeze):
[0072] The image is input into a residual convolutional network, and global average pooling is applied at the feature layer level. Spatial dimensional features compressed to The channel descriptors are used to obtain statistical information for each channel globally.
[0073] Emotional channel stimulation (excitation):
[0074] Two fully connected layers are used to perform a nonlinear transformation on the channel descriptors to learn the dependencies between channels, and a sigmoid activation function is used to generate weight coefficients for each channel. These weight coefficients reflect the influence of image features such as color, contrast, and composition on emotional expression.
[0075] Feature reconstruction and output:
[0076] The generated weight coefficients are multiplied and weighted with the corresponding channels of the original feature map to strengthen the feature channels that contribute to emotional expression, suppress irrelevant noise, and finally output the emotional feature E that represents the emotional atmosphere of the image.
[0077] S2: Complete the external general knowledge graph using the graph attention network to obtain a dense knowledge graph; input the object information into the dense knowledge graph to query entity relationships and obtain relationship knowledge vectors.
[0078] The dense knowledge graph used in this invention is completed in advance or in real time through the following process:
[0079] Subgraph extraction: First, the object labels obtained in S1 are used as anchor entities. The associated entities and attributes in the k-th order neighborhood are retrieved from external general knowledge graphs such as ConceptNet to construct the initial sparse semantic subgraph.
[0080] Graph attention reasoning: Utilizing graph attention networks to perform feature evolution on sparse subgraphs. Calculate the attention score between the central entity i and its neighboring nodes j. .
[0081] Link prediction and aggregation: Calculate the attention coefficients between nodes and deduce the visually invisible implicit logical relationships by aggregating neighborhood node information (this step is the core of the completion), that is, through the normalized attention weights. By aggregating information from neighboring nodes, implicit logical relationships that are not visually presented can be predicted (e.g., predicting the association between "riding" or "repairing" from "person" and "bicycle"), transforming the originally fragmented or sparse knowledge structure into a dense knowledge graph with rich logical connections.
[0082] like Figure 2 As shown, the process of obtaining the relational knowledge vector includes:
[0083] Object information is input as entity information into a dense knowledge graph completed by a graph attention network, and multiple candidate entity relationships related to the current set of objects are extracted through semantic retrieval.
[0084] Information gain filtering is performed on multiple entity relationships to obtain the optimal entity relationship. The specific process of information gain filtering on multiple entity relationships is as follows:
[0085] First, the various candidate entity relations output by the dense knowledge graph are defined as the candidate set R, and the image features V extracted in step S1 are introduced as screening evidence. Then, the contribution of each candidate relation to the current image scene description is evaluated using information theory criteria. Specifically, the information gain value of each candidate relation is obtained by calculating the prior entropy H(R) and the conditional entropy H(R|V) under known image features of each candidate relation. During this process, the system automatically identifies and removes common-sense redundant information with low information gain (judged by a preset threshold) such as generalized relationships like "the object is in the scene". By comparing the magnitude of the IG value, it accurately selects the relationship path with the highest entropy reduction and the ability to eliminate the uncertainty of the current visual scene as the optimal entity relationship, that is, it selects one or more relationship paths with the highest gain value as the optimal entity relationship.
[0086] After obtaining the optimal entity relationship, a semantic mapping mechanism is used to input the textual description of this optimal relationship (e.g., "person-cycling-bicycle") into a pre-trained semantic encoder for vectorization. Finally, a learnable linear projection layer is used. Adjust its feature dimensions to place it in the same vector space as image features and emotion features, and finally generate a relational knowledge vector K that represents deep logical relationships.
[0087] S3: The Transformer decoder is used to process the relational knowledge vector, image features, and sentiment features to obtain the image description text.
[0088] S31: Convert the word sequence generated at the current time step t into a word embedding vector through the embedding layer, and combine it with position encoding for representation to obtain the word embedding vector.
[0089] At each time step t in the decoder's text generation, the input information at that time is first obtained, which is the word vector obtained by transforming the word generated in the previous time step through the word embedding layer, and the positional encoding information is superimposed to obtain the word embedding vector.
[0090] S32: Input the word embedding vector into the masked multi-head self-attention layer of the Transformer decoder to obtain the hidden state vector of the current step by capturing the internal temporal dependencies of the generated text sequence. .
[0091] Subsequently, the input vector is fed into the Masked Multi-Head Self-Attention layer of the Transformer decoder. By capturing the temporal dependencies within the generated text sequence, the hidden state vector for the current step is extracted. .
[0092] S33: Utilizing a learnable linear transformation matrix For the hidden state vector Perform projection mapping to generate query vectors for interacting with multimodal features.
[0093] Finally, a learnable linear transformation matrix is used. The hidden state vector is projected and mapped using the following formula: This generates a query vector Q for interactive queries with multimodal features. This query vector Q carries the model's "semantic expectation" of the direction of subsequent text generation at the current time step.
[0094] S34: Based on the query vector, the MCAM module is used to process the relational knowledge vector, image features, and sentiment features to obtain the fused context vector.
[0095] This invention designs an MCAM module, which differs from traditional attention mechanisms. It can simultaneously receive three features—image, knowledge, and sentiment—as keys and interact with the current state (query) of the decoder. Through cross-attention calculation, it dynamically generates context-dependent fusion weights for each feature, thereby achieving adaptive deep feature fusion and ultimately generating high-quality descriptive text.
[0096] like Figure 3 As shown, the specific processing procedure and data flow are as follows:
[0097] The relational knowledge vector, image features, and sentiment features are respectively input into the feature interaction layer for processing, generating an intermediate knowledge vector. intermediate image vectors and intermediate sentiment vector Specifically:
[0098] For each feature, the system generates a key vector and a value vector using the corresponding linear transformation matrix, and then performs a dot product operation between the query vector Q and the key vector. The feature interaction layer calculation process is represented as follows:
[0099]
[0100]
[0101]
[0102] in, and These are the first and second linear transformation matrices corresponding to each mode. The scaling factor is used. After processing and Softmax normalization, the attention distribution of the query vector to each feature is calculated. Then, the value vectors are weighted and aggregated to generate intermediate image vectors, intermediate sentiment vectors, and intermediate knowledge vectors. This process achieves a preliminary alignment of visual space, emotional atmosphere, and common sense logic in the current context.
[0103] A dynamic weight generation layer is used to process the intermediate knowledge vector, intermediate image vector, and intermediate sentiment vector respectively to obtain the knowledge weights. Image weights And emotional weight Specifically:
[0104] The system adaptively adjusts the contribution of each modal component through a gating mechanism; that is, for each intermediate vector, it uses linear projection mapping to generate an unnormalized energy score. The dynamic weight generation layer is represented as:
[0105]
[0106] in, Let represent the unnormalized energy score of the i-th intermediate vector. These are the first, second, and third weight matrices, respectively. Represents the query vector. Represents the intermediate vector of the i-th path. Let V represent the bias term, and let E and K be the image feature, sentiment feature, and relational knowledge vectors, respectively.
[0107] The energy scores are normalized using the Softmax function to obtain the final weights:
[0108]
[0109] This weight reflects the model's dependence on different information at the current generation step; for example, the sentiment weight when generating emotional modifiers. The value will increase adaptively; this layer enables the model to adaptively adjust its dependence on visual space, sensory atmosphere, or logical knowledge when generating different types of words.
[0110] A weighted fusion layer is used to sum the intermediate features based on knowledge weights, image weights, and sentiment weights to obtain a fused context vector. The weighted fusion layer is represented as:
[0111]
[0112] The generated fusion context vector contains the most critical multimodal evidence at the current moment.
[0113] S35: A feedforward neural network is used to process the fused context vector to obtain the image description text.
[0114] fusion context vector The input decoder uses a feedforward network (FFN) layer. This layer further enhances feature representation through two linear transformations and activation functions, the process of which is represented as follows:
[0115]
[0116] Subsequently, a linear layer is used to map the processed features to the vocabulary space, and the probability distribution of each candidate word generated at the current time step is calculated using the Softmax function.
[0117]
[0118] The descriptive word for the current time step is determined through probability sampling. This output word will then serve as the input for the next time step, and the above process will be repeated until a complete descriptive text is generated.
[0119] In summary, this invention employs Faster-RCNN and ResNet-50 to extract image features and targets; utilizes graph attention networks to complete and learn representations of the knowledge graph, querying relationships between targets; identifies image sentiment by introducing a channel attention module into the ResNet-50 network; and innovatively integrates MCAM (Multimodal Cross-Attention Module) at the Transformer decoding end to dynamically fuse image features, knowledge vectors, and sentiment features to generate descriptive text. This invention systematically solves the problems of insufficient semantic understanding, lack of sentiment expression, and weak external knowledge support in existing technologies through the SDAM (Semantic Dynamic Fusion Attention Framework), generating highly accurate, complete, and emotionally rich image descriptions with promising application prospects.
[0120] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An image description method integrating knowledge graphs and emotion recognition, characterized in that, Includes the following steps: S1: Acquire the image to be processed and perform feature extraction and object detection to obtain object information, image features, and sentiment features of the image; S2: Complete the external general knowledge graph using the graph attention network to obtain a dense knowledge graph; input the object information into the dense knowledge graph to query entity relationships and obtain relation knowledge vectors; S3: The Transformer decoder is used to process the relational knowledge vector, image features, and sentiment features to obtain the image description text.
2. The image description method integrating knowledge graph and emotion recognition according to claim 1, characterized in that, The process of feature extraction and object detection in an image includes: The Faster R-CNN object detection model, which combines ResNet-50 residual network, is used to process the image to obtain object information; The regression head is used to accurately correct the position of the target bounding box, and the high-dimensional visual feature vectors corresponding to each target are extracted from the RoI Pooling layer and defined as image features. A convolutional neural network with an integrated channel attention module is used to process images to obtain emotional features.
3. The image description method integrating knowledge graph and emotion recognition according to claim 1, characterized in that, Step S2, the process of obtaining the relational knowledge vector includes: Using object information as semantic anchors, we retrieve associated entities and initial relationships in an external general knowledge graph and construct a sparse semantic subgraph. A dense knowledge graph is obtained by using a graph attention network to perform link prediction and aggregation on sparse subgraphs. Object information is input as entity information into a dense knowledge graph, and multiple candidate entity relationships are extracted through semantic retrieval. Information gain filtering is performed on multiple candidate entity relationships to obtain the optimal entity relationship; The selected optimal entity relationships are vectorized and mapped to the same dimensional space as the image features to generate a relationship knowledge vector K.
4. The image description method integrating knowledge graph and emotion recognition according to claim 3, characterized in that, The process of information gain filtering for multiple candidate entity relationships includes: Define a set of multiple candidate entity relations as a candidate relation set R; Using image features V as visual evidence, we establish a mapping logic between candidate relation sets and image content, calculate the conditional probability distribution P(R|V) of each candidate relation in the current image context, and calculate the prior entropy H(R) of the candidate relation set R to characterize the prior uncertainty of the relation in the general knowledge graph. Calculate the conditional entropy H(R|V) of the candidate relation set R under the condition of known image features V; The information gain value of each candidate relation is calculated using the formula IG(R) = H(R) - H(R|V); candidate entity relations whose information gain value does not reach the threshold are eliminated. The remaining candidate entity relationships are sorted from high to low according to their information gain values, and one or more relationship paths with the highest gain values are selected as the optimal entity relationships.
5. The image description method integrating knowledge graph and emotion recognition according to claim 1, characterized in that, Step S3, which involves processing the relational knowledge vector, image features, and sentiment features using a Transformer decoder, includes: The word sequence generated at the current time step t is converted into a word embedding vector through an embedding layer, and then represented by positional encoding to obtain the word embedding vector; By inputting the word embedding vector into the masked multi-head self-attention layer of the Transformer decoder, the hidden state vector of the current step is obtained by capturing the internal temporal dependencies of the generated text sequence. The hidden state vector is projected onto a learnable linear transformation matrix to generate a query vector for interaction with multimodal features. Based on the query vector, the MCAM module is used to process the relational knowledge vector, image features, and sentiment features to obtain a fused context vector; A feedforward neural network is used to process the fused context vector to obtain the image description text.
6. The image description method integrating knowledge graph and emotion recognition according to claim 5, characterized in that, The MCAM module's processing of relational knowledge vectors, image features, and sentiment features includes: The feature interaction layer processes the relational knowledge vector, image features, and sentiment features based on the query vector to generate intermediate knowledge vector, intermediate image vector, and intermediate sentiment vector. A dynamic weight generation layer is used to process the intermediate knowledge vector, intermediate image vector, and intermediate sentiment vector respectively to obtain knowledge weight, image weight, and sentiment weight. A weighted fusion layer is used to perform weighted summation of relational knowledge vectors, image features, and sentiment features based on knowledge weights, image weights, and sentiment weights to obtain a fused context vector.
7. The image description method integrating knowledge graph and emotion recognition according to claim 6, characterized in that, The process by which the feature interaction layer processes the relational knowledge vector, image features, and sentiment features based on the query vector includes: The feature interaction layer utilizes the query vector to perform parallel cross-attention calculations with the three feature paths respectively, achieving semantic alignment between heterogeneous modalities. The processing procedure is as follows: intermediate image vector : ； Intermediate sentiment vector : ； intermediate knowledge vector : ； Where Q is the query vector; V, E, and K are the image feature, sentiment feature, and relational knowledge vectors, respectively; and For the first and second learnable linear transformation matrices, Scaling factor express Activation function.
8. The image description method integrating knowledge graph and emotion recognition according to claim 6, characterized in that, The process of using a dynamic weight generation layer to process the intermediate knowledge vector, intermediate image vector, and intermediate sentiment vector includes: The dynamic weight generation layer dynamically calculates the contribution weight of each intermediate vector in the current generation step through a gating mechanism. The processing procedure is expressed as follows: For each intermediate vector, an unnormalized energy score is generated using linear projection mapping. : ； in, Let represent the unnormalized energy score of the i-th intermediate vector. These are the first, second, and third weight matrices, respectively. Represents the query vector. Represents the intermediate vector of the i-th path. Let V, E, and K represent the bias term, and V, E, and K be the image feature, sentiment feature, and relational knowledge vectors, respectively. The energy scores are normalized using the Softmax function to obtain the final weights.