A multi-modal social relation extraction method based on hypergraph attention neural network

Abstract

The application discloses a kind of multi-modal social relationship extraction methods based on hypergraph attention neural network, comprising: obtaining multi-modal dataset;Multi-modal dataset is extracted, and text features and image features are obtained;Based on text features and image features, node feature matrix is obtained, and according to node feature, hypergraph correlation matrix is obtained;Node feature matrix and hypergraph correlation matrix are input into hypergraph neural network, and high-order complex relationship between nodes is learned, and the updated hypergraph node feature is pooled, and hypergraph feature is obtained;Hypergraph feature is denoised using cross diffusion attention module, and hypergraph feature representation is obtained;Hypergraph feature and text features and image features are spliced, and final features are obtained;Based on final features, social relationship prediction results are obtained.The application constructs multi-modal hypergraph, and proposes hypergraph attention neural network, and effectively learns high-order complex relationship between multi-modal data.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing and multimodal social relationship extraction, and in particular relates to a multimodal social relationship extraction method based on a hypergraph attention neural network. Background Art

[0002] Social relations are specific relationships in daily human life. They define the connection between two people in the real or virtual world. Social relation extraction (SRE) aims to infer social relations between two people from text, personal photo albums, or movies. This method has proven to have significant practical value. For example, it can capture social connections, enabling machines to better understand human behavior. In recent years, SRE has attracted increasing research interest. Fairclough extracted social relations from Weibo text. Du extracted social relations from the Four Classical Classics of Chinese Literature. Du mined social relations from Weibo text. Zhang proposed learning facial representations to capture facial attributes and performing pairwise face reasoning to predict relationships. These studies all focus on unimodal information and ignore the high coupling of multimodal information. Multimodal social relation extraction methods can distinguish social relations by adding age-dependent visual cues provided by facial information. Therefore, the multidimensional information provided by facial features can be used to enrich unimodal textual information, thereby improving performance. MNRE is the first dataset for multimodal relation extraction. Existing few-shot multimodal fusion has been studied in the field of visual question answering, including image captioning, action recognition, and sentiment analysis. Studies have shown that the performance of these tasks can be improved by fusing information from different modalities in a few-shot learning. Recent works, such as MNRE and HVPNet, have focused on multimodal relation extraction techniques and demonstrated the effectiveness of aggregating image information for relation extraction. MKGformer and MOREformer promote the multi-level fusion of visual and textual information, processing and integrating information from both modalities. DGF-PT represents a recent advancement that adopts a hint-based autoregressive encoder that combines two different types of prefix adjustments to promote stronger connections between entity pairs, textual context, and images / objects. However, existing methods ignore the multiple relationships of different entity pairs in a sentence due to the variations in entity pairs. In multimodal relation extraction tasks, images can provide valuable information. However, the potential of leveraging image information in different ways for different entity pairs has not been fully explored.

[0003] However, social relationship extraction methods based on multimodal information still face several key challenges. Due to the differences between modalities, information fusion is difficult. Insufficient fusion fails to fully utilize the assistance between modalities, while excessive fusion discards the original unimodal information. Furthermore, relationships exist in an unbalanced distribution, which leads to recognition bias between label-rich and label-poor relationships. Furthermore, existing methods can only capture shallow relationships between multimodal information, failing to capture higher-order, complex relationships between them. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a multimodal social relationship extraction method based on a hypergraph attention neural network, which addresses this gap by capturing high-order correlations between entity pairs and related image objects, and integrating intra-modal and inter-modal correlations that are truly useful for each entity pair.

[0005] The present invention provides a multimodal social relationship extraction method based on a hypergraph attention neural network, comprising:

[0006] Acquire multimodal datasets;

[0007] Performing feature extraction on the multimodal dataset to obtain text features and image features;

[0008] Based on the text features and image features, a node feature matrix is ​​obtained, and according to the node features, a hypergraph association matrix is ​​obtained;

[0009] Input the node feature matrix and the hypergraph association matrix into the hypergraph neural network to learn the high-order complex relationships between nodes, perform pooling processing on the updated hypergraph node features, and obtain hypergraph features;

[0010] Denoising the hypergraph features using a cross-diffusion attention module to obtain a hypergraph feature representation;

[0011] Concatenate the hypergraph features, the text features, and the image features to obtain final features;

[0012] Based on the final features, a social relationship prediction result is obtained.

[0013] Optionally, performing feature extraction on the multimodal dataset to obtain text features and image features includes:

[0014] Extracting text features of the multimodal dataset using a text encoder;

[0015] The FaceNet model is used to extract image features of the multimodal dataset.

[0016] Optionally, extracting image features of the multimodal dataset using a FaceNet model includes:

[0017] Inputting the multimodal dataset into the FaceNet model to obtain facial features of the head entity and the tail entity;

[0018] The facial features of the head entity and the tail entity are normalized and spliced ​​to obtain the image features.

[0019] Optionally, obtaining a hypergraph association matrix based on the text features and the image features includes:

[0020] Determining node features based on the text features and image features;

[0021] According to the node feature, the node feature and the K nearest neighbor nodes of the node feature are calculated by Euclidean distance to form a hyperedge feature;

[0022] The hypergraph association matrix is ​​obtained according to the node features and the hyperedge features.

[0023] Optionally, the node feature matrix and the hypergraph association matrix are input into a hypergraph neural network to learn high-order complex relationships between nodes, and the updated hypergraph node features are pooled to obtain hypergraph features, including:

[0024] Acquiring initial node features according to the multimodal dataset;

[0025] Performing feature conversion on the initial node features through a learnable filter matrix to obtain node features after feature conversion;

[0026] Obtaining hyperedge features based on the node features after feature conversion and the hypergraph association matrix;

[0027] Aggregating the hyper-edge features to obtain updated node features;

[0028] Based on the updated node features and hyperedge features, learning high-order complex relationships between nodes;

[0029] The updated hypergraph node features are pooled to obtain hypergraph features.

[0030] Optionally, performing denoising on the hypergraph features using a cross-diffusion attention module to obtain a hypergraph feature representation includes:

[0031] Computing affinities between different tokens within the modality of the hypergraph features and text features via self-attention;

[0032] Normalizing the affinity to obtain a normalized affinity matrix;

[0033] obtaining pairwise affinities between tokens of two different modalities based on the normalized affinity matrix;

[0034] Based on the pairwise affinity, obtaining contextual information of two different modalities;

[0035] The context information is integrated to obtain the hypergraph feature representation.

[0036] Compared with the prior art, the present invention has the following advantages and technical effects:

[0037] 1. This paper constructs a multimodal hypergraph and proposes a hypergraph attention neural network to effectively learn high-order complex relationships between multimodal data.

[0038] 2. This paper combines a hypergraph neural network with a cross-diffusion attention mechanism to effectively reduce the noise generated during hypergraph learning while enhancing the affinity between hypergraph features and text semantics. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0040] Figure 1 This is a flow chart of a multimodal social relationship extraction method based on a hypergraph attention neural network according to an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of hyperedge generation according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of a super-edge convolutional layer according to an embodiment of the present invention;

[0043] Figure 4 Schematic diagram of the cross-diffusion attention mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] This paper proposes a multimodal social relationship extraction method based on hypergraph attention neural network. Figure 1As shown, the specific steps include:

[0047] Acquire multimodal datasets;

[0048] Perform feature extraction on multimodal datasets to obtain text features and image features;

[0049] Based on text features and image features, obtain the node feature matrix, and according to the node features, obtain the hypergraph association matrix;

[0050] Input the node feature matrix and hypergraph association matrix into the hypergraph neural network to learn the high-order complex relationships between nodes, and perform pooling on the updated hypergraph node features to obtain hypergraph features;

[0051] Use the cross-diffusion attention module to denoise the hypergraph features and obtain the hypergraph feature representation;

[0052] Combine hypergraph features, text features, and image features to obtain the final features;

[0053] Based on the final features, the social relationship prediction results are obtained.

[0054] Specifically, the present invention first constructs a multimodal hypergraph for the data, in which each sentence (global information of the sentence, head entity, tail entity) and the corresponding image (and the face in the image) are represented as a hypergraph. Next, the model automatically learns the edges and weights of the hypergraph to learn a better structure and capture the potential associations between entities and relationships. The model takes a sentence and a set of images as input, and generates feature representations of the head / tail entities and facial images through a sentence encoder and a face encoder. The sentence encoder extracts text features through a pre-trained language model. The face encoder generates facial features by extracting bounding boxes and corresponding visual features from a pre-trained face recognition model. The generated features are then used to construct a multimodal hypergraph, and the high-order complex relationships between multimodal information are obtained through a hypergraph convolutional neural network. The cross-diffusion attention mechanism is then used to reduce the impact of noise generated during the hypergraph learning process, while enhancing the affinity between the hypergraph features and the text semantics.

[0055] Furthermore, feature extraction is performed on the multimodal dataset to obtain text features and image features, including:

[0056] Use text encoders to extract text features from multimodal datasets;

[0057] The FaceNet model is used to extract image features from multimodal datasets.

[0058] Furthermore, the FaceNet model is used to extract image features of multimodal datasets, including:

[0059] Input the multimodal dataset into the FaceNet model to obtain the facial features of the head entity and the tail entity;

[0060] The facial features of the head entity and the tail entity are normalized and concatenated to obtain image features.

[0061] Specifically, the encoder

[0062] (1) Sentence Encoder

[0063] In the sentence encoder, each sentence is encoded into a vector. This paper uses the pre-trained language model BERT as the language encoder. Given a sentence w1,w2,...,w consisting of m tokens m , where the names of the characters have been masked. The present invention constructs a token sequence s′=[CLS],w1,w2,...,w m ,[SEP], where [CLS] and [SEP] are special tokens introduced in BERT. BERT takes the token sequence s' as input and outputs a 768-dimensional vector sequence The present invention uses a fully connected layer to transform the first vector V [CLS] Processing to obtain B = tanh (W1V [CLS] +b1).

[0064] (2) Facial Information Encoder

[0065] For two given images i h and i t , respectively including the head entity h and the tail entity t, the present invention can be applied to R 1792 We get two vectors f h and f t In order to obtain high-level features, the present invention adds a fully connected layer and a batch normalization layer to obtain the final facial features of the two entities. More precisely, the two vectors v h and v t Representing the final facial features of h and t, respectively, are represented by f h and f t Refined.

[0066] v h =BatchNorm(W2f h +b2) (1)

[0067] v t =BatchNorm(W3f t +b3) (2)

[0068] Where W2,W3∈R 512×1792 ,b2,b3∈R512 These are all trainable parameters.

[0069] Finally, the present invention is to v h and v t Perform L2 normalization and connect them to get the final face feature vector F h,t , defined as

[0070] F h,t =[L2Norm(v h ); L2Norm(v t )] (3)

[0071] Furthermore, based on the text features and the image features, obtaining the hypergraph association matrix includes:

[0072] Determine node features based on text features and image features;

[0073] According to the node features, the node features and the K nearest neighbor nodes of the node features are calculated through the Euclidean distance to form the hyperedge features;

[0074] Obtain the hypergraph association matrix based on node features and hyperedge features.

[0075] Specifically, the construction of a multimodal hypergraph is different from that of a simple graph. Hyperedges in a hypergraph connect two or more vertices. The hypergraph is defined as G = (V, E, W), which contains a vertex set V and a hyperedge set E. The hyperedge weight matrix W = diag (w1, w2, ..., w |E| ) is the hyperedge e j (j=1,2,...,|E|) assigns the corresponding weight w j The hypergraph G can be represented by a |V|×|E| incidence matrix H, whose entries are defined as:

[0076]

[0077] In this way, each hyperedge e connects all associated nodes v, indicating the correlation between them. The degree of nodes and hyperedges in the hypergraph can be calculated based on the hypergraph association matrix H. The degree of a node is defined as the number of hyperedges containing the node, and the degree of a hyperedge is defined as the number of nodes contained in the hyperedge, which are defined as:

[0078]

[0079] Given the features of each vertex, the second class E can be generated by finding the neighbors of each vertex in the feature space featureFeatures. Different strategies can be used here. Given a vertex as the centroid, you can connect its k nearest neighbors in the feature space with hyperedges, or select all neighbors within a distance d from the centroid (including the centroid), as follows Figure 2 As shown. The present invention constructs a hypergraph based on the Euclidean distance to calculate the distance between two features. In the hypergraph construction, each vertex represents a visual object or text feature, and each hyperedge is formed by connecting a vertex and its K nearest neighbors, which forms N hyperedges connecting K+1 vertices. Therefore, the present invention obtains the association matrix H∈R N×N , where the N×(K+1) term is equal to 1 and the other terms are equal to 0.

[0080] Furthermore, the node feature matrix and the hypergraph association matrix are input into the hypergraph neural network to learn the high-order complex relationships between nodes. The updated hypergraph node features are pooled to obtain the hypergraph features including:

[0081] Obtain initial node features based on the multimodal dataset;

[0082] Perform feature transformation on the initial node features through a learnable filter matrix to obtain the node features after feature transformation;

[0083] According to the node features after feature conversion and the hypergraph association matrix, the hyperedge features are obtained;

[0084] Aggregate the hyper-edge features to obtain updated node features;

[0085] Based on the updated node features and hyperedge features, high-order complex relationships between nodes are learned;

[0086] The updated hypergraph node features are pooled to obtain hypergraph features.

[0087] Specifically, in the extraction of multimodal social relations, this paper constructs a hypergraph attention neural network model, such as Figure 1 As shown. The multimodal dataset includes a text dataset and an image dataset, each of which contains multiple nodes with features. Then, multiple hyperedge structure groups are constructed based on the complex correlation of the multimodal dataset. The present invention connects these hyperedge groups to generate a hypergraph adjacency matrix H, and then inputs the hypergraph adjacency matrix H and the node features into the model. Specifically, a hyperedge convolution layer f(X, W, Θ) can be constructed as follows.

[0088]

[0089] Among them, X (l) is the hypergraph signal of the lth layer, the initial value is the feature matrix input X, and σ is the nonlinear activation function. v and D ewhere represents the node degree matrix and the hyperedge degree matrix, respectively, and plays a normalization role in Equation (7). W represents the hyperedge weight matrix, which is initialized to the identity matrix. Θ is a parameter that needs to be learned during training.

[0090] The hypergraph convolutional neural network model is based on spectral convolution on the hypergraph. Here, the present invention further studies the characteristics of the model that utilizes high-order correlations between data. Figure 3 As shown in , the hyperedge convolution layer can perform node-edge-node transformation and better refine features by using the multimodal hypergraph structure. More specifically, first, the learnable filter matrix Θ (l) C1-dimensional feature X of the initial node (l) Processing is performed to extract C2-dimensional features. Then, by comparing with H T ∈R E×N Multiplication is used to collect node features based on hyperedges to form hyperedge features Finally, by multiplying the matrix H, the related hyper-edge features are aggregated to output the new node feature X (l+1) Therefore, the hyperedge convolution layer can effectively extract high-order correlations on the hypergraph through node-edge-node transformation. For each multimodal hypergraph, after l layers of feature propagation and aggregation, all node features on the constructed multimodal hypergraph can be calculated. The present invention uses mean pooling to learn node representations to obtain the final multimodal hypergraph information representation h G .

[0091]

[0092] Furthermore, the cross-diffusion attention module is used to denoise the hypergraph features, and the hypergraph feature representation is obtained, including:

[0093] Compute the affinity between different tokens within the modality of hypergraph features and text features via self-attention;

[0094] Normalize the affinity to obtain a normalized affinity matrix;

[0095] Based on the normalized affinity matrix, pairwise affinity between tokens of two different modalities is obtained;

[0096] Based on pairwise affinity, contextual information of two different modalities is obtained;

[0097] Integrate contextual information to obtain hypergraph feature representation.

[0098] Specifically, in fact, when the hypergraph convolutional neural network processes multimodal data, it brings high-order complex relationship information, but also brings noise and loss of some original modal information. To solve this problem, the present invention uses the cross-diffusion attention mechanism (CDA) to reduce the noise and redundant information generated in the process of the hypergraph convolutional neural network acquiring high-order complex relationship features, while retaining more text modal semantic information through text semantic guidance. For the self-attention mechanism, let X t ∈R n×d ,h G ∈R n×d Denote the text semantics and hypergraph modal features, respectively, where n represents the number of tokens in the two modalities and d represents the feature dimension of the tokens. To capture the dependencies between tokens within each modality, the present invention uses a self-attention architecture. Specifically, the affinity between different tokens in the text modality is first calculated.

[0099]

[0100] where Q t and K t By X t Perform two linear transformations to obtain: Indicates the scale factor. t Based on this, the present invention obtains the context-aware representation U of the token in modality t by using a message passing mechanism. t =S t V t Similarly, the present invention can obtain the S of the hypergraph feature v and U v .

[0101] The key to the cross-diffusion attention mechanism is to communicate information between tokens of different modalities. The traditional way to solve this problem is to design a cross-attention mechanism. Previous work defines the cross-attention mechanism (CA) as:

[0102]

[0103] U t→v =S t→v V v (11)

[0104] Similarly, we can get S v→t and U v→t Here, the present invention can be noted that the calculation of cross affinity is the core step of CA. However, although Q t and K v They come from different modalities and are transformed into the original input features X t 、h G But Q tand K v The modal / domain gap between is still not negligible. This inherent modal gap makes the does not reliably reflect the pairwise affinity S between tokens of different modalities t→v .

[0105] To overcome the above problems, this paper proposes a new cross-diffusion attention mechanism with reference to the regularized diffusion process. Unlike conventional CA, which defines cross affinity in feature space, CDA is defined in metric space, such as Figure 4 As shown. The present invention can first obtain the affinity S between modality t / modality v tokens through self-attention in formula (9) t / S v Then according to S t / S v Define S t→v Specifically, let and represents the normalized affinity matrix, where D t and D v is an angle matrix. The present invention defines cross-diffusion attention as follows:

[0106]

[0107] U t→v =S t→v V v (13)

[0108] Where A=S t +S v , the parameter μ∈(0,1) represents the balancing hyperparameter. It can be initialized to the identity matrix I, or to the initial affinity matrix obtained by other methods, such as the cross affinity of formula (10). is the unit matrix I. Similarly, the present invention can obtain S v→t and U v→t By using the self-attention and cross-diffusion attention modules, the present invention can obtain the context information representation of two modalities {U t ,U t→v},{U v ,U v→t By integrating them together, the present invention can obtain a more reliable and effective representation of the two modalities H t ,H v ∈R n×d .

[0109] H t =C t (U t ||U t→v) (14)

[0110] H v =C v (U v ||U v→t ) (15)

[0111] Among them, || represents the splicing operation. C t (·) and C v (·) represents two 1×1 convolutional layers with different parameters. The cross-diffusion attention module converts H t and H v Aggregated together through a feedforward neural network, the final feature Z is obtained.

[0112] Z=FFN(LN(C(H t ||H v ))) (16)

[0113] Where C(·) represents a 1×1 convolutional layer. FFN represents two fully connected layers with a nonlinear activation function GELU. LN represents the inter-layer normalization operation.

[0114] Given a support set S and a query set Q. Each instance is represented as follows.

[0115] L = LayerNorm[B; Z; F h,t ] (17)

[0116] In the N-Way-K-shot (N relationship classes, K instances per relationship class) few-shot scenario, each of the N relationship classes in the support set S has a prototype. The prototype of the mth relationship class is calculated as follows:

[0117]

[0118] To predict social relations among N relation classes, the prototype network computes the query instance L (Q) and each prototype P in the support set m (S),m∈{1,2,...,N}, and apply softmax on the distance vector to generate the probability distribution of social relations, and select the category with the largest probability as the predicted value of the query instance. Formally, given a query instance q, a relationship class r m The probability on is defined as:

[0119]

[0120] The effects of this embodiment are described in detail below through experiments:

[0121] This example considered two different few-shot learning settings in its experiments: 3-way-1-shot and 3-way-3-shot. In all experiments, this example used prediction accuracy as the evaluation metric. This example compared the performance of FL-MSRE image sampling using two different methods with the baseline on all three datasets. To mitigate the impact of random factors on the experimental results, this example repeated the experiment 10 times during the test phase for each setting and calculated the mean accuracy.

[0122] Table 1

[0123]

[0124] The results are shown in Table 1. For FC-TF, the model of this embodiment achieved an absolute improvement of 3.98% and 4.23% over the baseline Proto(BERT) in both few-shot scenario settings. In addition, when the number of training instances decreases, the performance of the model of this embodiment is more stable than the baseline. For the other two smaller datasets, OM-TF and DRC-TF, the model of this embodiment achieved more significant improvements over the baseline using any method for image sampling. In particular, since most social relationships in DRC-TF are highly correlated with role attributes, the model of this embodiment achieved the most significant improvement over the baseline.

[0125] In order to verify the role of each module in the model proposed in this embodiment, this embodiment conducted a series of ablation experiments under the 3-way-1-shot setting in a few-sample scenario.

[0126] Table 2

[0127]

[0128] The results, as shown in Table 2, show that removing any module leads to a decline in model performance. The hypergraph convolutional neural network plays a particularly important role. Without the hypergraph convolution operation, the model's performance on all datasets degrades significantly. This demonstrates that simply incorporating visual feature information and performing simple cross-modal interactive learning is insufficient. The introduction of the hypergraph learning module enables the model to better learn high-order, complex relationships between multimodal data, improving accuracy by 0.25%, 2.23%, and 0.81% on the three datasets, respectively. Furthermore, the introduction of the cross-diffusion attention mechanism also contributes to the improved model performance. When learning high-order, complex relationships between multimodal data, the model is prone to noise and loss of original modal input information. The cross-diffusion attention mechanism can, to a certain extent, reduce the noise generated during the hypergraph learning process and enhance the affinity between hypergraph nodes and the original text modal information, reducing the loss of original text modal information. This improves model performance on all three datasets and enhances generalization.

[0129] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multimodal social relationship extraction method based on hypergraph attention neural network, characterized by: include: Acquire multimodal datasets; Performing feature extraction on the multimodal dataset to obtain text features and image features; Performing feature extraction on the multimodal dataset to obtain text features and image features includes: Extracting text features of the multimodal dataset using a text encoder; Extracting image features of the multimodal dataset using a FaceNet model; The image features of the multimodal dataset extracted using the FaceNet model include: Inputting the multimodal dataset into the FaceNet model to obtain facial features of the head entity and the tail entity; Inputting the facial features of the head entity and the tail entity into a fully connected layer and a batch normalization layer to obtain the final facial features of the two entities; Performing L2 normalization on the final facial features and splicing to obtain the image features; Based on the text features and image features, a node feature matrix is ​​obtained, and according to the node features, a hypergraph association matrix is ​​obtained; Input the node feature matrix and the hypergraph association matrix into the hypergraph neural network to learn the high-order complex relationships between nodes, perform pooling processing on the updated hypergraph node features, and obtain hypergraph features; The cross-diffusion attention module is used to perform denoising on the hypergraph features to obtain a hypergraph feature representation, specifically including: Computing affinities between different tokens within the modality of the hypergraph features and text features via self-attention; Normalizing the affinity to obtain a normalized affinity matrix; obtaining pairwise affinities between tokens of two different modalities based on the normalized affinity matrix; Based on the pairwise affinity, obtaining contextual information of two different modalities; Integrating the context information to obtain the hypergraph feature representation; Concatenate the hypergraph feature representation with the text feature and the image feature to obtain a final feature; Based on the final features, a social relationship prediction result is obtained.

2. The multimodal social relationship extraction method based on hypergraph attention neural network according to claim 1 is characterized in that: Based on the text features and the image features, obtaining a hypergraph association matrix includes: Determining node features based on the text features and image features; According to the node feature, the node feature and the K nearest neighbor nodes of the node feature are calculated by Euclidean distance to form a hyperedge feature; The hypergraph association matrix is ​​obtained according to the node features and the hyperedge features.

3. The multimodal social relationship extraction method based on hypergraph attention neural network according to claim 2 is characterized in that: The node feature matrix and the hypergraph association matrix are input into the hypergraph neural network to learn the high-order complex relationships between nodes. The updated hypergraph node features are pooled to obtain the hypergraph features, including: Acquiring initial node features according to the multimodal dataset; Performing feature conversion on the initial node features through a learnable filter matrix to obtain node features after feature conversion; Obtaining hyperedge features based on the node features after feature conversion and the hypergraph association matrix; Aggregating the hyper-edge features to obtain updated node features; Based on the updated node features and hyperedge features, learning high-order complex relationships between nodes; The updated hypergraph node features are pooled to obtain hypergraph features.

Images