Multimodal social relation extraction method based on hypergraph attention neural network

Abstract

The invention discloses a multi-modal social relation extraction method based on a hypergraph attention neural network. The method comprises the following steps: acquiring a multi-modal data set; performing feature extraction on the multi-modal data set to obtain text features and image features; obtaining a node feature matrix based on the text features and the image features, and obtaining a hypergraph incidence matrix according to the node features; inputting the node feature matrix and the hypergraph incidence matrix into a hypergraph neural network, learning a high-order complex relationship between nodes, and performing pooling processing on updated hypergraph node features to obtain hypergraph features; performing denoising processing on the hypergraph features by using a cross-divergent attention module to obtain hypergraph feature representation; splicing the hypergraph features, the text features and the image features to obtain final features; and obtaining a social relation prediction result based on the final features. According to the method, the multi-modal hypergraph is constructed, the hypergraph attention neural network is provided, and the high-order complex relation between the multi-modal data is effectively learned.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing and multimodal social relationship extraction, and in particular relates to a multimodal social relationship extraction method based on a hypergraph attention neural network. Background Art

[0002] Social relations are specific relationships in human daily life. They define the connection between two people in the real world or the virtual world. Social relation extraction (SRE) aims to infer the social relations between two people from text, personal albums, or movies, etc. Practice has proved that this method has important practical value. For example, it can capture social connections and enable machines to better understand human behavior. In recent years, SRE has attracted more and more research interests. Fairclough extracted social relations from Weibo text. Du extracted social relations from the four classic masterpieces of Chinese literature. Du mined social relations from Weibo text. Zhang proposed to capture facial attributes by learning face representations and perform pairwise face reasoning for relationship prediction. These studies all start from unimodal information and ignore the high coupling of multimodal information. Multimodal social relation extraction methods can distinguish social relations by adding age-varying visual cues provided by facial information. Therefore, the multidimensional information provided by facial features can be used to enrich unimodal text information, thereby improving performance. MNRE is the first dataset for multimodal relation extraction. Existing few-shot multimodal fusion has been studied in the field of visual question answering, including image captioning, action recognition, sentiment analysis, etc. Studies have shown that the performance of these tasks can be improved by fusing information from different modalities in few-shot learning. Recent works, such as MNRE and HVPNet, have focused on multimodal relation extraction techniques and demonstrated the effectiveness of aggregating image information for relation extraction. MKGformer and MOREformer promote multi-level fusion of visual and textual information, processing and integrating information from both modalities. DGF-PT represents a recent advance that adopts a hint-based autoregressive encoder that combines two different types of prefix adjustments to promote stronger connections between entity pairs, textual context, and images / objects. However, existing methods ignore the multiple relationships of different entity pairs in a sentence, which is caused by the variations in entity pairs. In multimodal relation extraction tasks, images can provide valuable information. However, the potential of leveraging image information in different ways for different entity pairs has not been fully explored.

[0003] However, there are still some key problems in social relationship extraction methods based on multimodal information. Due to the differences between modalities, it is difficult to fuse information. Insufficient fusion cannot fully utilize the assistance between modalities, while over-fusion will discard the original unimodal information. In addition, relations exist in an unbalanced distribution, which leads to recognition bias between label-rich and label-poor relations. At the same time, existing methods can only capture shallow relationships between multimodal information, but cannot obtain high-order complex relationships between multimodal information. Summary of the invention

[0004] To solve the above technical problems, the present invention proposes a multimodal social relationship extraction method based on a hypergraph attention neural network, which addresses this gap by capturing high-order correlations between entity pairs and related image objects, and integrating intra-modal and inter-modal correlations that are truly useful for each entity pair.

[0005] The present invention provides a multimodal social relationship extraction method based on a hypergraph attention neural network, comprising:

[0006] Obtain multimodal datasets;

[0007] Performing feature extraction on the multimodal data set to obtain text features and image features;

[0008] Based on the text features and the image features, a node feature matrix is ​​obtained, and according to the node features, a hypergraph association matrix is ​​obtained;

[0009] Input the node feature matrix and the hypergraph association matrix into the hypergraph neural network, learn the high-order complex relationship between nodes, perform pooling processing on the updated hypergraph node features, and obtain the hypergraph features;

[0010] De-noising the hypergraph features using a cross-diffusion attention module to obtain a hypergraph feature representation;

[0011] Concatenate the hypergraph feature with the text feature and the image feature to obtain a final feature;

[0012] Based on the final features, a social relationship prediction result is obtained.

[0013] Optionally, performing feature extraction on the multimodal dataset to obtain text features and image features includes:

[0014] Extracting text features of the multimodal dataset using a text encoder;

[0015] The FaceNet model is used to extract image features of the multimodal dataset.

[0016] Optionally, extracting image features of the multimodal dataset using the FaceNet model includes:

[0017] Inputting the multimodal data set into the FaceNet model to obtain facial features of the head entity and the tail entity;

[0018] The facial features of the head entity and the tail entity are normalized and spliced ​​to obtain the image features.

[0019] Optionally, based on the text features and the image features, obtaining a hypergraph association matrix includes:

[0020] Determine node features based on the text features and image features;

[0021] According to the node feature, the node feature and K nearest neighbor nodes of the node feature are calculated by Euclidean distance to form a hyperedge feature;

[0022] The hypergraph association matrix is ​​obtained according to the node features and the hyperedge features.

[0023] Optionally, the node feature matrix and the hypergraph association matrix are input into a hypergraph neural network to learn high-order complex relationships between nodes, and the updated hypergraph node features are pooled to obtain the hypergraph features, including:

[0024] According to the multimodal data set, obtaining initial node features;

[0025] Performing feature conversion on the initial node features through a learnable filter matrix to obtain node features after feature conversion;

[0026] Acquire hyper-edge features according to the node features after the feature conversion and the hyper-graph association matrix;

[0027] Aggregating the hyper-edge features to obtain updated node features;

[0028] Based on the updated node features and hyperedge features, learning high-order complex relationships between nodes;

[0029] The updated hypergraph node features are pooled to obtain hypergraph features.

[0030] Optionally, performing denoising processing on the hypergraph features by using a cross-diffusion attention module to obtain a hypergraph feature representation includes:

[0031] Computing affinities between different tokens within the modality of the hypergraph features and text features via self-attention;

[0032] Normalizing the affinity to obtain a normalized affinity matrix;

[0033] Based on the normalized affinity matrix, obtaining pairwise affinities between tokens of two different modalities;

[0034] Based on the pairwise affinity, obtaining context information of two different modalities;

[0035] The context information is integrated to obtain the hypergraph feature representation.

[0036] Compared with the prior art, the present invention has the following advantages and technical effects:

[0037] 1. This paper constructs a multimodal hypergraph and proposes a hypergraph attention neural network to effectively learn the high-order complex relationships between multimodal data.

[0038] 2. The present invention combines the hypergraph neural network with the cross-diffusion attention mechanism, which effectively reduces the noise impact generated during the hypergraph learning process and enhances the affinity between the hypergraph features and the text semantics. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 It is a flow chart of a multimodal social relationship extraction method based on a hypergraph attention neural network according to an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of hyperedge generation according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of a super edge convolutional layer of an embodiment of the present invention;

[0043] Figure 4 Schematic diagram of a cross-diffusion attention mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] This paper proposes a multimodal social relationship extraction method based on hypergraph attention neural network. Figure 1As shown, the specific steps include:

[0047] Acquire multimodal datasets;

[0048] Perform feature extraction on multimodal datasets to obtain text features and image features;

[0049] Based on text features and image features, a node feature matrix is ​​obtained, and based on the node features, a hypergraph association matrix is ​​obtained;

[0050] Input the node feature matrix and the hypergraph association matrix into the hypergraph neural network to learn the high-order complex relationship between nodes, perform pooling on the updated hypergraph node features, and obtain the hypergraph features;

[0051] The cross-diffusion attention module is used to denoise the hypergraph features and obtain the hypergraph feature representation;

[0052] Concatenate the hypergraph features, text features, and image features to obtain the final features;

[0053] Based on the final features, the social relationship prediction results are obtained.

[0054] Specifically, the present invention first constructs a multimodal hypergraph for the data, in which each sentence (global information of the sentence, head entity, tail entity) and the corresponding image (and the face in the image) are represented as a hypergraph. Next, the model automatically learns the edges and weights of the hypergraph to learn a better structure and capture the potential associations between entities and relationships. The model takes a sentence and a set of images as input, and generates feature representations of head / tail entities and facial images through a sentence encoder and a face encoder. The sentence encoder extracts text features through a pre-trained language model. The face encoder generates facial features by extracting bounding boxes and corresponding visual features from a pre-trained face recognition model. Then, a multimodal hypergraph is constructed through the generated features, and a high-order complex relationship between multimodal information is obtained through a hypergraph convolutional neural network. Then, the cross-diffusion attention mechanism is used to reduce the noise impact generated during the hypergraph learning process, while enhancing the affinity between the hypergraph features and the text semantics.

[0055] Furthermore, feature extraction is performed on the multimodal dataset to obtain text features and image features including:

[0056] Extract text features from multimodal datasets using text encoders;

[0057] The FaceNet model is used to extract image features from multimodal datasets.

[0058] Furthermore, the FaceNet model is used to extract image features of multimodal datasets including:

[0059] Input the multimodal dataset into the FaceNet model to obtain the facial features of the head entity and the tail entity;

[0060] The facial features of the head entity and the tail entity are normalized and concatenated to obtain image features.

[0061] Specifically, the encoder

[0062] (1) Sentence Encoder

[0063] In the sentence encoder, each sentence is encoded into a vector. The present invention uses the pre-trained language model BERT as the language encoder. Given a sentence consisting of m tokens w1,w2,...,w m , where the names of the characters have been masked. The present invention constructs a token sequence s′=[CLS],w1,w2,...,w m ,[SEP], where [CLS] and [SEP] are special tokens introduced in BERT. BERT takes the token sequence s' as input and outputs a 768-dimensional vector sequence The present invention uses a fully connected layer to transform the first vector V [CLS] After processing, we get B = tanh(W1V [CLS] +b1).

[0064] (2) Facial Information Encoder

[0065] For two given images i h and i t , respectively including the head entity h and the tail entity t, the present invention can be applied to R 1792 We get two vectors f h and f t To obtain high-level features, the present invention adds a fully connected layer and a batch normalization layer to obtain the final facial features of the two entities. More precisely, the two vectors v h and v t Representing the final facial features of h and t, respectively, are represented by f h and f t Refined.

[0066] v h =BatchNorm(W2f h +b2) (1)

[0067] v t =BatchNorm(W3f t +b3) (2)

[0068] Where W2,W3∈R 512×1792 ,b2,b3∈R512 These are all trainable parameters.

[0069] Finally, the present invention is to v h and v t Perform L2 normalization and connect them to get the final face feature vector F h,t , defined as

[0070] F h,t =[L2Norm(v h ); L2Norm(v t )] (3)

[0071] Further, based on the text features and the image features, obtaining the hypergraph association matrix includes:

[0072] Determine node features based on text features and image features;

[0073] According to the node features, the node features and the K nearest neighbor nodes of the node features are calculated through the Euclidean distance to form a hyperedge feature;

[0074] According to the node features and hyperedge features, the hypergraph association matrix is ​​obtained.

[0075] Specifically, the construction of a multimodal hypergraph is different from that of a simple graph. The hyperedges in a hypergraph connect two or more vertices. The hypergraph is defined as G = (V, E, W), which contains a vertex set V and a hyperedge set E. The hyperedge weight matrix W = diag (w1, w2, ..., w |E| ) is the hyperedge e j (j=1,2,...,|E|) assign the corresponding weight w j The hypergraph G can be represented by a |V|×|E| incidence matrix H, whose entries are defined as:

[0076]

[0077] In this way, each hyperedge e connects all associated nodes v, indicating the correlation between them. The degree of nodes and hyperedges in the hypergraph can be calculated according to the hypergraph association matrix H. The degree of a node is defined as the number of hyperedges containing the node, and the degree of a hyperedge is defined as the number of nodes contained in the hyperedge, which are defined as:

[0078]

[0079] Given the features of each vertex, the second class E can be generated by finding the neighbors of each vertex in the feature space featureFeatures. Different strategies can be used here. Given a vertex as the centroid, you can connect its k nearest neighbors in the feature space with hyperedges, or select all neighbors within a distance d from the centroid (including the centroid), such as Figure 2 As shown. The present invention constructs a hypergraph by calculating the distance between two features based on the Euclidean distance. In the hypergraph construction, each vertex represents a visual object or text feature, and each hyperedge is formed by connecting a vertex and its K nearest neighbors, which forms N hyperedges connecting K+1 vertices. Therefore, the present invention obtains the association matrix H∈R N×N , where N×(K+1) terms are equal to 1 and the other terms are equal to 0.

[0080] Furthermore, the node feature matrix and the hypergraph association matrix are input into the hypergraph neural network to learn the high-order complex relationship between nodes, and the updated hypergraph node features are pooled to obtain the hypergraph features including:

[0081] According to the multimodal data set, initial node features are obtained;

[0082] Perform feature transformation on the initial node features through a learnable filter matrix to obtain the node features after feature transformation;

[0083] According to the node features after feature conversion and the hypergraph association matrix, the hyper-edge features are obtained;

[0084] Aggregate the hyper-edge features to obtain updated node features;

[0085] Based on the updated node features and hyperedge features, learn the high-order complex relationships between nodes;

[0086] The updated hypergraph node features are pooled to obtain hypergraph features.

[0087] Specifically, in multimodal social relationship extraction, this paper constructs a hypergraph attention neural network model, such as Figure 1 As shown. The multimodal dataset includes a text dataset and an image dataset, each of which contains multiple nodes with features. Then, multiple hyper-edge structure groups are constructed according to the complex correlation of the multimodal dataset. The present invention connects these hyper-edge groups to generate a hypergraph adjacency matrix H, and then inputs the hypergraph adjacency matrix H and the node features into the model. Specifically, a hyper-edge convolution layer f(X, W, Θ) can be constructed as shown in the following formula.

[0088]

[0089] Among them, X (l) is the hypergraph signal of the lth layer, the initial value is the feature matrix input X, and σ is the nonlinear activation function. v and D eThey represent the node degree matrix and the hyperedge degree matrix respectively, and play a normalization role in equation (7). W represents the hyperedge weight matrix, which is initialized to the unit matrix. Θ is the parameter to be learned during the training process.

[0090] The hypergraph convolutional neural network model is based on spectral convolution on the hypergraph. Here, the present invention further studies the characteristics of the model that utilizes high-order correlations between data. Figure 3 As shown in , the hyper-edge convolution layer can perform node-edge-node transformation, and the multi-modal hypergraph structure can better refine the features. More specifically, first, the learnable filter matrix Θ (l) C1-dimensional feature X of the initial node (l) Processing is performed to extract C2-dimensional features. Then, by comparing with H T ∈R E×N Multiplication is used to collect node features based on hyperedges to form hyperedge features Finally, the related hyper-edge features are aggregated by multiplying the matrix H to output the new node feature X (l+1) Therefore, the hyper-edge convolution layer can effectively extract high-order correlations on the hypergraph through node-edge-node transformation. For each multimodal hypergraph, after l layers of feature propagation and aggregation, all node features on the constructed multimodal hypergraph can be calculated. The present invention uses mean pooling to obtain the final multimodal hypergraph information representation h for the learned node representation. G .

[0091]

[0092] Furthermore, the cross-diffusion attention module is used to denoise the hypergraph features, and the hypergraph feature representation is obtained including:

[0093] Compute the affinity between different tokens within the modality of hypergraph features and text features via self-attention;

[0094] Normalize the affinity to obtain a normalized affinity matrix;

[0095] Based on the normalized affinity matrix, pairwise affinities between tokens of two different modalities are obtained;

[0096] Based on pairwise affinity, contextual information of two different modalities is obtained;

[0097] Integrate context information to obtain hypergraph feature representation.

[0098] Specifically, in fact, when the hypergraph convolutional neural network processes multimodal data, it brings high-order complex relationship information, but also brings noise and loss of some original modal information. In order to solve this problem, the present invention uses the cross-diffusion attention mechanism (CDA) to reduce the noise and redundant information generated in the process of the hypergraph convolutional neural network acquiring high-order complex relationship features, and at the same time retains more text modal semantic information through text semantic guidance. For the self-attention mechanism, let X t ∈R n×d ,h G ∈R n×d Represent the text semantics and hypergraph modal features respectively, where n represents the number of tokens in the two modalities and d represents the feature dimension of the tokens. In order to capture the dependencies of tokens in each modality, the present invention uses a self-attention architecture. Specifically, the affinity between different tokens in the text modality is first calculated.

[0099]

[0100] Where Q t and K t By X t Perform two linear transformations to obtain: Indicates the scale factor. t Based on this, the present invention obtains the context-aware representation U of the token in modality t by using a message passing mechanism. t =S t V t Similarly, the present invention can obtain the S of the hypergraph feature v and U v .

[0101] The key to the cross-diffusion attention mechanism is to communicate information between tokens of different modalities. The traditional way to solve this problem is to design a cross-attention mechanism. Previous work defines the cross-attention mechanism (CA) as:

[0102]

[0103] U t→v =S t→v V v (11)

[0104] Similarly, we can get S v→t and U v→t Here, the present invention can be noted that the calculation of cross affinity is the core step of CA. However, although Q t and K v They come from different modalities and are transformed into the original input features X t 、h G But Q tand K v The modal / domain gap between them is still not negligible. This inherent modal gap makes the does not reliably reflect pairwise affinities S between tokens of different modalities t→v .

[0105] In order to overcome the above problems, this paper proposes a new cross-diffusion attention mechanism with reference to the regularized diffusion process. Different from the conventional CA which defines cross affinity in feature space, CDA is defined in metric space, such as Figure 4 As shown. The present invention can first obtain the affinity S between modality t / modality v tokens through self-attention in formula (9): t / S v Then according to S t / S v Definition S t→v Specifically, suppose and represents the normalized affinity matrix, where D t and D v is an angle matrix. The present invention defines cross diffusion attention as follows:

[0106]

[0107] U t→v =S t→v V v (13)

[0108] Where A=S t +S v , the parameter μ∈(0,1) represents the balancing hyperparameter. It can be initialized to the unit matrix I, or to the initial affinity matrix obtained by other methods, such as the cross affinity matrix of formula (10). is the unit matrix I. Similarly, the present invention can obtain S v→t and U v→t By using the self-attention and cross-diffusion attention modules, the present invention can obtain the context information representation of two modalities {U t ,U t→v},{U v ,U v→t By integrating them together, the present invention can obtain a more reliable and effective representation of the two modes H t ,H v ∈R n×d .

[0109] H t =C t (U t ||U t→v) (14)

[0110] H v =C v (U v ||U v→t ) (15)

[0111] Among them, || represents the concatenation operation. t (·) and C v (·) represents two 1×1 convolutional layers with different parameters. The cross-diffusion attention module converts H t and H v Aggregated together through a feedforward neural network, the final feature Z is obtained.

[0112] Z=FFN(LN(C(H t ||H v ))) (16)

[0113] Where C(·) represents a 1×1 convolutional layer. FFN represents two fully connected layers with nonlinear activation function GELU. LN represents inter-layer normalization operation.

[0114] Given a support set S and a query set Q. Each instance is represented as follows.

[0115] L = LayerNorm[B; Z; F h,t ] (17)

[0116] In the N-Way-K-shot (N relationship classes, K instances per relationship class) few-shot scenario setting, each of the N relationship classes in the support set S has a prototype. The prototype of the mth relationship class is calculated as follows:

[0117]

[0118] To predict social relations among N relation classes, the prototype network computes the query instance L (Q) and each prototype P in the support set m (S),m∈{1,2,...,N}, and apply softmax on the distance vector to generate the probability distribution of social relations, and select the category with the largest probability as the predicted value of the query instance. Formally, given a query instance q, a relationship class r m The probability of is defined as:

[0119]

[0120] The effect of this embodiment is described in detail below through experiments:

[0121] This embodiment considers two different few-shot learning settings in the experiment: 3-way-1-shot and 3-way-3-shot. In all experiments, this embodiment uses prediction accuracy as the evaluation metric. This embodiment compares the performance of FL-MSRE image sampling using two different methods with the baseline on all three datasets. In order to reduce the impact of random factors on the experimental results, this embodiment repeats the experiment 10 times in the test phase under each setting and calculates the mean of the accuracy.

[0122] Table 1

[0123]

[0124] The results are shown in Table 1. For FC-TF, the model of this embodiment achieves an absolute improvement of 3.98% and 4.23% over the baseline Proto(BERT) in both few-sample scenario settings. In addition, when the number of training instances decreases, the performance of the model of this embodiment is more stable than the baseline. For the other two smaller datasets OM-TF and DRC-TF, the model of this embodiment achieves more significant improvements over the baseline using any method for image sampling. In particular, since most of the social relations in DRC-TF are highly correlated with role attributes, the model of this embodiment achieves the most significant improvement over the baseline.

[0125] In order to verify the role of each module in the model proposed in this embodiment, this embodiment conducted a series of ablation experiments under the 3-way-1-shot setting in a few-sample scenario.

[0126] Table 2

[0127]

[0128] The results are shown in Table 2. Removing any module will lead to a decrease in model performance. Among them, the hypergraph convolutional neural network plays a relatively important role. Without the hypergraph convolution operation, the performance of the model on all data sets decreases significantly. This shows that it is not enough for the model to introduce visual feature information and perform simple cross-modal interactive learning. The introduction of the hypergraph learning module enables the model to better learn the high-order complex relationships between multimodal data, and the accuracy on the three data sets is improved by 0.25%, 2.23% and 0.81% respectively. In addition, the introduction of the cross-diffusion attention mechanism also contributes to the improvement of model performance. When the model learns the high-order complex relationships between multimodal data, it is easy to generate noise and the loss of the original modal input information. The cross-diffusion attention mechanism can reduce the noise generated in the hypergraph learning process to a certain extent and enhance the affinity between the hypergraph nodes and the original text modal information, reducing the loss of the original text modal information, so that the model performance is improved on the three data sets and the generalization is better.

[0129] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multimodal social relationship extraction method based on hypergraph attention neural network, characterized in that: include: Acquire multimodal datasets; Performing feature extraction on the multimodal data set to obtain text features and image features; Based on the text features and the image features, a node feature matrix is ​​obtained, and according to the node features, a hypergraph association matrix is ​​obtained; Input the node feature matrix and the hypergraph association matrix into the hypergraph neural network, learn the high-order complex relationship between nodes, perform pooling processing on the updated hypergraph node features, and obtain the hypergraph features; De-noising the hypergraph features using a cross-diffusion attention module to obtain a hypergraph feature representation; Concatenate the hypergraph feature with the text feature and the image feature to obtain a final feature; Based on the final features, a social relationship prediction result is obtained.

2. According to claim 1, a multimodal social relationship extraction method based on hypergraph attention neural network is characterized in that: Performing feature extraction on the multimodal data set to obtain text features and image features includes: Extracting text features of the multimodal dataset using a text encoder; The FaceNet model is used to extract image features of the multimodal dataset.

3. According to claim 2, a multimodal social relationship extraction method based on hypergraph attention neural network is characterized in that: The image features of the multimodal dataset extracted using the FaceNet model include: Inputting the multimodal data set into the FaceNet model to obtain facial features of the head entity and the tail entity; The facial features of the head entity and the tail entity are normalized and spliced ​​to obtain the image features.

4. According to claim 1, a multimodal social relationship extraction method based on hypergraph attention neural network is characterized in that: Based on the text features and the image features, obtaining a hypergraph association matrix includes: Determine node features based on the text features and image features; According to the node feature, the node feature and K nearest neighbor nodes of the node feature are calculated by Euclidean distance to form a hyperedge feature; The hypergraph association matrix is ​​obtained according to the node features and the hyperedge features.

5. According to claim 4, a multimodal social relationship extraction method based on hypergraph attention neural network is characterized in that: The node feature matrix and the hypergraph association matrix are input into the hypergraph neural network to learn the high-order complex relationship between nodes, and the updated hypergraph node features are pooled to obtain the hypergraph features, including: According to the multimodal data set, obtaining initial node features; Performing feature conversion on the initial node features through a learnable filter matrix to obtain node features after feature conversion; Acquire hyper-edge features according to the node features after the feature conversion and the hyper-graph association matrix; Aggregating the hyper-edge features to obtain updated node features; Based on the updated node features and hyperedge features, learning high-order complex relationships between nodes; The updated hypergraph node features are pooled to obtain hypergraph features.

6. According to claim 5, a multimodal social relationship extraction method based on hypergraph attention neural network is characterized in that: The cross-diffusion attention module is used to perform denoising on the hypergraph features to obtain the hypergraph feature representation, including: Computing affinities between different tokens within the modality of the hypergraph features and text features via self-attention; Normalizing the affinity to obtain a normalized affinity matrix; Based on the normalized affinity matrix, obtaining pairwise affinities between tokens of two different modalities; Based on the pairwise affinity, obtaining context information of two different modalities; The context information is integrated to obtain the hypergraph feature representation.

Images