Multi-dimensional character relationship discovery method based on multi-modal information fusion

Abstract

The invention relates to the technical field of people relationship mining, in particular to a multi-dimensional people relationship discovery method based on multi-modal information fusion, which comprises the following steps: extracting text, visual and audio modal features from a video; according to the Euclidean distance of the modal features, optimal alignment path optimization and nonlinear time adjustment are carried out, and the modal features of time step alignment are obtained; performing self-attention and cross-attention mechanism processing on the modal features to obtain multi-modal representations of texts, vision and audios; multi-modal representation of texts, vision and audios is carried out, and a graph structure network of a social relation, a standing relation and a co-occurrence relation is established; determining the social relationship closeness degree, the standing consistency degree and the co-occurrence strength of each person in the video according to the graph structures of the social relationship, the standing relationship and the co-occurrence relationship; according to the method, the understanding accuracy of the multi-dimensional character relationship can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of character relationship mining, and in particular to a multi-dimensional character relationship discovery method based on multimodal information fusion. Background Art

[0002] The discovery of person relationships plays a vital role in social network analysis, media content analysis, and public opinion monitoring. By revealing the relationship patterns between people, tasks such as user portrait construction, behavior prediction, and social influence assessment can be performed more accurately. Therefore, the development of accurate person relationship extraction technology not only has important applications in social science research, but also has great potential in the fields of business and public security. In this context, video data, as an intuitive and vivid source of information, has become an important carrier for person relationship discovery. Compared with static text, video not only contains multimodal information such as vision and audio, but also has temporal continuity, which can more realistically reflect the behavior, emotions, and interactive characteristics of people. However, most of the existing person relationship analysis relies on text data, and there are relatively few studies on systematic person relationship extraction from videos. This is mainly because the complexity of video data makes the identification and analysis of person relationships face multiple challenges. Current video person relationship analysis usually only uses information from a single modality, such as using only visual features or text features to infer the social relationships of people, and lacks the coordinated use of multimodal information. Moreover, most of these analysis methods only target person relationships in a single dimension, such as focusing only on social relationships. This single modality and dimension limits the breadth and depth of the analysis, making it difficult to accurately reveal the complex network of character relationships.

[0003] Although multimodal and multidimensional character relationship analysis can effectively make up for the above shortcomings in theory, it also faces a series of technical challenges in its implementation:

[0004] Complexity and inconsistency of multimodal information: The information formats of modalities such as text, vision, and audio are different, and the spatial and temporal distribution of data is different. This heterogeneity requires not only precise alignment in time sequence, but also compatibility and fusion in feature space during the analysis process. In addition, information from different modalities may be redundant and conflicting. For example, emotional expressions in text and speech may be inconsistent, and facial expressions in vision may contradict the content of speech. These inconsistencies between modalities will affect the accuracy of the analysis results.

[0005] There are differences in multi-dimensional relationships: The relationship between characters does not only exist in a single dimension, but may also cover multiple dimensions such as social, stance, and co-occurrence. The characteristics of each dimension are significantly different. For example, social relationships rely more on visual body language and voice interaction in audio, while stance relationships rely more on the representation of textual emotions and opinions. How to achieve effective collaborative analysis in multi-dimensional features while balancing the contribution of different dimensions to relationship inference is a challenging task. Summary of the invention

[0006] In view of the above problems, the present invention provides a multi-dimensional character relationship discovery method based on multi-modal information fusion, which solves the technical problems of limitations of single-modal data analysis and shortcomings of single-dimensional relationship analysis in the prior art.

[0007] The present invention provides a multi-dimensional character relationship discovery method based on multimodal information fusion, comprising the following steps:

[0008] Step S1, extracting text modality features, visual modality features and audio modality features of each character in each time period from the video;

[0009] Step S2: Based on the Euclidean distances of the text modality features, the visual modality features, and the audio modality features, optimal alignment path optimization and nonlinear time adjustment are performed to obtain time-step aligned text modality features, visual modality features, and audio modality features;

[0010] Step S3, performing self-attention and cross-attention mechanism processing on the time-step aligned text modality features, visual modality features and audio modality features to obtain a multimodal representation of text, vision and audio;

[0011] Step S4: establishing a graph structure network of social relations, stance relations and co-occurrence relations based on the multimodal representation of text, vision and audio;

[0012] Step S5: Determine the closeness of the social relationship, the consistency of the stance, and the co-occurrence strength of each character in the video based on the graph structure of the social relationship, stance relationship, and co-occurrence relationship.

[0013] Preferably, step S1 specifically includes:

[0014] Step S1-1, using an automatic speech recognition method to transcribe the audio in the video into text, then performing word segmentation, stop word removal and lemmatization on the text, inputting the obtained text sequence into a BERT-based model to extract text modal features;

[0015] Step S1-2, performing face detection and recognition on the video frame to obtain the identity features of the bounding box corresponding to each face, and extracting facial expression features and body movement features, and weightedly fusing the identity features, facial expression features and body movement features to obtain visual modality features;

[0016] Step S1-3: extract basic audio features from the audio in the video, integrate the basic audio features using the sentiment classification model, and obtain audio modal features.

[0017] Preferably, step S2 specifically includes:

[0018] Step S2-1, using Euclidean distance to calculate the sequence formed by the modal features to construct a distance matrix; constructing a cumulative distance matrix according to the distance matrix;

[0019] Step S2-2, performing an optimal alignment path on the cumulative distance matrix, and finding the optimal alignment path by minimizing the path in the cumulative distance matrix;

[0020] Step S2-3, performing nonlinear time adjustment on the optimal alignment path, and resampling the sequence formed by the modal features based on the optimal alignment path so that their time steps are aligned; finally, obtaining text modal features, visual modal features, and audio modal features with time step alignment.

[0021] Preferably, step S2-1 specifically includes:

[0022] The modal characteristic sequences are set as follows: F T (t) = {F T (1), F T (2), ..., F T (T)},F V ={F V (1), F V (2), ..., F V (T)} and F A ={F A (1), F A (2), ..., F A (T)},F T (T), F V (T) and F A (T) represents the text, visual and audio modality features in time segment T, respectively, and T represents the total number of time segments;

[0023] Determine the distance matrix D(t1, t2, t3), which represents the Euclidean distance of the three modes at the time segments t1, t2, and t3. The expression is:

[0024] D(t1, t2, t3)=||FT (t1)-F V (t2)|| 2 +||F V (t2)-F A (t3)|| 2 +||F A (t3)-F T (t1)|| 2

[0025] Among them, ||·|| 2 Indicates the square of the magnitude of the vector; F T (t), F V (t), F A (t) are the text, visual and audio modality features of the t time segment, respectively, where t represents the index of the divided time segment;

[0026] According to the distance matrix D(t1, t2, t3), the cumulative distance matrix C(t1, t2, t3) is constructed to represent the cumulative distances of the three modes in the time segments t1, t2, and t3. The expression is:

[0027] C(t1,t2,t3)=D(t1,t2,t3)+min{C(t1-1,t2,t3),C(t1,t2-1,t3),C(t1,t2,t3-1 ), C(t1-1, t2-1, t3), C(t1-1, t2, t3-1), C(t1, t2-1, t3-1), C(t1-1, t2-1, t3-1)}

[0028] Among them, min{·} means taking the minimum value.

[0029] Preferably, step S3 specifically includes:

[0030] Step S3-1, using a self-attention mechanism to process the text modality features, visual modality features and audio modality features respectively, and obtaining a feature weight adjustment result of each modality;

[0031] Step S3-2: using a cross-attention mechanism to cross-calculate the feature weight adjustment results of the text, vision and audio modalities to obtain the cross-attention features of the text, vision and audio modalities;

[0032] Step S3-3, performing residual connection and layer normalization calculations on the cross-attention features respectively to obtain the final multimodal representation of text, vision and audio.

[0033] Preferably, step S3-1 specifically includes:

[0034] For the features of each modality after alignment Define the query, key, and value matrices Q, K, V of the self-attention mechanism:

[0035] Q=W Q F,K=W K F,V=W V F

[0036] Among them, W Q , W K and W v is the projection matrix of self-attention, F represents the features of any modality; by calculating the similarity between the query and the key, the attention score matrix is ​​obtained, which is expressed as:

[0037]

[0038] The output F of the self-attention mechanism attn include They represent the feature weight adjustment results of text, visual and audio modalities respectively, and d represents the feature dimensions of Q, K and V;

[0039] Step S3-2 specifically includes:

[0040] For text modal It is used as the query Q of the cross attention mechanism T , and the features of the visual and audio modalities are used as the key K of the cross attention mechanism respectively. V , K A Sum value V V and V A , the expression is:

[0041]

[0042] in, Represents the projection matrix in the cross-attention mechanism;

[0043] Cross-attention between textual and visual modalities It is expressed as:

[0044]

[0045] Among them, Cross-Attention(·) represents the cross attention calculation, K V The transpose of , d represents the dimension of key and query;

[0046] Cross-attention between text and audio modalities It is expressed as:

[0047]

[0048] The cross-attention feature output for text modality is:

[0049]

[0050] The other modalities are calculated in the same way to obtain the cross-attention features of text, visual and audio modalities.

[0051] Step S3-3 specifically includes:

[0052] The computational expression for the multimodal representation of text is:

[0053]

[0054] LayerNorm(·) represents the layer normalization calculation; other modalities are calculated in the same way to obtain the final multimodal representation of text, vision and audio.

[0055] Preferably, step S4 specifically includes:

[0056] Step S4-1, obtaining a multimodal social feature vector of a person based on the multimodal representation, and mapping the social feature vector of the person into a social relationship network;

[0057] Step S4-2, verifying the consistency of the emotion and stance information extracted from the text modality in the visual and audio modalities, generating an emotion feature vector for each modality, and determining a stance relationship network based on the emotion feature vector;

[0058] Step S4-3: Determine the co-occurrence intensity between characters based on the frequency and duration of the characters appearing simultaneously in the video, and then determine the co-occurrence relationship network.

[0059] Preferably, step S4-1 specifically includes:

[0060] Generate multimodal social feature vectors of characters Indicates character i p Social Behavior:

[0061]

[0062] Among them, W social (·) represents mapping multimodal features into social feature vectors, α T , α V , α A Respectively represent the weights of the three modalities in the character's social feature vector;

[0063] Mapping the social feature vector of a person into a graph structure, the expression is:

[0064] Wij =β1·frequency ij +β2·emotion ij

[0065] Among them, W ij represents the weight of the edge between node i and node j in the graph structure, β1 and β2 are the weights of interaction frequency and emotional consistency, ij Represents the interaction frequency between node i and node j, emotion ij Represents the sentiment consistency between node i and node j;

[0066] To infer the strength of social relationships, the expression is:

[0067]

[0068] in, represents the updated social feature vector of node i, neighbors(i) represents the set of neighbor nodes of node i, AGGREGATE(·) is the aggregation function, W represents the weight matrix of the graph neural network, b represents the bias term of the graph neural network, σ nl represents a nonlinear activation function;

[0069] Finally, a social relationship network G is constructed S .

[0070] Preferably, step S4-2 specifically includes:

[0071] Generate sentiment feature vectors for each modality:

[0072]

[0073] Among them, E T , E V , E A Represent the sentiment feature vectors of text, vision and audio respectively, Represents the projection matrices of text, visual and audio features projected onto the corresponding sentiment feature vectors;

[0074] Calculate the comprehensive sentiment feature vector of node i and the stance feature vector L of node i i , the expression is:

[0075] E i =β T ·E T +β V ·E V +β A ·E a

[0076] Li =f emotion (E i )

[0077] Among them, β T , β V , β A Respectively represent the weights of the three modes in the character's emotional feature vector, f emotion (·) is the stance classification function;

[0078] To carry out stance propagation, the expression is:

[0079]

[0080] in, represents the updated stance feature vector of node i, L j Represents the stance feature vector of node i’s neighbors;

[0081] Finally, the position relationship network G is constructed L ;

[0082] Step S4-3 specifically includes:

[0083] Calculate the co-occurrence strength I between node i and node j ij :

[0084] I ij =τ1·frequency ij +τ2·time ij

[0085] Among them, frequency ij represents the interaction frequency between node i and node j, time ij represents the co-occurrence duration between node i and node j, τ1 and τ2 are the proportional weights of interaction frequency and co-occurrence duration respectively;

[0086] Finally, the co-occurrence relationship network G is constructed I .

[0087] Preferably, step S5 specifically includes: based on the constructed social relationship network G S , Position relationship network G L and the co-occurrence network G I , by G S The weight of the middle edge determines the closeness of the social relationship between the characters in the video; L The similarity of the middle edge determines the consistency of the position of the characters in the video; G I The weight of the middle edge determines the co-occurrence strength of the characters in the video.

[0088] Compared with the prior art, the present invention has at least the following beneficial effects:

[0089] (1) This invention significantly improves the comprehensiveness and accuracy of information processing by realizing the deep integration of three modalities: text, vision, and audio. The multimodal Transformer model is based on the self-attention mechanism, which can intelligently adjust the weights of different modal features and effectively capture the interdependence between modalities, making the performance of tasks such as sentiment analysis and stance recognition more accurate and robust, overcoming the limitations of single-modal information processing.

[0090] (2) The present invention provides a relationship analysis model with three dimensions: social, stance, and co-occurrence, which greatly enriches the depth and breadth of character relationship analysis. By applying multimodal information to feature construction in different dimensions, it is possible to comprehensively characterize the interaction patterns between characters, not only identifying direct social interactions, but also analyzing the similarity of stances and attitudes and the regularity of co-occurrence.

[0091] (3) The present invention uses graph neural networks to infer multi-dimensional relationships, significantly enhancing the reasoning ability and mining depth of relationship analysis. Through the relationship propagation and feature aggregation mechanism, it can perform global reasoning based on known local information, effectively discover and understand potential complex relationship patterns, and improve the overall understanding and accuracy of multi-dimensional character relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0093] Figure 1 A flowchart of a multi-dimensional character relationship discovery method based on multi-modal information fusion provided by the present invention;

[0094] Figure 2 This is a schematic diagram of the multi-dimensional character relationship discovery method based on multi-modal information fusion of the present invention. DETAILED DESCRIPTION

[0095] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0096] The present invention aims to provide a method for discovering character relationships based on multimodal information fusion and multidimensional relationship analysis, overcoming the limitations of single-modal data analysis and the shortcomings of single-dimensional relationship analysis in the prior art. By integrating the information of the three modalities of text, vision and audio in the video, accurate analysis of the social relationships, position relationships and co-occurrence relationships of the characters is achieved, thus improving the accuracy and comprehensiveness of mining complex character relationships.

[0097] In order to illustrate the effectiveness of the method proposed by the present invention, the above technical solution of the present invention is described in detail below through a specific embodiment. Figure 1 As shown, a multi-dimensional character relationship discovery method based on multimodal information fusion is disclosed, and the specific implementation steps are as follows:

[0098] Step S1: extracting text modality features, visual modality features and audio modality features of each character in each time period from the video.

[0099] First, multimodal features are extracted for each character and each time period in the video data. Each character will have a separate set of multimodal features. For ease of representation, the character identification will be omitted below.

[0100] (1) Extracting text modal features: The present invention uses automatic speech recognition (ASR) technology to transcribe the audio in the input video into text, and then performs preprocessing such as word segmentation, stop word removal, and morphological restoration on the text to normalize the text structure. Next, the BERT-based model is used to extract text modal features. The steps of extracting text modal features by the BERT-based model include:

[0101] Text embedding: text sequence T = {t1, t2, ...t n} is input into the BERT-based model to generate a word-level embedding vector E T ={e1, e2, ..., e n}, where t n is the nth token, e n is the embedding corresponding to the nth token, n is the total number of words in the text sequence. In addition, a special global tag [CLS] is added in the embedding stage as the overall representation of the sequence.

[0102] Sentiment and Opinion Analysis: Embedding Vector E T Input to the BERT-based model and extract the [CLS] vector e from the last layer of the model CLS , used to represent the semantics of the entire sequence, e CLS Sentiment analysis is performed through the sentiment classification head, and the expression is:

[0103] F T =σ(W s ·e CLS +b s )

[0104] Among them, F T is the text modality feature, W s and b s is the weight and bias of the classification head, and σ is the activation function. Text modality feature F T Provides information about the character's emotional tendencies and attitudes.

[0105] (2) Extracting visual modal features: The present invention performs multi-level feature extraction on facial expressions and body movements of characters in video frames.

[0106] First, face detection and recognition are performed. For face detection, the present invention uses the YOLO model to detect the face area in the video frame and outputs a bounding box set {B1, B2, ..., B m}, where B m is the mth bounding box, m is the number of detected faces, each bounding box contains the box coordinates and face confidence, the i-th f The facial feature vector of the bounding box in, The i f The horizontal and vertical coordinates of the bounding box, The i f The width and height of the bounding box, For the i f The face confidence of the bounding box.

[0107] The face area of ​​each detected bounding box is cropped and identified, and feature extraction and identity authentication are performed through the ArcFace model. ArcFace uses weighted cosine similarity to measure feature consistency, and the identity feature is F f , the expression is:

[0108]

[0109] Among them, F f For identity characteristics, For the i f The face area corresponding to the bounding box, ArcFace(·) represents the ArcFace identity recognition model, W f is the weight matrix of ArcFace, θ represents the feature vector x i and W f The angle between is a fixed angular interval used to increase the discrimination between different categories.

[0110] Then, facial expression and body movement feature extraction is performed. For facial expression feature extraction, OpenFace is used to extract the set of facial key points in the face area of ​​the bounding box. k , the facial expression features F are obtained through the dynamic changes of these key points e :

[0111] F e =g(F k )

[0112] Among them, g(·) represents the mapping function based on key point features.

[0113] Use OpenPose to extract the position information of joint points in the video frame A k , extract body movement features F through body movement model a :

[0114] F a =h(A k )

[0115] Where h(·) represents the mapping function for posture feature extraction.

[0116] Visual modality feature F v It is a weighted fusion of multi-level features:

[0117] F V =ω1·F f +ω2·F e +ω3·F a

[0118] Among them, ω1, ω2, and ω3 are the weights of identity features, facial expression features, and body movement features, respectively.

[0119] (3) Extracting audio modal features: The present invention uses the OpenSMILE tool to extract basic audio features from the audio, and then generates the emotional features of the overall audio through the emotion classification model.

[0120] Basic audio characteristics F b Indicates basic elements of conversation such as pitch, speed and volume.

[0121] The basic audio features F are transformed into b Integrate into a unified emotion vector as the audio modality feature F A , in order to align with the sentiment features of textual and visual modalities.

[0122] The final multimodal features of different time periods are expressed as: F T (t), F V (t), FA (t), where t represents the time segment index of the division.

[0123] Step S2: Based on the Euclidean distances of the text modal features, visual modal features and audio modal features, optimal alignment path optimization and nonlinear time adjustment are performed to obtain time-step aligned text modal features, visual modal features and audio modal features.

[0124] Since there may be slight temporal offsets in the features of different modalities, especially between visual and audio emotional features, after the synchronization is completed, the dynamic time warping (DTW) algorithm is used to perform nonlinear time alignment on the feature sequences of different modalities to ensure their precise consistency in time steps for subsequent multimodal fusion and analysis.

[0125] In this step, we first construct a trimodal distance matrix and set the trimodal feature sequence to F T (t) = {F T (1), F T (2), ..., F T (T)},F V ={F V (1), F V (2), ..., F V (T)} and F A ={F A (1), F A (2), ..., F A (T)}, respectively representing the sequence of text, visual and audio modal features in the time dimension, where T represents the total number of time segments. Construct a three-dimensional distance matrix D(t1, t2, t3) to represent the comprehensive distance of the three modalities at time steps t1, t2, and t3. The comprehensive distance can be calculated using the square of the Euclidean distance, expressed as:

[0126] D(t1, t2, t3)=||F T (t1)-F V (t2)|| 2 +||F V (t2)-F A (t3)|| 2 +||F A (t3)-F T (t1)|| 2

[0127] Among them, ||·|| 2 It represents the square of the modulus of the calculated vector, which is equivalent to the square of the Euclidean distance between eigenvectors. T (t), FV (t), F A (t) are the text, visual and audio modality features of time segment t respectively, and t represents the index of the divided time segment.

[0128] Calculate the cumulative distance matrix. According to the distance matrix D(t1, t2, t3), construct the cumulative distance matrix C(t1, t2, t3), which is recursively defined as:

[0129] C(t1,t2,t3)=D(t1,t2,t3)+min{C(t1-1,t2,t3),C(t1,t2-1,t3),C(t1,t2,t3-1 ), C(t1-1, t2-1, t3), C(t1-1, t2, t3-1), C(t1, t2-1, t3-1), C(t1-1, t2-1, t3-1)}

[0130] Among them, min{·} represents taking the minimum value. The matrix recursively calculates the cumulative minimum path in three-dimensional space to ensure that the final path finds the global optimal alignment relationship between the three modes.

[0131] The optimal alignment path is found by minimizing the path in the cumulative distance matrix. The path represents the best alignment relationship between the three modes at the time step. p is the index of the path point, K p is the total length of the path, that is, the path contains K p The number of time steps, Respectively represent the indexes of the path points in the time segments t1, t2, and t3.

[0132] The optimal alignment path is nonlinearly adjusted in time, and the feature sequences of the three modalities are resampled based on the path P so that their time steps are accurately aligned, and the text modal features, visual modal features, and audio modal features aligned in time steps are determined.

[0133] Step S3: Process the time-step aligned text modal features, visual modal features, and audio modal features using self-attention and cross-attention mechanisms to obtain a multimodal representation of text, vision, and audio.

[0134] After completing the alignment of text, vision and audio modalities, the present invention realizes the fusion of multimodal features through self-attention and cross-attention mechanisms. It achieves the retention of each feature representation in each modality, while enabling the information between the modalities to fully interact, providing rich multimodal features for the subsequent multi-level relationship extraction module.

[0135] (1) Use the self-attention mechanism to process single-modal features: First, self-attention is performed on the time-step aligned text modal features, visual modal features, and audio modal features to adjust the feature weights within each modality and highlight the important feature information in the modality. Define the query, key, and value matrices Q, K, V of the self-attention mechanism:

[0136] Q=W Q F,K=W K F,V=W V F

[0137] Among them, W Q , W K and W V is the projection matrix of self-attention, and F represents the features of any modality. By calculating the similarity between the query and the key, the attention score matrix is ​​obtained:

[0138]

[0139] Where d is the scaling factor of the feature dimension, which is used to stabilize the gradient. This attention score reflects the importance of each feature, thereby adjusting the feature expression within each modality. Finally, the output F of the self-attention mechanism attn Respectively represent That is, the feature weight adjustment result in each mode.

[0140] (2) The modal cross-attention mechanism is used to achieve inter-modal interaction. After self-attention processing, in order to further complement and enhance the features of different modalities, the modal cross-attention mechanism is introduced so that the features of different modalities can interact across modalities. It is used as the query Q of the cross attention mechanism T , and the features of the visual and audio modalities are used as the key K of the cross attention mechanism respectively. V , K A Sum value V V and V A , the expression is:

[0141]

[0142] in, Represents the projection matrix in the criss-cross attention mechanism.

[0143] Similarly, similar settings are made for the visual and audio modalities, so that each modality can perform attention calculations on the features of the other modalities. For example, the cross attention of the text modality and the visual modality It is expressed as:

[0144]

[0145] Among them, Cross-Atttention(·) represents the cross attention calculation, K V The transpose of , d represents the dimension of key and query.

[0146] Cross-attention between text and audio modalities It is expressed as:

[0147]

[0148] Finally, the cross-attention feature output of the text modality is:

[0149]

[0150] Similarly, other modalities can be calculated in the same way to obtain cross-attention features The features generated by the cross-attention mechanism contain not only information within the modality, but also complementary information from other modalities.

[0151] (3) Residual connection and layer normalization. In order to improve the stability of the model and maintain the balance of the features of each modality, the cross-attention features of each modality are finally subjected to residual connection and layer normalization. For example, the calculation expression of the multimodal representation of text is:

[0152]

[0153] LayerNorm(·) represents the layer normalization calculation. The same is true for other modalities, and the final multimodal representation of text, vision and audio is obtained.

[0154] Step S4: Based on the multimodal representation of text, vision and audio, a graph structure network of social relations, stance relations and co-occurrence relations is established.

[0155] In this step, the module of the present invention uses the multimodal representation to infer the complex relationships between characters from the three dimensions of sociality, stance, and co-occurrence. Through the in-depth analysis of multimodal features and the construction of graph structure, the system can fully reveal the characters' interaction patterns, stance tendencies, and co-occurrence strengths.

[0156] (1) Social relationship analysis, extracting information related to social interaction from the multimodal representation. The visual modality provides features of physical contact, eye contact, and expression consistency; the audio modality provides features of conversation frequency and voice emotion; and the text modality provides features of language interaction. By combining these features, a multimodal social feature vector of the character is generated. Indicates character i p Social Behavior:

[0157]

[0158] Among them, W social (·) represents mapping multimodal features into social feature vectors, α T , α V , α A Respectively represent the weights of the three modalities in the character's social feature vector.

[0159] Then, the social feature vector of the character is mapped into a graph structure. The nodes in the graph represent the characters, and the information such as interaction frequency and emotional consistency is used as the weight of the edge. The expression is:

[0160] W ij =β1·frequency ij +β2·emotion ij

[0161] Among them, W ij represents the weight of the edge between node i and node j in the graph structure, β1 and β2 are the weights of interaction frequency and emotional consistency, ij Represents the interaction frequency between node i and node j, emotion ij Represents the sentiment consistency between node i and node j.

[0162] Finally, on the constructed graph structure representing social relationships, a graph neural network (such as GraphSAGE) is used to infer the strength of social relationships. The network aggregates the feature information of neighbor nodes, infers the potential social connections between characters, and reveals the changes in the strength of social relationships:

[0163]

[0164] in, represents the updated social feature vector of node i, reflecting the strength of the social relationship of the node, S j represents the social feature vector of node j, neighbors(i) represents the set of neighbor nodes of node i, that is, the set of all nodes connected to node i by edges, and represents other people who have direct social relationships with i. AGGREGATE(·) is an aggregation function, which is used to aggregate all neighbor node features of node i into an aggregate representation. W represents the weight matrix of the graph neural network, which is a learnable parameter matrix. It performs a linear transformation on the aggregated neighbor features to adjust the weights of different feature dimensions. b represents the bias term of the graph neural network, which is a learnable bias vector that helps adjust the output of the linear transformation to make the updated features more stable. σ nlrepresents a nonlinear activation function, which introduces nonlinearity to the updated features, so that the model can better express complex social relationships.

[0165] (2) Stance relationship analysis: First, we verify the consistency of the sentiment and stance information extracted from the text modality in the visual and audio modalities, and generate sentiment feature vectors for each modality:

[0166]

[0167] Among them, E T , E V , E A Represent the sentiment feature vectors of text, vision and audio respectively, Represents the projection matrices of text, visual and audio features onto the corresponding sentiment feature vectors respectively.

[0168] The consistency check is performed by calculating the cosine similarity between each feature vector to ensure a comprehensive and accurate understanding of the character's position.

[0169] Then we get the comprehensive emotional feature vector E of node i i , classify and obtain the position feature vector L of node i i , the expression is:

[0170] E i =β T ·E T +β V ·E V +β A ·E A

[0171] L i =f emotion (E i )

[0172] Among them, β T , β V , β A Respectively represent the weights of the three modes in the character's emotional feature vector, f emotion (·) is the stance classification function.

[0173] Finally, stance propagation is performed, using graph neural networks to spread the stance feature vectors of different characters:

[0174]

[0175] in, represents the updated stance feature vector of node i, L j Represents the stance feature vector of node i’s neighbors.

[0176] In this way, it is possible to identify groups with similar positions to a certain character, or to reveal the distribution and relationships of opposing camps.

[0177] (3) Co-occurrence relationship analysis: Based on face detection in the visual modality, the system constructs a character co-occurrence matrix to record the frequency and duration of each pair of characters appearing simultaneously in the video. The audio modality is used to verify the conversation during the co-occurrence period to ensure that co-occurrence is not only physical co-frame, but also includes actual language interaction, and obtains the co-occurrence intensity I between node i and node j. ij :

[0178] I ij =τ1·frequency ij +τ2·time ij

[0179] Among them, frequency ij represents the interaction frequency between node i and node j, time ij represents the co-occurrence duration between node i and node j, τ1 and τ2 are the proportional weights of interaction frequency and co-occurrence duration, respectively.

[0180] After being processed by the above multi-level relationship extraction module, the social relationship network G is finally constructed. S , Position relationship network G L and the co-occurrence network G I .

[0181] Step S5, determining the closeness of social relationships, the consistency of stances and the strength of co-occurrence of characters in the video from the graph structure of the social relationships, stance relationships and co-occurrence relationships.

[0182] The social relationship network G S , Position relationship network G L and the co-occurrence network G I middle:

[0183] Social NetworkG S : The nodes represent characters, and the weights of the edges represent the closeness of the social relationships between the characters.

[0184] Position relationship network G L : The nodes represent the characters’ positions, and the similarity of the edges represents the degree of consistency of the positions.

[0185] Co-occurrence relationship network G I : The nodes represent characters, and the edge weights represent the co-occurrence strength between characters.

[0186] In this step, based on the constructed social relationship network G S , Position relationship network G L and the co-occurrence network GI , by G S The weight of the edges in GL determines the closeness of the social relationship between the characters in the video; the similarity of the edges in GL determines the consistency of the positions of the characters in the video; I The weight of the middle edge determines the co-occurrence strength of the characters in the video.

[0187] In summary, the present invention provides a multi-dimensional character relationship discovery method based on multimodal information fusion, which aims to mine more comprehensive and accurate character relationships from video data. The method integrates multiple technologies such as multimodal feature extraction, time alignment, feature fusion, and multi-dimensional relationship analysis to form an end-to-end character relationship analysis system. The overall process first extracts three modal information of text, vision, and audio from the video, and performs feature processing and alignment on these modalities. After fusion of the multimodal Transformer model, a unified multimodal feature representation is generated. On this basis, the system further infers the social relationships, position relationships, and co-occurrence relationships of the characters through technologies such as graph neural networks and propagation models to fully reveal the interaction and position information between the characters. This multimodal collaborative analysis improves the ability to characterize complex character relationships and breaks through the limitations of single modal analysis.

[0188] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A multi-dimensional character relationship discovery method based on multimodal information fusion, characterized in that: The following steps are involved: Step S1, extracting text modality features, visual modality features and audio modality features of each character in each time period from the video; Step S2: Based on the Euclidean distances of the text modality features, the visual modality features, and the audio modality features, optimal alignment path optimization and nonlinear time adjustment are performed to obtain time-step aligned text modality features, visual modality features, and audio modality features; Step S3, performing self-attention and cross-attention mechanism processing on the time-step aligned text modality features, visual modality features and audio modality features to obtain a multimodal representation of text, vision and audio; Step S4: establishing a graph structure network of social relations, stance relations and co-occurrence relations based on the multimodal representation of text, vision and audio; Step S5: Determine the closeness of the social relationship, the consistency of the stance, and the co-occurrence strength of each character in the video from the graph structure network of the social relationship, stance relationship, and co-occurrence relationship.

2. The multi-dimensional character relationship discovery method based on multimodal information fusion according to claim 1 is characterized in that: Step S1 specifically includes: Step S1-1, using an automatic speech recognition method to transcribe the audio in the video into text, then performing word segmentation, stop word removal and lemmatization on the text, inputting the obtained text sequence into a BERT-based model to extract text modal features; Step S1-2, performing face detection and recognition on the video frame to obtain the identity features of the bounding box corresponding to each face, and extracting facial expression features and body movement features, and weightedly fusing the identity features, facial expression features and body movement features to obtain visual modality features; Step S1-3: extract basic audio features from the audio in the video, integrate the basic audio features using the sentiment classification model, and obtain audio modal features.

3. The multi-dimensional character relationship discovery method based on multimodal information fusion according to claim 2 is characterized in that: Step S2 specifically includes: Step S2-1, using Euclidean distance to calculate the sequence formed by the modal features to construct a distance matrix; constructing a cumulative distance matrix according to the distance matrix; Step S2-2, performing an optimal alignment path on the cumulative distance matrix, and finding the optimal alignment path by minimizing the path in the cumulative distance matrix; Step S2-3, performing nonlinear time adjustment on the optimal alignment path, and resampling the sequence formed by the modal features based on the optimal alignment path so that their time steps are aligned; finally, obtaining text modal features, visual modal features, and audio modal features with time step alignment.

4. The multi-dimensional character relationship discovery method based on multimodal information fusion according to claim 3 is characterized in that: Step S2-1 specifically includes: The modal characteristic sequences are set as follows: F T (t) = {F T (1),F T (2),…,F T (T)},F V ={F V (1),F V (2),…,F V (T)} and F A ={F A (1),F A (2),…,F A (T)},F T (T), F V (T) and F A (T) represents the text, visual and audio modality features in time segment T, respectively, and T represents the total number of time segments; Determine the distance matrix D(t1, t2, t3), which represents the Euclidean distance of the three modes at the time segments t1, t2, and t3. The expression is: <h2 style=";text-align:left;direction:ltr">D(t1,t2,t3)=||F<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> (t1)-F<h2 style=";text-align:left;direction:ltr"> V <h2 style=";text-align:left;direction:ltr"> (t2)||<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +||F<h2 style=";text-align:left;direction:ltr"> V <h2 style=";text-align:left;direction:ltr"> (t2)-F<h2 style=";text-align:left;direction:ltr"> A <h2 style=";text-align:left;direction:ltr"> (t3)||<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +||F<h2 style=";text-align:left;direction:ltr"> A <h2 style=";text-align:left;direction:ltr"> (t3)-F<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> (t1)||<h2 style=";text-align:left;direction:ltr"> 2 Among them, ||·|| 2 Indicates the square of the magnitude of the vector; F T (t),F V (t),F A (t) are the text, visual and audio modality features of the t time segment, respectively, where t represents the index of the divided time segment; According to the distance matrix D(t1, t2, t3), the cumulative distance matrix C(t1, t2, t3) is constructed to represent the cumulative distances of the three modes in the time segments t1, t2, and t3. The expression is: C(t1,t2,t3)=D(t1,t2,t3)+min{C(t1-1,t2,t3),C(t1,t2-1,t3),C(t1,t2,t3-1 ),C(t1-1,t2-1,t3),C(t1-1,t2,t3-1),C(t1,t2-1,t3-1),C(t1-1,t2-1,t3-1)} Among them, min{·} means taking the minimum value.

5. The multi-dimensional character relationship discovery method based on multi-modal information fusion according to claim 4 is characterized in that: Step S3 specifically includes: Step S3-1, using a self-attention mechanism to process the text modality features, visual modality features and audio modality features respectively, and obtaining a feature weight adjustment result of each modality; Step S3-2: using a cross-attention mechanism to cross-calculate the feature weight adjustment results of the text, vision and audio modalities to obtain the cross-attention features of the text, vision and audio modalities; Step S3-3, performing residual connection and layer normalization calculations on the cross-attention features respectively to obtain the final multimodal representation of text, vision and audio.

6. The multi-dimensional character relationship discovery method based on multi-modal information fusion according to claim 5 is characterized in that: Step S3-1 specifically includes: For the features of each modality after alignment Define the query, key, and value matrices Q, K, V of the self-attention mechanism: Q=W Q F,K=W K F,V=W V F Among them, W Q , W K and W V is the projection matrix of self-attention, F represents the features of any modality; by calculating the similarity between the query and the key, the attention score matrix is ​​obtained, which is expressed as: The output F of the self-attention mechanism attn include They represent the feature weight adjustment results of text, visual and audio modalities respectively, and d represents the feature dimensions of Q, K and V; Step S3-2 specifically includes: For text modal It is used as the query Q of the cross attention mechanism T , and the features of the visual and audio modalities are used as the key K of the cross attention mechanism respectively. V , K A Sum value V V and V A , the expression is: in, Represents the projection matrix in the cross-attention mechanism; Cross-attention between textual and visual modalities It is expressed as: Among them, Cross-Attention(·) represents the cross attention calculation, K V The transpose of , d represents the dimension of key and query; Cross-attention between text and audio modalities It is expressed as: The cross-attention feature output for text modality is: The other modalities are calculated in the same way to obtain the cross-attention features of text, visual and audio modalities. Step S3-3 specifically includes: The computational expression for the multimodal representation of text is: LayerNorm(·) represents the layer normalization calculation; other modalities are calculated in the same way to obtain the final multimodal representation of text, vision and audio.

7. The multi-dimensional character relationship discovery method based on multi-modal information fusion according to claim 6 is characterized in that: Step S4 specifically includes: Step S4-1, obtaining a multimodal social feature vector of a person based on the multimodal representation, and mapping the social feature vector of the person into a social relationship network; Step S4-2, verifying the consistency of the emotion and stance information extracted from the text modality in the visual and audio modalities, generating an emotion feature vector for each modality, and determining a stance relationship network based on the emotion feature vector; Step S4-3: Determine the co-occurrence intensity between characters based on the frequency and duration of the characters appearing simultaneously in the video, and then determine the co-occurrence relationship network.

8. The multi-dimensional character relationship discovery method based on multi-modal information fusion according to claim 7 is characterized in that: Step S4-1 specifically includes: Generate multimodal social feature vectors of characters Indicates character i p Social Behavior: Among them, W social (·) represents mapping multimodal features into social feature vectors, α T , α V , α A Respectively represent the weights of the three modalities in the character's social feature vector; Mapping the social feature vector of a person into a graph structure, the expression is: W ij =β1·frequency ij +β2·emotion ij Among them, W ij represents the weight of the edge between node i and node j in the graph structure, β1 and β2 are the weights of interaction frequency and emotional consistency, ij Represents the interaction frequency between node i and node j, emotion ij Represents the sentiment consistency between node i and node j; To infer the strength of social relationships, the expression is: in, represents the updated social feature vector of node i, neighbors(i) represents the set of neighbor nodes of node i, AGGREGATE(·) is the aggregation function, W represents the weight matrix of the graph neural network, b represents the bias term of the graph neural network, σ nl represents a nonlinear activation function; Finally, a social relationship network G is constructed S .

9. The multi-dimensional character relationship discovery method based on multi-modal information fusion according to claim 8 is characterized in that: Step S4-2 specifically includes: Generate sentiment feature vectors for each modality: Among them, E T , E V , E A Represent the sentiment feature vectors of text, vision and audio respectively, Represents the projection matrices of text, visual and audio features projected onto the corresponding sentiment feature vectors; Calculate the comprehensive sentiment feature vector of node i and the stance feature vector L of node i i , the expression is: E i =b T ·E T +b V ·E V +b A ·E A L i =f emotion (E i ) Among them, β T , β V , β A Respectively represent the weights of the three modes in the character's emotional feature vector, f emotion (·) is the stance classification function; To carry out stance propagation, the expression is: in, represents the updated stance feature vector of node i, L j Represents the stance feature vector of node i’s neighbors; Finally, the position relationship network G is constructed L ; Step S4-3 specifically includes: Calculate the co-occurrence strength I between node i and node j ij : I ij =τ1·frequency ij +τ2·time ij Among them, frequency ij represents the interaction frequency between node i and node j, time ij represents the co-occurrence duration between node i and node j, τ1 and τ2 are the proportional weights of interaction frequency and co-occurrence duration respectively; Finally, the co-occurrence relationship network G is constructed I .

10. The multi-dimensional character relationship discovery method based on multi-modal information fusion according to claim 9 is characterized in that: Step S5 specifically includes: Based on the constructed social relationship network G S , Position relationship network G L and the co-occurrence network G I , by G S The weight of the middle edge determines the closeness of the social relationship between the characters in the video; L The similarity of the middle edge determines the consistency of the position of the characters in the video; G I The weight of the middle edge determines the co-occurrence strength of the characters in the video.

Images