Manifold learning and hierarchical fusion network-based multi-modal humor recognition method and system

Abstract

This invention discloses a multimodal humor recognition method and system based on manifold learning and hierarchical fusion networks, belonging to the field of natural language processing technology. Multimodal humor recognition utilizes multiple different perceptual modalities to identify and understand humor. Accurately identifying humor in multimodal information can help improve the performance of social media recommendation systems. By recognizing and understanding humor, it is also possible to more accurately capture people's emotional states. This invention aims to overcome the limitations of existing technologies and provide an innovative multimodal humor recognition method. This method models the consistency between modalities through two hierarchical fusion networks. Furthermore, since manifold learning can map high-dimensional data to low-dimensional data and reflect the essential structural features of the original high-dimensional data, this method also utilizes the local linear embedding method in manifold learning to transform visual and textual features into low-dimensional manifold features to reduce noise and redundancy, thereby improving the performance of multimodal humor recognition.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a multimodal humor recognition method and system based on manifold learning and hierarchical fusion networks. Background Technology

[0002] The understanding and research of humor have been extensively explored, encompassing multiple disciplines such as linguistics, psychology, and sociology. Humor is a complex social phenomenon, often relying on multiple perceptual modalities, including language, vision, and sound; therefore, single-modality humor recognition is often insufficient. For example, understanding a humorous remark requires considering not only the linguistic features of the text but also the tone of voice and facial expressions. With the development of deep learning and multimodal technologies, multimodal humor recognition has gradually become a highly regarded research direction. By combining different perceptual modalities such as text, images, and audio, multimodal humor recognition aims to improve the understanding and generation of humor. Accurately identifying humor in multimodal information can help improve the performance of social media recommendation systems, and by recognizing and understanding humor, it is also possible to more accurately capture people's emotional states.

[0003] In recent years, with the development of neural networks, many neural network-based methods have been applied to multimodal humor recognition tasks, and multimodal humor recognition technology has also made rapid progress. Given multimodal information, the model learns the important information in each modality and fuses it to determine the expressed emotion. However, existing methods often ignore noise and redundancy in visual information, which can affect the accuracy of the model in judging humorous emotions. Summary of the Invention

[0004] In view of this, the present invention proposes a multimodal humor recognition method and system based on manifold learning and hierarchical fusion networks. It performs consistency modeling between modalities through two hierarchical fusion networks, and also utilizes the local linear embedding method in manifold learning (manifold learning can map high-dimensional data to low-dimensional data and reflect the essential structural features of the original high-dimensional data) to transform visual and text features into low-dimensional manifold features, so as to reduce noise and redundancy, thereby improving the performance of multimodal humor recognition.

[0005] Therefore, the present invention adopts the following technical solution:

[0006] On one hand, this invention discloses a multimodal humor recognition method based on manifold learning and hierarchical fusion networks, the method comprising:

[0007] Multimodal humor recognition data is acquired and encoded to obtain multimodal humor features. The multimodal humor recognition data consists of text-image pairs. The multimodal humor features include text features and visual features.

[0008] Calculate the micro-level consistency score between text features and visual features;

[0009] The text and images are modeled and corresponding dependency graphs are generated. Based on the constructed text dependency graph and visual dependency graph, the text features and visual features are obtained using a graph attention network. The local linear embedding method is used to reduce the dimensionality of the text features and visual features. The enhanced cross-modal consistency score between the reduced text features and visual features is calculated.

[0010] Humor sentiment is classified based on the micro-level consistency score and the enhanced level cross-modal consistency score, and the result of the humor sentiment classification includes humor and non-humor.

[0011] Furthermore, the multimodal humor recognition data is encoded, including: encoding the text using the pre-trained model BERT and encoding the image using the visual encoder Vision Transformer.

[0012] Furthermore, the micro-level consistency score between text features and visual features is calculated, including:

[0013] A multi-head cross-attention mechanism is adopted to map textual and visual features to the same semantic space;

[0014] The text features are updated by connecting all the heads through two layers of multilayer perceptrons and residual connections, and then passing through a regularization layer.

[0015] A set of consistency scores is calculated using the softmax function, and these scores are used to measure the importance of each word in the humor recognition task, denoted as:

[0016]

[0017] in, For the updated text features; W m ∈R d×1 b m ∈R n For learnable parameters, Q c [i,j] represents the similarity score between the i-th text tag and the j-th image region.

[0018] Furthermore, local linear embedding methods are used to reduce the dimensionality of text features, including:

[0019] Using Euclidean distance as the metric, the distance to the text feature t is calculated. i k nearest neighbors in, Text features are obtained using graph attention networks based on constructed text dependency graphs and visual dependency graphs;

[0020] Find the local covariance matrix And calculate the corresponding weight coefficient vector:

[0021]

[0022] From the weight coefficient vector W i Construct the weight coefficient matrix W, and calculate the matrix.

[0023] The matrix formed by the second eigenvector to the (m+1)th eigenvector is the output low-dimensional sample set matrix T′=(y2,y3,…,y m+1 ).

[0024] Furthermore, the dimensions of the manifold features are between 2 and 10.

[0025] Furthermore, the enhanced cross-modal consistency score between the dimensionality-reduced text features and visual features is calculated, including:

[0026]

[0027] in, Specifically, Q e ∈R n×r , T′∈R n×m , I′∈R r×m Where n is the number of tags in the sentence, r is the number of image regions, and m is the vector dimension of the feature representation after dimensionality reduction using the local linear embedding method.

[0028] Furthermore, humor sentiment classification is performed based on the micro-level consistency score and the enhanced cross-modal consistency score, including:

[0029] The micro-level consistency score and the enhancement-level cross-modal consistency score are concatenated and fused, and finally input into the linear layer to become a one-dimensional vector.

[0030] Use the softmax function to normalize this one-dimensional vector.

[0031] In another aspect, the present invention also discloses a multimodal humor recognition system based on manifold learning and hierarchical fusion networks, the system comprising:

[0032] A multimodal data acquisition module is used to acquire multimodal humor recognition data and encode the multimodal humor recognition data to obtain multimodal humor features. The multimodal humor recognition data is a text-image pair; the multimodal humor features include text features and visual features.

[0033] The micro-level consistency calculation module is used to calculate the micro-level consistency score between text features and visual features;

[0034] The enhanced cross-modal consistency calculation module models and generates corresponding dependency graphs for text and images respectively. Based on the constructed text dependency graph and visual dependency graph, it uses a graph attention network to obtain text features and visual features. It uses a local linear embedding method to reduce the dimensionality of text features and visual features, and calculates the enhanced cross-modal consistency score between the reduced text features and visual features.

[0035] The humor sentiment classification prediction module classifies humor sentiment based on the micro-level consistency score and the enhanced level cross-modal consistency score. The humor sentiment classification result includes humorous and non-humorous content.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] This invention utilizes hierarchical semantics between textual and visual modalities to jointly model humor recognition with consistency at the micro and augmentation levels. By employing a manifold learning method, this invention can reduce the dimensionality of visual and auditory features while preserving key structural features, thus reducing computational cost and minimizing the impact of noise and redundancy. This invention can effectively improve the accuracy of humor sentiment classification. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating the multimodal humor recognition method based on manifold learning and hierarchical fusion networks in an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the local embedding method in an embodiment of the present invention;

[0041] Figure 3 This is a framework diagram of the multimodal humor recognition model in an embodiment of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] This invention proposes an innovative multimodal humor recognition method. First, BERT and VisionTransformer pre-trained models are used to encode text and images respectively, generating corresponding textual and visual features. These features have contextual and positional relationships. Then, at the micro-level, the consistency relationship between textual and visual features is modeled. In this part, a multi-head cross-attention mechanism is used to calculate the similarity score between textual and visual features to measure their consistency. Next, the textual and visual features are processed separately to obtain their internal dependencies, constructing a finer-grained dependency graph. An enhancement level is then constructed to model deeper consistency relationships between textual and visual features. Furthermore, in this part, a local linear embedding method is applied to perform manifold dimensionality reduction on the updated textual and visual features to reduce noise and redundancy. The consistency scores obtained from the two levels are concatenated and fused, and finally input into the linear layer to become a one-dimensional vector. Then, the softmax function is used to normalize this one-dimensional vector.

[0045] like Figure 1 As shown in the figure, a multimodal humor recognition method based on manifold learning and hierarchical fusion network in an embodiment of the present invention specifically includes the following steps:

[0046] S1. Given an input text image pair, generate corresponding text features and visual features using a text encoder and a visual encoder, respectively.

[0047] For text, the text encoder BERT transforms the text input into a numerical input {T1, T2, ..., T}. n}, T∈R n×d Where n is the number of tags in the sentence, and d is the dimension of the vector representation.

[0048] For the image, first resize the image to 224×224, then divide each image into r regions and reconstruct them into a sequence, denoted as {p1, p2, ..., p...}. r Then, a Vision Transformer is used for encoding, and the encoded visual features are represented as I = {V1, V2, ..., V}. r}, I∈R r×d Where r represents the number of each region in the image, and d is the dimension of the vector representation.

[0049] S2, Calculation of micro-level cross-modal consistency score:

[0050] Similarity scores are used to measure the semantic consistency between images and text sequences.

[0051] First, an n-head cross-attention mechanism is used to map the text modality and the image modality to the same semantic space, represented as:

[0052]

[0053] Where i represents the i-th attention head, I∈R r×d and T∈R n×d Given the text and visual features, These are the query, key, and value weight matrices, respectively. In particular,

[0054] Then, all the heads are connected together through two layers of multilayer perceptrons and residual connections, and the updated text features are obtained after passing through a regularization layer, as follows:

[0055]

[0056] In this context, LN stands for regularization layer, MLP stands for multilayer perceptron layer, and contact represents the splicing operation.

[0057] A set of consistency scores is calculated using the softmax function, and these scores are used to measure the importance of each word in the humor recognition task, denoted as:

[0058]

[0059] Among them, W m ∈Rd×1 b m ∈R n For learnable parameters, Specifically, Q c [i,j] represents the similarity score between the i-th text tag and the j-th image region.

[0060] S3, Calculation of Enhanced Cross-Modal Consistency Score:

[0061] The input text and images are modeled and corresponding dependency graphs are generated. Based on the constructed text dependency graph and visual dependency graph, a graph attention network is used to obtain text features and visual features. The local linear embedding method is used to reduce the dimensionality of the text features and visual features. The enhanced cross-modal consistency score between the reduced text features and visual features is calculated.

[0062] Understanding humor requires a foundation of understanding textual information, making the acquisition of syntactic information particularly important. This invention utilizes Spacy to extract the dependencies between each tag in the text, treating each tag as a graph node and the dependencies between different graph nodes as edges, iteratively constructing a complete text dependency graph.

[0063] Taking the construction of a text dependency graph as an example:

[0064]

[0065] Where θ l ∈R 2d W l ∈R d×d These are the learnable parameters in the L-th layer of the graph attention network. This represents the dependency relationship between the i-th graph node and its neighbor node j.

[0066] Furthermore, the initial text features are defined as t0, and the text feature input generated by the graph attention network in each layer is the output of the graph attention network in the previous layer, as follows:

[0067]

[0068] Finally, all updated text features are obtained.

[0069] Similarly, a visual dependency graph is constructed based on the relative positions between each image region, where each graph node is represented by a segmented image region, and the edges of the graph are constructed according to the positional relationships of each region. The steps for constructing the visual dependency graph are as follows:

[0070]

[0071] Where θ l ∈R 2d W l ∈R d×d These are the learnable parameters in the L-th layer of the graph attention network. This represents the adjacency relationship between the i-th graph node and its neighbor node j.

[0072] Similarly, we define the initial visual feature as v0, and the visual feature input generated by the graph attention network in each layer is the output of the graph attention network in the previous layer, as follows:

[0073]

[0074] Finally, all updated visual features are obtained.

[0075] S4. Dimensionality reduction process of the local linear embedding method:

[0076] Manifold learning can map high-dimensional data to low-dimensional data and reflect the essential structural features of the original high-dimensional data. In this embodiment of the invention, the local linear embedding method in manifold learning is used to transform visual and text features into low-dimensional manifold features to reduce noise and redundancy, thereby improving the performance of multimodal humor recognition.

[0077] Local linear embedding (LOE) achieves dimensionality reduction by minimizing the similarity difference between high-dimensional and low-dimensional data points. The specific steps are as follows:

[0078] S41. Using Euclidean distance as the metric, calculate the sum t. i k nearest neighbors

[0079] S42. Find the local covariance matrix. And calculate the corresponding weight coefficient vector:

[0080]

[0081] S43, from the weight coefficient vector W i Construct the weight coefficient matrix W, and calculate the matrix.

[0082] S44. The matrix formed by the second eigenvector to the m+1 eigenvectors is the output low-dimensional sample set matrix T′=(y2,y3,…,y m+1 Similarly, the position mapping of the visual modality is used as input, and I′ is obtained using the same method.

[0083] Calculate the semantic consistency score s e as follows:

[0084]

[0085] in, Specifically, Q e ∈R n×r , T′∈R n×m , I′∈R r×m Where n is the number of tags in the sentence, r is the number of image regions, and m is the vector dimension of the feature representation after dimensionality reduction using the local linear embedding method.

[0086] S5. Fusion of consistency scores at two levels to predict humor sentiment classification:

[0087] The final prediction result for humor sentiment classification is obtained through micro-level cross-modal consistency scores and enhancement-level cross-modal consistency scores. The humor sentiment classification result includes both humorous and non-humorous aspects. Specifically, the consistency scores obtained from the two levels are concatenated and fused, and then input into a linear layer to form a one-dimensional vector. This one-dimensional vector is then normalized using the softmax function.

[0088]

[0089] p f =softmax(W f I+b f );

[0090] Among them W y ∈R 2×2r W f ∈R d×1 b y ∈R 2 b f ∈R r Here are the learnable parameters, ⊙ represents the element-wise vector product, where r represents the number of image regions and d is the dimension of the vector representation.

[0091] Finally, the model is optimized using the cross-entropy loss commonly used in classification tasks as follows:

[0092]

[0093] Where y represents the real label. Let be the probability of predicting the label for the i-th text-image pair.

[0094] In the above embodiments, hierarchical semantics between text and visual modalities are utilized to jointly model humor recognition with consistency at the micro and augmentation levels. This invention, by using a manifold learning method, can reduce the dimensionality of visual and auditory features while preserving key structural features, thus reducing computational load and minimizing the impact of noise and redundancy. This invention can effectively improve the accuracy of humor sentiment classification.

[0095] Corresponding to the above-described multimodal humor recognition method, this embodiment of the invention also provides a multimodal humor recognition system, which includes:

[0096] A multimodal data acquisition module is used to acquire multimodal humor recognition data and encode the multimodal humor recognition data to obtain multimodal humor features. The multimodal humor recognition data is a text-image pair; the multimodal humor features include text features and visual features.

[0097] The micro-level consistency calculation module is used to calculate the micro-level consistency score between text features and visual features;

[0098] The enhanced cross-modal consistency calculation module models and generates corresponding dependency graphs for text and images respectively. Based on the constructed text dependency graph and visual dependency graph, it uses a graph attention network to obtain text features and visual features. It uses a local linear embedding method to reduce the dimensionality of text features and visual features, and calculates the enhanced cross-modal consistency score between the reduced text features and visual features.

[0099] The humor sentiment classification prediction module classifies humor sentiment based on the micro-level consistency score and the enhanced level cross-modal consistency score. The humor sentiment classification result includes humorous and non-humorous content.

[0100] The multimodal humor recognition system of this invention is described simply because it corresponds to the multimodal humor recognition method in the above embodiments. For related similarities, please refer to the description of the multimodal humor recognition method in the above embodiments, which will not be described in detail here.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal humor recognition method based on manifold learning and hierarchical fusion networks, characterized in that, The method includes: Multimodal humor recognition data is acquired and encoded to obtain multimodal humor features. The multimodal humor recognition data consists of text-image pairs. The multimodal humor features include text features and visual features. Calculate the micro-level consistency score between text features and visual features; The text and images are modeled and corresponding dependency graphs are generated. Based on the constructed text dependency graph and visual dependency graph, the text features and visual features are obtained using a graph attention network. The local linear embedding method is used to reduce the dimensionality of the text features and visual features. The enhanced cross-modal consistency score between the reduced text features and visual features is calculated. Humor sentiment is classified based on the micro-level consistency score and the enhanced level cross-modal consistency score, and the result of the humor sentiment classification includes humor and non-humor.

2. The multimodal humor recognition method according to claim 1, characterized in that, Encoding the multimodal humor recognition data includes: encoding the text using the pre-trained model BERT and encoding the image using the visual encoder VisionTransformer.

3. The multimodal humor recognition method according to claim 1, characterized in that, Calculate the micro-level consistency score between text features and visual features, including: A multi-head cross-attention mechanism is used to map text and images to the same semantic space; The text features are updated by connecting all the heads through two layers of multilayer perceptrons and residual connections, and then passing through a regularization layer. A set of consistency scores is calculated using the softmax function, and these scores are used to measure the importance of each word in the humor recognition task, denoted as: in, For the updated text features; W m ∈R d×1 b m ∈R n For learnable parameters, Q c [i,j] represents the similarity score between the i-th text tag and the j-th image region.

4. The multimodal humor recognition method according to claim 1, characterized in that, Dimensionality reduction of text features is achieved using local linear embedding methods, including: Using Euclidean distance as the metric, the distance to the text feature t is calculated. i k nearest neighbors in, Text features are obtained using graph attention networks based on constructed text dependency graphs and visual dependency graphs; Find the local covariance matrix And calculate the corresponding weight coefficient vector: From the weight coefficient vector W i Construct the weight coefficient matrix W, and calculate the matrix. The matrix formed by the second eigenvector to the (m+1)th eigenvector is the output low-dimensional sample set matrix T′=(y2,y3,…,y m+1 ).

5. The multimodal humor recognition method according to claim 4, characterized in that, The dimensions of the manifold features range from 2 to 10.

6. The multimodal humor recognition method according to claim 4 or 5, characterized in that, Calculate the enhanced cross-modal consistency score between the dimensionality-reduced text features and visual features, including: in, Specifically, Q e ∈R n×r , T′∈R n×m , I′∈R r×m , where n is the number of tags in the sentence, r is the number of image regions, and m is the vector dimension of the feature representation after dimensionality reduction using the local linear embedding method.

7. The multimodal humor recognition method according to claim 1, characterized in that, Humor sentiment classification is performed based on the micro-level consistency score and the enhanced cross-modal consistency score, including: The micro-level consistency score and the enhancement-level cross-modal consistency score are concatenated and fused, and finally input into the linear layer to become a one-dimensional vector. Use the softmax function to normalize this one-dimensional vector.

8. A multimodal humor recognition system based on manifold learning and hierarchical fusion networks, characterized in that, The system includes: A multimodal data acquisition module is used to acquire multimodal humor recognition data and encode the multimodal humor recognition data to obtain multimodal humor features. The multimodal humor recognition data is a text-image pair; the multimodal humor features include text features and visual features. The micro-level consistency calculation module is used to calculate the micro-level consistency score between text features and visual features; The enhanced cross-modal consistency calculation module models text and images separately and generates corresponding dependency graphs. Based on the constructed text dependency graph and visual dependency graph, it uses a graph attention network to obtain text features and visual features. It uses a local linear embedding method to reduce the dimensionality of text features and visual features and calculates the enhanced cross-modal consistency score between the reduced text features and visual features. The humor sentiment classification prediction module classifies humor sentiment based on the micro-level consistency score and the enhanced level cross-modal consistency score. The humor sentiment classification result includes humorous and non-humorous content.

Images