Multimodal semantic understanding method, device and equipment based on bidirectional dynamic interaction

Abstract

The invention relates to a multimodal semantic understanding method, device and equipment based on bidirectional dynamic interaction. The method comprises the following steps: acquiring a memetic sample, and constructing a joint learning model containing feature extraction, fusion and multi-task classification modules; through a feature extraction module, a multi-mode large model is guided to generate visual, hatred and metaphor descriptions through thinking chain prompt, through a pre-training multi-mode encoder, the descriptions and corresponding mode data are subjected to fusion coding to obtain fusion coding vectors, through a feature fusion module, the vectors are subjected to normalization and unified dimension mapping, and the fusion coding vectors are subjected to fusion coding. The method comprises the following steps: performing element-by-element difference, global difference feature calculation, gating weighted summation and residual superposition to obtain global fusion features, synchronously classifying hatred and metaphor information based on the features, and outputting a classification result to a to-be-classified model factor through a combined loss function training model containing two types of loss. By adopting the method, hatred model factors packaged by the metaphor can be accurately identified, and an efficient technical support is provided for compliance treatment of a social media platform.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method, apparatus and device for multimodal semantic understanding based on bidirectional dynamic interaction. Background Technology

[0002] Memes, as popular discourse symbols in the internet age, spread rapidly among groups through image-text collages (emojis). Their meaning generation relies on the dynamic coupling of text, visual symbols, and contextual information, thus placing higher demands on the cross-modal understanding capabilities of algorithms. Existing research on meme understanding mainly focuses on hate detection and metaphor detection: the former aims to identify offensive content to maintain platform security, while the latter attempts to reveal unconventional concept mappings to explain rhetorical strategies. Although these paradigms have achieved considerable results in filtering harmful information and uncovering latent intentions, they neglect the overlap and conflict between the two types of signals in the semantic space, as well as their complementarity in rhetorical structure: hate often uses metaphors to circumvent censorship, while the ambiguity of metaphors can amplify or weaken the intensity of hate. Existing research trains the two tasks in isolation, resulting in metaphor features failing to provide disambiguation cues for hate judgment, and hate labels failing to constrain the overgeneralization of metaphor detection. This information fragmentation makes it difficult for models to capture complex rhetoric that is "superficially metaphorical but contains hate," thus limiting their applicability and interpretability in broader social scenarios. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, and device for multimodal semantic understanding based on bidirectional dynamic interaction to address the aforementioned technical problems.

[0004] A multimodal semantic understanding method based on bidirectional dynamic interaction, the method comprising: Acquire meme samples, which include image data and text data; Construct a joint learning model, which includes a feature extraction module, a feature fusion module, and a multi-task classification module; Through the feature extraction module, a multimodal large model is generated based on the thought chain prompt to generate visual descriptions of the image data and hate descriptions and metaphor descriptions of the text data. The pre-trained multimodal encoder is used to fuse and encode each type of description with the corresponding modal data to obtain the corresponding fused encoding vector. Through the feature fusion module, the fused encoding vectors are normalized and mapped to a unified dimension. The vectors after mapping between any two classes are differencing element by element. The global difference feature is obtained based on the mean of the difference results between each vector and other vectors. The gating weight of each vector is obtained based on the global difference feature. The mapped vectors are weighted and summed based on the gating weight. The summation is then superimposed with the residual of the normalized result to obtain the global fused feature. The multi-task classification module classifies hate information and metaphorical information synchronously based on the global fusion features. The joint learning model is trained using meme samples and a joint loss function to obtain a trained joint learning model; the joint loss function includes hate classification loss and metaphor classification loss. The memes to be classified are input into the trained joint learning model, which outputs the classification results of hate information and metaphor information.

[0005] A multimodal semantic understanding device based on bidirectional dynamic interaction, the device comprising: A sample acquisition module is used to acquire meme samples, which include image data and text data; The model building module is used to construct a joint learning model, which includes a feature extraction module, a feature fusion module, and a multi-task classification module. The feature extraction module generates visual descriptions of the image data and hate descriptions and metaphor descriptions of the text data based on thought chain prompts and a multimodal large model. A pre-trained multimodal encoder is used to fuse and encode each type of description with its corresponding modality data to obtain corresponding fused encoding vectors. The feature fusion module normalizes and unifies the dimension mapping of each fused encoding vector. For any two mapped vectors, element-wise difference is performed. A global difference feature is obtained based on the mean of the difference results between each vector and other vectors. A gating weight is obtained for each vector based on the global difference feature. The mapped vectors are weighted and summed based on the gating weight, and then superimposed with the residual of the normalized result to obtain the global fused feature. The multi-task classification module simultaneously classifies hate information and metaphor information based on the global fused feature. The model training module is used to train the joint learning model using meme samples and a joint loss function to obtain a trained joint learning model; the joint loss function includes hate classification loss and metaphor classification loss. The output module is used to input the memes to be classified into the trained joint learning model and output the classification results of hate information and metaphor information.

[0006] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Acquire meme samples, which include image data and text data; Construct a joint learning model, which includes a feature extraction module, a feature fusion module, and a multi-task classification module; Through the feature extraction module, a multimodal large model is generated based on the thought chain prompt to generate visual descriptions of the image data and hate descriptions and metaphor descriptions of the text data. The pre-trained multimodal encoder is used to fuse and encode each type of description with the corresponding modal data to obtain the corresponding fused encoding vector. Through the feature fusion module, the fused encoding vectors are normalized and mapped to a unified dimension. The vectors after mapping between any two classes are differencing element by element. The global difference feature is obtained based on the mean of the difference results between each vector and other vectors. The gating weight of each vector is obtained based on the global difference feature. The mapped vectors are weighted and summed based on the gating weight. The summation is then superimposed with the residual of the normalized result to obtain the global fused feature. The multi-task classification module classifies hate information and metaphorical information synchronously based on the global fusion features. The joint learning model is trained using meme samples and a joint loss function to obtain a trained joint learning model; the joint loss function includes hate classification loss and metaphor classification loss. The memes to be classified are input into the trained joint learning model, which outputs the classification results of hate information and metaphor information.

[0007] The aforementioned method, apparatus, and device are based on bidirectional dynamic interaction for multimodal semantic understanding. By acquiring meme samples containing both image and text data, a joint learning model integrating feature extraction, fusion, and multi-task classification modules is constructed. Mind chain prompts guide the large multimodal model to generate targeted semantic descriptions. Fusion encoding is achieved through a pre-trained multimodal encoder. Then, through normalization, unified dimensionality mapping, differential feature calculation, gated weighted fusion, and residual superposition, a global fusion feature that balances cross-modal information and unique value is obtained. Finally, based on this feature, hatred and metaphor classification are simultaneously completed, and the model is trained using a joint loss function containing both types of losses. This enables hatred and metaphor representations to dynamically interact and mutually calibrate classification boundaries, effectively capturing complex rhetoric that is "superficially metaphorical but inherently hateful." It solves the problems that metaphor features cannot disambiguate hatred judgments and that hatred labels cannot constrain the generalization of metaphor detection, while also improving the classification accuracy and robustness of both tasks, enhancing the applicability and interpretability of meme semantic understanding. The embodiments of the present invention can be applied to practical scenarios such as harmful information filtering and network content security supervision on social media platforms, accurately identify hate memes disguised by metaphors, provide efficient technical support for platform compliance governance, and reduce manual review costs. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating a multimodal semantic understanding method based on bidirectional dynamic interaction in one embodiment; Figure 2This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0010] In one embodiment, such as Figure 1 As shown, a multimodal semantic understanding method based on bidirectional dynamic interaction is provided, including the following steps: Step 102: Obtain meme samples, which contain image data and text data.

[0011] A meme is a popular discourse symbol that spreads among internet users in the form of a collage of images and text. Its meaning generation depends on the dynamic coupling of text, visual symbols, and contextual information. Image data consists of the visual elements (such as graphics and images) contained in a meme, while text data consists of the accompanying textual content (such as slogans and annotations). Meme samples come from a pre-constructed set of meme samples, and each meme sample in the set carries tag information, including hate information tags and metaphorical information tags.

[0012] Step 104: Construct a joint learning model, which includes a feature extraction module, a feature fusion module, and a multi-task classification module.

[0013] Joint learning models refer to model frameworks that can simultaneously carry out training and inference for multiple related tasks, improving overall performance through information interaction between tasks; the feature extraction module is used to extract valuable feature information from raw data, the feature fusion module is used to integrate features from different sources and modalities into unified features, and the multi-task classification module is used to simultaneously complete the two tasks of hate information classification and metaphor information classification.

[0014] Step 106: Through the feature extraction module, the multimodal large model is guided by the thought chain prompt to generate visual descriptions of image data and hate descriptions and metaphor descriptions of text data. The pre-trained multimodal encoder is used to fuse and encode each type of description with the corresponding modal data to obtain the corresponding fused encoding vector.

[0015] Mind chain hints refer to a hint strategy that prompts a large model to gradually uncover deeper information through multi-step guided queries. Multimodal large model (MLLM) refers to an artificial intelligence model that can process multiple modal data such as images and text and generate corresponding outputs. Pre-trained multimodal encoders (such as CLIP models) refer to tools that, after being trained on large-scale data, can map different modal data and their related descriptions into high-dimensional numerical vectors.

[0016] It is understandable that by guiding the large model to generate accurate semantic descriptions through thought chain prompts, deep contextual information is added to the modal data, solving the problem of cross-modal coupling of meme semantic dependence. By using a multimodal encoder to achieve the fusion encoding of data and description, it is ensured that the features simultaneously contain original modal information and semantic association information, providing high-quality feature input for subsequent feature fusion and dual-task classification, and improving the model's ability to understand cross-modal semantics.

[0017] Step 108: Through the feature fusion module, normalize and unify the dimension mapping of each fused encoding vector, perform element-wise difference on any two types of mapped vectors, obtain the global difference feature based on the mean of the difference results of each vector and other vectors, obtain the gating weight of each vector based on the global difference feature, perform weighted summation of the mapped vectors based on the gating weight, and then superimpose it with the residual of the result after normalization to obtain the global fused feature.

[0018] Normalization refers to adjusting the mean and variance of features to bring different modal features to the same numerical scale. Unified dimensionality mapping refers to converting features of different dimensions into features of the same dimension through linear transformation. Element-wise differencing involves calculating the difference between corresponding elements of two class vectors to characterize the differences between features. Global difference features are the mean of the differences between a given feature and all other features, used to quantify the total amount of unique information contained in that feature. Gated weights are weight coefficients calculated based on global difference features, used to characterize the contribution of each feature. Residual stacking is the operation of stacking the processed original features with the fused features to preserve the original information. Global fusion features are the final features that integrate all modal features and semantic information after the above series of processing, used for subsequent classification tasks.

[0019] Understandably, normalization and unified dimensional mapping solve the scale imbalance problem caused by traditional multimodal feature splicing, ensuring that features of different modalities can be effectively fused. By using the difference concept and gated weight calculation, overlapping and redundant information between features is removed, while retaining the unique value information of each feature. This guides the model to focus on the core classification criteria. Residual superposition avoids excessive loss of original information during the fusion process and stabilizes the feature distribution. The final global fused feature takes into account cross-modal information, semantic association information, and unique value information, providing accurate and comprehensive feature support for dual-task classification and improving the accuracy and robustness of the classification results.

[0020] Step 110: Through the multi-task classification module, hate information and metaphor information are classified synchronously based on global fusion features.

[0021] Hate information classification refers to the task of determining whether a meme contains hate-related content such as offensiveness or discrimination. Metaphorical information classification refers to the task of determining whether a meme contains unconventional conceptual mappings (i.e., metaphorical rhetoric). Based on global fusion features, this method achieves simultaneous classification of both tasks, enabling hate and metaphor classification to share cross-modal semantic information. This solves the information fragmentation problem caused by the isolated processing of the two tasks in existing technologies. Hate classification can use metaphor features to determine whether it is "hate wrapped in metaphor," while metaphor classification can use hate features to constrain overgeneralization. The two tasks provide disambiguation cues to each other, improving their respective classification accuracy, especially enhancing the ability to identify composite rhetorical memes that are "superficially metaphorical but contain hate."

[0022] Step 112: Train the joint learning model using meme samples and the joint loss function to obtain the trained joint learning model.

[0023] The joint loss function includes hate classification loss and metaphor classification loss. Hate classification loss measures the difference between the hate classification result and the true label. Metaphor classification loss measures the difference between the metaphor classification result and the true label.

[0024] It is understandable that by binding the training of the two types of tasks together through the joint loss function, the model can simultaneously optimize the parameters of hate classification and metaphor classification during the training process, realizing the dynamic interaction between hate and metaphor representations; the decrease in loss of one type of task will drive the feature optimization of the other type of task, avoiding the bias caused by single task training, solving the problem of insufficient generalization ability caused by isolated task training in existing technologies, and improving the robustness and adaptability of the model in complex meme scenarios.

[0025] Step 114: Input the meme to be classified into the trained joint learning model, and output the classification results of hate information and metaphor information.

[0026] The meme to be classified refers to unknown meme data that needs to be determined whether it contains hate information or metaphorical information; the classification result refers to the model's output judgment result that the meme to be classified belongs to hate / non-hate or metaphorical / non-metaphorical.

[0027] It is understood that the method of the present invention uses multi-perspective knowledge enhancement and high-dimensional similarity redundancy removal to enable hatred and metaphorical representations to interact dynamically during the training process, serving as soft cues to mutually calibrate the task classification boundaries, thereby simultaneously improving the accuracy and robustness of both tasks.

[0028] The aforementioned multimodal semantic understanding method based on bidirectional dynamic interaction acquires meme samples containing both image and text data, constructs a joint learning model integrating feature extraction, fusion, and multi-task classification modules, and guides the large multimodal model to generate targeted semantic descriptions using thought chain prompts. Fusion encoding is achieved through a pre-trained multimodal encoder, and then, through normalization, unified dimensionality mapping, differential feature calculation, gated weighted fusion, and residual superposition, a global fusion feature that balances cross-modal information and unique value is obtained. Finally, based on this feature, hatred and metaphor classification are simultaneously completed, and the model is trained using a joint loss function containing both types of losses. This method enables hatred and metaphor representations to dynamically interact and mutually calibrate classification boundaries, effectively capturing complex rhetoric that is "superficially metaphorical but inherently hateful." It solves the problems that metaphor features cannot disambiguate hatred judgments and that hatred labels cannot constrain the generalization of metaphor detection, while also improving the classification accuracy and robustness of both tasks, enhancing the applicability and interpretability of meme semantic understanding. The embodiments of the present invention can be applied to practical scenarios such as harmful information filtering and network content security supervision on social media platforms, accurately identify hate memes disguised by metaphors, provide efficient technical support for platform compliance governance, and reduce manual review costs.

[0029] In one embodiment, the thought chain prompt includes: a first query for requesting the multimodal large model to describe the visual content of the image data; a second query for requesting the multimodal large model to analyze hate information by combining the image data, text data, and the visual description output after the first query; and a third query for requesting the multimodal large model to analyze metaphorical information by combining the image data, text data, and the visual description output after the first query.

[0030] In this embodiment, to obtain high-quality and information-rich task prompts, the present invention uses CoT (Chain-of-Thought) prompts to guide a multimodal large model in extracting deep cross-modal information. The present invention constructs the following thought framework: ; ; ; in Representing memes, each meme From an image and a text Composition. Each prompt is designed as follows: Please briefly describe the content in the image. Please analyze the hate information in the meme in conjunction with the image, the text, and the image description. Please analyze the metaphorical information in this meme in conjunction with the image, the text, and the image description. In one embodiment, a pre-trained multimodal encoder is used to fuse various descriptions with corresponding modal data to obtain corresponding fused encoding vectors. This includes: the pre-trained multimodal encoder is a CLIP model; image data and visual descriptions are input into the image encoder of the CLIP model, and a first fused encoding vector is output; text data and hate descriptions are input into the text encoder of the CLIP model, and a second fused encoding vector is output; and text data and metaphorical descriptions are input into the text encoder of the CLIP model, and a third fused encoding vector is output.

[0031] In this embodiment, a pre-trained CLIP model is used as a multimodal information encoder to obtain the contextual information of the input meme: ; ; ; ; in, Represents the number of samples. Represents the vector dimension. Indicates visual description, Describing hate, It indicates a metaphorical description.

[0032] In one embodiment, normalizing and unifying the dimension mapping of each fused coding vector includes: normalizing the first fused coding vector, the second fused coding vector, and the third fused coding vector respectively through a LayerNorm layer, wherein the second fused coding vector and the third fused coding vector use the same learnable scaling and translation parameters, and the first fused coding vector uses independent learnable scaling and translation parameters; and mapping the normalized first fused coding vector, the second fused coding vector, and the third fused coding vector to the same preset dimension through three independent linear transformation layers respectively.

[0033] In this embodiment, to address the problems of modal scale imbalance caused by simple splicing of traditional multimodal features and the large amount of overlapping and redundant information in features from different perspectives and modalities, this invention introduces strategies such as differential attention mechanism, gated fusion, and residual connection, and improves feature fusion efficiency by using lightweight linear layers and activation functions.

[0034] To address the numerical scale differences among features of different modalities, this invention defines LayerNorm layers for text features and visual features respectively: ; ; in, , These are the mean and variance of the feature, respectively. For learnable scaling and translation parameters, This is a numerically stable term; after normalization, the feature mean is 0 and the variance is 1, ensuring that text and visual features participate in fusion at the same scale.

[0035] Mapping different features to a unified dimension, the mathematical expression is: ; ; ; Where W is the weight matrix and b is the bias term.

[0036] In one embodiment, element-wise differencing of any two mapped vectors includes: calculating the element-wise absolute difference between the two mapped vectors, normalizing the absolute difference using a sigmoid activation function, and obtaining the differencing result; the differencing result is: ; in, For the first i Class and the j The difference result of the vectors after class mapping. For the first i Class-mapped vectors For the first j Class-mapped vectors This is the Sigmoid activation function.

[0037] In this embodiment, the gating weights of each feature are calculated using the difference concept. This method quantifies the difference information between features from different perspectives within the same modality, removes redundant overlapping parts between features, and retains the unique and task-valued differentiated information of each perspective. This guides the model to focus its attention highly on new information that contributes to core tasks such as classification and regression. First, the element-wise difference between features is calculated, and combined with the Sigmoid activation normalization difference result, the redundancy between features is characterized.

[0038] In one embodiment, obtaining the gating weights of each vector based on the global difference features includes: performing a linear transformation on the global difference features, and then normalizing them using the Softmax function to obtain the gating weights corresponding to each mapped vector; the gating weights are used to characterize the unique information contribution of the corresponding vector.

[0039] In this embodiment, the average of the difference results of the i-th feature and all other viewpoint features is taken to obtain the global difference feature of that feature, quantifying the total amount of new information it contains: ; By applying linear transformation and Softmax normalization, the global difference features are converted into gating weights for each feature, as shown in the following formula: .

[0040] In one embodiment, the global fusion feature is obtained by weighted summation of the mapped vectors based on gating weights and then superimposed with the residual of the normalized result. This includes: weighted summation of all mapped vectors based on the gating weights corresponding to each mapped vector to obtain gated fusion features; concatenating the normalized results of all fusion encoded vectors to obtain the original feature concatenation result; processing the original feature concatenation result sequentially through a linear layer and a GELU activation function to obtain residual supplementary features; superimposing the residual supplementary features with the gating fusion features, and then stabilizing the feature distribution through a LayerNorm layer to obtain the global fusion feature.

[0041] In this embodiment, the mapped features are weighted and summed based on the gating weights to obtain the fused features, and the mathematical expression is as follows: ; To avoid excessive discarding of original feature information in gated fusion, this invention introduces residual connections. The original normalized features are concatenated, then activated by a linear layer and GELU, and finally superimposed with the residual features from the gated fusion. The distribution is then stabilized using LayerNorm. The mathematical expression is: .

[0042] In one embodiment, both the hate classification loss and the metaphor classification loss are cross-entropy losses; the joint loss function is the superposition of the hate classification loss and the metaphor classification loss.

[0043] In this embodiment, the present invention defines two classifiers for classifying hatred and metaphors. The classification process can be represented as follows: ; ; The loss function is calculated as follows: ; ; in Represents a meme dataset, Represents the cross-entropy loss function. These represent the actual value and the predicted value, respectively.

[0044] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0045] In one embodiment, a multimodal semantic understanding device based on bidirectional dynamic interaction is provided, comprising: The sample acquisition module is used to acquire meme samples, which include image data and text data; The model building module is used to construct a joint learning model, which includes a feature extraction module, a feature fusion module, and a multi-task classification module. The feature extraction module generates visual descriptions of image data and hate / metaphorical descriptions of text data based on thought chain prompts, using a pre-trained multimodal encoder to fuse each type of description with its corresponding modality data, resulting in a fused encoding vector. The feature fusion module normalizes and unifies the dimensionality mapping of each fused encoding vector, performs element-wise differencing on any two mapped vectors, obtains global difference features based on the mean of the differences between each vector and other vectors, obtains the gating weights for each vector based on the global difference features, weights the mapped vectors based on the gating weights, and then superimposes the sum with the normalized residual to obtain the global fused features. The multi-task classification module simultaneously classifies hate and metaphorical information based on the global fused features. The model training module is used to train a joint learning model using meme samples and a joint loss function to obtain a trained joint learning model; the joint loss function includes hate classification loss and metaphor classification loss. The output module is used to input the memes to be classified into the trained joint learning model and output the classification results of hate information and metaphor information.

[0046] Specific limitations regarding the bidirectional dynamic interaction-based multimodal semantic understanding device can be found in the limitations of the bidirectional dynamic interaction-based multimodal semantic understanding method described above, and will not be repeated here. Each module in the aforementioned bidirectional dynamic interaction-based multimodal semantic understanding device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0047] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 2As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal semantic understanding method based on bidirectional dynamic interaction. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0048] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0049] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0050] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0051] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for multi-modal semantic understanding based on bidirectional dynamic interaction, characterized in that, The method comprises: acquiring a meme sample containing image data and text data; constructing a joint learning model comprising a feature extraction module, a feature fusion module, and a multi-task classification module; using the feature extraction module, guiding a multi-modal large model to generate a visual description of the image data and a hatred description and a metaphor description of the text data based on a thinking chain prompt, and using a pre-trained multi-modal encoder to fuse and encode each type of description with the corresponding modal data to obtain a corresponding fusion encoding vector; using the feature fusion module, normalizing and uniformly dimension mapping each fusion encoding vector, performing element-by-element difference on any two types of mapped vectors, obtaining a global difference feature according to the mean of the difference results of each vector and other vectors, obtaining a gating weight of each vector according to the global difference feature, and performing weighted summation on the mapped vectors based on the gating weight, and then adding the residual error after normalization processing to obtain a global fusion feature; using the multi-task classification module, synchronously classifying hatred information and metaphor information according to the global fusion feature; training the joint learning model using the meme sample and a joint loss function to obtain a trained joint learning model; the joint loss function comprises a hatred classification loss and a metaphor classification loss; inputting a meme to be classified into the trained joint learning model to output a hatred information classification result and a metaphor information classification result.

2. The method of claim 1, wherein, The thinking chain prompt comprises: a first query for requesting the multi-modal large model to describe the visual content of the image data; a second query for requesting the multi-modal large model to analyze the hatred information in combination with the image data, the text data, and the visual description output after the first query; a third query for requesting the multi-modal large model to analyze the metaphor information in combination with the image data, the text data, and the visual description output after the first query.

3. The method of claim 1, wherein, The use of a pre-trained multi-modal encoder to fuse and encode each type of description with the corresponding modal data to obtain a corresponding fusion encoding vector comprises: the pre-trained multi-modal encoder is a CLIP model; inputting the image data and the visual description into the image encoder of the CLIP model to output a first fusion encoding vector; inputting the text data and the hatred description into the text encoder of the CLIP model to output a second fusion encoding vector; inputting the text data and the metaphor description into the text encoder of the CLIP model to output a third fusion encoding vector.

4. The method of claim 1, wherein, The normalization and uniform dimension mapping of each fusion encoding vector comprises: performing normalization on the first fusion encoding vector, the second fusion encoding vector, and the third fusion encoding vector through a LayerNorm layer, wherein the second fusion encoding vector and the third fusion encoding vector use the same learnable scaling parameter and translation parameter, and the first fusion encoding vector uses an independent learnable scaling parameter and translation parameter; mapping the normalized first fusion encoding vector, the second fusion encoding vector, and the third fusion encoding vector to the same preset dimension through three independent linear transformation layers.

5. The method of claim 1, wherein, The element-by-element difference on any two types of mapped vectors comprises: The element-wise absolute value difference of any two mapped vectors is calculated, and the absolute value difference is normalized by a Sigmoid activation function to obtain a difference result; the difference result is: wherein, is the first i class mapping post-vector, j difference result of the first class mapping post-vector, i is the first class mapping post-vector, j is the first class mapping post-vector, is a Sigmoid activation function.

6. The method of claim 1, wherein, The global difference feature is linearly transformed, and then normalized by a Softmax function to obtain the gating weight corresponding to each mapped vector; the gating weight is used to represent the unique information contribution degree of the corresponding vector. The mapped vectors are weighted and summed based on the gating weight, and then stacked with the normalized result residual to obtain the global fusion feature, which includes:

7. The method of claim 1, wherein, The mapped vectors are weighted and summed based on the gating weight corresponding to each mapped vector to obtain the gated fusion feature. The normalized results of all fusion encoding vectors are spliced to obtain an original feature splicing result. The original feature splicing result is sequentially processed by a linear layer and a GELU activation function to obtain a residual supplementary feature. The residual supplementary feature is stacked with the gated fusion feature, and then the feature distribution is stabilized by a LayerNorm layer to obtain the global fusion feature. The hatred classification loss and the metaphor classification loss are both cross-entropy losses; the joint loss function is the superposition of the hatred classification loss and the metaphor classification loss.

8. The method of claim 1, wherein, The device includes:

9. A device for multi-modal semantic understanding based on bidirectional dynamic interaction, characterized in that, A sample acquisition module for acquiring a meme sample, the meme sample containing image data and text data; A model construction module for constructing a joint learning model, the joint learning model including a feature extraction module, a feature fusion module, and a multi-task classification module; the feature extraction module is used to guide a multi-modal large model to generate visual descriptions of the image data and hatred descriptions and metaphor descriptions of the text data based on a thinking chain prompt, and to fuse encode each type of description with the corresponding modal data using a pre-trained multi-modal encoder to obtain corresponding fusion encoding vectors; the feature fusion module is used to normalize and uniformly dimension map each fusion encoding vector, perform element-wise differencing on any two mapped vectors, obtain a global difference feature based on the mean of the difference result of each vector and other vectors, obtain the gating weight of each vector based on the global difference feature, and weighted sum the mapped vectors based on the gating weight, and then stack the normalized result residual to obtain the global fusion feature; the multi-task classification module is used to simultaneously classify hatred information and metaphor information based on the global fusion feature; A model training module for training the joint learning model using the meme sample and the joint loss function to obtain a trained joint learning model; the joint loss function includes a hatred classification loss and a metaphor classification loss; A result output module for inputting a meme to be classified into the trained joint learning model and outputting a hatred information classification result and a metaphor information classification result. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 8. 10.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-9. ​

Images