Method, device and equipment for multi-modal semantic understanding based on bidirectional dynamic interaction

Abstract

The application relates to a multi-modal semantic understanding method, device and equipment based on bidirectional dynamic interaction. The method comprises the following steps: acquiring a meme sample, and constructing a joint learning model comprising a feature extraction, fusion and multi-task classification module; a multi-modal large model is guided to generate visual, hatred and metaphor descriptions through a thinking chain prompt by the feature extraction module; the descriptions are fused and encoded with corresponding modal data by a pre-trained multi-modal encoder to obtain a fusion encoding vector; after normalization and uniform dimension mapping of the vector, global fusion features are obtained through element-by-element difference, global difference feature calculation, gated weighted summation and residual superposition; hatred and metaphor information are classified based on the features; the model is trained by a joint loss function comprising two types of losses; and a classification result is output for a to-be-classified meme. The method can accurately identify hatred memes wrapped in metaphors, and provides efficient technical support for compliance management of social media platforms.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a multi-modal semantic understanding method, device and equipment based on bidirectional dynamic interaction. BACKGROUND

[0002] Meme, as a popular discourse symbol in the Internet era, spreads rapidly among groups in the form of graphic collages (meme packs), and its meaning generation relies on the dynamic coupling of text, visual symbols and contextual information, thus putting higher requirements on the cross-modal understanding ability of algorithms. Existing researches on meme understanding mainly focus on hate detection and metaphor detection: the former aims to identify offensive content to maintain platform safety, and the latter tries to reveal unconventional concept mapping to explain rhetorical strategies. Although the above paradigms have achieved remarkable results in harmful information filtering and potential intent mining, they ignore the overlap and conflict of the two types of signals in the semantic space and the complementarity of the two in the rhetorical structure: hate often uses metaphor to avoid censorship, while the ambiguity of metaphor may amplify or weaken the intensity of hate. Existing researches train the two tasks separately, resulting in the inability of metaphor features to provide disambiguation clues for hate judgment, and the inability of hate labels to constrain the overgeneralization of metaphor detection. This information fragmentation makes it difficult for the model to capture the complex rhetoric of "surface metaphor, internal hatred", thus limiting its applicability and explainability in more extensive social scenarios. SUMMARY

[0003] Therefore, it is necessary to provide a multi-modal semantic understanding method, device and equipment based on bidirectional dynamic interaction to solve the above technical problems.

[0004] A multi-modal semantic understanding method based on bidirectional dynamic interaction, the method comprising:

[0005] obtaining a meme sample, the meme sample containing image data and text data;

[0006] constructing a joint learning model, the joint learning model comprising a feature extraction module, a feature fusion module and a multi-task classification module;

[0007] guiding a multi-modal large model to generate visual descriptions of the image data and hate descriptions and metaphor descriptions of the text data based on thought chain prompts through the feature extraction module, and using a pre-trained multi-modal encoder to fuse and encode each type of description with the corresponding modal data to obtain corresponding fusion encoding vectors;

[0008] The feature fusion module is configured to normalize each of the fused encoding vectors, map the normalized vectors to a uniform dimension, perform element-wise difference on any two of the mapped vectors, obtain global difference features according to mean values of difference results of each vector and other vectors, obtain gating weights of the vectors according to the global difference features, perform weighted summation on the mapped vectors based on the gating weights, and obtain global fusion features by superimposing the result residual after normalization processing;

[0009] The multi-task classification module is configured to simultaneously perform hatred information classification and metaphor information classification according to the global fusion features.

[0010] The model training module is configured to train the joint learning model by using the meme samples and a joint loss function, and obtain a trained joint learning model; the joint loss function includes a hatred classification loss and a metaphor classification loss.

[0011] The result output module is configured to input a meme to be classified into the trained joint learning model, and output a hatred information classification result and a metaphor information classification result.

[0012] A multi-modal semantic understanding device based on bidirectional dynamic interaction, the device comprising:

[0013] A sample acquisition module configured to acquire meme samples, the meme samples including image data and text data;

[0014] A model construction module configured to construct a joint learning model, the joint learning model including a feature extraction module, a feature fusion module, and a multi-task classification module; the feature extraction module is configured to guide a multi-modal large model to generate visual descriptions of the image data and hatred descriptions and metaphor descriptions of the text data based on a thinking chain prompt, fuse each type of description with corresponding modal data by using a pre-trained multi-modal encoder, and obtain corresponding fused encoding vectors; the feature fusion module is configured to normalize each of the fused encoding vectors, map the normalized vectors to a uniform dimension, perform element-wise difference on any two of the mapped vectors, obtain global difference features according to mean values of difference results of each vector and other vectors, obtain gating weights of the vectors according to the global difference features, perform weighted summation on the mapped vectors based on the gating weights, and obtain global fusion features by superimposing the result residual after normalization processing; and the multi-task classification module is configured to simultaneously perform hatred information classification and metaphor information classification according to the global fusion features.

[0015] A model training module configured to train the joint learning model by using the meme samples and a joint loss function, and obtain a trained joint learning model; the joint loss function includes a hatred classification loss and a metaphor classification loss.

[0016] A result output module configured to input a meme to be classified into the trained joint learning model, and output a hatred information classification result and a metaphor information classification result.

[0017] A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:

[0018] Obtaining a meme sample containing image data and text data;

[0019] Constructing a joint learning model comprising a feature extraction module, a feature fusion module and a multi-task classification module;

[0020] Through the feature extraction module, based on the thought chain prompt, guiding the multi-modal large model to generate the visual description of the image data and the hatred description and the metaphor description of the text data, using the pre-trained multi-modal encoder to fuse and encode each type of description with the corresponding modal data respectively to obtain the corresponding fusion encoding vector;

[0021] Through the feature fusion module, normalizing and uniformly dimension mapping each fusion encoding vector, performing element-by-element difference on any two types of mapped vectors, obtaining global difference features according to the mean of the difference results of each vector and other vectors, obtaining the gating weight of each vector according to the global difference features, weighted summing the mapped vectors based on the gating weight, and then stacking the residual error after normalization processing to obtain the global fusion features;

[0022] Through the multi-task classification module, classifying the hatred information and the metaphor information simultaneously according to the global fusion features;

[0023] Using the meme sample and the joint loss function to train the joint learning model to obtain the trained joint learning model; the joint loss function comprises a hatred classification loss and a metaphor classification loss;

[0024] Inputting the meme to be classified into the trained joint learning model to output the hatred information classification result and the metaphor information classification result.

[0025] The above-mentioned multi-modal semantic understanding method, device and equipment based on bidirectional dynamic interaction. By acquiring a meme sample containing image data and text data, a joint learning model integrating feature extraction, fusion and multi-task classification modules is constructed, a multi-modal large model is guided to generate a targeted semantic description by means of a thinking chain prompt, fusion coding is realized by means of a pre-trained multi-modal encoder, and finally, global fusion features considering cross-modal information and unique values are obtained through normalization, unified dimension mapping, differential feature calculation, gated weighted fusion and residual superposition. Finally, the classification of hatred and metaphor is completed based on the features, and the model is trained by a joint loss function containing two types of losses. The hatred and metaphor representation dynamic interaction and mutual calibration of the classification boundary can effectively capture the complex rhetoric of "surface metaphor, internal hatred", which not only solves the problem that metaphor features cannot disambiguate hatred judgment and hatred labels are difficult to constrain metaphor detection generalization, but also improves the classification accuracy and robustness of the two types of tasks, and enhances the applicability and explainability of meme semantic understanding. The embodiment of the present application can be applied to harmful information filtering of social media platforms, network content safety supervision and other practical scenarios, accurately identifies hatred memes wrapped in metaphors, provides efficient technical support for platform compliance governance, and reduces the cost of manual review. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 FIG. 1 is a flowchart of a multi-modal semantic understanding method based on bidirectional dynamic interaction in one embodiment;

[0027] Figure 2 FIG. 4 is an internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0029] In one embodiment, as shown in FIG. 1, a multi-modal semantic understanding method based on bidirectional dynamic interaction is provided, comprising the following steps: Figure 1

[0030] Step 102, acquiring a meme sample, the meme sample containing image data and text data.

[0031] ​Meme refers to a popular discourse symbol spread among Internet communities in the form of collage, and its meaning generation relies on the dynamic coupling of text, visual symbols and contextual information; image data is the visual element (such as graphics, pictures, etc.) contained in the meme, and text data is the text content (such as slogans, annotations, etc.) matched in the meme. The meme sample comes from a pre-constructed meme sample set, and each meme sample in the sample set carries label information, including hate information label and metaphor information label.

[0032] In step 104, a joint learning model is constructed, which includes a feature extraction module, a feature fusion module and a multi-task classification module.

[0033] The joint learning model refers to a model framework capable of simultaneously carrying out multiple related task training and reasoning, and improving overall performance through information interaction between tasks; the feature extraction module is used to extract valuable feature information from original data, the feature fusion module is used to integrate features of different sources and different modalities into unified features, and the multi-task classification module is used to simultaneously complete the two tasks of hate information classification and metaphor information classification.

[0034] In step 106, the feature extraction module is used to guide the multi-modal large model to generate visual description of image data and hate description and metaphor description of text data based on the thought chain prompt, and a pre-trained multi-modal encoder is used to fuse and encode each type of description and corresponding modal data to obtain corresponding fusion encoding vectors.

[0035] The thought chain prompt refers to a prompt strategy that promotes the large model to gradually dig deep information through multi-step guided queries, the multi-modal large model (MLLM) refers to an artificial intelligence model capable of processing image, text and other multi-modal data and generating corresponding output, and the pre-trained multi-modal encoder (such as CLIP model) refers to a tool trained on large-scale data, capable of mapping different modal data and their related descriptions into high-dimensional numerical vectors.

[0036] It can be understood that the thought chain prompt is used to guide the large model to generate accurate semantic description, which supplements deep contextual information for modal data, solves the problem of meme semantic dependence on cross-modal coupling, uses the multi-modal encoder to realize fusion encoding of data and description, ensures that the features contain both modal original information and semantic associated information, provides high-quality feature input for subsequent feature fusion and double-task classification, and improves the understanding ability of the model for cross-modal semantics.

[0037] At step 108, the feature fusion module is used to normalize each fusion encoding vector, map the dimensions uniformly, perform element-by-element difference between any two types of mapped vectors, obtain global difference features according to the mean of the difference results of each vector and other vectors, obtain the gating weight of each vector according to the global difference features, weight sum the mapped vectors based on the gating weight, and then superimpose the result residual after normalization processing to obtain the global fusion features.

[0038] Normalization refers to a processing method that adjusts the mean and variance of features to make different modal features in the same numerical scale. Uniform dimension mapping refers to an operation of converting features of different dimensions to the same dimension through linear transformation. Element-by-element difference refers to difference calculation of corresponding elements of two types of vectors, which is used to characterize the difference between features. Global difference features refer to the mean of the difference results of a certain feature and all other features, which is used to quantify the total amount of unique information contained in the feature. Gating weight refers to a weight coefficient calculated based on the global difference features, which is used to represent the contribution of each feature. Residual superposition refers to the operation of superimposing the results of the original features after processing with the fusion features to retain the original information. Global fusion features refer to the final features that integrate all modal features and semantic information after a series of processing, which are used for subsequent classification tasks.

[0039] It can be understood that normalization and uniform dimension mapping solve the scale imbalance problem caused by traditional multi-modal feature concatenation, ensure effective fusion of different modal features, remove overlapping redundant information between features through difference and gating weight calculation, retain unique valuable information of each feature, guide the model to focus on the core classification basis, residual superposition avoids excessive loss of original information during fusion, stabilizes the feature distribution, and finally obtains global fusion features that take into account cross-modal information, semantic association information and unique valuable information, providing precise and comprehensive feature support for dual-task classification, and improving the accuracy and robustness of the classification results.

[0040] At step 110, the multi-task classification module is used to simultaneously classify the hate information and metaphor information according to the global fusion features.

[0041] The hate information classification refers to a task of judging whether the meme contains offensive, discriminatory or other hate-related content. The metaphor information classification refers to a task of judging whether the meme contains unconventional concept mapping (i.e. metaphor rhetoric). The dual-task synchronous classification based on the global fusion feature enables the hate classification and the metaphor classification process to share cross-modal semantic information, solves the information fragmentation problem caused by the isolated processing of the two types of tasks in the prior art, the hate classification can determine whether it is "hate wrapped in metaphor" with the aid of the metaphor feature, the metaphor classification can constrain over-generalization with the aid of the hate feature, the two types of tasks provide disambiguation clues to each other, and the classification accuracy of each type of task is improved, and the recognition ability of the "surface metaphor and internal hate" compound rhetoric meme is especially enhanced.

[0042] In step 112, the joint learning model is trained by using the meme sample and the joint loss function, and a trained joint learning model is obtained.

[0043] The joint loss function includes a hate classification loss and a metaphor classification loss. The hate classification loss refers to an index for measuring the difference between the hate classification result and the true label. The metaphor classification loss refers to an index for measuring the difference between the metaphor classification result and the true label.

[0044] It can be understood that the training of the two types of tasks is bound by the joint loss function, so that the parameters of the hate classification and the metaphor classification are simultaneously optimized in the training process, and the dynamic interaction of the hate and metaphor representation is realized; the loss of one type of task drives the feature optimization of the other type of task, avoids the bias caused by single-task training, solves the problem of insufficient generalization ability caused by isolated task training in the prior art, and improves the robustness and adaptability of the model in complex meme scenarios.

[0045] In step 114, the meme to be classified is input into the trained joint learning model, and the hate information classification result and the metaphor information classification result are output.

[0046] The meme to be classified refers to unknown meme data that needs to be judged whether it contains hate information and metaphor information; the classification result refers to the judgment result of the model that the meme to be classified belongs to hate / non-hate, metaphor / non-metaphor.

[0047] It can be understood that the method of the present application enables the hate and metaphor representation to dynamically interact in the training process through multi-perspective knowledge enhancement and high-dimensional similarity de-redundancy, and mutually calibrates the task classification boundary as a soft prompt, so as to improve the accuracy and robustness of the two tasks at the same time.

[0048] In the above multi-modal semantic understanding method based on bidirectional dynamic interaction, by acquiring a meme sample containing image data and text data, a joint learning model integrating feature extraction, fusion and multi-task classification modules is constructed, a large multi-modal model is guided to generate a targeted semantic description by means of a thought chain prompt, fusion coding is realized by means of a pre-trained multi-modal encoder, and then normalized, unified dimension mapping, differential feature calculation, gate weighted fusion and residual superposition are performed to obtain global fusion features considering cross-modal information and unique value, and finally, hatred and metaphor classification are simultaneously completed based on the features, and the model is trained by a joint loss function containing two types of losses. The hatred and metaphor representation dynamic interaction, mutual calibration of classification boundaries, effective capture of the complex rhetoric of "surface metaphor, internal hatred", not only solve the problems that metaphor features cannot disambiguate hatred judgment and hatred labels are difficult to constrain metaphor detection generalization, but also improve the classification accuracy and robustness of the two types of tasks, and enhance the applicability and explainability of meme semantic understanding. The embodiment of the present application can be applied to harmful information filtering of social media platforms, network content safety supervision and other practical scenarios, accurately identifies hatred memes wrapped in metaphors, provides efficient technical support for platform compliance governance, and reduces the cost of manual review.

[0049] In one embodiment, the thought chain prompt includes: a first query for requesting the multi-modal large model to describe the visual content of the image data; a second query for requesting the multi-modal large model to analyze hatred information in combination with the image data, the text data and the visual description output after being guided by the first query; and a third query for requesting the multi-modal large model to analyze metaphor information in combination with the image data, the text data and the visual description output after being guided by the first query.

[0050] In the present embodiment, in order to obtain high-quality and information-rich task prompt information, the present application uses a CoT (Chain-of-Thought, thought chain) prompt to guide the multi-modal large model to extract cross-modal deep information. The present application constructs the following thought framework:

[0051] ;

[0052] ;

[0053] ;

[0054] wherein represents a meme, and each meme is composed of an image and a text . Each prompt is designed as follows:

[0055] Please briefly describe the content in the image.

[0056] : Please analyze the hate information of the meme in conjunction with the image, the text and the image description information

[0057] : Please analyze the metaphorical information of this meme inconjunction with the image, the text and the image description information

[0058] In one embodiment, the pre-trained multi-modal encoder is used to fuse and encode each type of description with the corresponding modal data to obtain the corresponding fusion encoding vector, including: the pre-trained multi-modal encoder is a CLIP model; the image data and the visual description are input into the image encoder of the CLIP model, and the first fusion encoding vector is output; the text data and the hate description are input into the text encoder of the CLIP model, and the second fusion encoding vector is output; the text data and the metaphorical description are input into the text encoder of the CLIP model, and the third fusion encoding vector is output.

[0059] In this embodiment, the pre-trained CLIP model is used as a multi-modal information encoder to obtain the context information of the input meme:

[0060] ;

[0061] ;

[0062] ;

[0063] ;

[0064] wherein, represents the number of samples, represents the vector dimension, represents the visual description, representing a hate speech description, representing a metaphorical description.

[0065] In one embodiment, the normalization and uniform dimension mapping of each fusion encoding vector comprises: normalizing the first fusion encoding vector, the second fusion encoding vector and the third fusion encoding vector respectively through a LayerNorm layer, wherein the second fusion encoding vector and the third fusion encoding vector adopt the same learnable scaling parameter and translation parameter, and the first fusion encoding vector adopts an independent learnable scaling parameter and translation parameter; mapping the normalized first fusion encoding vector, the second fusion encoding vector and the third fusion encoding vector to the same preset dimension through three independent linear transformation layers respectively.

[0066] In the embodiment, in order to solve the problems of modal scale imbalance caused by simple splicing of traditional multi-modal features, and a large amount of overlapping redundant information in features of different views and different modalities, the application introduces differential attention mechanism, gating fusion, residual connection and other strategies, and realizes the improvement of feature fusion efficiency through a light linear layer and an activation function.

[0067] In order to solve the numerical scale difference of different modal features, the application defines a LayerNorm layer for text features and visual features respectively:

[0068] ;

[0069] ;

[0070] wherein, , are the mean and variance of the features respectively, are learnable scaling and translation parameters, is a numerical stability term; the normalized feature mean is 0 and the variance is 1, which ensures that the text and visual features participate in fusion at the same scale.

[0071] The different features are mapped to a uniform dimension, and the mathematical expression is:

[0072] ;

[0073] ;

[0074] ;

[0075] wherein, W is a weight matrix, and b is a bias term.

[0076] In one embodiment, the element-wise difference between any two mapped vectors includes: calculating the element-wise absolute value difference between any two mapped vectors, normalizing the absolute value difference by combining a Sigmoid activation function to obtain a difference result; the difference result is:

[0077] ;

[0078] wherein, is the difference result of the i-th class and the j-th class mapped vector, i is the i-th class mapped vector, j is the j-th class mapped vector, is the difference result of the i-th class and the j-th class mapped vector, i is the i-th class mapped vector, is the j-th class mapped vector, j is the difference result of the i-th class and the j-th class mapped vector, is the Sigmoid activation function.

[0079] In this embodiment, the gating weight of each feature is calculated using the difference idea. This method quantifies the difference information between different view features in the same modality, removes the redundant part of the overlapping features, and retains the differentiated information unique to each view and with task value, thereby guiding the model to highly concentrate attention on new information that contributes to the core task of classification, regression, etc. First, the element-wise difference between features is calculated, and the difference result is normalized by combining the Sigmoid activation to characterize the redundancy between features.

[0080] In one embodiment, the gating weight of each vector obtained according to the global difference feature includes: linearly transforming the global difference feature, and then normalizing by the Softmax function to obtain the gating weight corresponding to each mapped vector; the gating weight is used to represent the unique information contribution degree of the corresponding vector.

[0081] In this embodiment, the global difference feature of the i-th feature is obtained by averaging the difference results of the i-th feature and all other view features, and the total amount of new information contained is quantified:

[0082] ;

[0083] The global difference feature is converted into the gating weight of each feature by linear transformation and Softmax normalization, and the formula is as follows:

[0084] .

[0085] In one embodiment, the global fusion feature is obtained by weighting and summing the mapped vectors based on the gating weights, and then superimposing the result residual after normalization processing. The global fusion feature includes: weighting and summing all mapped vectors based on the gating weights corresponding to each mapped vector to obtain a gating fusion feature; splicing the normalized results of all fusion encoding vectors to obtain an original feature splicing result; sequentially processing the original feature splicing result through a linear layer and a GELU activation function to obtain a residual supplementary feature; superimposing the residual supplementary feature and the gating fusion feature, and then stabilizing the feature distribution through a LayerNorm layer to obtain the global fusion feature.

[0086] In this embodiment, the fusion feature is obtained by weighting and summing the mapped features based on the gating weights, and the mathematical expression is:

[0087] ;

[0088] To avoid excessive loss of original feature information by gating fusion, the present application introduces a residual connection. After splicing the original normalized features, the features are processed through a linear layer and a GELU activation function, and then superimposed with the gating fusion feature, and then the distribution is stabilized through a LayerNorm, and the mathematical expression is:

[0089] .

[0090] In one embodiment, the hate classification loss and the metaphor classification loss are both cross-entropy losses; and the joint loss function is the superposition of the hate classification loss and the metaphor classification loss.

[0091] In this embodiment, the present application defines two classifiers for hate and metaphor classification, and the classification process can be represented as:

[0092] ;

[0093] ;

[0094] The loss function is calculated as follows:

[0095] ;

[0096] ;

[0097] wherein represents a meme data set, represents a cross-entropy loss function, respectively represent the true value and the predicted value.

[0098] It should be understood that, although Figure 1The steps in the flowchart are shown in sequence according to the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order limitation for the execution of the steps, and the steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or at least part of the sub-steps or stages of other steps.

[0099] In one embodiment, a multi-modal semantic understanding device based on bidirectional dynamic interaction is provided, comprising:

[0100] A sample acquisition module is configured to acquire a meme sample, the meme sample including image data and text data.

[0101] A model construction module is configured to construct a joint learning model, the joint learning model including a feature extraction module, a feature fusion module, and a multi-task classification module. The feature extraction module is configured to guide a multi-modal large model to generate a visual description of the image data and a hatred description and a metaphor description of the text data based on a thinking chain prompt, to fuse and encode each type of description with corresponding modal data using a pre-trained multi-modal encoder to obtain corresponding fusion encoding vectors. The feature fusion module is configured to normalize and uniformly map each fusion encoding vector, to perform element-by-element difference on any two types of mapped vectors, to obtain global difference features according to the mean of the difference results of each vector and other vectors, to obtain gating weights of each vector according to the global difference features, to perform weighted summation on the mapped vectors based on the gating weights, and to add the result residual after normalization processing to obtain global fusion features. The multi-task classification module is configured to simultaneously classify hatred information and metaphor information according to the global fusion features.

[0102] A model training module is configured to train the joint learning model using the meme sample and a joint loss function to obtain a trained joint learning model. The joint loss function includes a hatred classification loss and a metaphor classification loss.

[0103] A result output module is configured to input a meme to be classified into the trained joint learning model to output a hatred information classification result and a metaphor information classification result.

[0104] Specific limitations of the device for multi-modal semantic understanding based on bidirectional dynamic interaction can be seen from the limitations of the method for multi-modal semantic understanding based on bidirectional dynamic interaction described above, which will not be repeated here. Each module in the device for multi-modal semantic understanding based on bidirectional dynamic interaction described above can be realized by software, hardware, and combinations thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0105] In one embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 2 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with external terminals through network connections. The computer program is executed by the processor to implement a method for multi-modal semantic understanding based on bidirectional dynamic interaction. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball, or touchpad arranged on the shell of the computer device, or can be an external keyboard, touchpad, or mouse, etc.

[0106] Those skilled in the art can understand that Figure 2 The structure shown in the above embodiment is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in the above-mentioned embodiments.

[0108] Each technical feature of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0109] The above embodiments only express several implementation ways of the present application, and the description is more specific and detailed, but it should not be understood as a limitation to the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for multi-modal semantic understanding based on bidirectional dynamic interaction, characterized in that, The method comprises: acquiring a meme sample containing image data and text data; constructing a joint learning model comprising a feature extraction module, a feature fusion module, and a multi-task classification module; using the feature extraction module, guiding a multi-modal large model to generate a visual description of the image data and a hatred description and a metaphor description of the text data based on a thinking chain prompt, and using a pre-trained multi-modal encoder to fuse and encode each type of description with the corresponding modal data to obtain a corresponding fusion encoding vector; using the feature fusion module, normalizing and uniformly dimension mapping each fusion encoding vector, performing element-by-element difference on any two types of mapped vectors, obtaining a global difference feature according to the mean of the difference results of each vector and other vectors, obtaining a gating weight of each vector according to the global difference feature, and performing weighted summation on the mapped vectors based on the gating weight, and then adding the residual error after normalization processing to obtain a global fusion feature; using the multi-task classification module, synchronously classifying hatred information and metaphor information according to the global fusion feature; training the joint learning model using the meme sample and a joint loss function to obtain a trained joint learning model; the joint loss function comprises a hatred classification loss and a metaphor classification loss; inputting a meme to be classified into the trained joint learning model to output a hatred information classification result and a metaphor information classification result.

2. The method of claim 1, wherein, The thinking chain prompt comprises: a first query for requesting the multi-modal large model to describe the visual content of the image data; a second query for requesting the multi-modal large model to analyze the hatred information in combination with the image data, the text data, and the visual description output after the first query; a third query for requesting the multi-modal large model to analyze the metaphor information in combination with the image data, the text data, and the visual description output after the first query.

3. The method of claim 1, wherein, The use of a pre-trained multi-modal encoder to fuse and encode each type of description with the corresponding modal data to obtain a corresponding fusion encoding vector comprises: the pre-trained multi-modal encoder is a CLIP model; inputting the image data and the visual description into the image encoder of the CLIP model to output a first fusion encoding vector; inputting the text data and the hatred description into the text encoder of the CLIP model to output a second fusion encoding vector; inputting the text data and the metaphor description into the text encoder of the CLIP model to output a third fusion encoding vector.

4. The method of claim 1, wherein, The normalization and uniform dimension mapping of each fusion encoding vector comprises: performing normalization on the first fusion encoding vector, the second fusion encoding vector, and the third fusion encoding vector through a LayerNorm layer, wherein the second fusion encoding vector and the third fusion encoding vector use the same learnable scaling parameter and translation parameter, and the first fusion encoding vector uses an independent learnable scaling parameter and translation parameter; mapping the normalized first fusion encoding vector, the second fusion encoding vector, and the third fusion encoding vector to the same preset dimension through three independent linear transformation layers.

5. The method of claim 1, wherein, The element-by-element difference on any two types of mapped vectors comprises: The element-wise absolute value difference of any two mapped vectors is calculated, and the absolute value difference is normalized by a Sigmoid activation function to obtain a difference result; the difference result is: wherein, is the first i class mapping post-vector, j difference result of the first class mapping post-vector, i is the first class mapping post-vector, j is the first class mapping post-vector, is a Sigmoid activation function.

6. The method of claim 1, wherein, The global difference feature is linearly transformed, and then normalized by a Softmax function to obtain the gating weight corresponding to each mapped vector; the gating weight is used to represent the unique information contribution degree of the corresponding vector. The mapped vectors are weighted and summed based on the gating weight, and then stacked with the normalized result residual to obtain the global fusion feature, which includes:

7. The method of claim 1, wherein, The mapped vectors are weighted and summed based on the gating weight corresponding to each mapped vector to obtain the gated fusion feature. The normalized results of all fusion encoding vectors are spliced to obtain an original feature splicing result. The original feature splicing result is sequentially processed by a linear layer and a GELU activation function to obtain a residual supplementary feature. The residual supplementary feature is stacked with the gated fusion feature, and then the feature distribution is stabilized by a LayerNorm layer to obtain the global fusion feature. The hatred classification loss and the metaphor classification loss are both cross-entropy losses; the joint loss function is the superposition of the hatred classification loss and the metaphor classification loss.

8. The method of claim 1, wherein, The device includes:

9. A device for multi-modal semantic understanding based on bidirectional dynamic interaction, characterized in that, A sample acquisition module for acquiring a meme sample, the meme sample containing image data and text data; A model construction module for constructing a joint learning model, the joint learning model including a feature extraction module, a feature fusion module, and a multi-task classification module; the feature extraction module is used to guide a multi-modal large model to generate visual descriptions of the image data and hatred descriptions and metaphor descriptions of the text data based on a thinking chain prompt, and to fuse encode each type of description with the corresponding modal data using a pre-trained multi-modal encoder to obtain corresponding fusion encoding vectors; the feature fusion module is used to normalize and uniformly dimension map each fusion encoding vector, perform element-wise differencing on any two mapped vectors, obtain a global difference feature based on the mean of the difference result of each vector and other vectors, obtain the gating weight of each vector based on the global difference feature, and weighted sum the mapped vectors based on the gating weight, and then stack the normalized result residual to obtain the global fusion feature; the multi-task classification module is used to simultaneously classify hatred information and metaphor information based on the global fusion feature; A model training module for training the joint learning model using the meme sample and the joint loss function to obtain a trained joint learning model; the joint loss function includes a hatred classification loss and a metaphor classification loss; A result output module for inputting a meme to be classified into the trained joint learning model and outputting a hatred information classification result and a metaphor information classification result. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 8. 10.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-9. ​

Images