A brain-like cross-modal information interaction fusion method for video satire analysis

Abstract

The application relates to the field of electric digital data processing, in particular to a brain-like cross-modal information interaction and fusion method for video satire analysis. The method comprises the following steps: S100, acquiring image modal characteristics, text modal characteristics and voice modal characteristics of a target video; S200, performing interaction within and between the image modal characteristics, the text modal characteristics and the voice modal characteristics of the target video to obtain feature vectors corresponding to the image modal, the text modal and the voice modal of the target video; S300, respectively performing splicing, self-attention and attention pooling processing on the feature vectors corresponding to the image modal, the text modal and the voice modal of the target video to obtain target matrices and target vectors corresponding to the image modal, the text modal and the voice modal of the target video; and S400, judging whether the target video is a satire video by using the target vectors. The application realizes satire detection of a video.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electric digital data processing, in particular to a brain-like cross-modal information interaction fusion method for video sarcasm analysis. BACKGROUND

[0002] Sarcasm is a social mechanism for indirectly conveying criticism or humorously masking embarrassment, and is considered less aggressive and more polite than confrontation. Sarcasm detection is of great significance in customer service, opinion mining, online harassment detection and other tasks that require understanding of people's true intentions. A foreign literature entitled Mimicking the Brain's Cognition of Sarcasm From Multidisciplines for Twitter Sarcasm Detection published in IEEE Transactions on Neural Networks and Learning Systems in June 2021 discloses a sarcasm detection method suitable for single images, but it is only suitable for sarcasm detection of single images and is not suitable for sarcasm detection of videos. How to detect sarcasm in videos is a problem to be solved. SUMMARY

[0003] The present application aims to provide a brain-like cross-modal information interaction fusion method for video sarcasm analysis, which realizes sarcasm detection of videos.

[0004] According to the present application, a brain-like cross-modal information interaction fusion method for video sarcasm analysis is provided, comprising the following steps:

[0005] S100, obtaining image modal features, text modal features and speech modal features of a target video, wherein the image modal features include a spatial memory matrix, a spatial guide vector, a time memory matrix and a time guide vector, the text modal features include a text memory matrix and a text guide vector, and the speech modal features include a speech memory matrix and a speech guide vector.

[0006] S200, interacting the image modal features, the text modal features and the speech modal features of the target video within and between modalities to obtain feature vectors corresponding to the image modal, the text modal and the speech modal of the target video.

[0007] S300, respectively, the feature vectors corresponding to the image modalities, text modalities and speech modalities of the target video are spliced, self-attention and attention pooling processing is performed to obtain the target matrix corresponding to the image modalities, text modalities and speech modalities of the target video; the target matrix corresponding to the image modalities, text modalities and speech modalities of the target video is subjected to self-attention and attention pooling processing to obtain a target vector.

[0008] S400, using the target vector to determine whether the target video is a satirical video.

[0009] Compared with the prior art, the present application has at least the following beneficial effects:

[0010] The present application is aimed at satirical detection of videos, and the present application first obtains features of different modalities of a target video, including image modality features, text modality features and speech modality features of the target video; by interacting within and between the three modalities, feature vectors corresponding to the three modalities are obtained; based on the feature vectors corresponding to the three modalities, the present application realizes determination of whether the target video is a satirical video.

[0011] Moreover, the image modality features of the target video of the present application consider both spatial dimension features and time dimension features, the features are considered comprehensively, the comprehensiveness of feature extraction of the target video is improved, and the accuracy of determination of whether the target video is a satirical video is improved. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0013] Figure 1 The flowchart of the brain-like cross-modality information interaction fusion method for video satire analysis provided by the embodiments of the present application. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0015] According to the present application, a brain-like cross-modality information interaction fusion method for video satire analysis is provided, as shown in Figure 1As shown, comprising the following steps:

[0016] S100, obtaining image modality features, text modality features and speech modality features of the target video, the image modality features comprising a spatial memory matrix, a spatial guide vector, a time memory matrix and a time guide vector, the text modality features comprising a text memory matrix and a text guide vector, and the speech modality features comprising a speech memory matrix and a speech guide vector.

[0017] According to the present application, whether the target video is a satire video is analyzed based on three modalities of the target video, the three modalities being image modality, text modality and speech modality.

[0018] According to the present application, the method for obtaining the spatial memory matrix, the spatial guide vector, the time memory matrix and the time guide vector comprises:

[0019] S110, obtaining an image memory matrix corresponding to each frame of image of the target video.

[0020] It should be understood that the target video comprises multiple frames of images, and each frame of image has a corresponding image memory matrix. The method for obtaining the image memory matrix corresponding to each frame of image is a prior art, which can refer to a foreign document entitled Mimicking the Brain’s Cognition of Sarcasm From Multidisciplines for Twitter Sarcasm Detection (brain cognition inspired multidisciplinary Twitter sarcasm detection) published in IEEE Transactions on Neural Networks and Learning Systems in June 2021 (hereinafter referred to as the first document).

[0021] S120, splicing the image memory matrices corresponding to each frame of image of the target video to obtain an image memory tensor of the target video.

[0022] According to the present application, if the i-th frame of image of the target video is adjusted to 331x331 and divided into 11x11 regions, then the image memory matrix corresponding to the i-th frame of image of the target video is d img is the dimension of the image feature vector corresponding to a single region, then the image memory matrices corresponding to the L frames of images included in the target video are spliced to obtain an image memory tensor of the target video

[0023] S130, performing a pooling operation on the image memory tensor in the spatial dimension to obtain a spatial memory matrix, and performing an average pooling operation on the spatial memory matrix to obtain a spatial guide vector.

[0024] According to the application, the T img is subjected to a pooling operation in the spatial dimension, and a spatial memory matrix is obtained The T is subjected to an average pooling operation, and a spatial guide vector is obtained

[0025] S140, the image memory tensor is subjected to a pooling operation in the time dimension, and a time memory matrix is obtained; the time memory matrix is subjected to an average pooling operation, and a time guide vector is obtained.

[0026] According to the application, the T img is subjected to a pooling operation in the time dimension, and a time memory matrix is obtained The T is subjected to an average pooling operation, and a spatial guide vector is obtained

[0027] According to the application, the method for obtaining the text memory matrix and the text guide vector comprises:

[0028] S150, the subtitle memory matrix corresponding to each frame image of the target video is obtained according to the subtitles corresponding to each frame image of the target video.

[0029] The method for obtaining the subtitle memory matrix corresponding to each frame image is a prior art, which can be referred to the first document.

[0030] S160, the subtitle memory matrices corresponding to each frame image of the target video are spliced, and a subtitle memory tensor of the target video is obtained.

[0031] According to the application, if the subtitle memory matrix corresponding to the i-th frame image of the target video is n is the number of words included in the subtitle corresponding to the i-th frame image of the target video, d caption is the dimension of the text feature vector corresponding to each word, and the obtained subtitle memory tensor of the target video is

[0032] S170, the subtitle memory tensor is subjected to sequence modeling by using an LSTM, and a text memory matrix is obtained; the text memory matrix is subjected to an average pooling operation, and a text guide vector is obtained.

[0033] According to the application, the T caption is subjected to sequence modeling by using an LSTM, and a text memory matrix M caption is obtained The M caption is subjected to an average pooling operation, and a text guide vector v caption is obtained.

[0034] According to the application, the method for obtaining the speech memory matrix and the speech guide vector comprises the following steps:

[0035] S180, obtaining the MGCC features corresponding to the speech of each frame of the target video; and splicing the MGCC features corresponding to the speech of each frame of the target video in chronological order to obtain the two-dimensional features of the target video.

[0036] The skilled in the art knows that any method for obtaining the MGCC features corresponding to the speech in the prior art falls within the protection scope of the application. The two-dimensional features of the target video in the application are N represents the number of frames into which the speech signal corresponding to the target video is divided, d speech represents the dimension of the single-frame speech signal feature.

[0037] S190, performing feature extraction on the two-dimensional features of the target video using a CNN to obtain a speech memory matrix; and performing average pooling operation on the speech memory matrix to obtain a speech guide vector.

[0038] In the application, the CNN is used to perform feature extraction on S to obtain a speech memory matrix M speech , M speech =CNN(S); and average pooling operation is performed on M speech to obtain a speech guide vector v speech . Optionally, the CNN is composed of 6 convolutional layers and 3 pooling layers, and a 3*3 convolution kernel and a 2*2 pooling kernel are adopted, wherein the number of channels of the first convolutional layer and the second convolutional layer is 32, the number of channels of the third convolutional layer and the fourth convolutional layer is 64, and the number of channels of the fourth convolutional layer and the fifth convolutional layer is 128.

[0039] S200, performing intra-modal and inter-modal interaction on the image modal features, the text modal features and the speech modal features of the target video to obtain the feature vectors corresponding to the image modal, the text modal and the speech modal of the target video.

[0040] According to the application, the intra-modal interaction on the image modal features of the target video comprises the following steps:

[0041] S210, processing the spatial memory matrix and the temporal memory matrix using a gating mechanism respectively to obtain a first spatial memory matrix and a first temporal memory matrix processed using the gating mechanism.

[0042] The purpose of the gating mechanism is to reduce the irrelevant elements in the matrix and amplify the valuable components, and it is commonly used to reduce noise and eliminate redundancy; the process of processing using the gating mechanism is the prior art, and can be referred to the first document.

[0043] According to the application, the spatial memory matrix and the temporal memory matrix are processed using the gating mechanism performing processing to obtain a first spatial memory matrix processed using a gating mechanism and a first temporal memory matrix G() represents performing gating mechanism processing.

[0044] S220, performing processing on the first spatial memory matrix and the spatial guide vector using a guided attention mechanism to obtain a first spatial vector; performing processing on the first temporal memory matrix and the temporal guide vector using the guided attention mechanism to obtain a first temporal vector.

[0045] The guided attention mechanism is used to extract elements related to the guide vector from the matrix to form a new vector; the process of using the guided attention mechanism for processing is prior art, which can be referred to in the first document.

[0046] According to the present application, the first spatial memory matrix and the spatial guide vector are processed using a guided attention mechanism to obtain a first spatial vector GA() represents performing guided attention mechanism processing. The first temporal memory matrix and the temporal guide vector are processed using the guided attention mechanism to obtain a first temporal vector

[0047] S230, performing processing on the first spatial memory matrix and the first spatial vector using a multi-hop mechanism to obtain a second spatial memory matrix; performing processing on the first temporal memory matrix and the first temporal vector using the multi-hop mechanism to obtain a second temporal memory matrix.

[0048] The multi-hop mechanism is used to generate a new matrix by using the guided attention mechanism to scan the matrix multiple times; the process of using the multi-hop mechanism for processing is prior art, which can be referred to in the first document.

[0049] According to the present application, the first spatial memory matrix and the first spatial vector are processed using a multi-hop mechanism to obtain a second spatial memory matrix MH() represents performing processing using a multi-hop mechanism; the first temporal memory matrix and the first temporal vector are processed using the multi-hop mechanism to obtain a second temporal memory matrix

[0050] S240, respectively performing attention pooling processing on the second spatial memory matrix and the second temporal memory matrix to obtain a second spatial vector and a second temporal vector.

[0051] Attention pooling is to condense all vectors in the matrix into a vector according to their importance; the process of attention pooling is prior art, which can be found in the first document.

[0052] According to the present application, the second spatial memory matrix and the second temporal memory matrix are subjected to attention pooling processing, respectively, to obtain the second spatial vector and the second temporal vector AP() represents the attention pooling processing.

[0053] S250, the second spatial vector and the second temporal vector are fused to obtain the image feature vector of the target video.

[0054] According to the present application, the second spatial vector and the second temporal vector are subjected to splicing processing to obtain || represents splicing; the image feature vector of the target video is obtained W is a learnable weight parameter, and b is a learnable bias parameter.

[0055] According to the present application, the interaction of the text modal feature of the target video within the modal includes:

[0056] S260, the text memory matrix is processed using a gating mechanism to obtain the first text memory matrix processed using the gating mechanism.

[0057] According to the present application, the text memory matrix M caption is processed using a gating mechanism to obtain the first text memory matrix processed using the gating mechanism

[0058] S270, the first text memory matrix and the text guide vector are processed using a guided attention mechanism to obtain the first text vector.

[0059] According to the present application, the first text memory matrix and the text guide vector v caption are processed using a guided attention mechanism to obtain the first text vector

[0060] S280, the first text memory matrix and the first text vector are processed using a multi-hop mechanism to obtain the second text memory matrix.

[0061] According to the present application, the first text memory matrix and the first text vector are processed using a multi-hop mechanism to obtain the second text memory matrix

[0062] S290, performing attention pooling processing on the second text memory matrix to obtain a text feature vector.

[0063] According to the present application, the second text memory matrix is subjected to attention pooling processing to obtain a text feature vector

[0064] The processing procedure of the interaction within the modal for the voice modal feature of the target video is the same as that of the interaction within the modal for the text modal feature of the target video, which will not be repeated here.

[0065] The above-mentioned interaction within the modal is single-modal interaction. The interaction of the present application also includes double-modal interaction, triple-modal interaction and quadruple-modal interaction. According to the present application, the interaction between the image modal feature, the text modal feature and the voice modal feature of the target video includes:

[0066] S291, obtaining the sum of any two of the space guide vector, the time guide vector, the text guide vector and the voice guide vector, and performing interaction between the sum of the any two guide vectors and any one of the space memory matrix, the time memory matrix, the text memory matrix and the voice memory matrix.

[0067] According to the present application, the above-mentioned interaction procedure is double-modal interaction. It should be understood that for any one of the space memory matrix, the time memory matrix, the text memory matrix and the voice memory matrix, the number of corresponding double-modal interactions is Each double-modal interaction generates one feature vector; therefore, the double-modal interaction stage of the present application generates a total of 4*6=24 feature vectors.

[0068] S292, obtaining the sum of any three of the space guide vector, the time guide vector, the text guide vector and the voice guide vector, and performing interaction between the sum of the any three guide vectors and any one of the space memory matrix, the time memory matrix, the text memory matrix and the voice memory matrix.

[0069] According to the present application, the above-mentioned interaction procedure is triple-modal interaction. It should be understood that for any one of the space memory matrix, the time memory matrix, the text memory matrix and the voice memory matrix, the number of corresponding triple-modal interactions is Each triple-modal interaction generates one feature vector; therefore, the triple-modal interaction stage of the present application generates a total of 4*4=16 feature vectors.

[0070] S293, obtain the sum of spatial guidance vector, temporal guidance vector, textual guidance vector and speech guidance vector, and use the sum of spatial guidance vector, temporal guidance vector, textual guidance vector and speech guidance vector to interact with any one of the memory matrices of spatial memory matrix, temporal memory matrix, textual memory matrix and speech memory matrix.

[0071] According to the present invention, the above-described interaction process is a four-modal interaction. It should be understood that for any memory matrix among the spatial memory matrix, temporal memory matrix, text memory matrix, and speech memory matrix, the number of corresponding dual-modal interactions is 1, and each four-modal interaction generates 1 feature vector; therefore, the four-modal interaction stage of the present invention generates a total of 4 feature vectors.

[0072] It should be noted that the feature vectors generated during the single-modal interaction, dual-modal interaction, triple-modal interaction, and quadruple-modal interaction processes mentioned above all belong to the feature vectors corresponding to the image modality, text modality, and speech modality of the target video.

[0073] S300 performs concatenation, self-attention, and attention pooling on the feature vectors corresponding to the image modality, text modality, and speech modality of the target video, respectively, to obtain the target matrices corresponding to the image modality, text modality, and speech modality of the target video; and performs self-attention and attention pooling on the target matrices corresponding to the image modality, text modality, and speech modality of the target video to obtain the target vector.

[0074] According to the present invention, all feature vectors corresponding to a certain mode among the image mode, text mode, and speech mode of the target video are concatenated to obtain the mode matrix M′ corresponding to that mode. mode For the mode matrix M′ mode After using a self-attention mechanism, attention pooling is applied to condense it into a vector v′. mode To represent the first-order understanding of this mode, i.e., v′ mode =AP(SA(M′) mode Thus, we obtain three vectors corresponding to the three modes, and denote the new matrix M″ formed by the three vectors as the target matrix.

[0075] To achieve second-order understanding of cross-modal information and avoid misinterpreting it as other semantically similar information, such as interpreting sarcasm as lying, the same structure is used to aggregate all modal information into a final vector (i.e., the target vector) for the final implicit semantic information identification, that is, for determining whether the final target video is a satirical video. Here, the target vector v″ = AP(SA(M″)).

[0076] S400, use the target vector to determine whether the target video is a satirical video.

[0077] After the target vector is obtained, a process of judging whether the target video is a satire video by using the target vector is prior art, which can be seen from the first document, and will not be repeated here.

[0078] The present application is directed to satire detection of a video, and the present application first obtains different modal features of a target video, including image modal features, text modal features and voice modal features of the target video; the three kinds of modal features are interacted within and between modalities to obtain feature vectors corresponding to the three kinds of modalities; and based on the feature vectors corresponding to the three kinds of modalities, the present application realizes judgment of whether the target video is a satire video. Moreover, the image modal features of the target video of the present application consider both spatial dimension features and time dimension features, the features are considered more comprehensively, the comprehensiveness of feature extraction of the target video is improved, and the accuracy of judgment of whether the target video is a satire video is improved.

[0079] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration, but not for limiting the scope of the present application. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A brain-like cross-modal information interaction fusion method for video satire analysis, characterized in that, The method comprises the following steps: S100, obtaining image modality features, text modality features and speech modality features of a target video, wherein the image modality features comprise a spatial memory matrix, a spatial guide vector, a time memory matrix and a time guide vector, the text modality features comprise a text memory matrix and a text guide vector, and the speech modality features comprise a speech memory matrix and a speech guide vector; S200, performing intra-modality and inter-modality interaction on the image modality features, the text modality features and the speech modality features of the target video to obtain feature vectors corresponding to the image modality, the text modality and the speech modality of the target video; S300, performing splicing, self-attention and attention pooling processing on the feature vectors corresponding to the image modality, the text modality and the speech modality of the target video respectively to obtain target matrices corresponding to the image modality, the text modality and the speech modality of the target video; and performing self-attention and attention pooling processing on the target matrices corresponding to the image modality, the text modality and the speech modality of the target video to obtain a target vector; S400, judging whether the target video is a satire video by using the target vector; The intra-modality interaction on the image modality features of the target video comprises: S210, processing the spatial memory matrix and the time memory matrix respectively by using a gating mechanism to obtain a first spatial memory matrix and a first time memory matrix processed by using the gating mechanism; S220, processing the first spatial memory matrix and the spatial guide vector by using a guided attention mechanism to obtain a first spatial vector, and processing the first time memory matrix and the time guide vector by using the guided attention mechanism to obtain a first time vector; S230, processing the first spatial memory matrix and the first spatial vector by using a multi-hop mechanism to obtain a second spatial memory matrix, and processing the first time memory matrix and the first time vector by using the multi-hop mechanism to obtain a second time memory matrix; S240, performing attention pooling processing on the second spatial memory matrix and the second time memory matrix respectively to obtain a second spatial vector and a second time vector; S250, fusing the second spatial vector and the second time vector to obtain an image feature vector of the target video.

2. The brain-like cross-modal information interaction fusion method for video satire analysis according to claim 1, characterized in that, The method for obtaining the spatial memory matrix, the spatial guide vector, the time memory matrix and the time guide vector comprises: S110, obtaining image memory matrices corresponding to each frame of images of a target video; S120, splicing the image memory matrices corresponding to each frame of images of the target video to obtain an image memory tensor of the target video; S130, performing a pooling operation on the image memory tensor in a spatial dimension to obtain a spatial memory matrix, and performing an average pooling operation on the spatial memory matrix to obtain a spatial guide vector; S140, performing a pooling operation on the image memory tensor in a time dimension to obtain a time memory matrix, and performing an average pooling operation on the time memory matrix to obtain a time guide vector.

3. The brain-like cross-modal information interaction fusion method for video satire analysis according to claim 1, characterized in that, The method for obtaining the text memory matrix and the text guide vector comprises: S150, obtaining a subtitle memory matrix corresponding to each frame of images of a target video according to subtitles corresponding to each frame of images of the target video; S160, splicing the subtitle memory matrix corresponding to each frame image of the target video to obtain a subtitle memory tensor of the target video; S170, performing sequence modeling on the subtitle memory tensor by using an LSTM to obtain a text memory matrix; performing average pooling operation on the text memory matrix to obtain a text guide vector.

4. The brain-like cross-modal information interaction fusion method for video satire analysis according to claim 3, characterized in that, The interaction of the text modal feature of the target video in the modal includes: S260, processing the text memory matrix using a gating mechanism to obtain a first text memory matrix processed using the gating mechanism; S270, processing the first text memory matrix and the text guide vector using a guide attention mechanism to obtain a first text vector; S280, processing the first text memory matrix and the first text vector using a multi-hop mechanism to obtain a second text memory matrix; S290, performing attention pooling processing on the second text memory matrix to obtain a text feature vector.

5. The brain-like cross-modal information interaction fusion method for video satire analysis according to claim 1, characterized in that, The method for obtaining the speech memory matrix and the speech guide vector includes: S180, obtaining the MGCC features corresponding to each frame of speech of the target video; splicing the MGCC features corresponding to each frame of speech of the target video in chronological order to obtain a two-dimensional feature of the target video; S190, performing feature extraction on the two-dimensional feature of the target video using a CNN to obtain a speech memory matrix; performing average pooling operation on the speech memory matrix to obtain a speech guide vector.

6. The brain-like cross-modal information interaction fusion method for video satire analysis according to claim 1, characterized in that, The interaction of the image modal feature, the text modal feature and the speech modal feature of the target video between the modal includes: S291, obtaining a sum of any two guide vectors in the space guide vector, the time guide vector, the text guide vector and the speech guide vector, and interacting the sum of the any two guide vectors with any one of the space memory matrix, the time memory matrix, the text memory matrix and the speech memory matrix; S292, obtaining a sum of any three guide vectors in the space guide vector, the time guide vector, the text guide vector and the speech guide vector, and interacting the sum of the any three guide vectors with any one of the space memory matrix, the time memory matrix, the text memory matrix and the speech memory matrix; S293, obtaining a sum of the space guide vector, the time guide vector, the text guide vector and the speech guide vector, and interacting the sum of the space guide vector, the time guide vector, the text guide vector and the speech guide vector with any one of the space memory matrix, the time memory matrix, the text memory matrix and the speech memory matrix.

Images