A short video event detection method and device based on multi-modal representation learning
By employing a multimodal representation learning method and utilizing a bidirectional long short-term memory network and a multimodal attention fusion network, the problem of insufficient utilization of multimodal information in short video event detection is solved, achieving efficient detection of short video content.
Patent Information
- Application Number
- CN202310505779.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-05
- Publication Date
- 2026-05-15
- Estimated Expiration
- 2043-05-05
AI Technical Summary
How to quickly and efficiently utilize existing short video information to find the short video content that users need? Existing technologies struggle to effectively utilize multimodal information for event detection.
We employ a multimodal representation learning approach, which uses a bidirectional long short-term memory network to acquire latent sequence characteristics, constructs a latent sequence characteristic acquisition module, utilizes a recurrent interactive information embedding module to explore the relationships between multimodal data, combines a multimodal attention fusion network to mine local and global attention characteristics, and finally uses a classifier to calculate event category scores.
This approach fully utilizes multimodal information from short videos, uncovers potential correlations between preceding and following sequences, explores relationships between multimodal data, obtains attention-enhanced fusion feature representations, and improves the accuracy of short video event detection.
Smart Images
Figure CN116524407B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimedia and big data analysis technology, and in particular relates to a method and apparatus for short video event detection based on multimodal representation learning. Background Technology
[0002] With the rapid development of the short video industry, short video content analysis, represented by short video event detection, has received increasing attention. Short video event detection helps solve the challenges of short video regulation and ensures the industry's continued healthy development. However, with the ever-growing number of short videos and the increasing complexity and diversity of the information they contain, how to quickly and efficiently find the short videos that users need using existing short video information has become an urgent problem to be solved.
[0003] Currently, artificial intelligence (AI) technologies, represented by deep learning, are rapidly developing across various fields, including video information processing. Utilizing AI to solve short video event detection problems can not only advance the field of computer vision but also improve user experience, making it valuable for both research and practical applications. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention proposes a short video event detection method and apparatus based on multimodal representation learning, aiming to solve the problem of how to quickly and efficiently find the short videos needed by users using existing short video information.
[0005] To achieve the above objectives, this invention provides a short video event detection method based on multimodal representation learning, comprising the following steps:
[0006] A latent sequence feature acquisition module is constructed by using a bidirectional long short-term memory network. This module acquires the latent features of multimodal information in the preceding and following sequences of a short video, thereby obtaining visual and auditory representations containing latent audio-related information.
[0007] Based on the visual and auditory representations, a cyclic interaction information embedding module is constructed. A cyclic matrix is constructed for different modalities through the cyclic interaction information embedding module. Based on the cyclic matrix, a feature representation of the interaction information of potential elements between embedded modalities is obtained.
[0008] By mining the local and global attention of multimodal information through a multimodal attention fusion network, a fusion modal feature representation weighted by local and global attention characteristics is obtained;
[0009] The fused modal feature representation is input into a classifier to obtain an event category score, and the short video event detection result is obtained based on the score.
[0010] Preferably, the process of constructing the latent sequence feature acquisition module includes: inputting the visual modal features and auditory modal features of a short video, and encoding the visual modal features and auditory modal features through a bidirectional long short-term memory network, with the encoding formula being:
[0011] H v =Bi-LSTM(x v ;θ v )
[0012] H s =Bi-LSTM(x) s ;θ s )
[0013] in, Visual features; Auditory characteristics; D v and D s These represent the dimensions of visual and auditory features, respectively; Bi-LSTM is a bidirectional long short-term memory network. For visual features x v The visual encoding features obtained after training with Bi-LSTM, l v It is the length of the visual encoding feature sequence, d v It is the size of the visual encoding feature dimension; For auditory features x s The auditory coding features obtained after training with Bi-LSTM, l s It is the length of the auditory coding feature sequence, d s It is the size of the auditory coding feature dimension; θ v and θ s These are the network parameters to be learned.
[0014] Preferably, the method for constructing cyclic matrices for different modalities includes: mapping visual features and auditory features into a low-dimensional space; and constructing visual cyclic matrices and auditory cyclic matrices based on the visual projection vectors and auditory projection vectors mapped into the low-dimensional space.
[0015] Preferably, the method for obtaining the feature representation of the intermodal latent element interaction information based on the cyclic matrix includes: using matrix multiplication on the projection vector and the cyclic matrix to obtain the feature representation of the intermodal latent element interaction information.
[0016] Preferably, the calculation formula for obtaining the local attention characteristics is:
[0017] Q t =BN(Conv2(δ(BN(Conv1(F)) t ))))), t∈{v,s};
[0018] Where Conv1 means using 1*1 pointwise convolution to reduce the dimension of the input feature sum to 1 / r of the original, where r is the dimension scaling ratio; BN means BatchNorm layer; δ means ReLU activation function; Conv2 means restoring the feature dimension to the original input feature dimension through 1*1 pointwise convolution. This represents the captured modal features exhibiting local attention characteristics; d o This represents the dimension size after mapping to a lower-dimensional space.
[0019] Preferably, the formula for calculating the global attention characteristics is:
[0020] G=GAP[BN[Conv2(δ(BN(Conv1(GAP(F h )))))]];
[0021] Wherein, GAP represents global pooling operation; This is the overall modal feature representation obtained by cascading feature representations that embed latent element interaction information between modalities; This represents the captured global modal features with global attention properties.
[0022] Preferably, the calculation formula for obtaining the fused modal feature representation is as follows:
[0023]
[0024] in, and Visual and auditory features for embedding latent intermodal element interaction information; This represents the captured global modal features with global attention characteristics; This represents the modal features captured that exhibit local attention characteristics; σ represents element-wise multiplication; σ represents the sigmoid activation function.
[0025] Preferably, the formula for calculating the event category score is:
[0026]
[0027] in, Indicates the score for the event category. represents the parameters to be learned in the fully connected layer, and C represents the number of event categories.
[0028] The present invention also provides a short video event detection device based on multimodal representation learning. The device includes a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to cause the device to perform the method steps of any one of claims 1-8.
[0029] Compared with the prior art, the present invention has the following advantages and technical effects:
[0030] 1. This invention fully utilizes the multimodal information of short videos and uncovers the potential correlation between preceding and following sequences;
[0031] 2. This invention explores the relationships and coupling methods between multimodal data by constructing cyclic matrices for different modalities, mines the potential correlations between multimodal feature elements, and obtains feature representations that embed the interaction information of potential elements between modalities;
[0032] 3. This invention utilizes a multimodal attention fusion network to mine the local and global attention characteristics of multimodal information, and uses this to guide the fusion of multimodal information, obtaining attention-enhanced fusion feature representations. A classifier is then used to calculate event category scores. This provides a new approach to solving the problem of short video event detection. Attached Figure Description
[0033] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0034] Figure 1 This is a flowchart of the short video event detection method according to an embodiment of the present invention. Detailed Implementation
[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0036] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0037] Example 1
[0038] like Figure 1 As shown, this invention proposes a short video event detection method based on multimodal representation learning, comprising the following steps:
[0039] 101: Construct a latent sequence feature acquisition module and use a bidirectional long short-term memory network to acquire the latent features of visual and auditory modal information of short videos in the preceding and following sequences of short videos;
[0040] 102: Use the modal representation containing potential related information between previous and next frames obtained in step 101 to guide the learning of the cyclic interaction information embedding module, construct cyclic matrices for different modalities, fully explore the relationship between multimodal data and its coupling mode, mine the potential correlation between multimodal feature elements, and obtain feature representations of the interaction information of potential elements between embedded modalities.
[0041] 103: By mining the local and global attention of multimodal information through a multimodal attention fusion network, a fusion feature representation with enhanced local and global attention characteristics is obtained;
[0042] 104: The multimodal fusion features of short videos with potential association characteristics of multimodal information and attention enhancement obtained through training are input into the classifier to obtain event category scores, thus completing the short video event detection task.
[0043] The following section provides a further explanation of the scheme, using calculation formulas and examples. See the description below for details:
[0044] 201: Input the visual and auditory features of short videos, mine the potential information correlation between their preceding and following sequences, and obtain visual representations containing potential information correlation between preceding and following frames and auditory representations containing potential information correlation between preceding and following audio.
[0045] The visual characteristics of short videos are Auditory characteristics are l v and l s D represents the length of the frame sequence after downsampling the short video and the length of the graph sequence after converting the short video audio file into a spectrogram, respectively. v and D s These represent the dimensions of visual and auditory features, respectively.
[0046] To obtain latent feature representations containing correlations between preceding and following sequence information, a bidirectional long short-term memory network is used to construct a latent sequence feature acquisition module, enabling the encoding of latent information for two modalities:
[0047] H v =Bi-LSTM(x v ;θ v )
[0048] H s =Bi-LSTM(x s ;θ s )
[0049] Among them, Bi-LSTM is a bidirectional long short-term memory network; For visual features x v The visual encoding features obtained after training with Bi-LSTM, lv It is the sequence length, d v It is its dimensional size; For auditory features x s The auditory coding features obtained after training with Bi-LSTM, l s It is the sequence length, d s It is its dimensional size; θ v and θ S These are the network parameters to be learned. This module ultimately yields a visual representation H containing latent relevant information from previous and subsequent frames. v and auditory representation H containing potential information related to preceding and following audio. s .
[0050] 202: Use the modal representation containing potential related information between previous and next frames obtained in step 201 to guide the learning of the cyclic interaction information embedding module, construct cyclic matrices for different modalities, fully explore the relationship between multimodal data and its coupling mode, mine the potential correlation between multimodal feature elements, and obtain feature representations of the interaction information of potential elements between embedded modalities.
[0051] First, visual and auditory features are mapped to a low-dimensional space:
[0052] I v =h v W v T
[0053] I s =h s W s T
[0054] in, For H v Features of the transposed column vector; d is the visual projection vector mapped to a low-dimensional space. o The size of the mapped dimension; To make h v The mapping matrix used in the process of mapping to a lower-dimensional space; To make H s Features of the transposed column vector; d is the auditory projection vector mapped to a low-dimensional space. o The size of the mapped dimension; To make h s The mapping matrix used in the process of mapping to a lower-dimensional space. Then, using I... v and I s Construct visual cyclic matrix respectively and auditory loop matrix
[0055] L v =circ(I v )
[0056] L s =circ(I s )
[0057] Here, circ(·) represents the operation of cyclically shifting the positions of the elements inside the projection vector to the right and merging them into a matrix. Finally, to ensure that the elements in the projection vector and the cyclic matrix fully interact, matrix multiplication is used between the cyclic matrix and the projection vector to obtain a feature representation that embeds intermodal element interaction information. and The specific formula is as follows:
[0058] F v =I v L s
[0059] F s =I s L v
[0060] 203: In order to better integrate modal features with different semantic properties, a multimodal attention fusion network is used to mine the local attention and global attention of multimodal information, and finally obtain a fusion feature representation with enhanced local attention and global attention characteristics;
[0061] The feature representation with intermodal element interaction information obtained in step 202 and To obtain the overall feature representation of different modalities:
[0062]
[0063] in, d represents the overall feature representation after combining different modal representations. h =d o +d o It represents its dimension size; Concat(·) indicates a cascading operation.
[0064] The formula for calculating the local attention characteristics is as follows:
[0065] Q t =BN(Conv2(δ(BN(Conv1(F)) t ))))),t∈{v,s}
[0066] Here, Conv1 indicates the use of 1*1 pointwise convolution, which takes the input features... and The number of dimensions is reduced to 1 / r of the original, where r is the dimension scaling ratio; BN represents the BatchNorm layer; δ represents the ReLU activation function; Conv2 means that the feature dimension is restored to the original input feature dimension through 1*1 pointwise convolution. This represents the modal features captured that exhibit local attention characteristics.
[0067] The formula for calculating the global attention characteristic is as follows:
[0068] G=GAP[BN[Conv2(δ(BN(Conv1(GAP(F h )))))]]
[0069] Here, GAP represents the Global Average Pooling operation. Compared to the formula for capturing local attention features, the formula for capturing global attention features mainly adds a global pooling layer. This represents the overall modal characteristics obtained from cascaded operations. This represents the captured global modal features with global attention properties.
[0070] Finally, this module performs a fusion operation on the modal features with local attention and global attention characteristics obtained from the above process:
[0071]
[0072] in, This indicates element-wise multiplication; σ represents the sigmoid activation function. This yields a weighted fusion modal feature representation that incorporates both local and global attention characteristics.
[0073] 204: Utilize the fusion modality short video feature representation learned above to obtain event category scores and complete the short video event detection task;
[0074] The fusion modal features finally obtained from the above process are represented as The input is fed into a fully connected layer, and the event detection prediction result is obtained using the Softmax(·) function:
[0075]
[0076] in, These are the parameters to be learned in the fully connected layer, and C represents the number of event categories.
[0077] Two loss functions are used for training: classification loss and reconstruction loss. For the classification loss, binary cross-entropy loss is used.
[0078]
[0079] Where σ(·) is the sigmoid function; The value is 1 if the i-th short video sample belongs to the j-th class, and 0 otherwise. This represents the probability that the i-th short video sample is predicted to be of class j.
[0080] For low-rank constraints, to ensure that the short video content representation obtained after model training captures the characteristics between different modal elements without excessive redundant information, and to enhance the robustness of the model, ||·|| is used. * The kernel function constructs low-rank constraints:
[0081]
[0082] Finally, during the model training phase, the following objective function is minimized:
[0083]
[0084] Here, λ1 and λ2 are parameters that balance the different contributions of loss.
[0085] The model is trained by setting appropriate parameters, and the final results are evaluated using accuracy, recall, and mean average precision (mAP) as metrics.
[0086] Example 2
[0087] The present invention also proposes a short video event detection device based on multimodal representation learning. The device includes a processor and a memory. The memory stores program instructions, and the processor calls the program instructions stored in the memory to implement the above method.
[0088] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A short video event detection method based on multimodal representation learning, characterized in that, Includes the following steps: A latent sequence feature acquisition module is constructed by using a bidirectional long short-term memory network. This module acquires the latent features of multimodal information in the preceding and following sequences of a short video, thereby obtaining visual and auditory representations containing latent audio-related information. Based on the visual and auditory representations, a cyclic interaction information embedding module is constructed. A cyclic matrix is constructed for different modalities through the cyclic interaction information embedding module. Based on the cyclic matrix, a feature representation of the interaction information of potential elements between embedded modalities is obtained. The method for constructing cyclic matrices for different modalities includes: mapping visual features and auditory features to a low-dimensional space; and constructing visual cyclic matrices and auditory cyclic matrices based on the visual projection vectors and auditory projection vectors mapped to the low-dimensional space. The method for obtaining feature representations of embedded intermodal latent element interaction information based on the cyclic matrix includes: using matrix multiplication on the projection vector and the cyclic matrix to obtain feature representations of embedded intermodal latent element interaction information; By mining the feature representations of latent element interaction information between modalities through a multimodal attention fusion network, we obtain a fusion modal feature representation weighted by the local and global attention characteristics. The fused modal feature representation is input into a classifier to obtain an event category score, and the short video event detection result is obtained based on the score.
2. The short video event detection method based on multimodal representation learning according to claim 1, characterized in that, The process of constructing the latent sequence feature acquisition module includes: inputting the visual modal features and auditory modal features of a short video, and encoding the visual modal features and auditory modal features through a bidirectional long short-term memory network. The encoding formula is as follows: in, Visual features; Auditory characteristics; and These represent the dimensions of visual and auditory features, respectively; Bi-LSTM is a bidirectional long short-term memory network. Visual features Visual encoding features obtained after training with Bi-LSTM It is the length of the visual encoding feature sequence. It is the size of the visual encoding feature dimension; auditory characteristics Auditory coding features obtained after training with Bi-LSTM It is the length of the auditory coding feature sequence. It is the size of the auditory coding feature dimension; and These are the network parameters to be learned.
3. The short video event detection method based on multimodal representation learning according to claim 1, characterized in that, The formula for calculating the local attention characteristics is as follows: ； in, This means that using 1*1 pointwise convolution reduces the dimension of the input features to 1 / r of the original dimension, where r is the dimension scaling ratio; Indicates the BatchNorm layer; Represents the ReLU activation function; This means that the feature dimensions are restored to the original input feature dimensions through 1*1 pointwise convolution; This represents the modal features captured that exhibit local attention characteristics; This represents the dimension size after mapping to a lower-dimensional space.
4. The short video event detection method based on multimodal representation learning according to claim 1, characterized in that, The formula for calculating the global attention property is: ； in, This indicates a global pooling operation; This is the overall modal feature representation obtained by cascading feature representations that embed latent element interaction information between modalities; This represents the captured global modal feature representation with global attention properties. This means that using 1*1 pointwise convolution reduces the dimension of the input features to 1 / r of the original dimension, where r is the dimension scaling ratio; This means that the feature dimensions are restored to the original input feature dimensions through 1*1 pointwise convolution. Indicates the BatchNorm layer; Represents the ReLU activation function. The dimension size after mapping to a lower-dimensional space. The size of the dimension representing the overall feature representation.
5. The short video event detection method based on multimodal representation learning according to claim 1, characterized in that, The formula for calculating the fused modal feature representation is as follows: ； in, and Visual and auditory features for embedding latent intermodal element interaction information; This represents the captured global modal features with global attention characteristics; This represents the modal features captured that exhibit local attention characteristics; This indicates element-wise multiplication; This represents the sigmoid activation function.
6. The short video event detection method based on multimodal representation learning according to claim 1, characterized in that, The formula for calculating the event category score is as follows: ； in, Indicates the score for the event category. , This represents the parameters to be learned in the fully connected layer. Z represents the number of event categories, and Z is the fused modal feature representation. This represents the dimension size after mapping to a lower-dimensional space.
7. A short video event detection device based on multimodal representation learning, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the steps of the method according to any one of claims 1-6.