A video event recognition method based on audio-visual modal fusion

Abstract

The application belongs to the technical field of video time recognition, and particularly relates to a video event recognition method based on audio-visual modal fusion; the method comprises the following steps: acquiring a video to be recognized and dividing the video to obtain visual information and audio information; performing feature extraction on the visual information and the audio information respectively to obtain video features and audio features; enhancing the visual features by using an audio-adjusted visual information attention module according to the visual features and the audio features to obtain enhanced visual features; inputting the audio features and the enhanced visual features into a time attention module for processing to obtain time weights; inputting the audio features and the enhanced visual features into a channel attention module for processing to obtain channel weights; and processing the time weights and the channel weights by using a fusion attention module to obtain a video event recognition result; and the application can better recognize events in the audio-video dual modal condition.

Description

Technical Field

[0001] This invention belongs to the field of video time recognition technology, specifically relating to a video event recognition method based on audiovisual modal fusion. Background Technology

[0002] In today's era of information explosion, data from various events are often presented in multiple modalities. For example, news events can be described in text, images, and videos. All events correspond to specific categories. How to extract the most core information from events and combine it to determine the nature or category of the event has always been a focus in the field of event recognition. Among these, video event recognition is currently a major challenge in the field of machine learning, and its applications are very extensive. Specific research has covered public opinion monitoring, behavior analysis, car driving, and tracking social hotspots. In recent years, some researchers have organized image-type data from videos and trained event classification models, achieving good results. However, many processing methods are based on single-modal information. There are still many difficulties to overcome in classifying multimodal data, such as huge storage resources, complex computational load, and high-performance experimental environment requirements.

[0003] However, for video, event classification based solely on visual perspectives has many shortcomings. For example, classification performance is significantly affected when visual information is scarce. Video data typically consists of both visual and audio modal localization data. Therefore, considering feature extraction from both visual and auditory modalities allows for information complementarity. This benefits from describing the overall video information from two different perspectives. More broadly, collecting different modal data on the same subject from two or more sources can provide a more complete picture of the subject. This is one of the reasons why multimodal data fusion technology is currently a very popular research area.

[0004] Current research on video event recognition focuses on the extraction and utilization of information from multiple modalities. However, in the learning process of audiovisual modal data, the data formats, features, and paths are all different. To combine data from two modalities for joint training to achieve the best training effect, it is necessary to consider the differences in the time spent training the model, training batches, training times, and learning rates between the two modalities. Therefore, training two types of information together will result in unsatisfactory training effects for different modalities within the same time period. Furthermore, not all extracted information is useful, and focusing on the most effective information region for the relevant task is also very challenging.

[0005] In summary, current audio-visual modal fusion technologies for video event recognition still face significant challenges, necessitating a method that can reasonably fuse audio-visual modal information for video event recognition. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a video event recognition method based on audiovisual modal fusion, which includes:

[0007] S1: Acquire the video to be identified and segment it to obtain visual and audio information;

[0008] S2: Perform feature extraction on visual and audio information respectively to obtain video features and audio features;

[0009] S3: Based on visual and audio features, the audio-adjusted visual information attention module is used to enhance the visual features, resulting in enhanced visual features;

[0010] S4: Input the audio features and enhanced visual features into the temporal attention module for processing to obtain temporal weights;

[0011] S5: Input the audio features and enhanced visual features into the channel attention module for processing to obtain the channel weights;

[0012] S6: The fusion attention module is used to process the temporal weights and channel weights to obtain the video event recognition results.

[0013] Preferably, the process of extracting features from visual and audio information separately includes: extracting features from visual information using a ResNet network and a self-attention mechanism to obtain visual information; and extracting features from audio information using a VGGish model and a self-attention mechanism to obtain audio information.

[0014] Preferably, the process by which the audio-modulated visual information attention module enhances visual features includes: projecting visual features and audio features onto the same dimension; fusing the visual features and audio features projected onto the same dimension element by element; and processing the fused features using a non-linear layer to obtain enhanced visual features.

[0015] Preferably, the process by which the time attention module processes the audio features and the enhanced visual features includes: calculating the time weights of the audio features and the enhanced visual features respectively; and performing a weighted sum of the time weights of the audio features and the enhanced visual features to obtain the final time weights.

[0016] Furthermore, the formula for calculating the time weights of audio features is as follows:

[0017]

[0018]

[0019] Among them, z t Indicates the attention weight at intermediate time. This indicates that the parameter x is learned immediately. t Let represent the audio features at time t, b represent the second-time learning parameters, ReLU() represent the activation function, and S t The time weight of the audio features at time t is represented, and T represents the total number of time segments into which the video is divided.

[0020] Preferably, the process by which the channel attention module processes the audio features and the enhanced visual features includes: calculating the channel weights of the audio features and the channel weights of the enhanced visual features respectively; and performing a weighted sum of the channel weights of the audio features and the channel weights of the enhanced visual features to obtain the final channel weights.

[0021] Furthermore, the formula for calculating the channel weights of audio features is as follows:

[0022]

[0023]

[0024] Among them, z k Indicates the attention weight of the intermediate channel. Indicates the learning parameters for the first channel, x k Let b' represent the audio features of the k-th channel, b' represent the learning parameters of the second channel, and ReLU() represent the activation function. This represents the weight of the k-th channel of the audio feature, where K represents the number of channels.

[0025] Preferably, the process of using the fusion attention module to process the time weight and channel weight includes: calculating the comprehensive weight based on the time weight and channel weight; classifying the video events based on the comprehensive weight to obtain the video event recognition result.

[0026] Furthermore, the formula for calculating the overall weight is as follows:

[0027]

[0028]

[0029] in, Indicates the intermediate fusion attention weights. Indicated in the first fusion learning parameters, This represents the data input to the k-th channel at time t of the fusion attention module, where b″ represents the second fusion learning parameter. This indicates the overall weight.

[0030] The beneficial effects of this invention are as follows: This invention proposes a fusion network architecture for both visual and auditory modal information. This network can dynamically and adaptively extract data of the corresponding modality for each time period, and utilize the interaction of the two modal information to achieve better video event classification results. At the same time, this invention proposes a time- and channel-based attention module, which makes full use of the relationships within and between channels to capture the most effective information for video classification. This invention can better identify events in the case of audio and video dual modalities, thereby improving the accuracy of video event recognition. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the video event recognition model structure based on audiovisual modal fusion in this invention;

[0032] Figure 2 This is a schematic diagram of the audio adjustment and visual information enhancement process in this invention;

[0033] Figure 3 This is a schematic diagram of the fusion attention module structure in this invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] This invention proposes a video event recognition method based on audiovisual modal fusion, such as... Figure 1 As shown, the method includes the following:

[0036] This invention designs a video event recognition model based on audiovisual modal fusion. Its input is a video segment S, which contains visual information V and audio information A. After segmenting the video accordingly, V is formed. t and A t The overall expression is The final output obtained after model processing is:

[0037]

[0038] Here, E represents the total number of categories, and the final output label y represents the classification prediction for the current video.

[0039] The specific process of video event recognition implemented in this invention is as follows:

[0040] S1: Acquire the video to be identified and segment it to obtain visual and audio information.

[0041] To obtain the video to be identified, the video is first divided into T non-overlapping segments. Where V t and A t These represent the visual information and corresponding audio information at time t, respectively. The visual information is defined as... Auditory information is H represents the visual information at time T. v W v and D v These represent the height, width, and dimensions of the visual feature map, respectively. H represents the audio information at time T. a W a and D a These represent the height, width, and dimension of the audio feature map, respectively.

[0042] S2: Perform feature extraction on visual and audio information respectively to obtain video features and audio features.

[0043] Data from two modalities (U=2) – video and audio – is extracted. For each input segment, a feature extraction module is used to extract features from both visual and audio information. Specifically, convolutional neural networks are used to extract relevant features. In the visual feature extraction process, the ResNet model is used. This network, proposed by four scholars from Microsoft Research, utilizes a residual mechanism to significantly save time and space costs without affecting feature extraction accuracy. Thanks to its powerful feature extraction capabilities, it is now widely used in image feature extraction. Auditory feature extraction uses the VGGish model, which was pre-trained by Google on the Audioset dataset, a dataset containing large amounts of audio. This model is best suited for scene-based sound events. After extracting relevant features, considering the correlation between different segments in the corresponding segments, a self-attention mechanism is used on the extracted features. This method utilizes the relationship between segments within the same channel to improve the extracted features, resulting in new visual and audio features. The principle is expressed by the following formula:

[0044] Q, K, V = (W q x, W k x, W v x)

[0045]

[0046] Where Q, K, and V represent the query, key, and value generated by the transformation of the input vector x, respectively, and W q W k W v Let be the transformation matrix of the input x, d be the dimensions of Q and K, SA(x) be the self-attention function, and the output of the function is a weighted average vector V′. Video and audio features are fed into this function, and then a Dense layer is added for processing. The purpose of this layer is to ensure that each channel has the same dimension, thus obtaining the output video features. and audio features

[0047] S3: Based on visual and audio features, the audio-adjusted visual information attention module enhances the visual features to obtain enhanced visual features.

[0048] After extracting data from both modalities, the relationship between the modalities is used to model the visual information using audio information. This process is called modal conditioning. The most commonly used method for modal conditioning is the attention mechanism. This invention also applies this method to the video recognition process, using audio features to guide visual information, thereby improving the quality of visual features. The main process of this module is to use audio information to guide visual information in the channel dimension, thus obtaining enhanced visual features, which are more prominent and effective information features for video classification, thereby providing better information input for subsequent classification work.

[0049] After the feature extraction module has extracted features, the visual feature input is now defined as follows: Audio feature input is H and W are the height and width of the feature map, respectively; the process by which the audio-modulated visual information attention module enhances visual features includes:

[0050] like Figure 2 As shown, guided by audio information, visual and audio features are first projected into the same dimension to generate audio-guided feature maps and visual feature maps. Then, visual features are fused through element-wise multiplication. The implementation process is as follows:

[0051] First, the input audio and visual features are concatenated, then activated using the ReLU function to obtain f. a，v f a，v After learning the concatenated feature representations, enhanced visual features are finally generated, represented as:

[0052]

[0053]

[0054] Among them, f a，v This represents the feature obtained by cascading visual and audio information. Indicates enhanced visual features, δ a This indicates that average pooling is being performed, and w represents the learnable parameters.

[0055] S4: Input the audio features and enhanced visual features into the temporal attention module for processing to obtain temporal weights.

[0056] The main purpose of the temporal attention module is to assign corresponding weights to each time window. Each weight represents the contribution of the relevant segment to the classification, which can also be understood as the amount of classification information contained in that segment within the entire video. The temporal weights of the audio features and the enhanced visual features are calculated separately, and the output temporal weights are represented as s = {s1, s2, ..., s...}. t The formula for calculating the time weights of audio features is:

[0057]

[0058]

[0059] z t This indicates the intermediate temporal attention weights, which are the intermediate attention weights obtained after passing through the temporal attention module. This represents the parameters learned at the first moment, i.e., the parameters learned at each time interval through the time attention model, x. t Let x represent the audio features at time t, b represent the second-time learning parameters, and x represent the audio features at time t. t The parameters at time t represent the audio features learned through the temporal attention model, used for fine-tuning the results. ReLU() represents the activation function. t The time weight of the audio feature at time t is the weight after normalizing the intermediate attention weight, and T represents the total number of time segments into which the video is divided.

[0060] The formula for calculating the time weights of the enhanced video features is the same as the formula for calculating the time weights of the audio features, and will not be repeated here.

[0061] The temporal weights of the audio features and the enhanced visual features are weighted and summed to obtain the final temporal weights.

[0062]

[0063] Among them, o tempThis represents the temporal attention feature, calculated by weighting the visual and audio features according to their temporal weights. This represents the information weight of visual information within a given time period. This represents the information weight of visual information within a given time period. The visual features enhanced at time t, representing the input time attention module. This represents the audio features at time t of the input time attention module.

[0064] S5: Input the audio features and enhanced visual features into the channel attention module for processing to obtain the channel weights.

[0065] The channel attention module also calculates the contribution of features from corresponding time segments to overall event recognition. The channel attention mechanism compresses features, representing each channel numerically. It learns the importance of different channels by predicting the importance of the compressed features. This module also calculates the weight of each channel to represent the amount of information contained in the relevant segment for classification within the entire video. Similarly, it calculates the channel weights for audio features and the enhanced visual features separately. The formula for calculating the channel weights of audio features is as follows:

[0066]

[0067]

[0068] Among them, z k This indicates the intermediate channel attention weights, which are the intermediate attention weights obtained after passing through the channel attention module. This represents the first channel learning parameters, which are the parameters of each channel learned through the channel attention model, x. k Let represent the audio features of the k-th channel, b′ represent the learned parameters of the second channel (i.e., the parameters learned through the channel attention model), used to fine-tune the results, and ReLU() represent the activation function. The weight of the k-th channel of the audio feature is the weight after normalizing the intermediate attention weights, where K represents the number of channels.

[0069] The formula for calculating the channel weights of the enhanced video features is the same as the formula for calculating the channel weights of the audio features, and will not be repeated here.

[0070] The channel weights of the audio features and the enhanced visual features are weighted and summed to obtain the final channel weights.

[0071]

[0072] Among them, o modThis represents the channel attention feature, x', after weighted summation based on the channel weights of visual and audio features. 1 The enhanced visual features representing the input channel attention module, x' 2 This represents the audio characteristics of the input channel attention module.

[0073] S6: The fusion attention module is used to process the temporal weights and channel weights to obtain the video event recognition results.

[0074] The main function of this module is to fuse the weights calculated by the two attention modules. In video time recognition, sometimes visual information contains more key classification information, while sometimes auditory information contains more key information. Thanks to the combined effect of the two attention modules, the weight allocation for the corresponding segments can be more reasonable.

[0075] like Figure 3 As shown, the process of using the fusion attention module to process the time weights and channel weights includes: calculating the comprehensive weight based on the time weights and channel weights; specifically: the time weights and channel weights are weighted and summed to obtain the input of the fusion attention module, expressed as:

[0076]

[0077] Where x′ represents the input feature vector, This represents the weight parameters input to the Kth channel of the fusion attention module at time T.

[0078] The formula for calculating the overall weight is:

[0079]

[0080]

[0081] in, The intermediate fusion attention weights refer to the intermediate attention weights obtained after passing through the fusion attention module. This indicates the first fusion learning parameter, which is the parameter of each channel learned in the fusion attention module. This represents the data input to the k-th channel at time t of the fusion attention module, and b″ represents the second fusion learning parameter, i.e., the parameter learned through the fusion attention model. The overall weight represents the weight after normalizing the intermediate attention weights.

[0082]

[0083] Where o represents the combined weight value after concatenating visual and audio features, and concat() represents the concatenation operation.

[0084] Video events are classified based on comprehensive weights. The final output weight represents the predicted probability value for each event. If the corresponding event has the largest weight, the prediction is correct, and the video event recognition result is obtained.

[0085] In summary, this invention should pay attention to the relationship and interaction between the two modalities when fusing information from different modes. It uses audio information to obtain enhanced visual information features, thereby highlighting key visual features for video classification and focusing on the visual regions most relevant to video event classification. A temporal and channel attention module is proposed, which assigns corresponding weights to visual and audio information within each appropriate time window, using these weights as the basis for video classification. Therefore, this invention can better identify events in audio-visual dual-modal scenarios.

[0086] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video event recognition method based on audiovisual modal fusion, characterized in that, include: S1: Acquire the video to be identified and segment it to obtain visual and audio information; S2: Perform feature extraction on visual and audio information respectively to obtain video features and audio features; S3: Based on visual and audio features, the audio-adjusted visual information attention module is used to enhance the visual features, resulting in enhanced visual features; S4: Input the audio features and enhanced visual features into the temporal attention module for processing to obtain temporal weights; The temporal attention module processes audio features and enhanced visual features by: calculating the temporal weights of the audio features and the enhanced visual features respectively; weighted summing of the temporal weights of the audio features and the enhanced visual features to obtain the final temporal weight; the formula for calculating the temporal weight of the audio features is: ； ； in, Indicates the attention weight at intermediate time. This indicates that the parameters are learned immediately. This represents the audio characteristics at time t. This indicates the parameters learned at the second time step. This represents the activation function. This represents the time weight of the audio features at time t. This indicates the total number of time segments into which the video is divided; S5: Input the audio features and enhanced visual features into the channel attention module for processing to obtain channel weights. The process of the channel attention module processing the audio features and enhanced visual features includes: calculating the channel weights of the audio features and the enhanced visual features respectively; and performing a weighted sum of the channel weights of the audio features and the enhanced visual features to obtain the final channel weights. The formula for calculating the channel weights of the audio features is as follows: ； ； in, Indicates the attention weight of the intermediate channel. Indicates the learning parameters for the first channel. This represents the audio characteristics of the k-th channel. This represents the learning parameters for the second channel. This represents the activation function. This represents the weight of the k-th channel of the audio feature. Indicates the number of channels; S6: The fusion attention module is used to process the temporal weights and channel weights to obtain the video event recognition results.

2. The video event recognition method based on audiovisual modal fusion according to claim 1, characterized in that, The process of extracting features from visual and audio information separately includes: using a ResNet network and a self-attention mechanism to extract features from visual information to obtain visual information; and using a VGGish model and a self-attention mechanism to extract features from audio information to obtain audio information.

3. The video event recognition method based on audiovisual modal fusion according to claim 1, characterized in that, The process by which the audio-modulated visual information attention module enhances visual features includes: projecting visual and audio features onto the same dimension; fusing the visual and audio features projected onto the same dimension element by element; and processing the fused features using a non-linear layer to obtain enhanced visual features.

4. The video event recognition method based on audiovisual modal fusion according to claim 1, characterized in that, The process of using the fusion attention module to process the temporal weights and channel weights includes: calculating the comprehensive weight based on the temporal weights and channel weights; classifying video events based on the comprehensive weights to obtain the video event recognition results.

5. The video event recognition method based on audiovisual modal fusion according to claim 4, characterized in that, The formula for calculating the overall weight is: ； ； in, Indicates the intermediate fusion attention weights. Indicated in the first fusion learning parameters, This represents the data input to the k-th channel at time t of the fusion attention module. Indicates the second fusion learning parameter. This indicates the overall weight.

Images