Video data feature extraction method based on audio and video multi-mode time sequence prediction

Abstract

The invention discloses a video data feature extraction method based on audio and video multi-mode time sequence prediction, and the method comprises the steps: 1, obtaining a video data set through a video collection device, and constructing an audio and video double-flow data pair; 2, for each video frame in the video stream and each audio clip in the audio stream, adopting a series of data enhancement operations in respective modes, and converting a one-dimensional audio into a two-dimensional spectrogram; 3, constructing an audio and video multi-modal prediction model which comprises a video stream feature extraction network unit, an audio stream feature extraction network unit, a time sequence information aggregation network unit and a multi-modal interaction prediction network unit; and 4, according to the uncertain features obtained by the multi-modal interaction prediction, calculating the total loss of the multi-modal prediction of the audio and video and optimizing the network. According to the method, useful information of the video can be effectively mined in a self-supervised manner by utilizing the time sequence of the video and combining the interaction between audio and video double streams, so that the effectiveness of feature extraction is improved, and actual downstream tasks such as video understanding, sound source localization, anomaly detection and the like are facilitated.

Description

technical field [0001] The invention relates to the field of video data processing and analysis, in particular to a video data feature extraction method for audio and video multimodal time series prediction. Background technique [0002] In the context of today's Internet big data, it is becoming more and more important to process and analyze specific data. This kind of data analysis can also be called "representation learning" in some fields of artificial intelligence, that is, to extract useful information from data. Machine learning, especially deep learning algorithms, largely rely on data representation, so how to use the Internet Shanghai The self-supervised mining of its own potential effective information has attracted extensive attention of researchers. As we all know, human cognition is a reaction based on the combination of multiple modal information perceptions, in which the visual and auditory senses usually coexist with each other, for example, the wind whistl...

AI Technical Summary

Problems solved by technology

The single mode is often imitated in the image field, such as the generation of missing frames in video clips, video rate prediction, etc. However, for the understanding of video, it is far from enough to rely on a single mode of video stream, such as the laughing of the observed Sound and background cheerful music are helpful to improve the classification accuracy of a funny video; for the latter multimodal video representation learning, some researchers use the clustering results to construct pseudo-labels and guide each other in feature classification. The performance of the method depends heavily on the selected clustering method or the number of category clusters needs to be set in advance; in addition, most researchers use a multimodal representation learning method with another modality of audio flow or optical flow , where the extraction of optical flow information is time-consuming, and the quality of optical flow depends on the performance of the selected optical flow extraction network. For audio and video representation learning, timing alignment between two streams is usually used, and a large number of negative example pairs are introduced. Carry out self-supervised comparative learning, but the existing audio and video multi-modal data feature extraction methods ignore the timing between audio and video, that is, the connection between frames is not considered, and timing is exactly what video compares with The unique nature of the image itself, independent processing between frames will lead to the loss of some important time-series coherent information, making the machine's understanding of the video limited, and the noise information of the frame will easily reduce the accuracy of action recognition, while the loss of time-series information will reduce Accurate Classification of Video Actions and Noise Robustness of Reduced Models

Images