Target behavior perception method based on audio-video cross-modal feature expression

Abstract

The application belongs to the technical field of video information processing, and proposes a target behavior perception method of audio-video cross-modal feature expression. The method is realized based on a target behavior perception network model, and includes a video feature extraction network, a cross-modal information fusion network and a classifier. After spatial dimension reduction of video data, a transformation operation is used to convert the information mode into a frequency representation similar to audio, eliminating the dimensional difference and expression difference between the two modalities. On this basis, the audio-video combination is realized by combining intermediate fusion and late fusion, improving the cross-modal information fusion effect. The method can be simply extended and applied to other tasks that need to use audio-video fusion information, effectively solving the influence of the difference between audio and video on modal fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video information processing, and particularly relates to a target behavior perception method based on audio-video cross-modal feature expression. BACKGROUND

[0002] The related background of the present application mainly contains two aspects: one is a target behavior perception method based on deep learning, and the other is a cross-modal information fusion method.

[0003] The target behavior perception method based on deep learning can be mainly divided into single-modal method and multi-modal method according to the difference in the number of information modalities. The single-modal method simply uses the visual part of the video, and understands the video as a continuous video frame with correlation in time. On this basis, the mature structures in the image processing field such as convolutional neural network (CNN) structure and Transformer structure are adjusted and improved to form a single-modal behavior recognition method that can be applied to videos. Among them, the typical one is the SlowFast network structure proposed by Christoph Feichtenhofer et al. in the literature “SlowFast Networks for Video Recognition”. This method decomposes the video into slow-changing semantic information and fast-changing action information from the information perspective, and designs independent paths to extract the two kinds of information. Under the condition of using only video frames as input, excellent recognition effect is obtained. The multi-modal method aims to maximize the use of existing resources, and usually uses video frames in addition to optical flow (Optical Flow) or synchronous audio of the video as auxiliary information to provide multi-angle information for network model for content understanding. A typical method is the two-stream network structure based on cross-modal attention proposed by Lu Chi et al. in the literature “Two-Stream Video Classification with Cross-Modality Attention”. This method uses two symmetrical and identical convolutional networks as feature extraction branches to receive video frames and optical flow of the same video respectively, and cross-modal information interaction is performed through the designed cross-modal attention module (CMABlock) between them. Although the multi-modal method generally has the problems of complex model structure and large number of parameters, it is favored by more and more researches due to its generally better effect than the single-modal method.

[0004] The core problem faced by any multimodal method is how to effectively integrate and utilize the differentiated representation information from different modalities. The fusion method can be mainly divided into three categories according to the timing of the fusion of different data modes in the model: early fusion, intermediate fusion and late fusion. Early fusion is generally carried out at the data level, and cross-modal fusion is achieved by splicing different modal input before model processing. This method requires the network model to learn the way to distinguish the modal difference from the mixed data input, and the effect is often not satisfactory due to the lack of effective guidance for model learning in design. In recent years, few methods have adopted this strategy; intermediate and late fusion intend to utilize the feature extraction process of deep neural networks to carry out cross-modal information fusion at the feature level to reduce the influence of the difference between modalities. Before fusion, different modal input data will go through a separate feature extraction process for feature mapping, and fusion is usually achieved through feature splicing, attention mechanism, etc. The late fusion method performs a one-time fusion on the deep feature representation before and after the classifier. A typical method is the TSN structure proposed by Limin Wang et al. in the literature “Temporal segment networks: Towards good practices for deep action recognition”. This method processes different modal inputs separately to obtain independent class predictions, and finally averages the prediction results of different modalities to obtain a comprehensive prediction. The intermediate fusion method emphasizes repeated cross-modal information interaction during feature extraction. A typical method is the MBT structure proposed by Arshanagrani et al. in the literature “Attention Bottlenecks for Multimodal Fusion”, which utilizes the Transformer self-attention mechanism to realize free interaction between different modal information during feature extraction. Intermediate fusion is more complex than late fusion structure, but practice shows that it has relatively better fusion effect, and both are widely used by recent methods.

[0005] In current target behavior perception methods based on audio-video combination, the video part is mostly processed using classic single-modal methods, and the video frames are directly input into the network for feature extraction after time sampling or spatial division. The audio part generally borrows from the conventional methods in the natural language processing field, and the audio is transformed into a frequency domain representation spectrogram for image processing. The transformation of audio changes the original representation, causing a difference in expression between video content; video information records changes in spatial and temporal dimensions, while audio as a single-dimensional data in the time dimension lacks spatial information, and there is an obvious dimensional difference between the two. The existence of two differences makes it difficult for existing methods to accurately establish a corresponding relationship between audio and video, and cross-modal information is not fully utilized. SUMMARY

[0006] In order to improve the accuracy of behavior recognition task, a target behavior perception method of audio-video cross-modal feature expression is proposed to eliminate the representation difference and dimension difference between audio and video in the process of audio-video cross-modal fusion. The method decomposes the spatial dimension of video in the mode fusion by partition mapping, thereby eliminating the dimension difference between modes. Meanwhile, the video is unified to the frequency representation similar to the audio by using the partition time-frequency transformation, thereby relieving the modal difference between the two. On this basis, the cross-modal feature fusion is performed by using the self-attention mechanism, and the comprehensive feature representation is obtained to perform behavior prediction.

[0007] The technical scheme of the present application is as follows:

[0008] A target behavior perception method of audio-video cross-modal feature expression is realized based on a target behavior perception network model. The target behavior perception network model comprises a video feature extraction network, a cross-modal information fusion network and a classifier.

[0009] Specifically, the method comprises the following steps:

[0010] Step 1: sampling a video segment X containing audio, and separating to obtain a video part X V and an audio part X A input into the target behavior perception network model;

[0011] Step 2: inputting the video part X V and the audio part X A into the cross-modal information fusion network; the video part X V is preprocessed, and then subjected to spatial dimension reduction, time-frequency transformation and video feature mapping to obtain a video mapping; the audio part X A is subjected to audio short-time Fourier transformation and audio feature mapping to obtain an audio mapping; and the video mapping and the audio mapping are fused to obtain fused audio-video features;

[0012] Step 3: inputting the video part X V into the video feature extraction network; the video part X V is preprocessed, and then subjected to video feature extraction to obtain video features;

[0013] Step 4: inputting the video features and the fused audio-video features into the classifier after splicing to obtain the behavior classification result of the video segment X containing audio; and optimizing the network parameters of the target behavior perception network model according to the classification loss function until the model converges.

[0014] The target behavior perception network model is as a whole Y:

[0015]

[0016] where h(·) is the classifier, is the feature concatenation operation, f(·) is the video feature extraction network, which is implemented based on 3D convolutional network 3D ResNet; g(·) is the cross-modal information fusion network; g(·) is further decomposed as:

[0017]

[0018] where r(·) is the audio feature mapping network, which is implemented based on 2D convolutional network ResNet; s(·) is the time-frequency transform and video feature mapping network, which is implemented based on shallow 2D convolutional network; T(·) is the information fusion network, which is implemented based on Transformer encoder.

[0019] The audio part X A is obtained by short-time Fourier transform to get the frequency domain representation is then input to the audio feature mapping network r(·) to get the corresponding audio mapping F A .

[0020] The video part X V The process of obtaining the video mapping is as follows: the video part X V is spatially decomposed to obtain N blocks, denoted as is the video frame sequence composed of the i-th block;

[0021] The block set obtained after spatial decomposition is each local video frame sequence is performed grayscale transformation:

[0022]

[0023] where are the red channel frame sequence, green channel frame sequence, and blue channel frame sequence of the video part X V respectively; the video frame sequence corresponding to each partition i is obtained by taking the mean value on the spatial local part of each frame:

[0024]

[0025] where p is the size of each block; (u, v) is the pixel in the u-th column and v-th row within the i-th block; is the representation of the motion in the time sequence for different positions in the video space; is a one-dimensional vector of length t, t is the number of sampled video frames; the Fourier transform based on the time window maps the time sequence representation of to the frequency representation:

[0026]

[0027]

[0028] where ω(·) is a window function; the transformed result is with two dimensions, the first dimension n represents the window sequence sampled continuously in time, and the second dimension ω represents the frequency components obtained after each window is transformed; after converting the time sequence changes of each block divided video frame into frequency representation, further convolution operation is used to extract features to obtain the corresponding video mapping where N is the total number of divided blocks, and s is the length of the one-dimensional feature vector corresponding to each block division.

[0029] In the cross-modal information fusion network, the audio mapping F A is copied N times to match the shape of the video mapping F V , and the joint feature X is obtained after splicing operation. The learnable Token vector is set together with the joint feature X F is input into the information fusion network; under the self-attention framework, the information of the joint feature X T is constantly converged to X T , so that it has global information expression capability, and the corresponding output vector of X V is the final result of the cross-modal information fusion network g(·); the final result of the cross-modal information fusion network g(·) and the video feature f(X V ) obtained by the video feature extraction network f(·) are spliced and fused again and input into the classifier, and after dimension reduction by the fully connected layer in the classifier and activation by the SoftMax function, the class prediction p=[p1,p2,…,p K ] is obtained.

[0030] The whole video target behavior perception network is trained by using multi-classification cross-entropy loss:

[0031]

[0032] where K is the total number of classes, y is the class true value label of the sample, and p is the output prediction of the classifier.

[0033] The application provides a target behavior perception method for audio-video cross-modal feature expression. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The application provides a target behavior perception method for audio-video cross-modal feature expression.

[0035] Figure 2 The application provides a target behavior perception method for audio-video cross-modal feature expression. DETAILED DESCRIPTION

[0036] The application provides a target behavior perception method for audio-video cross-modal feature expression.

[0037] As shown in the accompanying drawings and technical solutions, the application provides a target behavior perception method for audio-video cross-modal feature expression. Figure 1 The application provides a target behavior perception method for audio-video cross-modal feature expression.

[0038] The application provides a target behavior perception method for audio-video cross-modal feature expression. V The application provides a target behavior perception method for audio-video cross-modal feature expression. A The application provides a target behavior perception method for audio-video cross-modal feature expression.

[0039]

[0040] The application provides a target behavior perception method for audio-video cross-modal feature expression. The application provides a target behavior perception method for audio-video cross-modal feature expression. V The application provides a target behavior perception method for audio-video cross-modal feature expression. A The application provides a target behavior perception method for audio-video cross-modal feature expression.

[0041] The application provides a target behavior perception method for audio-video cross-modal feature expression. V The application provides a target behavior perception method for audio-video cross-modal feature expression.

[0042] The cross-modal information fusion network g(·) receives the video part and the audio part at the same time, and focuses on modeling and describing the motion and changes in the time and space dimensions in the video based on the establishment of the cross-modal association between the video and the audio. These information helps to provide key discrimination clues for the approximate behaviors that are easy to be misclassified.

[0043] For the audio part X of the input cross-modal information fusion network g(·) A , the short-time Fourier transform (STFT) is used to obtain the spectrogram frequency domain representation Then, the 2D convolution network is used to extract the corresponding audio mapping F A . According to the foregoing analysis, the audio mapping F at this time A is established on the frequency representation of the audio, which is different from the visual features extracted directly on the time-space representation in terms of dimension and expression. In order to eliminate the influence of these differences on cross-modal fusion, the video part X V is first densely decomposed into a plurality of blocks in space, denoted as wherein represents the frame sequence formed by applying the division to all video frames. In the subsequent process, the synchronous feature mapping will be performed in blocks, and then the visual feature information corresponding to different spatial positions will be obtained. This reduces the spatial dimension of the video information while preserving the local features of the space to some extent, but the global position information is weakened due to the block division. Therefore, we introduce the Transformer structure based on the self-attention mechanism in the cross-modal information fusion network to reconstruct the global information.

[0044] Then, in order to handle the representation difference between the audio and video inputs, the present application proposes to extend the idea of short-time Fourier transform of audio to the processing of video, so as to input the video information into the network for feature extraction in a similar frequency domain representation manner. Since frequency mainly describes the change rule of objects in the time dimension, the color information is relatively redundant when considering the motion of the video object. Therefore, for each local frame sequence in the block set , the grayscale transformation is performed first:

[0045]

[0046] wherein, are the red, green and blue channel frame sequences of the video X V , respectively.

[0047] Considering that the spatial local pixels corresponding to each block often contain the same semantic object, when describing the change in the time dimension, it can be approximately regarded as a whole to reduce the overall calculation amount. Therefore, for the frame sequence corresponding to each divided block i The mean value is calculated on the spatial local part of each frame:

[0048]

[0049] where p is the size of each block. At this time is the description of the motion in the time series for different locations in the video space. is a one-dimensional vector with a length of t, and t is the number of sampled video frames. On this basis, the Fourier transform based on the time window is used to map the change description to the frequency description:

[0050]

[0051]

[0052] where ω(·) is a window function. The transformed also presents a two-dimensional spectrogram. The first dimension n represents the window sequence sampled continuously in time, and the second dimension ω represents the frequency components obtained after each window is transformed. After converting the time series changes of each block divided video into frequency representation, further convolution operation is used to extract features to obtain the corresponding video mapping where n is the total number of divided blocks, and d is the length of the vector.

[0053] After the foregoing process, the audio and video information can be unified in the same dimension and description for feature mapping. In further cross-modal fusion operation, the audio feature F A will be replicated N times to match the shape of the video mapping F V , and then a splicing operation is used on both to establish a joint feature The Transformer encoder structure based on attention mechanism is used to perform intermediate fusion on the joint feature X F , and a learnable label vector is input together, and the corresponding output vector is used as the final result of the cross-modal information fusion network g(·), and the output of the video feature extraction network f(·) is spliced again to realize late fusion. After dimension reduction by the fully connected layer and activation by the SoftMax function, the behavior class prediction p = [p1, p2, …, p K ] is obtained. The overall network is trained using multi-classification cross-entropy loss:

[0054]

[0055] where K is the total number of classes, and y is the class label of the sample.

[0056] The specific structure of the target behavior perception network model is as follows: Figure 2As shown, the target behavior perception network model includes a video feature extraction network, a cross-modal information fusion network and a classifier. The video part obtains video features through the video feature extraction network; the video part and the audio part are input into the cross-modal information fusion network; the cross-modal information fusion network includes an audio feature mapping network, a time-frequency transformation and video feature mapping network and an information fusion network. The audio part is input into the audio feature mapping network to obtain audio mapping; the video part obtains video mapping through the time-frequency transformation and video feature mapping network; the time-frequency transformation and video feature mapping network include block division, gray transformation and mean value, time-frequency transformation, convolution operation, and the video part extracts video mapping by sequentially passing through the above network layers; the video mapping and the audio mapping are jointly input into the information fusion network, and after intermediate fusion, the fused audio-video features are obtained, and then the video features are post-fused and input into the classifier to obtain the classification result.

[0057] First, a batch of video data is sampled from the training set, and the video part and the audio part are separated for separate processing. The video part is sampled at a fixed frame rate to obtain video frames and is copied into two identical parts, one of which is directly input into the single-modal video feature extraction network f(·) to directly extract video features; the other is used for the cross-modal information fusion network g(·). First, it is densely decomposed into several blocks in space, then the pixel mean value is calculated after gray transformation, and further time-frequency transformation is performed to obtain the local frequency spectrum representation of the video information, which has a similar form as the audio spectrogram after short-time Fourier transform. The video spectrogram and the audio spectrogram are mapped to feature vectors using a convolution network, and the feature vectors of the two are concatenated and input into the Transformer encoder as a vector sequence to obtain the cross-modal feature vector. Finally, the single-modal video feature vector and the cross-modal feature vector are concatenated, and the dimension is reduced by using a fully connected layer, and then the video behavior prediction is obtained by activating the SoftMax function. The network is trained using a loss function to have feature extraction and class prediction capabilities.

[0058] In summary, the present application proposes a target behavior perception method of audio-video cross-modal feature expression, which effectively solves the influence of the difference between audio and video on modal fusion. It is different from the existing methods in the following aspects: first, it is necessary to provide explicit action information to improve the perception ability of the model, but the present application proposes to use the frequency spectrum representation of the time sequence video frame to describe the object motion, instead of the commonly used optical flow representation; second, in the way of integrating motion information in the model, the method of the present application uses motion information as a bridge to establish the association between audio and video, instead of using it as an independent information mode to participate in fusion, which is more in line with the nature of the association between modes; in addition, different cross-modal fusion strategies are adopted for different modal combinations instead of synchronous fusion, which improves the efficiency of cross-modal information fusion.

Claims

1. A target behavior perception method based on cross-modal feature representation of audio and video, characterized in that, This is implemented based on a target behavior perception network model, which includes a video feature extraction network, a cross-modal information fusion network, and a classifier. Specifically, the steps include the following: Step 1: Sample the video segment X containing audio and separate the video portion X. V With audio section X A Input is fed into the target behavior-aware network model; Step 2: Video section X V And audio part X A Input to a cross-modal information fusion network; Video component X V After preprocessing, the video map is obtained through spatial dimensionality reduction, time-frequency transformation, and video feature mapping; the audio part X A The audio mapping is obtained after audio short-time Fourier transform and audio feature mapping; Step 3: Video Section X V Input is fed into the video feature extraction network; Video section X V Video features are obtained through video feature extraction after preprocessing; Step 4: After concatenating the video features and the fused audio-video features, input them into the classifier to obtain the behavior classification result of the video segment X containing audio; optimize the network parameters of the target behavior perception network model according to the classification loss function until the model converges; the overall target behavior perception network model is Y: Where h(·) is the classifier, For feature concatenation operation, f(·) is the video feature extraction network, implemented using 3D ResNet based on 3D convolution; g(·) is the cross-modal information fusion network; g(·) is further decomposed into: Where r(·) is the audio feature mapping network, implemented based on the 2D convolutional feature extraction network ResNet; s(·) is the time-frequency transformation and video feature mapping network, implemented based on a shallow 2D convolutional network; and T(·) is the information fusion network, implemented based on the Transformer encoder.

2. The target behavior perception method for audio and video cross-modal feature representation according to claim 1, characterized in that, The audio portion X A Frequency domain representation obtained by short-time Fourier transform The input is then fed into the audio feature mapping network r(·) to obtain the corresponding audio mapping F. A .

3. The target behavior perception method for audio and video cross-modal feature representation according to claim 2, characterized in that, The video portion X V The process of obtaining the video mapping is as follows: Video portion X V The space is decomposed into N blocks, denoted as The video frame sequence consisting of the i-th block; The block set obtained after spatial decomposition Each local video frame sequence Perform grayscale transformation: in, The video portion X V The red channel frame sequence, green channel frame sequence, and blue channel frame sequence; the video frame sequence corresponding to all partition blocks i. Calculate the mean value locally in each frame: Where p is the size of each block; (u,v) is the number of pixels in the u-th column and v-th row within the i-th block; This represents the motion of different locations in the video space over a time series. Let be a one-dimensional vector of length t, where t is the number of sampled video frames; the Fourier transform based on the time window will... Timing representation mapped to frequency representation: Where ω(·) is the window function; the transformed... It has two dimensions: the first dimension n represents the window sequence sampled continuously in time, and the second dimension ω represents the frequency components obtained after transforming each window; after converting the temporal changes of video frames under each block into frequency representations, convolution operations are further used to extract features to obtain the corresponding video mapping. Where N is the total number of blocks, and d is the length of the one-dimensional feature vector corresponding to each block.

4. The target behavior perception method for audio and video cross-modal feature representation according to claim 3, characterized in that, In the cross-modal information fusion network, audio mapping F A Perform N copies to match video mapping F V The shape of the two is combined to obtain a joint feature. Set up learnable token vectors Together with joint features Common inputs are fed into the information fusion network; features X are jointly input within a self-attention framework. F Information is constantly being sent to X T Convergence enables it to express global information, ultimately X T The corresponding output vector serves as the final result of the cross-modal information fusion network g(·); The final result of the cross-modal information fusion network g(·) and the video part X V Video features f(X) obtained by video feature extraction network f(·) V After being concatenated and fused again, the data is input into the classifier. After dimensionality reduction by the fully connected layer and activation by the SoftMax function, the class prediction p = [p1, p2, ..., p] is obtained. K ].

5. The target behavior perception method for audio and video cross-modal feature representation according to claim 3 or 4, characterized in that, The video target behavior perception network is trained using multi-class cross-entropy loss. Where K is the total number of categories, y is the true label of the sample category, and p is the classifier's output prediction.

Images