A cross-encoder model for classification

Abstract

A cross-encoder model for classification, for noisy and chaotic audio-video environment, join attention model to improve the feature extraction ability of noisy audio-video. Depend on cross-encoder model to realize the information interaction and fusion before multi-modal, and relative to use multiple single modal auto-encoder. The model optimizes and improves the auto-encoder, proposes the concept of cross-encoder; through the use of cross-encoder and attention mechanism to improve the accuracy of classification; improve the relevance between multiple modal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and in particular to a mutual encoder model for classification. BACKGROUND

[0002] Multi-modal machine learning aims to establish a model capable of processing and associating multi-modal information, learning feature information of different modalities through different network modules, and training the network accordingly. In order to improve the classification accuracy, the common classification models at present include: a classification model based on an autoencoder, a classification model based on attention, etc.

[0003] Common autoencoders include VAE (Variational Auto-Encoder, VAE), CVAE (Conditional VAE), VQ-VA (Vector Quantised-Variational AutoEncoder), etc. An autoencoder learns the features of information by reconstructing the input information, and improves the robustness of the autoencoder and the ability of feature extraction by adding noise. The current autoencoder classification model separately encodes multiple modalities and then fuses them, which makes the calculation too complex and causes additional performance overhead for the server.

[0004] Attention obtains the internal relationship of a sequence by calculating the attention of the sequence itself and assigning different weight information to different elements. The self-attention mechanism focuses on global correlation information by calculating the mapping relationship between the query vector Q (Query), the key matrix K (Key), and the value matrix V (Value), wherein Q=K=V, i.e. the embedded vector after simple splicing of audio and video, and each unit in the sequence and all units in the sequence are attention calculated. The attention model can well capture the characteristics and improve the classification accuracy, but the current network using the attention model alone often has insufficient classification accuracy and cannot fully utilize the performance of the attention model. SUMMARY

[0005] To solve the problems in the prior art, the present application proposes a mutual encoder model for classification using an attention model and a mutual encoder. For a noisy and chaotic audio and video environment, an attention model is added to improve the feature extraction capability of the noisy audio and video. The mutual encoder model is used to realize information interaction and fusion before multi-modal, and relatively uses multiple separate modal autoencoders.

[0006] A mutual encoder model for classification, which comprises a self-attention unit, a deep residual network unit, an attention unit, a classification unit, a convolution unit, and a mutual encoder.

[0007] The self-attention unit is connected to the mutual encoder through the convolution unit; the deep residual network unit is connected to the mutual encoder, and the output of the mutual encoder is connected to the classification unit through the attention unit.

[0008] Further, in each component, the self-attention unit allocates different attention weights to different contents; the deep residual network unit performs residual learning; the classification unit classifies features; the convolution unit extracts local features; and the mutual encoder encodes and decodes multi-modal information.

[0009] Further, the mutual encoder comprises an encoder unit and a decoder unit.

[0010] Further, the encoder receives an input and processes an output to the decoder, and then the output of the encoder and the decoder is combined to obtain the output of the mutual encoder.

[0011] Further, the received input is compiled and converted into a signal form for communication, transmission and storage, and the audio and video modalities in the mutual encoder are jointly trained in the same mutual encoder.

[0012] Further, the decoder unit is a device for decoding digital audio and video data streams into analog audio and video signals, and the audio and video modalities in the mutual encoder are each used with a separate decoder unit.

[0013] The present application has the following beneficial effects:

[0014] 1) The self-encoder is optimized and improved, and the concept of mutual encoder is proposed;

[0015] 2) The classification accuracy is improved by using the mutual encoder and the attention mechanism;

[0016] 3) The correlation between multiple modalities is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A mutual encoder model for classification in the embodiment of the present application is constituted.

[0018] Figure 2 An EAE working principle flow in the embodiment of the present application.

[0019] Figure 3 A confusion matrix without introducing a mutual encoder model in the embodiment of the present application.

[0020] Figure 4 A confusion matrix with introducing a mutual encoder model in the embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions of the present application will be further described in detail below with reference to the accompanying drawings of the specification.

[0022] The model structure includes Self-attention (self-attention unit), ResNet50 (Residual Network 50 unit), attention (attention unit), Classification layer (classification unit), Convolutional layer (convolution unit), EAE (Each encoder). The function of each component is described as follows:

[0023] 1) Self-attention (self-attention unit): Different attention weights are assigned to different content. The audio hidden layer vector obtained from the OpenL3 network is fed into the self-attention layer for acoustic feature extraction. After information extraction, the acoustic feature can pay more attention to feature information.

[0024] 2) ResNet50 (Residual Network 50 unit): The pre-trained model ResNet50 is used as a visual feature extractor in this model. Residual learning is easier than direct learning of original features. When the residual is 0, the stacking layer only makes an identity mapping, and at least the network performance will not decrease. In fact, the residual will not be 0, which will also make the stacking layer learn new features based on the input features, thus having better performance. The ResNet50 network structure used in this model is composed of 49 convolutional layers and a fully connected layer. The convolution kernel size has three types, 7x7, 1x1, and 3x3. The activation function of the convolutional layer is a nonlinear Relu activation function. In order to maintain consistency with the acoustic module, the data dimension output by the visual module is also set to 16x10x1024, which means that the model expects the input image sequence to be represented as 10 frames and each frame as a 1024-dimensional visual embedding vector.

[0025] 3) Attention (attention unit): Attention mechanism can solve the problem of information overload and improve the ability of neural network to process information by focusing on target information. It can be divided into two categories: one is focused attention, which is a top-down active attention method with predetermined purpose and task dependence, actively focusing attention on a certain object; the other is saliency attention, which is a bottom-up passive attention method driven by external stimuli, without active intervention and task dependence. In the research of neural network, focused attention is mostly used. Attention mechanism is essentially addressing. Given a task-related query vector Query, the relevance of Query and Key matrix Key is calculated to obtain the weight coefficient of each Key corresponding to Value, and then the weighted sum of each Value is obtained to get the Attention value. In this model, the video embedding vector output by the visual module is essentially used as the query vector Query in the attention mechanism, and the audio embedding vector output by the mutual encoder module is used as the Key and Value in the attention mechanism.

[0026] 4) Classification layer (classification unit): classifying network features, this paper is a ten-class task, which divides features into ten categories, and the 10 categories of scenes can be divided into three categories: indoor, outdoor and traffic places.

[0027] 5) Convolutional layer: further deep extraction of features and feature extraction of scene features, and effectively reduce the computational complexity of the network. Convolutional layer uses convolution operation to extract features from input data. For an input feature map, the size of the output feature map is determined by the convolution kernel, step and padding (Padding) three parameters. Among them, the convolution kernel is used to extract the input features. When a convolutional neural network has more convolution kernels, the more feature maps it produces, the more complex the internal structure of the network; the step is the distance that the convolution kernel moves on the input matrix during feature extraction; compared with the pixel points in the middle of the feature map, the edge pixel points are calculated very few times in the convolution, which will cause the edge information to be easily lost, and the padding item is to fill the edge of the input matrix to ensure that the edge pixel information can also be fully utilized. The biggest feature of convolutional layer is its powerful feature learning ability. Although only some simple feature information can be obtained from the original data in the first convolutional layer, high-level feature information can be extracted from the original features through continuous training and back propagation, avoiding complex feature engineering. Convolution kernel has local properties, which only focuses on local features regardless of the location of the features, while the degree of locality is determined by the size of the convolution kernel. Generally, the size of the convolution kernel is much smaller than the input size, so many connections in the convolutional neural network are actually sparse connections, which requires fewer trainable parameters, and improves the efficiency of the ordinary neural network model by a large margin. Another biggest feature of convolutional neural network is weight sharing, which is realized through the sliding of convolution kernel, the purpose is to extract the same kind of features in different positions of the model input, which is also the reason why convolutional neural network has translation invariance.

[0028] 6) EAE (inter-encoder): encodes and decodes the input modal information, so that the original data is mapped to a low-dimensional vector space to obtain low-dimensional hidden layer features. The low-dimensional hidden layer features of the two modalities are simply spliced as the output features of the inter-encoder, and the output features of the inter-encoder are assisted by the visual features through the attention mechanism.

[0029] The EAE in the model includes Encoder and Decoder, and the working process is as follows Figure 2 , which is described as follows:

[0030] 1) Encoder, which encodes signals (such as bit streams) or data into a signal form that can be used for communication, transmission and storage. The audio and video modalities in EAE are trained with the same Encoder.

[0031] 2) Decoder (decoder unit), hardware / software device that decodes digital audio / video data stream into analog audio / video signal, EAE audio and video modalities are decoded by separate decoders for each modality. Unlike ordinary variational autoencoder, the reconstruction target of mutual encoder is the information of another modality. That is, the input audio modality information, the reconstruction target is the visual modality information; input visual modality information, the reconstruction target is the audio modality information.

[0032] Experiments show that the introduction of mutual encoder can improve the accuracy of classification task, Figure 3 The accuracy of the acoustic auxiliary vision model, that is, the model without introducing mutual encoder, is 87.01%, while Figure 4 The accuracy of the model with the introduction of mutual encoder is 89.72%. Mutual encoder has certain improvement on the accuracy of classification task. From the two confusion matrix diagrams, it can be seen that mutual encoder has obvious improvement on the accuracy of four scene classifications of airport, tram, street with pedestrians and shopping mall.

[0033] The above is only the preferred embodiment of the present application, the protection scope of the present application is not limited to the above-mentioned embodiment, any equivalent modification or change made by the ordinary skilled in the art according to the disclosed content of the present application shall be included in the protection scope recorded in the claims.

Claims

1. An inter-encoder model, characterized in that: the model structure comprises a self-attention unit, a deep residual network unit, a classification unit, an attention unit, a convolution unit, and an inter-encoder; the self-attention unit is connected to the inter-encoder through the convolution unit; the deep residual network unit is connected to the inter-encoder, and the output of the inter-encoder is connected to the classification unit through the attention unit; the self-attention unit delivers the audio hidden layer vector obtained from the OpenL3 network to the self-attention layer for acoustic feature extraction; after extraction, the acoustic feature can pay more attention to feature information; in the deep residual network unit, a pre-trained model ResNet50 is used as a visual feature extractor, and the output data dimension is set to 16x10x1024, meaning that the image sequence input by the model is represented as 10 frames of 1024-dimensional visual embedding vectors; the classification unit classifies the acoustic feature and the visual embedding vector into ten categories, and the ten categories are divided into three categories of indoor, outdoor, and traffic place.

2. The cross-encoder model of claim 1, wherein: In each component, the self-attention unit allocates different attention weights to different contents; the deep residual network unit performs residual learning; the classification unit classifies features; and the convolution unit extracts local features. The inter-encoder encodes and decodes multi-modal information.

3. The cross-encoder model of claim 1, wherein: The inter-encoder comprises an encoder unit and a decoder unit.

4. The cross-encoder model of claim 3, wherein: The encoder receives input and processes output to the decoder, and then the output of the encoder and the decoder is combined to obtain the output of the inter-encoder.

5. The cross-encoder model of claim 3, wherein: The encoder unit processes the received input and converts it into a signal form for communication, transmission, and storage; in the inter-encoder, the audio and video modalities jointly train the same inter-encoder.

6. The cross-encoder model of claim 3, wherein: The decoder unit is a device that decodes digital audio and video data streams into analog audio and video signals; in the inter-encoder, the audio and video modalities each use a separate decoder unit.

7. The cross-encoder model of claim 1, wherein: The ResNet50 network structure used in the deep residual network unit is composed of 49 convolution layers and one fully connected layer, and the convolution kernel size has three types, 7x7, 1x1, and 3x3; the activation function of the convolution layer is a nonlinear Relu activation function; and the output data dimension is set to 16x10x1024.

Images