An audio-image cross-modal retrieval method based on dual learning

Abstract

The present application belongs to the technical field of deep learning, and relates to an audio-image cross-modal retrieval method based on dual learning. The present application realizes audio-image cross-modal retrieval, and solves the problem that the correspondence relationship of multi-modal data is difficult to be labeled using a traditional machine learning method. In data feature extraction, VGG adopts a small convolution kernel and a pooling layer, a deeper layer, and more channel numbers, and more channel numbers represent richer image features, and meanwhile, the increase of the channel number enables more information to be extracted. A feature conversion neural network is constructed, dual learning is introduced into model learning, two cross-modal tasks are trained, and a new idea is provided for a deep learning method of cross-modal data.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, and relates to an audio-image cross-modal retrieval method based on dual learning. BACKGROUND

[0002] With the rapid expansion of the amount of Internet information and the rapid development of communication technology, in the Internet with a large amount of data, more and more users use search tools to quickly and conveniently find network information. The information content on the Internet has gradually changed from text form to multi-modal information such as audio, image and video. Therefore, the retrieval demand of users is not limited to simple text search, and the demand for audio and image search is increasing. At present, the audio retrieval applied on the market is commonly used for song recognition and voice conversion, and the picture retrieval is commonly used for similar picture search. These retrieval methods exist in single mode or simple conversion with text, and the cross-modal retrieval method between audio and image is less studied. Using the cross-modal retrieval between audio and image can realize scene music matching according to pictures, finding voice description of corresponding pictures or identifying picture information described according to audio, and using the retrieval method can connect audio and pictures with each other, and even can be used on voiceless disabled people to find corresponding voice expression through pictures they pay attention to. The method can explore a new field of cross-modal audio recognition and image recognition.

[0003] In the supervised learning of the traditional machine learning method, if a neural network model needs to be trained, data needs to be labeled. For example, in the audio-image cross-modal retrieval, audio and corresponding images need to be paired to train the neural network model. Due to the characteristics of the neural network, under certain conditions, the larger the amount of data is, the better the model effect will be. Therefore, a large amount of manpower and time is needed to label the data. In addition, due to the large amount of time required to identify audio data, the relationship between audio and image is ambiguous, and manual calibration will cause differences and other problems, which ultimately leads to the difficulty of using the traditional machine learning method to complete the training of the audio-image cross-modal retrieval model. If a semi-supervised learning method is used, although unlabeled data can be used to generate pseudo-labels for training and learning, to a certain extent, the cost of human resources is reduced, but the quality of the pseudo-labels cannot be distinguished, and therefore the influence on the model training is large.

[0004] Dual learning is proposed by Microsoft Asia Research Institute Machine Learning Group in NIPS 2016, which is first used to solve the bilingual translation problem, and finally improves the model effect through the cooperation between multiple tasks. With the continuous development in recent years, dual learning has changed from a method to improve the model to a new learning paradigm. Unlike supervised learning and semi-supervised learning, dual learning does not need to label the data, but forms a closed loop through two learning tasks, and can evaluate the pseudo-label generated in the learning process. Through the evaluation reward, the model is guided to learn. In addition, unlike multi-task learning, the two learning tasks in dual learning have no requirements for the input space, and even if the two types of input are completely different, learning can be performed. This feature well meets the needs of cross-modal data and enables two types of unlabeled data to learn correspondingly. SUMMARY

[0005] In order to realize the cross-modal retrieval of audio-image, solve the problem that the corresponding relationship of multi-modal data is difficult to label using traditional machine learning methods, the application proposes an audio-image cross-modal retrieval method based on dual learning, which combines deep learning technology, and finally provides a theoretical basis and practical experience for the application of dual learning technology in audio-image cross-modal retrieval.

[0006] The technical scheme provided by the application for the above technical problems is as follows:

[0007] An audio-image cross-modal retrieval method based on dual learning, comprising the following steps:

[0008] Step one, for a piece of audio data A randomly selected from the audio database, after pretreatment, STFT (short time Fourier transform) is performed, and then CNN (convolutional neural network) model and Bi-LSTM (bidirectional long short term memory network) model are used to extract audio features A x .

[0009] Step two, for an image data B randomly selected from the image database, after pretreatment, VGG16 model is used to extract image features B x .

[0010] Step three, construct an audio, image similarity algorithm to calculate the similarity between audio features and image features.

[0011] Step four, construct an audio-image, image-audio feature conversion neural network model.

[0012] Step five, combine the similarity calculation method in step three, and use dual learning to train the neural network in step four.

[0013] Step six, using the model trained in step five, create an audio-image cross-modal retrieval system, which can search for related images by inputting audio, or find related audio data by inputting images.

[0014] The specific steps of feature extraction of audio data in step one are as follows:

[0015] 1.1 Preprocess the randomly selected audio data A by pre-emphasis, framing and windowing, etc. to obtain audio data A1.

[0016] 1.2 Obtain the power vector and phase vector of the preprocessed audio data A1 by STFT. The formula is as follows: Where x(t), τ and ω are the input audio data A1, the phase vector and power vector to be extracted, respectively.

[0017] 1.3 Use the CNN model to extract features from the phase vector and power vector obtained in 1.2, and finally use a fully connected layer to connect them to obtain the audio feature vector A2.

[0018] 1.4 Based on the natural time sequence of audio data, further input the audio vector feature A2 obtained from the CNN model in 1.3 into the Bi-LSTM model to extract the audio feature A x .

[0019] The specific steps of feature extraction of image data in step two are as follows:

[0020] 2.1 If B is not a three-channel image but a black and white image, first convert it to a three-channel image data B1 before inputting it into the VGG16 model.

[0021] 2.2 Input the preprocessed image data B1 into the VGG16 model to extract the image feature B x .

[0022] The specific steps of the audio and image similarity calculation method in step three are as follows:

[0023] 3.1 Data normalization is performed on audio one feature A x1 and audio two feature A x2 to obtain A′ x1 and A′ x2 .

[0024] 3.2 Use the DTW algorithm to calculate the similarity of the two audio feature sequences. Let the length of feature sequence A′ x1 be M, and the length of feature sequence A′ x2Length N, first construct DTW regular path W = (w1, w2, w3, …, w k ), where Max (|X|, |Y|) ≤ k ≤ |X| + |Y|. w k The form is w (i, j), i represents the i subscript in X, j represents the j subscript in Y, where i and j are monotonically increasing, and the regular path W specifies that w1 = (1, 1) must be used as the beginning, and w k = (|X|, |Y|) as the end, each coordinate in X and Y needs to appear in the regular path.

[0025] 3.3 Find the shortest regular path D (i, j) = Dist (i, j) + min [D (i-1, j), D (i, j-1)]

[0026] 3.4 Take the shortest regular path length as the similarity measure of the audio feature sequence, the smaller the path length, the higher the similarity.

[0027] 3.5 For image similarity, since the corresponding strictness of audio and image data is low, only the similarity of two images with similar features needs to be compared, without high similarity in two-dimensional vector space. Therefore, the same algorithm is used to calculate the similarity of image features.

[0028] The specific steps of the audio-image, image-audio feature neural network model construction method are as follows:

[0029] 4.1 Build a basic LSTM neural network as an audio-image feature conversion model, where the input vector dimension is the audio feature dimension m in step 1, the LSTM layer is set to α1, and the output vector dimension is set to O1.

[0030] 4.2 Design a linear fully connected layer, the input vector dimension is O1, and the output vector dimension is the feature dimension n of the image. Connect this linear layer with the LSTM layer in 4.1 to realize the input of the m-dimensional audio feature vector, and the neural network can convert it to an n-dimensional image feature.

[0031] 4.3 Build another linear fully connected layer, the input vector dimension is the image data feature n, use the Relu activation function, and the output vector dimension is n'. This compresses the image data feature to a certain extent, reducing the LSTM layer neural network training time.

[0032] 4.4 Build a basic LSTM neural network as an image-audio feature conversion model, where the input vector dimension is the compressed feature dimension n' of the image in 4.3, the LSTM layer is set to α2, and the output vector dimension is set to O2.

[0033] 4.5 Design a linear fully connected layer with input vector dimension O2 and output vector dimension m, which is the feature dimension of audio. Connect this linear layer with the LSTM layer in 4.4, so that the neural network can convert an input image feature vector of dimension n to an audio feature of dimension m.

[0034] The specific steps of the neural network training method based on dual learning are as follows:

[0035] 5.1 The audio-image feature conversion neural network constructed in step 4 is denoted as P(·|s; θ AB ), and the image-audio feature conversion neural network is denoted as P(·|s; θ BA ). Where s is the input of the neural network, θ AB and θ BA are the parameters of the neural network.

[0036] 5.2 Let the audio feature dataset be D A , and the image feature dataset be D B . First, start with audio-image feature conversion, take an audio feature s from D A , input it into the neural network P(·|s; θ AB ), and get s mid as the intermediate output in dual learning. According to step 4, s mid is the image feature.

[0037] 5.3 Input s mid into the neural network P(·|s; θ BA ) to restore it to an audio feature and get s’.

[0038] 5.4 Input s and s’ into the similarity calculation method designed in step 3 to get the reward r.

[0039] 5.5 Since r can be regarded as a function of s, s mid , s’, and the neural network parameters θ AB and θ BA , the parameters of the two neural network models can be adjusted by derivation and policy gradient method to maximize the reward r.

[0040] 5.6 Take an image feature from the image feature dataset D B , and repeat steps 5.2-5.5 similar to the audio feature training method.

[0041] 5.7 Take data alternately from D A and D B to train the neural network model until the reward r converges, and complete the model training.

[0042] The beneficial results of the present application are as follows:

[0043] The present application realizes an audio-image cross-modal retrieval, solves the problem that the correspondence relationship of multi-modal data is difficult to be labeled using a traditional machine learning method. In data feature extraction, VGG adopts a small convolution kernel and a pooling layer, a deeper layer and more channels, and more channel numbers represent richer image features, and meanwhile, the increase of the channel number enables more information to be extracted. A feature conversion neural network is constructed, dual learning is introduced into model learning, two cross-modal tasks are trained, and a new idea is provided for a deep learning method of cross-modal data. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a dual learning-based audio-image cross-modal retrieval flowchart provided by the present application.

[0045] Figure 2 It is an audio feature extraction schematic diagram of the present application.

[0046] Figure 3 It is an image feature extraction schematic diagram of the present application.

[0047] Figure 4 It is a neural network structure diagram of audio-image feature mutual conversion of the present application.

[0048] Figure 5 It is a flowchart of training a neural network based on dual learning of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present application more clear, specific embodiments of the present application will be described in detail below with reference to the drawings.

[0050] The audio and image data are subjected to feature extraction, two types of feature conversion neural networks are constructed, and the neural network is trained by using dual learning technology, and finally a method capable of mutual retrieval through audio and image data is realized. As shown in Figure 1 The present application provides a dual learning-based audio-image cross-modal retrieval flowchart, which specifically comprises the following steps:

[0051] Step one, for a piece of audio data A randomly selected from an audio database, after pre-processing, STFT (short-time Fourier transform) is performed, and then a CNN (convolutional neural network) model and a Bi-LSTM (bidirectional long short-term memory network) model are used to extract audio features A x .

[0052] Step two, for an image data B randomly selected from an image database, after pre-processing, a VGG16 model is used to extract image features Bx .

[0053] Step three, construct audio, image similarity algorithm, calculate the similarity between audio features and image features.

[0054] Step four, construct audio-image, image-audio feature conversion neural network model.

[0055] Step five, combined with the similarity calculation method in step three, use dual learning to train the neural network in step four.

[0056] Step six, using the model trained in step five, create an audio-image cross-modal retrieval system, which can search for related images by inputting audio, or find related audio data by inputting images.

[0057] As Figure 2 The audio feature extraction schematic diagram in step one is as follows:

[0058] 1.1 Pre-emphasize, frame and window the randomly selected audio data A for preprocessing, and obtain audio data A1. After pre-emphasis processing, the high frequency resolution of the audio data is further increased; the input audio signal in STFT is required to be stationary, in order to reduce the possible instability in the audio data, the audio data needs to be frame processed, so that the frames are smoothly transitioned; window processing can emphasize the sound waveform and weaken the rest of the waveform.

[0059] 1.2 The preprocessed audio data A1 is input into STFT to obtain its power vector and phase vector. As shown in the following formula: Wherein, x(t), τ and ω are the input audio data A1, the phase vector and the power vector to be extracted. In specific implementation, for 1 min of audio data, the frame number of about 16s is extracted, and two 1024-dimensional phase vectors and power vectors are obtained.

[0060] 1.3 Use the CNN model to extract features from the phase vector and power vector obtained in 1.2, and finally use a fully connected layer to connect them to obtain the audio feature vector A2. After the two 1024-dimensional phase vectors and power vectors obtained in 1.2 are respectively extracted by the CNN model and connected, a 1024-dimensional audio feature vector can be obtained.

[0061] 1.4 Based on the natural time sequence of audio data, the audio vector feature A2 obtained from the CNN model in 1.3 is further input into the LSTM model to extract the audio feature A xThe 1024-dimensional audio feature vector obtained from step 1.3 is further input into a Bi-LSTM model for training, and finally an audio feature with time sequence characteristics is obtained.

[0062] Figure 3 The flowchart for image feature extraction in step two is as follows:

[0063] 2.1 If B is not a three-channel image but a black-and-white image, it is first pre-processed and converted into a three-channel image data B1 before being input into the VGG16 model.

[0064] 2.2 The pre-processed image data B1 is input into the VGG16 model to extract the image feature B x The VGG16 contains 13 convolutional layers, 5 max-pooling layers and 3 fully connected layers. For an image data, the specific content of the extracted feature vector includes: twice 64-kernel convolution, once max-pooling; twice 128-kernel convolution, once max-pooling; three times 256-kernel convolution, once max-pooling; three times 512-kernel convolution, once max-pooling; three times 512-kernel convolution, once max-pooling; three times full connection, and finally a 1000-dimensional image feature vector is obtained.

[0065] Figure 4 The neural network structure diagram for audio-image feature conversion in step four is as follows:

[0066] 4.1 A basic LSTM neural network is constructed as an audio-image feature conversion model, wherein the input vector dimension is the audio feature dimension m in step one, the LSTM layer number is set to a1, and the output vector dimension is set to O1.

[0067] 4.2 A linear fully connected layer is designed, with an input vector dimension of O1 and an output vector dimension of the image feature dimension n. The linear layer is connected with the LSTM layer in 4.1, so that the neural network can convert the m-dimensional audio feature vector into an n-dimensional image feature.

[0068] 4.3 Another linear fully connected layer is constructed, with an input vector dimension of the image data feature n, using a Relu activation function, and an output vector dimension of n'. The image data feature is compressed to a certain extent to reduce the LSTM layer neural network training time.

[0069] 4.4 A basic LSTM neural network is constructed as an image-audio feature conversion model, wherein the input vector dimension is the compressed image feature dimension n' in 4.3, the LSTM layer number is set to a2, and the output vector dimension is set to O2.

[0070] 4.5 Design a linear fully connected layer with input vector dimension O2 and output vector dimension m, which is the feature dimension of audio. Connect this linear layer with the LSTM layer in 4.4, so that the neural network can convert an input image feature vector of dimension n to an audio feature of dimension m.

[0071] Figure 5 The flow chart for training the neural network based on dual learning in step five is as follows:

[0072] 5.1 Let the audio-image feature conversion neural network constructed in step four be P(·|s; θ AB ), and the image-audio feature conversion neural network be P(·|s; θ BA ). Where s is the input of the neural network, and θ AB and θ BA are the parameters of the neural network.

[0073] 5.2 Let the audio feature dataset be D A , and the image feature dataset be D B . First, take an audio feature s from D A , input it into the neural network P(·|s; θ AB ), and get s mid as the intermediate output of dual learning. According to step four, s mid is an image feature.

[0074] 5.3 Input s mid into the neural network P(·|s; θ BA ) to restore it to an audio feature, and get s’.

[0075] 5.4 Input s and s’ into the similarity calculation method designed in step three to get the reward r.

[0076] 5.5 Since r can be regarded as a function of s, s mid , s’, and the neural network parameters θ AB and θ BA , the parameters of the two neural network models can be adjusted by derivation and policy gradient method to maximize the reward r.

[0077] 5.6 Take an image feature from the image feature dataset D B , and repeat steps 5.2-5.5 similarly to the audio feature training method.

[0078] 5.7 Take data alternately from D A and D B to train the neural network model until the reward r converges, and complete the model training.

Claims

1. An audio-image cross-modal retrieval method based on dual learning, characterized in that, Includes the following steps: Step 1: For a randomly selected audio data segment A from the audio database, preprocess it, perform STFT, and then use a CNN model and a Bi-LSTM model to extract audio features A. x ; Step 2: For an image data B randomly selected from the image database, after preprocessing, the VGG16 model is used to extract image features B. x ; Step 3: Construct audio and image similarity algorithms to calculate the similarity between audio features and between image features; Step 4: Construct a neural network model for audio-image and image-audio feature conversion; Step 5: Combining the similarity calculation method in Step 3, use dual learning to train the neural network in Step 4; The specific steps are as follows: 5.1 The audio-image feature conversion neural network constructed in step four is denoted as P(·|s;θ). AB The image-to-audio feature conversion neural network is denoted as P(·|s;θ). BA ); where s is the neural network input, θ AB and θ BA For neural network parameters; 5.2 Let the audio feature dataset be D. A The image feature dataset is D B First, we start with audio-to-image feature conversion, from D... A Extract an audio feature s from the input and input it into the neural network P(·|s;θ). AB ), to obtain s mid As an intermediate output of dual learning, according to step four, s mid Image features; 5.3 will s mid Input into neural network P(·|s;θ) BA This allows us to restore the audio features, resulting in s. ’ ; 5.4 s and s ’ The input is fed into the similarity calculation method designed in step three to obtain the reward r; 5.5 Since r is considered as s, s mid s ’ and neural network parameters θ AB and θ BA Therefore, the parameters of the two neural network models are adjusted by taking the derivative and the policy gradient to maximize the reward r. 5.6 From image feature dataset D B Extract an image feature from the image and repeat steps 5.2-5.5, similar to the audio feature training method. 5.7 Alternating from D A and D B Extract data from the sample and train the neural network model until the reward r converges, thus completing the model training. Step six: Using the model trained in step five, create an audio-image cross-modal retrieval system that can search for related images by inputting audio, or find related audio data by inputting images.

2. The audio-image cross-modal retrieval method based on dual learning as described in claim 1, characterized in that, In step one, feature extraction is performed on the audio data. The specific steps are as follows: 1.1 Perform pre-emphasis, framing, and windowing preprocessing on randomly selected audio data A to obtain audio data A1; 1.2 The preprocessed audio data A1 is processed through an STFT to obtain its power vector and phase vector, as shown in the following equation: Where x(t), τ and ω are the input audio data A1, the phase vector to be extracted, and the power vector, respectively; 1.3 The CNN model is used to extract features from the phase vector and power vector obtained in 1.2, and finally a fully connected layer is used to connect them to obtain the audio feature vector A2; 1.4 Based on the inherent temporal nature of audio data, the audio vector feature A2 obtained from the CNN model in 1.3 is further input into the Bi-LSTM model to extract the audio feature A. x .

3. The audio-image cross-modal retrieval method based on dual learning as described in claim 2, characterized in that, The specific steps of the audio and image similarity calculation method in step three are as follows: 3.1 Audio Feature A x1 With audio feature A x2 Perform data normalization to obtain A ′ x1 And A ′ x2 ; 3.2 The DTW algorithm is used to calculate the similarity between two audio feature sequences. Let feature sequence A be... ′ x1 The feature sequence A is of length M. ′ x2 For a length of N, first construct a DTW regularized path W = (w1, w2, w3, ..., w k ), where Max(|X|,|Y|)≤k≤|X|+|Y|; w k The form is w(i,j), where i represents the i-th index in X and j represents the j-th index in Y, where i and j are monotonically increasing, and the regularized path W must start with w1 = (1,1) and end with w1. k = (|X|,|Y|) as the ending, and each coordinate in X and Y must appear in the regular path; 3.3 Find the shortest regular path D(i,j) = Dist(i,j) + min[D(i-1,j),D(i,j-1)] 3.4 The shortest regular path length is used as the similarity criterion for audio feature sequences. The smaller the path length, the higher the similarity. 3.5 For image similarity, since the correspondence between audio and image data is not very strict, it is only necessary to compare whether two images have similar features, without needing to be highly similar in two-dimensional vector space; Therefore, the same algorithm is used to calculate the similarity of image features.

4. The audio-image cross-modal retrieval method based on dual learning as described in claim 3, characterized in that, The neural network model for audio-image and image-audio feature conversion in step four is as follows: 4.1 Construct a basic LSTM neural network as an audio-image feature conversion model, where the input vector dimension is the same as the audio feature extraction dimension m in step one, the number of LSTM layers is set to α1, and the output vector dimension is set to O1; 4.2 Design a linear fully connected layer with an input vector dimension of O1 and an output vector dimension of n (the feature dimension of the image). Connect this linear fully connected layer to the LSTM layer in 4.1 to realize that the neural network can transform an input m-dimensional audio feature vector into an n-dimensional image feature. 4.3 Construct another linear fully connected layer, with the input vector dimension being the image data features n, using the ReLU activation function, and the output vector dimension being n. ‘ This allows for the compression of image data features, reducing the training time of the LSTM layer neural network. 4.4 Constructing a basic LSTM neural network as an image-to-audio feature conversion model, where the input vector dimension is the same as the feature dimension n of the compressed image described in 4.

3. ‘ The number of LSTM layers is set to α2, and the output vector dimension is set to O2; 4.5 Design a linear fully connected layer with an input vector dimension of O2 and an output vector dimension of the audio feature dimension m; connect the linear layer to the LSTM layer in 4.4 to realize that the neural network can transform an input n-dimensional image feature vector into an m-dimensional audio feature vector.

5. The audio-image cross-modal retrieval method based on dual learning as described in any of claims 1 to 4, characterized in that, In step two, feature extraction is performed on the image data. The specific steps are as follows: 2.1 If B is not a three-channel image, but a black and white image, then it should be preprocessed and converted into three-channel image data B1 before being input into the VGG16 model; 2.2 Input the preprocessed image data B1 into the VGG16 model to extract image features B. x .

Images