A method for image and sound retrieval based on mutual information variational autoencoder

Abstract

The image-sound retrieval method based on mutual information variational autoencoder of the present invention first converts the sound into a spectrogram, extracts features of the sound and the face, and then encodes the extracted face image and spectrogram features through two different mutual information variational autoencoders. The encoding result encodes the face features into modality-shared face features and modality-specific face features, and encodes the sound features into modality-shared sound features and modality-specific sound features; through feature distillation, the modality-specific feature σ is filtered out. f ,σ v , retaining the modal shared feature μ f ,μ v , then resample the features of the two different feature spaces and reconstruct them into new face Z1 and sound Z2 through the same decoder, and enhance μ by mutual information and mean square error loss function. f and μ v The reconstructed image and sound feature pair indicates that the original image and sound are from the same person. The method of the present invention effectively improves the accuracy of voice and face retrieval and the execution efficiency of the system.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, machine vision and transmembrane state retrieval, and in particular to an image and sound retrieval method based on mutual information variational autoencoder. Background Art

[0002] Previous studies have shown that a person's appearance can be inferred from their voice. When we make a phone call, we usually form a rough idea of ​​the other person's appearance based on their voice. This is because age, gender, mouth shape, facial bone structure, all of these affect the sound a person makes. In addition, language, accent, and speed usually reflect a person's ethnicity, region, and cultural characteristics. Artificial intelligence networks make inferences based on the correlation between voice and appearance. The core task of transmodal face and voice retrieval can be regarded as cross-modal biometric recognition and matching, that is, given a facial image or video, determine which of the given audio it corresponds to; or given an audio clip of a voice, determine which of two or more facial images or videos it corresponds to.

[0003] At the same time, image generation technology based on variational auto-encoders (VAEs) is becoming increasingly mature. VAEs are an important type of generative model. Proposed in 2013 by Diederik P. Kingma, Max Welling, and others, VAEs have been widely used in image generation, compression, steganography, and other fields. In the cross-modal field of audio and video, VAEs can split raw data into modality-specific data and modality-shared data. Furthermore, mutual information (MI) is a measure of the degree of mutual dependence between two random variables. Specifically, for two random variables, MI is the "amount of information" gained by observing information about one random variable after obtaining information about the other. This concept originated from information theory. Introducing mutual information into cross-modal audio and image retrieval tasks aims to improve retrieval accuracy and efficiency.

[0004] Traditional image and sound retrieval technology has the following two main problems and the difficulty in solving the above problems is: First, it is difficult to align image and sound features. Traditional methods usually use contrastive learning methods to narrow the distance between sounds and images from the same source (the same person) when performing sound and image retrieval, while separating sound and image features that do not belong to the same source. This requires carefully matched positive and negative sample pairs. The selection of these sample pairs must meet both the requirements of breadth and the uniqueness of special samples, which are difficult and costly to produce; second, previous sound and image matching methods directly match the features extracted from images and sounds, ignoring the "noise" in the features, that is, the modality-specific features. Modality-specific features are features from both sound and image modalities that cannot be used for matching. The retrieval accuracy will be further reduced due to this.

[0005] Solving the above problems is of great significance to improving retrieval accuracy and efficiency. Summary of the Invention

[0006] The present invention provides an image and sound retrieval method based on mutual information variational autoencoder. It is a cross-modal feature extraction neural network image and sound mutual retrieval method combined with mutual information variational autoencoder. When performing mutual retrieval of face images and sounds, VAE encoding and feature distillation technology are used to obtain modal shared features, mutual information loss, bulldozer loss and other technical means, which effectively improves the accuracy of sound and face retrieval and the system execution efficiency.

[0007] The technical solutions of the present invention are as follows:

[0008] The image and sound retrieval method based on mutual information variational autoencoder of the present invention comprises the following steps: S1. preprocessing the face image and sound, converting the sound into a spectrogram and extracting sound features; S2. sending the face image and sound into a feature extraction network for feature extraction, encoding the extracted features through a variational autoencoder (VAE), and the encoded mean corresponds to the modal shared image feature μ f and modal shared acoustic features μ v , the variance corresponds to the modality-specific features, and then through feature distillation, the modality-specific features are discarded and only the modality-shared image features μ are retained. f and modal shared acoustic features μ v ; S3. Share the output modality of S2 with the image feature μ f and modal shared acoustic features μ v Splice and decode and reconstruct through the same universal variational autoencoder (VAE) decoder to obtain the reconstructed face and voice pairs Z1, Z2; S4. Calculate the shared image feature μ after encoding and feature distillation f and shared sound features μ vMutual information loss value, bulldozer distance loss, calculate the feature pair μ f , μ v The mean square error loss between the reconstructed face and voice pairs Z1 and Z2 is calculated, the KL divergence loss value is calculated, and the weighted average of the above losses is calculated as the final loss function of the network; S5. The output of S4 is used as the optimization target, and the Adam optimizer is used for iterative optimization.

[0009] Preferably, in the above-mentioned image-sound retrieval method based on mutual information variational autoencoder, in step S1, the modality-specific feature is the modality-specific image feature σ f and the mode-specific sound characteristics σ v .

[0010] Preferably, in the above-mentioned image-sound retrieval method based on mutual information variational autoencoder, in step S1, the face image is randomly rotated and sheared, the sound waveform is standardized, and a spectrogram is generated by fast Fourier transform, and then the spectrogram is sharpened to suppress background noise.

[0011] Preferably, in the above-mentioned image-sound retrieval method based on mutual information variational autoencoder, in step S2, the facial image and spectrogram features after feature extraction are encoded by two different mutual information variational autoencoders.

[0012] Preferably, in the above-mentioned image-sound retrieval method based on mutual information variational autoencoder, in step S4, μ is enhanced by mutual information and mean square error loss function. f and μ v The reconstructed image-sound feature pair indicates that the original image and sound come from the same person.

[0013] According to the technical solution of the present invention, the beneficial effects produced are:

[0014] The image and sound retrieval method based on mutual information variational autoencoder of the present invention uses an unsupervised training method and does not rely on carefully designed positive and negative sample pairs, which reduces the data preprocessing operations in the training process, reduces the algorithm complexity, and greatly saves training costs; at the same time, the method adopts a brain-inspired algorithm to simulate the working mode of the cerebral cortex when processing auditory and visual signals - the cerebral cortex that processes visual signals and the cerebral cortex that processes auditory signals partially overlap, and the image features and sound features are further decomposed into modality-shared features and modality-specific features through VAE (variational autoencoder) and then distilled and matched, successfully avoiding the influence of modality-specific features on the matching effect and greatly increasing the accuracy of the retrieval results.

[0015] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below with reference to the accompanying drawings and through specific embodiments: BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the specific implementation or the description of the prior art.

[0017] Figure 1 is a flow chart of the image and sound retrieval method based on mutual information variational autoencoder of the present invention;

[0018] Figure 2 It is a network structure diagram involved in the method of the present invention. DETAILED DESCRIPTION

[0019] To make the purpose, technical methods and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not intended to limit the present invention.

[0020] The image and sound retrieval method based on mutual information variational autoencoder of the present invention uses deep learning technology to learn facial images and speech spectra. Figure 2 The corresponding relationship between them. Figure 1 and Figure 2 As shown, the method, from start to finish, specifically includes the following steps:

[0021] S1. Preprocess the facial image and sound, converting the sound into a spectrogram and extracting sound features. Specifically, the facial image and spectrogram are preprocessed to enhance the data. The facial image is subjected to random transformations such as rotation and shearing. The sound waveform is normalized and a spectrogram is generated using a Fast Fourier Transform. The spectrogram is then sharpened to suppress background noise.

[0022] S2. The face image and sound are fed into the feature extraction network for feature extraction. The extracted features are encoded by a variational autoencoder (VAE). The encoded mean corresponds to the modality-shared feature, and the variance corresponds to the modality-specific feature. The modality-shared feature is the modality-shared image feature μ f and modal shared acoustic features μ v , the variance corresponds to the modality-specific feature, which is the modality-specific image feature σ f and the mode-specific sound characteristics σ v ; Then, through feature distillation, the modality-specific features are discarded and only the modality-shared features μ are retained f and μ v Perform subsequent matching operations.

[0023] S3. Share the output modal features μ of S2 f and μ v The face and voice are concatenated and decoded and reconstructed through the same VAE universal decoder to obtain the reconstructed face and voice pair (Z1, Z2). (That is, the reconstructed face is recorded as Z1 and the voice is recorded as Z2).

[0024] S4. Calculate the shared image features μ after encoding and feature distillation f Shares features μ with sound v Mutual information loss value, bulldozer distance loss, calculate the feature pair (μ f , μ v ) and the mean square error loss between the reconstructed face and voice pair (Z1, Z2), calculate the KL divergence loss value, and calculate the weighted average of the above losses as the final loss function of the network.

[0025] In this step, the feature pairs (μ f , μ v ) and the mean square error loss between the reconstructed face and voice pair (Z1, Z2), i.e., the Euclidean distance, while calculating the shared image feature μ after encoding and feature distillation. f Shares features μ with sound v The mutual information loss value, bulldozer distance loss, and KL divergence loss value are then weighted averaged to form the final loss function of the network.

[0026] S5. Take the output of S4 as the optimization target and use the Adam optimizer for iterative optimization.

[0027] The image-sound retrieval method based on mutual information variational autoencoder of the present invention first converts the sound into a spectrogram, extracts features of the sound and the face, and then encodes the extracted face image and spectrogram features through two different mutual information variational autoencoders. The encoding result encodes the face features into modality-shared face features and modality-specific face features, and encodes the sound features into modality-shared sound features and modality-specific sound features; through feature distillation, the modality-specific feature σ is filtered out. f , σ v , retaining the modal shared feature μ f , μ v After that, the features of the two different feature spaces are resampled and reconstructed into new face Z1 and voice Z2 through the same universal decoder, and μ is enhanced by mutual information and mean square error loss function. f and μ v The reconstructed image-sound feature pair indicates that the original image and sound come from the same person.

[0028] The method of the present invention uses the mutual information algorithm to measure the correlation between two random distributions, introduces relevant concepts of information theory into the field of deep learning, proposes the concept of mutual information loss, and improves the accuracy of matching; refers to the patterns of information perception in different visual and auditory areas of the cerebral cortex, proposes the concept of "modal sharing" matching, extracts features from sound and image, and then uses VAE to encode features. After encoding, the features are distilled to preserve the features shared by each modality and discard the features specific to the modality; introduces the bulldozer distance loss function, and uses the sinkhorn (an optimal transmission algorithm) algorithm to implement the bulldozer loss; through multiple experiments, it is proved that the KL divergence coefficient is optimal when it is 0.001.

[0029] The specific implementation scheme of the image and sound retrieval method based on mutual information variational autoencoder proposed in this invention is described as follows: Figure 2 .

[0030] 1) In the preprocessing stage, the training data is prepared first. Facial images are in ".jpg" format, and sounds are in ".wav" format. Then, sound processing software is used to read the ".wav" files and convert them into corresponding sound waves. Then, the sound waves are converted into a spectrogram using a fast Fourier transform. The horizontal axis of the spectrogram represents time, the vertical axis represents frequency, and the coordinate values ​​represent the strength of the speech.

[0031] 2) Use pre-trained feature extraction network to extract features from face images and sounds. ResNet50 is used for face image feature extraction network, and VGG-M is used for sound feature extraction network. The final output is 512-dimensional feature vector V f , V v . V f and V v Represent the feature vector of the face image and the feature vector of the spectrogram respectively. f and V v They are sent to the VAE sound encoder and VAE image encoder for feature encoding respectively. After encoding, the modality-shared face image features, modality-shared sound features, modality-specific face image features, and modality-specific sound features are obtained, which are denoted as μ f , μ v , σ f , σ v . Feature distillation is then performed to retain the modality shared feature μ f , μ v , filter out the modality-specific features σ f , σ v .

[0032] 3) According to the annotation, the modal shared feature μ of the output of S2 is f , μ vSpliced ​​into a feature pair (μ f , μ v ), μ is decoded by the same universal decoder f , μ v Decode and reconstruct the face image using the reparameterization algorithm in the VAE variational autoencoder. The reconstructed face image is denoted as Z1, the voice image is denoted as Z2, and the face-voice pair is denoted as (Z1, Z2).

[0033] 4) Calculate the feature μ f and μ v Mutual information loss MiLoss, calculate μ f and μ v The bulldozer distance loss WassLoss is calculated to calculate the reconstructed data pairs (Z1, Z2) and (μ f , μ v ) between the two VAE encoders encoder-V and encoder-A, and calculate the KL divergence loss value KLloss of each. Finally, the weighted total loss function TotalLoss is calculated;

[0034] Where KLLoss=KLV+KLA,

[0035]

[0036] Where σ and μ are the variance and mean of the feature distribution in the corresponding feature space, respectively.

[0037] Right now

[0038]

[0039]

[0040] Mutual information MiLoss can be expressed by the following formula:

[0041] H(X,Y)=H(X)+H(Y|X)=H(Y)+H(Y|X)

[0042] In the above formula, X and Y represent μ respectively. f and μ v

[0043] The bulldozer distance loss W measures the distance between two probability distributions and is defined as follows:

[0044]

[0045] Π(P1, P2) is the set of all possible joint distributions of P1 and P2. For each possible joint distribution γ, we can sample (x, y)~γ to obtain a sample x and y, and calculate the distance between the two samples ||xy||. Therefore, we can calculate the expected value of the distance between the two samples under the joint distribution γ, E(x, y)~γ[||xy||]. The lower bound of this expected value in all possible joint distributions, infγ~Π(P1, P2)E(x, y)~γ[||xy||], is the bulldozer distance. In practice, the SinkHorn (an optimal transmission) algorithm is used to approximate the bulldozer distance loss WassLoss.

[0046] 5) Use two Adam optimizers to optimize the above losses.

[0047] This preferred embodiment was trained and tested on the celebrity speech datasets VoxCeleb1 and VoxCeleb2, and the experimental results were evaluated using the currently recognized evaluation standard mRR (mean retrieval rank). The actual test results are shown in Table 1, indicating that the method proposed in the present invention has achieved the current leading detection accuracy.

[0048] Table 1 is a table of experimental results for evaluating the method of the preferred embodiment of the present invention.

[0049]

[0050] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the scope of protection of the present claims. Other embodiments and combinations of implementations of the present invention are also within the scope of protection of the present invention.

Claims

1. An image-sound retrieval method based on mutual information variational autoencoder, characterized in that: The following steps are involved: S1. Preprocess the facial image and voice, convert the voice into a spectrogram and extract voice features; S2. The face image and the sound are fed into a feature extraction network for feature extraction. The extracted features are encoded by a variational autoencoder (VAE). The encoded mean corresponds to the modal shared image feature μ f and modal shared acoustic features μ v , the variance corresponds to the modality-specific features, and then through feature distillation, the modality-specific features are discarded and only the modality-shared image features μ are retained. f and the modes share the acoustic feature μ v ; S3. Convert the output of S2 to the modality shared image feature μ f and the modes share the acoustic feature μ v The faces and voices are concatenated and decoded and reconstructed through the same universal variational autoencoder (VAE) decoder to obtain the reconstructed face and voice pairs Z1 and Z2. S4. Calculate the shared image features μ after encoding and feature distillation f and shared sound features μ v Mutual information loss value, bulldozer distance loss, calculate the feature pair μ f , μ v The mean square error loss between the reconstructed face and voice pairs Z1 and Z2 is calculated, the KL divergence loss value is calculated, and the weighted average of the above losses is calculated as the final loss function of the network; S5. Take the output of S4 as the optimization target and use the Adam optimizer for iterative optimization.

2. The image-sound retrieval method based on mutual information variational autoencoder according to claim 1, characterized in that: In step S1, the modality-specific feature is the modality-specific image feature σ f and the mode-specific sound feature σ v .

3. The image-sound retrieval method based on mutual information variational autoencoder according to claim 1, characterized in that: In step S1, the face image is subjected to random transformations such as rotation and shearing, the sound waveform is standardized, and the spectrogram is generated by fast Fourier transform, and then the spectrogram is sharpened to suppress background noise.

4. The image-sound retrieval method based on mutual information variational autoencoder according to claim 1, characterized in that: In step S2, the facial image and spectrogram features after feature extraction are encoded by two different mutual information variational autoencoders.

5. The image-sound retrieval method based on mutual information variational autoencoder according to claim 1, characterized in that: In step S4, the modality shared image feature μ is enhanced by mutual information and mean square error loss function. f and the modes share the acoustic feature μ v The reconstructed image-sound feature pair indicates that the original image and sound come from the same person.

Images