Cross-modal voice and face association method based on heterogeneous Hash network

Through a deep residual alignment network based on heterogeneous hash network, the speech and face features are mapped to the 128-dimensional hash code space, solving the problem of high overfitting and computing occupancy in the existing technology, and achieving more efficient Cross-modal feature alignment and matching effects.

CN118918924AActive Publication Date: 2024-11-08XIAN UNIV OF POSTS & TELECOMM

Patent Information

Application Number
CN202411097705.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-08
Estimated Expiration
2044-08-12

AI Technical Summary

Technical Problem

The existing cross-modal voice-face matching algorithm is prone to overfitting on training data, lacks generalization capabilities, and has high computing and memory usage when processing large-scale data, and has poor feature alignment effects.

Method used

Using a cross-modal deep residual alignment network based on heterogeneous hash network, the speech and face features are mapped to the 128-dimensional hash code space through multiple fully connected layers and batch normalization layers, using triple loss and The similarity loss optimization feature representation between modals reduces the computational and memory burden and improves generalization capabilities.

Benefits of technology

It significantly improves the accuracy of cross-modal matching, retrieval and verification, reduces computing and memory requirements, and improves the generalization ability and feature alignment of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918924A_ABST
    Figure CN118918924A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of communication, and discloses a cross-modal voice and face association method based on a heterogeneous Hash network, and the specific technical scheme is as follows: a cross-modal deep residual error alignment network based on Hash learning is provided, the network maps input features into Hash codes with fixed lengths, and the Hash codes are used as Hash codes; the method effectively reduces the use of the feature dimension and memory, greatly improves the calculation efficiency through hashing, enables the feature representation capability to be stronger through a deep network integrated with a residual structure, has good generalization, enables the Hamming distance of the voice and face features of the same identity to be closer in the Hash space through minimizing a loss function, and improves the recognition efficiency. The distance between voice and face features of different identities is farther, so that feature alignment of the voice and the face is achieved, cross-modal matching, verification and retrieval tasks are completed, and compared with an existing method, the accuracy of the model in the cross-modal matching, retrieval and verification tasks is obviously improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a cross-modal speech and face association method based on a heterogeneous hash network. Background Art

[0002] In modern society, identity recognition is a core requirement in many fields such as security, criminal investigation, monitoring, and personal services. Although traditional identification methods, such as those based on passwords, identity documents, or biometrics (fingerprints, facial recognition, etc.), meet the needs of identity authentication to a certain extent, they may have limitations in some cases, such as in scenarios where there is a lack of physical contact or other information.

[0003] Most of the existing cross-modal speech-face matching algorithms use feature extraction networks to extract features. In the feature alignment part, one or two fully connected layers are used to map the two modal features to the same feature space, and the loss function is minimized to make the class compact and the class expansion. However, using fewer fully connected layers may cause the model to overfit on the training data, and the generalization ability is insufficient, and it is not enough to achieve effective feature alignment. In addition, when the original features are mapped to the same dimensional space through the network, each sample feature is represented in the form of high-dimensional floating point numbers, which will undoubtedly increase memory usage and computational burden when processing large-scale data sets. Summary of the invention

[0004] In order to solve the technical problems existing in the prior art, the present invention provides a cross-modal deep residual alignment network based on hash learning. The network maps input features into hash codes of fixed length, effectively reducing the feature dimension and memory usage. At the same time, hashing greatly improves the computational efficiency. The deep network with integrated residual structure makes the feature representation capability stronger and has good generalization. By minimizing the loss function, the network can make the Hamming distance of speech and facial features of the same identity closer in the hash space, and the distance between speech and facial features of different identities farther, so as to achieve feature alignment of speech and face and complete cross-modal matching, verification and retrieval tasks.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is: a cross-modal speech and face association method based on heterogeneous hash network, the specific steps are as follows:

[0006] Constructing the dataset: Use the preprocessed VoxCeleb dataset to train and test the network. The voice face dataset includes voice clips and face images of 1225 identities, represented as Where i represents the i-th identity person, N = 1225. For each identity, the internal representation is in, represents the jth voice clip of the ith identity, where j ranges from 1 to m, i.e., the ith identity has a total of m voice samples; similarly, The jth face image frame represents the i-th identity, and j ranges from 1 to n, that is, the i-th identity has a total of n face image frames. In order to make a fair comparison, it is divided into training set, test set and validation set according to the division method of previous work.

[0007] Loading the dataset: Since triplet loss is needed later to back-propagate and optimize network parameters, a sufficient number of triplets (anchor, positive, negative) need to be selected in each iteration. Therefore, a sampler is added to ensure that the model can receive diverse samples during training. Each iteration contains samples of multiple different identities, and each identity has a fixed number of instances, so as to select more effective triplets for subsequent calculation of triplet loss. During network training, at each iteration, the data is presented in the form of a tuple, specifically Among them, v represents a voice clip, f represents a face image, and l is a label. If l i = l j , indicating v i ,f i With v j ,f j Belong to the same person, if l i ≠l j , are people of different identities, and batch represents the number of tuples in each iteration of the network.

[0008] Constructing speech feature extraction network A v (·): The ECAPA-TDNN network pre-trained on the VoxCeleb2 dataset is used as the speech feature extraction network. For the speech segment, its MFCC features are first extracted and normalized as the input of the ECAPA-TDNN network to extract features, and a 192-dimensional vector is output as the final speech feature.

[0009] Constructing face feature extraction network A F (·): Use the Inception-V1 network pre-trained on the VGGFace2 dataset as the face feature extraction network. The face image will output a 512-dimensional vector as the final face feature through the network.

[0010] Construct a cross-modal feature alignment network φ(·,θ)={φv(·,θv),φf(·,θf)}: The features of the two modalities obtained in the feature extraction stage are used as the input of the network, and the network automatically inputs the two modal features into its corresponding feature alignment subnetwork. In terms of network structure design, multiple fully connected layers FC(·) are used to achieve feature alignment. In order to reduce internal covariate shift, except for the last fully connected layer, each fully connected layer is followed by a batch normalization layer BN(·). Batch normalization can be expressed as follows:

[0011]

[0012] Among them, x i represents the features after the fully connected layer, μ B Represents the mean of all sample features in a batch on each feature channel, σ B Represents the variance of all sample features in a batch on each feature channel, and ε is a small constant for numerical stability to prevent division by zero. γ and β are learnable parameters used to scale and translate the normalized features to restore the expressive power of the model and allow the network to learn the best representation of the normalized features. At the same time, the batch-normalized features of the first layer of the network are residually connected with the features of the last layer, fusing the early layer and deep layer features, allowing the features captured by the early layer to be effectively integrated with the high-level features learned by the deep layer. The two modalities of the early layer are trained separately, and the weights of the two modalities are shared in the later layers, so that the cross-modal network can learn a more universal feature representation. Through this network, the original 512-dimensional face features and 192-dimensional voice features are mapped to a 128-dimensional space, and a hash function layer sgn(·) is added to the last layer of the network so that the network outputs a fixed-length hash code. Among them, sgn(·) is a sign function that returns 1 if the element is a positive number, otherwise it returns -1. The learned hash code representation can make the Hamming distance between the voice and face of the same identity closer in the hash space, and the distance between the voice and face of different identities farther. The final output of the network is:

[0013] V=sgn(φ v (A v (v i ),θ v )

[0014] F=sgn(φ f (A f (f i ),θ f )

[0015] Among them, φ v represents the speech feature alignment subnetwork in the constructed cross-modal feature alignment network, φ fA represents the face feature alignment subnetwork in the constructed cross-modal feature alignment network. v (v i ) is the speech feature obtained by the feature extraction network, A f (f i ) is the facial feature obtained through the feature extraction network, θ v ,θ f They represent the parameters learned by the two sub-networks in the cross-modal feature alignment network, and F and V are the hash codes of the two modalities finally obtained.

[0016] Construct loss function: According to the proposed triple construction method, N triplets (anchor, positive, negative) are constructed in a batch. The hash codes generated by the cross-modal feature alignment network will be used to calculate the triple loss to optimize the feature representation and enhance the model's discrimination ability. It is specifically expressed as:

[0017]

[0018] Among them, d(a i ,p i ) represents the Hamming distance between the anchor point and the positive sample, d(a i ,n i ) represents the Hamming distance between the anchor point and the negative sample. Margin is a hyperparameter used to control the distance difference between the positive and negative sample pairs. When calculating the loss, only the triplets that violate the boundary margin are calculated. This makes the model focus more on samples that are difficult to distinguish and ignores triplets that are very easy to distinguish. This helps reduce the computational burden and makes the training process more efficient, thereby improving model performance and generalization capabilities. Given the non-differentiability of the bitwise operation of the Hamming distance, the inner product is used to approximate the Hamming distance to ensure the differentiability of the loss function throughout the network training process, thereby effectively optimizing the feature representation and improving the alignment accuracy of the model.

[0019] In addition, based on the characteristics of hash codes, the inter-modal similarity loss is proposed, and the true similarity matrix S is constructed as follows:

[0020]

[0021] s ij Represents two samples v i ,f i If the labels of the two samples are consistent, the similarity is 1, otherwise, if the labels of the two samples are inconsistent, the similarity is -1. ij = 1, the hash code elements at the same position obtained by network mapping should be as similar as possible, otherwise, as dissimilar as possible; that is, when s ij =1, network output and The similarity should be as close to 1 as possible, otherwise, it tends to -1. Specifically, the inner product of two vectors is used to calculate the predicted similarity matrix Y:

[0022]

[0023] Where k is the hash code length, represents the hash code of the i-th speech sample obtained through network mapping, Represents the hash code of the jth face sample obtained by mapping. The similarity loss is expressed as follows:

[0024]

[0025] Among them, ||·|| is used to represent the distance between two points, using the Euclidean measurement method.

[0026] Overall loss function:

[0027]

[0028] Training: Since the feature extraction networks of the two modalities in the feature extraction stage use pre-trained networks, only the cross-modal feature alignment network is trained. The small batch training method is selected, and one batch of data is trained each time. The feature extraction network inputs the two modal features into their respective feature extraction subnetworks to extract features, and the extracted features are used by the cross-modal feature alignment network to learn a compact hash code representation. After multiple iterations, that is, backpropagation after calculating the loss function and updating the network parameters to minimize the loss function, the hash code output by the cross-modal feature alignment network can make the voice and face of the same identity as close as possible in the hash space, and the different identities as far away as possible. The hash codes of the two modalities learned in each iteration are expressed as:

[0029] B=[b1,b2,b3…,b n ] T ∈{+1,-1} n×k

[0030] Among them, B is the hash code matrix of speech or face obtained by network mapping, n is the batch size, which indicates the number of samples processed in each iteration; each element b in matrix B i , whether it is a speech or face sample, it is a hash code of length k, and each dimension of each hash code takes the value of +1 or -1.

[0031] The present invention proposes a cross-modal speech-face association technology based on hash learning, which can retrieve face pictures of the same identity from an image database according to an unknown speech for subsequent recognition tasks. In-depth studies in fields such as biology, cognitive science, and neuroscience have revealed the subtle connection between human voices and facial features. The vibration of the vocal cords and the facial structure of a person jointly act on the generation and transmission of sound, indicating that there is a certain correlation between the physiological / behavioral characteristics contained in the sound and facial features. This correlation provides a solid scientific basis for inferring facial features from sound; through experimental verification, the accuracy of the model of the present invention in cross-modal matching, retrieval, and verification tasks has been significantly improved compared with existing methods.

[0032] The present invention proposes a cross-modal deep residual alignment network based on hash learning. Using multiple linear layers and batch normalization layers, it can more effectively achieve the mapping from low-level features to high-level semantic features. On this basis, residual connections are used to fuse early layers with deep features. The model can learn low-level detailed features and high-level abstract features, and at the same time can eliminate the problem of gradient disappearance caused by the relatively deep network. During the network training process, the early layers are trained separately for the two modalities, and the later layers share weights for the two modalities. In this way, the multi-modal network can effectively utilize the unique information of each modality, while achieving cross-modal information integration and feature learning. Finally, a hash function is added to the last layer of the network to map the features into a compact hash code form, greatly improving the computational efficiency.

[0033] The present invention adopts a new method for constructing triplets. The present invention selects a more balanced strategy, that is, selects sample pairs where the distance between the anchor and the positive sample is less than the distance between the anchor and the negative sample, and this distance difference is less than a preset boundary value (margin). Specifically, the present invention selects negative samples that satisfy the condition 0 < d(a i , n i ) - d(a i , p i ) < margin. In this way, the present invention ensures that the negative samples in the triplets are neither too easy to distinguish nor too difficult, avoiding the collapse phenomenon during the network training process, that is, preventing the model from falling into local minima, thereby promoting the model to learn the distance metric more evenly in the feature space. Through experimental verification, the accuracy of the model of the present invention in cross-modal matching, retrieval, and verification tasks has been significantly improved compared with existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is the overall network framework diagram of the present invention.

[0035] Figure 2 is the t-SNE visualization diagram of the hash code of the present invention, Figure 2 (a) is the feature distribution diagram before learning. Figure 2 (b) is the feature distribution diagram after learning. DETAILED DESCRIPTION

[0036] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] like Figure 1 As shown in FIG. 1 , a cross-modal speech and face association method based on a heterogeneous hash network is described. The specific steps are as follows:

[0038] Constructing the dataset: Use the preprocessed VoxCeleb dataset to train and test the network. The voice face dataset includes voice clips and face images of 1225 identities, represented as Where i represents the i-th identity person, N = 1225. For each identity, the internal representation is in, represents the jth voice clip of the ith identity, where j ranges from 1 to m, i.e., the ith identity has a total of m voice samples; similarly, The jth face image frame represents the i-th identity, and j ranges from 1 to n, that is, the i-th identity has a total of n face image frames. In order to make a fair comparison, it is divided into training set, test set and validation set according to the division method of previous work.

[0039] Loading the dataset: Since triplet loss is needed later to back-propagate and optimize network parameters, a sufficient number of triplets (anchor, positive, negative) need to be selected in each iteration. Therefore, a sampler is added to ensure that the model can receive diverse samples during training. Each iteration contains samples of multiple different identities, and each identity has a fixed number of instances, so as to select more effective triplets for subsequent calculation of triplet loss. During network training, at each iteration, the data is presented in the form of a tuple, specifically Among them, v represents a voice clip, f represents a face image, and l is a label. If l i = l j , indicating v i ,f i With v j ,l j Belong to the same person, if l i ≠l j , are people of different identities, and batch represents the number of tuples in each iteration of the network.

[0040] Constructing speech feature extraction network A v (·): The ECAPA-TDNN network pre-trained on the VoxCeleb2 dataset is used as the speech feature extraction network. For the speech clip, its Mel-spectrogram features are first extracted and normalized as the input of the ECAPA-TDNN network to extract features, and a 192-dimensional vector is output as the final speech feature.

[0041] Constructing face feature extraction network A f (·): Use the Inception-V1 network pre-trained on the VGGFace2 dataset as the face feature extraction network. The face image will output a 512-dimensional vector as the final face feature through the network.

[0042] Construct a cross-modal feature alignment network φ(·,θ)={φv(·,θv),φf(·,θf)}: The features of the two modalities obtained in the feature extraction stage are used as the input of the network, and the network automatically inputs the two modal features into its corresponding feature alignment subnetwork. In terms of network structure design, multiple fully connected layers FC(·) are used to achieve feature alignment. In order to reduce internal covariate shift, except for the last fully connected layer, each fully connected layer is followed by a batch normalization layer BN(·). Batch normalization can be expressed as follows:

[0043]

[0044] Among them, x i represents the features after the fully connected layer, μ B Represents the mean of all sample features in a batch on each feature channel, σ BRepresents the variance of all sample features in a batch on each feature channel, and ε is a small constant for numerical stability to prevent division by zero. γ and β are learnable parameters used to scale and translate the normalized features to restore the expressive power of the model and allow the network to learn the best representation of the normalized features. At the same time, the batch-normalized features of the first layer of the network are residually connected with the features of the last layer, fusing the early layer and deep layer features, allowing the features captured by the early layer to be effectively integrated with the high-level features learned by the deep layer. The two modalities of the early layer are trained separately, and the weights of the two modalities are shared in the later layers, so that the cross-modal network can learn a more universal feature representation. Through this network, the original 512-dimensional face features and 192-dimensional voice features are mapped to a 128-dimensional space, and a hash function layer sgn(·) is added to the last layer of the network so that the network outputs a fixed-length hash code. Among them, sgn(·) is a sign function that returns 1 if the element is a positive number, otherwise it returns -1. The learned hash code representation can make the Hamming distance between the voice and face of the same identity closer in the hash space, and the distance between the voice and face of different identities farther. The final output of the network is:

[0045] V=sgn(φ v (A v (v i ),θ v )

[0046] F=sgn(φ f (A f (f i ),θ f )

[0047] Among them, φ v represents the speech feature alignment subnetwork in the constructed cross-modal feature alignment network, φ f A represents the face feature alignment subnetwork in the constructed cross-modal feature alignment network. v (v i ) is the speech feature obtained by the feature extraction network, A f (f i ) is the facial feature obtained through the feature extraction network, θ v ,θ f They represent the parameters learned by the two sub-networks in the cross-modal feature alignment network, and F and V are the hash codes of the two modalities finally obtained.

[0048] Construct loss function: According to the proposed triple construction method, N triplets (anchor, positive, negative) are constructed in a batch. The hash codes generated by the cross-modal feature alignment network will be used to calculate the triple loss to optimize the feature representation and enhance the model's discrimination ability. It is specifically expressed as:

[0049]

[0050] Among them, d(a i ,p i ) represents the Hamming distance between the anchor point and the positive sample, d(a i ,n i ) represents the Hamming distance between the anchor point and the negative sample. Margin is a hyperparameter used to control the distance difference between the positive and negative sample pairs. When calculating the loss, only the triplets that violate the boundary margin are calculated. This makes the model focus more on samples that are difficult to distinguish and ignores triplets that are very easy to distinguish. This helps reduce the computational burden and makes the training process more efficient, thereby improving model performance and generalization capabilities. Given the non-differentiability of the bitwise operation of the Hamming distance, the inner product is used to approximate the Hamming distance to ensure the differentiability of the loss function throughout the network training process, thereby effectively optimizing the feature representation and improving the alignment accuracy of the model.

[0051] In addition, based on the characteristics of hash codes, the inter-modal similarity loss is proposed, and the true similarity matrix S is constructed as follows:

[0052]

[0053] s ij Represents two samples v i ,f i If the labels of the two samples are consistent, the similarity is 1, otherwise, if the labels of the two samples are inconsistent, the similarity is -1. ij = 1, the hash code elements at the same position obtained by network mapping should be as similar as possible, otherwise, as dissimilar as possible; that is, when s ij =1, network output and The similarity should be as close to 1 as possible, otherwise, it tends to -1. Specifically, the inner product of two vectors is used to calculate the predicted similarity matrix Y:

[0054]

[0055] Where k is the hash code length, represents the hash code of the i-th speech sample obtained through network mapping, Represents the hash code of the j-th face sample obtained by mapping, and the similarity loss is expressed as follows:

[0056]

[0057] Among them, ||·|| is used to represent the distance between two points, using the Euclidean measurement method.

[0058] Overall loss function:

[0059]

[0060] Training: Since the feature extraction networks of the two modalities in the feature extraction stage use pre-trained networks, only the cross-modal feature alignment network is trained. The small batch training method is selected, and one batch of data is trained each time. The feature extraction network inputs the two modal features into their respective feature extraction subnetworks to extract features, and the extracted features are used to learn a compact hash code representation in the cross-modal feature alignment network. After multiple iterations, that is, back propagation and updating of network parameters after calculating the loss function to minimize the loss function, the hash code output by the cross-modal feature alignment network can make the voice and face of the same identity as close as possible in the hash space, and the different identities as far away as possible. The hash codes of the two modalities learned in each iteration are expressed as:

[0061] B=[b1,b2,b3…,b n ] T ∈{+1,-1} n×k

[0062] Where B is the hash code matrix of the voice or face obtained by network mapping, and n is the batch size, which indicates the number of samples processed in each iteration. Each element b in matrix B i , whether it is a speech or face sample, it is a hash code of length k, and each dimension of each hash code takes the value of +1 or -1.

[0063] All experiments in this paper were completed on the NVIDIA GeForce RTX 4090 graphics card, and the PyTorch deep learning framework and Python 3.9 environment were selected as the development basis; during the model training process, the Adam optimizer was used, and the learning rate was set to 1e-3, and the margin parameter was set to 1 to optimize the learning ability and generalization performance of the model. Finally, the Voxceleb dataset was used, divided into training set, validation set and test set, and the performance of the network was tested on cross-modal matching, retrieval and validation tasks.

[0064] 1. Cross-modal 1:2 matching: select a sample in the speech modality and two samples in the face image modality to find the face image with the same identity as the speech sample; conversely, use the face image modality as a query to find the speech segment with the same identity as the face image sample, and verify the performance of the model by calculating the matching accuracy, as shown in Table 1. Comparing the accuracy of the present invention with the results of other workers, the model of the present invention has been significantly improved.

[0065] Table 1 Accuracy of cross-modal 1:2 matching task

[0066] method VF FV SVHF 81.00 79.50 FV-CME 78.10 77.80 LAFV 78.20 78.60 PINs 83.80 DIMNet-I 83.45 83.52 LJE 85.18 85.42 0urs 85.58 85.68

[0067] 2. Cross-modal verification: Given a speech sample and a face sample, determine whether they belong to the same identity. This task uses the area under the curve (AUC) as the evaluation indicator. The comparison with previous work is shown in Table 2.

[0068] Table 2 AUC values ​​of cross-modal verification tasks

[0069] method AUC PINs 78.50 SSNet 78.80 DIMNet-I 82.50 Bi-Pcm-FST 85.00 Ours 86.93

[0070] 3. Cross-modal retrieval: Given a sample to be tested of one modality, it is necessary to query the positive samples that match the sample to be tested from the total dataset. This task will use mean average precision (mAP) as the evaluation indicator, and the comparison with previous work is shown in Table 3.

[0071] Table 3 Performance of mAP in cross-modal retrieval

[0072] method VF FV FV-CME 1.96 2.18 VFMR3 5.00 DIMNet-I 4.25 4.17 Bi-Pcm-FST 6.36 6.04 0urs 7.58 7.98

[0073] 4. Ablation experiment: The present invention compares the performance of networks with only two layers and multiple layers in cross-modal matching, verification and retrieval tasks. In addition, the performance of networks using cross-modal similarity loss and not using this loss is compared, as shown in Table 4.

[0074] Table 4 Evaluation of different deep networks on cross-modal matching, verification, and retrieval tasks

[0075]

[0076] In previous studies, floating point numbers were used to align the features of samples, and the similarity between samples was measured by calculating cosine similarity, Euclidean distance and other methods. The smaller the distance between samples, the higher the similarity between them; conversely, the larger the distance, the lower the similarity. The present invention proposes to map samples to hash codes of fixed length. Due to the binary characteristics of hash codes, their similarity can be quickly evaluated by calculating the inner product of the hash codes of two samples. In addition, hash codes can be represented by half-precision floating point numbers, which not only simplifies storage, but also significantly reduces the use of memory. By using hash codes, the present invention can significantly reduce the time complexity and space complexity of the algorithm, thereby improving the efficiency of the algorithm. For example, for two 256*128 voice and face feature matrices, if the similarity between samples is represented by calculating cosine similarity, approximately 5.01e+7 calculation operations are required, while the present invention can represent the similarity between samples by inner product, which requires approximately 1.61e+7 calculation operations.

[0077] 5. Network learning visualization: Figure 2 As shown, the present invention further uses the t-SNE algorithm to perform 2D visualization on the learned high-dimensional face-speech embedding features, where data points of the same color come from the same identity sample. Figure 2 (a) shows the feature distribution before training. It can be clearly seen that due to the heterogeneity of the two modal data, the same color is roughly divided into two parts, one is the voice sample and the other is the face sample, and the distribution is relatively scattered. Figure 2 (b) shows the results after training using the method of the present invention. The present invention not only eliminates the heterogeneity between modalities, but also makes the two modal samples of the same identity closer to each other, while the samples of different identities are far away from each other, indicating that the heterogeneous deep residual hash alignment network proposed in the present invention can effectively learn the association between modalities.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A cross-modal speech and face association method based on heterogeneous hashing network, It is characterized in that The specific steps are as follows: Step 1: Build a data set: Use the preprocessed data set to train and test the network; Step 2: Load the dataset: Add a sampler to receive samples during training. Each iteration contains samples of multiple different identities. Each identity contains a fixed number of instances. During network training, at each iteration, the data is presented in the form of a tuple, which is expressed as follows: Among them, v represents a voice clip, f represents a face image, and l is a label. If l i = l j , indicating v i ,f i With v j , f j belong to the same person; if l i ≠l j , then we assume that v i , f i With v j , f j People belonging to different identities, batch represents the number of tuples in each iteration of the network; Step 3: Establish speech feature extraction network A v (·): Use the ECAPA-TDNN model pre-trained on the VoxCeleb2 dataset as the speech feature extraction network. For the speech segment, first extract the Mel spectrum features of the speech and normalize them as the input of the ECAPA-TDNN model to extract features and output the final speech features; Step 4: Build face feature extraction network A f (·): Use the Inception-V1 network pre-trained on the VGGFace2 dataset as the face feature extraction network. The face image is output through the Inception-V1 network as a 512-dimensional vector as the final face feature; Step 5: Construct a cross-modal feature alignment network: The specific cross-modal feature alignment network expression is as follows: φ(·, θ)={φv(·, θv), φf(·, θf)} The speech features and face features obtained in the feature extraction stage are used as the input of the network. The cross-modal feature alignment network automatically inputs the speech features and face features into the corresponding feature alignment sub-network, and uses multiple fully connected layers FC(·) to achieve feature alignment. Except for the last fully connected layer, each fully connected layer is followed by a batch normalization layer BN(·). Batch normalization is expressed as follows: Among them, x i represents the features after the fully connected layer, μ B Represents the mean of all sample features in a batch on each feature channel, σ B Represents the variance of all sample features in a batch on each feature channel, ε is a small constant, and γ and β are learnable parameters; A hash function layer sgn(·) is added to the last layer of the cross-modal feature alignment network. The final output of the cross-modal feature alignment network is: V=sgn(φ v (A v (v i ),θ v ) F=sgn(φ f (A f (f i ),θ f ) where sgn(·) is a sign function, φ v represents the speech feature alignment subnetwork in the constructed cross-modal feature alignment network, φ f A represents the face feature alignment subnetwork in the constructed cross-modal feature alignment network. v (v i ) is the speech feature obtained by the feature extraction network, A f (f i ) is the facial feature obtained through the feature extraction network, θ v ,θ f They represent the parameters learned by the two sub-networks in the cross-modal feature alignment network, and F and V are the hash codes of the two modalities finally obtained.

2. According to claim 1, a cross-modal speech and face association method based on heterogeneous hash network, It is characterized in that In step 1, the dataset includes voice clips and face images of 1225 identities, and the dataset is represented as Where i represents the i-th identity person, N = 1225; Each identity is represented internally as in, represents the jth voice clip of the ith identity, where j ranges from 1 to m, i.e., the ith identity has a total of m voice samples; similarly, Represents the jth face image frame of the ith identity, where j ranges from 1 to n, i.e., the ith identity has a total of n face image frames.

3. According to claim 2, a cross-modal speech and face association method based on heterogeneous hashing network, It is characterized in that In step five, the batch-normalized features of the first layer of the network are residually connected to the features of the last layer to fuse the early layer features with the deep layer features. The two modalities of the early layer are trained separately, and the later layers share weights for the two modalities. Through this network, the original 512-dimensional facial features and 192-dimensional speech features are mapped to a 128-dimensional space.

4. According to claim 3, a method for associating cross-modal speech and face based on heterogeneous hashing network, It is characterized in that Construct a loss function, construct N triplets in a batch according to the triple construction method, and the N triplets are derived from the hash codes generated by the cross-modal feature alignment network and are used to calculate the triple loss to optimize the feature representation and enhance the model's distinguishing ability: Among them, d(a i , p i ) represents the Hamming distance between the anchor point and the positive sample, d(a i , n i ) represents the Hamming distance between the anchor point and the negative sample, and margin is a hyperparameter used to control the distance difference between the positive and negative sample pairs; In addition, based on the characteristics of hash codes, the inter-modal similarity loss is proposed, and the true similarity matrix S is constructed as follows: Among them, s ij Represents two samples v i , f i If the labels of the two samples are consistent, the similarity is 1. If the labels of the two samples are inconsistent, the similarity is -1. The inner product of the two vectors is used to calculate the predicted similarity matrix Y: Where k is the length of the hash code, represents the hash code of the i-th speech sample obtained through network mapping, Represents the hash code of the j-th face sample obtained by mapping, and the similarity loss is expressed as follows: Among them, ||·|| is used to represent the distance between two points, using the Euclidean metric method; The overall loss function is:

5. According to claim 4, a method for associating cross-modal speech and face based on heterogeneous hashing network, It is characterized in that Select the small batch training method to train the cross-modal feature alignment network, and train one batch of data each time. The specific expression is as follows: The feature extraction network inputs the two modal features into their respective feature extraction sub-networks to extract features. The extracted features are used to represent the hash codes learned by the cross-modal feature alignment network. After multiple iterations, that is, after calculating the loss function, backpropagation and updating the network parameters to minimize the loss function, the hash codes of the two modalities learned in each iteration are represented as: B=[b 1 ,b 2 ,b 3 ...,b n ] T ∈{+1,-1}n× k Among them, B is the hash code matrix of speech or face obtained by network mapping, n is the batch size, which indicates the number of samples processed in each iteration; each element b in matrix B i , whether it is a voice sample or a face sample, it is a hash code of length k, and each dimension of each hash code takes the value of +1 or -1.

Citation Information

Patent Citations

  • Face image retrieval method and device based on deep learning and Hash coding

    CN110175248A

  • Voice and face image matching method and device, storage medium and electronic equipment

    CN111507218A

  • Cross-modal retrieval method and device based on deep adversarial discrete hash learning

    CN111597298A

  • Cross-modal image audio retrieval method based on deep heterogeneous correlation learning

    CN113343014A

  • Cross-modal face retrieval hash method based on metric learning

    CN114067385A

Cited By

  • Voice and face cross-modal association method based on mutual information

    CN121637406A