Cross-face-voice verification method and system based on trimodal fusion contrastive learning

Abstract

This invention discloses a cross-face-voice verification method and system based on trimodal fusion contrastive learning. The method includes a training phase and a testing phase. The training phase includes the following steps: S1, constructing a training sample dataset; S2, building a trimodal fusion contrastive learning model; S3, loading pre-trained parameters to accelerate model fitting and improve model training efficiency; S4, setting the parameters required for training the trimodal fusion contrastive learning model; S5, iteratively training the trimodal fusion contrastive learning model and selecting the trained trimodal fusion contrastive learning model; the testing phase specifically involves loading the parameters of the trained trimodal fusion contrastive learning model into a randomly initialized trimodal fusion contrastive learning model to complete the task of biometric feature matching. The cross-face-voice verification method based on trimodal fusion contrastive learning proposed in this invention can effectively associate face and voice data, eliminating the semantic gap between deep features.

Description

Technical Field

[0001] The present invention relates to the field of biometric matching in pattern recognition, and in particular to a cross-face-voice verification method based on trimodal fusion contrast learning. Background Art

[0002] In recent years, with the development of artificial intelligence, biometric recognition technology has rapidly developed and is widely used in various scenarios. Among the many biometric features, facial and voice information are the most direct and active data forms for human-computer interaction. For example, applications that use visual information as biometric information, such as facial payment, provide people with more secure and accurate payment services. Voice information also has unique application value in areas such as smart homes, such as Apple's Siri and Baidu's smart speakers. This shows that the appropriate use of biomodal information can better achieve effective human-computer interaction and bring convenience to social life.

[0003] It's worth noting that neither visual nor auditory biometrics take into account the semantic correlation between the two. In fact, in the field of psychology, a psychology article published in Nature states that when people hear a voice, their brains automatically associate it with the corresponding visual information. In neurological research, MRI scans have shown that when people are exposed to unimodal visual information, the auditory center in the cerebral cortex is also activated. In the computer field, researchers at MIT have designed a product that, by simply inputting a 6-second voice clip into a neural network, can output a matching approximate facial image. These studies demonstrate the potential correlation between facial and voice data.

[0004] Inspired by the above research, cross-modal face and voice biometric recognition algorithms have attracted the attention of more and more researchers. Due to the heterogeneity of face and voice in terms of formal expression, the development of cross-modal face-voice biometric recognition methods is still in its early stages and still faces the following problems in face and voice association learning: (1) The training method of the cross-modal association model adopts the traditional batch training method. The selection of negative samples in cross-modal contrastive learning relies on random selection of the system and the number is limited by the batch size. The low data utilization leads to slow model fitting. (2) At the same time, the original features of face and voice data are obtained by different encoders. Due to the different forms of expression, they face the interference of data heterogeneity during association learning. The existing face-voice association method only focuses on the alignment of the surface features to solve this problem, and does not make up for the gap in deep semantics. Therefore, designing a general method to improve the fitting efficiency of cross-modal contrastive learning and promote the fusion and alignment of modal deep semantics has become an urgent problem to be solved. Summary of the Invention

[0005] To solve the above problems in the prior art, the present invention proposes a cross-face-voice verification method based on trimodal fusion contrastive learning, which includes a training phase and a testing phase. The training phase includes the following steps:

[0006] S1, build a training data set and obtain the initial features of the face and initial speech features Preprocess the input face images and voice clips to form a training data set where f i and v i represents the i-th face-speech sample pair, y i represents the identity label information of the i-th pair of data, and N represents the total number of training samples. According to the division of the cross-modal face-speech field on the Voxceleb1 dataset, paired and matched face-speech data are obtained from the training dataset by random sampling, and the initial facial features are extracted through the face feature convolutional network Fnet(·) Extracting initial speech features through speech feature convolutional network Vnet(·)

[0007] S2, build a trimodal fusion contrast learning model; the trimodal fusion contrast learning model includes a multimodal hybrid encoder and a feature classification layer; the multimodal hybrid encoder is used to and initial speech features Obtain facial modal features F i , speech modal features V i and identity modal features I i ;The feature classification layer is used for classification learning of identity modality;

[0008] S3, load pre-training parameters; load pre-training parameters of face feature convolutional network Fnet(·) and voice feature convolutional network Vnet(·);

[0009] S4, setting parameters required for trimodal fusion contrast learning model training; the parameters include learning rate, number of training rounds and batch size, and initializing queue length, momentum parameter, temperature parameter, modal fusion parameter and boundary value of trimodal internal fusion;

[0010] S5, using the training dataset The trimodal fusion contrastive learning model is iteratively trained based on the loss function. During the iteration process, the facial feature convolutional network Fnet(·), the speech feature convolutional network Vnet(·), and the multimodal hybrid encoder are updated by gradient backpropagation until the model reaches the preset training cycle. The trimodal fusion contrastive learning model with the highest score on the validation set is selected and saved as the trained trimodal fusion contrastive learning model.

[0011] Specifically, the testing phase involves loading the parameters of the trained trimodal fusion contrastive learning model into a randomly initialized trimodal fusion contrastive learning model, obtaining the results of the cross-modal face-voice verification and two-way matching tasks based on the test set, and completing the biometric matching task.

[0012] Preferably, in S1, the face-voice data is preprocessed as follows:

[0013] S11, using MTCNN face detection to obtain facial key points on the acquired facial image, cropping the image to the corresponding size, then performing de-meaning and normalization operations, and inputting the pre-processed image into the facial feature convolutional network to obtain the initial facial features;

[0014] S12, using the Mel-frequency cepstral coefficients of the acquired speech segment to obtain the spectrum of the short-term audio, and inputting the extracted spectrum into the speech feature convolutional network to obtain the initial speech features.

[0015] Preferably, in S2, a three-modal fusion contrast learning model is constructed as follows:

[0016] S21, construct a multimodal hybrid encoder Fusion(·), fuse the initial face features and the initial speech features, and obtain the face modal features F i , speech modal features V i and identity modal features I i , output to the feature classification layer;

[0017] S22, the feature classification layer performs the following operations:

[0018] S221, obtain the identity classification score and optimize the features of the identity modality using cross entropy loss;

[0019] S222, face modal feature F i , speech modal features V i and identity modal features I i Conduct cross-modal momentum comparative learning to explore the correlation between identity modal features, facial modal features, and voice modal features, promote the fusion and alignment of the three modalities, and bridge the semantic gap between facial data and voice data;

[0020] S223, the internal fusion loss of the three modalities, associates the features of the three modalities.

[0021] Preferably, the multimodal hybrid encoder of S21 is expressed as:

[0022] Fusion(x)=Drop(W2(ReLUσ(W1x)))

[0023] Where x represents the input, ReLU represents the activation function ReLU, W1 represents the weight of the first fully connected layer, W2 represents the weight of the second fully connected layer, σ represents the Batch Norm layer, which is mainly used to prevent model collapse, Drop(·) represents the Dropout operation, and the dimension of the fully connected layer changes from 512→256→128;

[0024] After the multimodal hybrid encoder, the face modality feature F of the i-th person in the training dataset i , speech modal features V i and identity modal features I i The construction process is expressed as:

[0025]

[0026] Among them, μ∈(0, 1) is a modal fusion parameter, which is mainly used to adjust the input ratio of features during the construction of identity modal features.

[0027] Preferably, the optimization process of S221 is represented by a loss function Expressed as:

[0028]

[0029]

[0030] Among them, CE(I i ,y i ) represents the cross entropy loss function, M represents the total number of training identity categories, It represents the probability of the identity modal feature being mapped to the sample label category c through a single fully connected layer, using the identity label as the category.

[0031] Preferably, the overall objective of the trimodal contrastive learning in S222 is expressed as:

[0032]

[0033] l con =l(I,F)+l(I,V)+l(F,V)+l(V,F)

[0034] Among them, m1 and m2 represent different modalities, (m1,m2)∈[(I,F),(I,V),(F,V),(V,F)], F, V and I represent all facial modality features F in the training set respectively. i , speech modal features V i and identity modal features I i l(I,F) represents the comparative learning goal between identity modality and face modality, l(I,V) represents the comparative learning goal between identity modality and voice modality, l(F,V) and l(V,F) represent the comparative learning goals between face modality and voice modality; and Represent samples of another modality with the same identity label as m1 and different identity label from m1 respectively; the positive examples of trimodal contrastive learning adopt the combination of identity modality features, face modality features and voice modality features, and the negative samples of trimodal contrastive learning are constructed in the form of momentum encoder and global queue, where the momentum encoder φ f (·) and φ v (·) is constructed by copying the parameters of the face feature convolutional network Fnet(·), the speech feature convolutional network Vnet(·) and the multimodal hybrid encoder Fusion(·).

[0035] Preferably, the S223 optimizes the internal alignment of the three modalities by minimizing the negative cosine similarity of the modal contents and setting a boundary value, so the target of the three modal content fusion is l rank (I,F,V) can be expressed as:

[0036] l rank (I,F,V)=max[0,-(sim(I,F)+sim(I,V))+λ]

[0037] Where λ is a boundary value hyperparameter and the cosine similarity metric is expressed as:

[0038]

[0039] where ||·||2 is the L2 norm, n1 and n2 represent modal features, (n1,n2)∈[(I,F),(I,V)].

[0040] The identity modality is used to bring the face modality and voice modality closer. At the same time, a certain degree of tolerance and relaxation are given by setting boundaries in the content fusion learning process, so that the model can maintain some fine-grained perception capabilities and further improve the general performance of the model. The three-modal fusion loss function Expressed as:

[0041]

[0042] Preferably, in S5, the face and voice feature convolutional network and the multimodal hybrid encoder use gradient backpropagation to update network parameters during training, and the cross-modal momentum encoder uses momentum update to update the encoder. The specific update strategy is expressed as follows: given the momentum coefficient η, the update process is expressed as:

[0043] θ f ,θ v ←optimizer(θ f ,θ v ,r)

[0044] ξ f ←ηξ f +(1-η)θ f

[0045] ξ v ←ηξ v +(1-η)θ v

[0046] Among them, the network parameters of the face feature convolution network Fnet(·) and the multimodal hybrid encoder Fusion(·) are θ f , the network parameters of speech feature convolutional network Vnet(·) and Fusion(·) are θ v , then the momentum encoder φ f (·) and φ v The network parameter of (·) is expressed as ξ f and ξ v ; optimizer is the optimizer, r is the learning rate of the optimizer, η∈(0,1] is a momentum coefficient;

[0047] Trimodal contrastive learning uses a global queue of faces and voices The global negative sample queue is stored in a way that is filled by the momentum encoder. The queue construction process is expressed as:

[0048] F i mom =φ f (f i ),V i mom =φ v (v i )

[0049]

[0050] Where K is the maximum length of the queue, mom represents the momentum feature generated by the momentum encoder, and the loss function of cross-modal face-speech contrastive learning is expressed as:

[0051]

[0052] Where τ is the temperature parameter, the contrastive learning loss function based on identity modality is expressed as:

[0053]

[0054] During the trimodal contrastive learning training process, the features in the queue do not participate in the model gradient backpropagation. Stopgrad is used to make the features lose their gradients, as shown below:

[0055] stopgrad(F mom ,V mom )

[0056] In the universal common subspace, the trimodal contrastive learning loss function can be expressed as:

[0057]

[0058] Finally, the overall training loss function of the cross-face-speech alignment and fusion model based on trimodal contrastive learning is yes:

[0059]

[0060] Preferably, in S3, the pre-training parameters of the facial feature convolutional network are obtained by performing face identity classification training based on the MS-1M dataset, and the pre-training parameters of the voice feature convolutional network are obtained by performing voice identity classification training based on the Voxceleb2 dataset.

[0061] The present invention also provides a cross-face-voice verification system based on trimodal fusion contrast learning, including a training module and a testing module, wherein the training module includes:

[0062] Preprocessing unit, builds training sample data set, obtains initial face features and initial speech features

[0063] A multimodal mixing unit for constructing the identity modality; and a feature classification layer for classification learning of the identity modality, wherein the feature classification includes an identity modality fusion optimization unit, a trimodal contrast learning alignment unit, and a trimodal internal fusion optimization unit.

[0064] Preloading unit, loading pre-training parameters of face feature convolutional network and speech feature convolutional network to speed up model fitting and improve model training efficiency;

[0065] A setting unit is used to set parameters required for training the trimodal fusion contrast learning model; the parameters include a learning rate, a number of training rounds, a batch size, and to initialize a queue length, a momentum parameter, a temperature parameter, a modal fusion parameter, and a boundary value of the trimodal internal fusion;

[0066] The iteration unit uses the training dataset to perform iterative training on the trimodal fusion contrastive learning model based on the loss function. During the iteration process, the parameters of the face and voice feature convolutional networks and the multimodal hybrid encoder are updated by gradient backpropagation until the model reaches the preset training cycle. The trimodal fusion contrastive learning model with the highest score on the validation set is selected and saved as the trained trimodal fusion contrastive learning model.

[0067] The testing module loads the parameters of the trained trimodal fusion contrastive learning model into a randomly initialized trimodal fusion contrastive learning model, obtains the results of the cross-modal face-voice verification and two-way matching tasks based on the test set, and completes the biometric matching task.

[0068] The above scheme describes the basic method of the present invention. Compared with the existing cross-face-voice biometric matching method, the cross-modal face-voice alignment and fusion method based on trimodal contrastive learning described in the present invention has the following beneficial effects:

[0069] (1) This paper proposes a cross-modal face-speech alignment and fusion scheme based on trimodal contrastive learning. This method uses the identity modality constructed by the fusion method as the identity modality. It adopts a supervised training method and uses identity labels to optimize feature representation, thereby enhancing the model fitting efficiency.

[0070] (2) The trimodal contrastive learning proposed in this invention is based on the cross-modal momentum contrastive learning method, which expands the traditional cross-modal contrastive learning method based on batch training. Due to the uniformity of the identity of modal data, face, voice and identity features can be regarded as natural positive examples. The cross-modal momentum encoder is constructed by parameter replication and momentum update, and the negative samples in the trimodal contrastive learning are extracted and filled in the global queue.

[0071] (3) In order to encourage the fusion of identity modality and good alignment with cross-modal data, this paper proposes a content fusion loss to associate trimodal features in order to further enhance the association ability and eliminate the semantic gap between deep features.

[0072] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A diagram showing the steps of the training phase according to an embodiment of the present invention;

[0074] Figure 2 is a detailed flow chart of an embodiment of the present invention;

[0075] Figure 3 Schematic diagram of the structure of a multimodal hybrid encoder according to an embodiment of the present invention;

[0076] Figure 4 Schematic diagram of a system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0077] See also Figure 1 FIG. 1 is a diagram showing the steps of the training phase according to an embodiment of the present invention, including:

[0078] S1, build a training sample data set to obtain the initial features of the face and initial speech features Preprocess the input face images and voice clips to form a training sample data set where f i and v i represents the i-th face-speech sample pair, y i represents the identity label information of the i-th pair of data, and N represents the total number of training samples. According to the division of the cross-modal face-speech field on the Voxceleb1 dataset, paired and matched face-speech data are obtained from the training dataset by random sampling, and the initial face features are extracted through the face feature convolutional network Fnet(·) Extracting initial speech features through speech feature convolutional network Vnet(·)

[0079] S2, constructing a trimodal fusion contrastive learning model; the trimodal fusion contrastive learning model includes a multimodal hybrid encoder and a feature classification layer; the multimodal hybrid encoder is used to construct the identity modality; the feature classification layer is used for classification learning of the identity modality;

[0080] S3, load pre-training parameters; load pre-training parameters of face feature convolutional network and speech feature convolutional network;

[0081] S4, setting parameters required for trimodal fusion contrast learning model training; the parameters include learning rate, number of training rounds and batch size, and initializing queue length, momentum parameter, temperature parameter, modal fusion parameter and boundary value of trimodal internal fusion;

[0082] S5, use the training dataset to perform iterative training on the trimodal fusion contrastive learning model based on the loss function. During the iteration process, the parameters of the face and voice feature convolutional network and the multimodal hybrid encoder are updated by gradient backpropagation until the model reaches the preset training cycle. The trimodal fusion contrastive learning model with the highest score on the validation set is selected and saved as the trained trimodal fusion contrastive learning model.

[0083] See also Figure 2 As shown in the detailed flow chart of the embodiment of the present invention, this method is mainly divided into three modules: pre-processing module, multimodal mixing module, identity modality fusion construction module, trimodal contrast learning alignment module and trimodal internal fusion optimization module:

[0084] (1) Preprocessing module, obtains initial facial features and initial speech features, and constructs a training dataset.

[0085] For the acquired facial image, MTCNN face detection (Zhang K, Zhang Z, Li Z, et al. Joint face detection and alignment using multitask cascaded convolutional networks [J]. IEEE signal processing letters, 2016, 23 (10): 1499-1503.) is used to obtain facial key points, and then the image is cropped. The cropped RGB face image is scaled to 112 * 112 * 3. For the facial image with acquired features, the corresponding image is de-meaned and normalized, and the pre-processed image is input into the ResNet-50 (Hu J, Shen L, Sun G. Squeeze-and-excitation networks [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7132-7141.) network to extract features and obtain the initial features of the facial image.

[0086] Mel-frequency cepstral coefficients (MFCCs) are used to extract speech segments. Mel-frequency cepstral coefficients perform a Fourier transform on a speech signal and use a triangular window function to obtain a spectrum that captures short-term audio information. The window frame length is set to 25 milliseconds, with a step size of 10 milliseconds. The extracted spectrum is then fed into a Thin-ResNet-34 (Arandjelovic R, Gronat P, Torii A, et al. NetVLAD: CNN architecture for weakly supervised place recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 5297-5307.) network to obtain initial speech features.

[0087] Face and voice data are input in pairs, and the training dataset is represented as where f i and v i represents the i-th face-speech sample pair, y i Represents the identity label information of the i-th pair of data, and N represents the total number of training samples.

[0088] The universal feature extraction network for face and speech is represented by the face encoder Fnet(·) and the speech encoder Vnet(·). In the cross-modal universal subspace, the feature extraction process can be expressed as:

[0089]

[0090] Specifically, the face encoder Fnet(·) uses the SE-ResNet-50 network encoder. For each input face image, the network outputs a 512-dimensional face feature.

[0091] The speech encoder uses a Thin-ResNet-34 network to extract features, and its output features are also 512-dimensional. This is to ensure consistency with the output facial features and facilitate subsequent feature fusion.

[0092] (2) Multimodal hybrid module, which fuses the initial face features and the initial speech features through the multimodal hybrid encoder Fusion(·).

[0093] Specific structure such as Figure 3 As shown, the mapping function of the encoder can accept inputs of different modalities as well as single-modal inputs and uniformly output features of the same dimension. The mapping function of the hybrid encoder is expressed as:

[0094] Fusion(x)=Drop(W2(ReLUσ(W1x)))

[0095] Where x represents the input, ReLU represents the activation function, W1 represents the weight of the first fully connected layer, W2 represents the weight of the second fully connected layer, σ represents the Batch Norm layer, which is mainly used to prevent model collapse, Drop(·) represents the Dropout operation, and the dimension of the fully connected layer changes from 512 to 256 to 128. After the multimodal hybrid encoder, the identity feature construction process is expressed as:

[0096]

[0097] Where I represents the identity modality, μ∈(0,1) is a modality fusion parameter, which is mainly used to adjust the input ratio of features during the construction of identity modality features.

[0098] (3) Identity modality fusion optimization module: This paper proposes the idea of ​​fusion construction of identity modalities in order to integrate the correlation and interactivity contained in the face and voice modalities into a unified multimodal common subspace. The construction of identity modalities is not only conducive to deep mining of the correlation between face and voice, but also establishes an intermediate modality to further narrow the heterogeneity gap between the face modality and the voice modality. Specifically, the construction of identity modalities mainly involves learning a multimodal hybrid encoder Fusion(·); further, to ensure the effectiveness of the fusion-constructed identity modality, the representation of the modality is enhanced by adding identity tags for supervised learning.

[0099] In the field of cross-modal face-speech association, adding identity semantic constraints is the most effective and direct way to promote the similarity of features of the same identity modality in a common subspace. Therefore, after passing the facial and voice features through a multimodal hybrid encoder, they are input into a fully connected layer with the number of categories, and the cross-entropy loss is used to optimize the features of the identity modality. The optimization process is expressed as:

[0100]

[0101] Where M represents the total number of training identity categories, It represents the probability of mapping the identity modal feature to the sample label category c after a single fully connected layer. The label information is actually the identity category. In the field of cross-modal face and voice, a person's identity information is used as a category.

[0102] (4) Trimodal contrastive learning module: In the cross-modal face and speech domain, the goal of contrastive learning is to effectively unify the alignment and association of cross-modal data. In the common subspace, the similarity between semantically consistent face and speech information is maximized, and the similarity between face and speech information with different identities is minimized. In order to enable face and speech features to be associated with identity fusion modality in the common subspace, face and speech features are also passed through the multimodal hybrid encoder Fusion(·) to obtain features of the same dimension. The process is expressed as follows:

[0103]

[0104] Where F represents the face modality and V represents the voice modality. The traditional cross-modal contrastive learning objective can be expressed as:

[0105] l con (F,V)=argmax[sim(F,V + )-sim(F,V - )]

[0106] l con (V,F)=argmax[sim(V,F + )-sim(V,F - )]

[0107] l cross-con =l con (F,V)+l con (V,F)

[0108] Where V + and V - They represent the sound samples with the same and different semantics as the face modality F, respectively. + and F - They represent face samples with the same and different semantics as the speech modality V, respectively. The facial features are obtained by the face encoder and the multimodal hybrid encoder in sequence, and the speech features are obtained by the speech encoder and the multimodal hybrid encoder in sequence. The sim(·,·) function represents the similarity of the two modal features through the dot product operation.

[0109] Furthermore, modality alignment is enhanced by maximizing the similarity between semantically consistent modalities while making semantically inconsistent modalities as unrelated as possible.

[0110] Trimodal fusion contrastive learning can be seen as an effective extension of cross-modal contrastive learning, with the goal of adding identity modality to further promote the alignment between face and voice modalities. Specifically, the trimodal contrastive learning method regards the identity modality as a new positive sample to expand the form of cross-modal contrastive learning, further explore the correlation between the identity modality and the face-voice modality, promote the fusion and alignment of the three modalities in the common common subspace, and bridge the semantic gap between face data and voice data, thereby improving the data utilization of the model and accelerating the model fitting speed. The trimodal contrastive learning objective based on identity modality can be expressed as:

[0111]

[0112] l con =l(I,F)+l(I,V)+l(F,V)+l(V,F)

[0113] Among them, m1 and m2 represent different modalities, (m1,m2)∈[(I,F),(I,V),(F,V),(V,F)], F, V and I represent all facial modality features F in the training set respectively. i , speech modal features V i and identity modal features I i l(I,F) represents the comparative learning goal between identity modality and face modality, l(I,V) represents the comparative learning goal between identity modality and voice modality, l(F,V) and l(V,F) represent the comparative learning goals between face modality and voice modality; and They represent samples of another modality with the same identity label as m1 and different identity label from m1, respectively.

[0114] Benefiting from the particularity of face-voice data collection, biometric features of the same identity have the same expression in semantic information. Therefore, trimodal contrastive learning uses natural positive pairs, namely, pairs of identity features, face features, and voice features. The construction of negative samples for trimodal contrastive learning adopts the form of momentum encoder and global queue. The present invention constructs momentum encoder φ by copying the parameters of cross-modal universal feature encoder Fnet(·), Vnet(·) and multimodal hybrid encoder Fusion(·) f (·) and φ v (·). Assume that the network parameters of the face general feature encoder Fnet(·) and the multimodal hybrid encoder Fusion(·) are uniformly expressed as θ f , the network parameters of Vnet(·) and Fusion(·) are uniformly expressed as θ v ,φ f (·) and φ v The network parameter of (·) is expressed as ξf and ξ v .

[0115] Furthermore, the face and voice feature convolutional networks and the multimodal hybrid encoder use gradient backpropagation to update network parameters during training, and the cross-modal momentum encoder uses momentum update to update the encoder. The specific update strategy is expressed as follows: Given the momentum coefficient η, the update process is expressed as:

[0116] θ f ,θ v ←optimizer(θ f ,θ v ,r)

[0117] ξ f ←ηξ f +(1-η)θ f

[0118] ξ v ←ηξ v +(1-η)θ v

[0119] Where optimizer is the optimizer, r is the learning rate of the optimizer, and η∈(0,1] is a momentum coefficient. Trimodal contrastive learning uses a global queue of face and voice to construct The global negative sample queue is stored in a way that is filled by the momentum encoder. The queue construction process is expressed as:

[0120] F i mom =φ f (f i ),V i mom =φ v (v i )

[0121]

[0122] Where K is the maximum length of the queue, mom represents the momentum feature generated by the momentum encoder, and the loss function of cross-modal face-speech contrastive learning is expressed as:

[0123]

[0124] Where τ is the temperature parameter, the contrastive learning loss function based on identity modality is expressed as:

[0125]

[0126] It is worth noting that during the trimodal contrastive learning training process, the features in the queue do not participate in the model gradient backpropagation, which includes the following operations:

[0127] stopgrad(F mom ,V mom )

[0128] In the universal common subspace, the trimodal contrastive learning loss function can finally be expressed as:

[0129]

[0130] (5) Trimodal internal fusion optimization module: Although the trimodal contrastive learning method is used in the common subspace to make the set positive example pairs and negative samples distinguished in terms of basic similarity and distance, the set identity modality is still relatively weak in the cross-modal semantic alignment link. On this basis, the present invention further proposes a trimodal fusion loss to narrow the distance between the three modalities. Specifically, the present invention optimizes the internal alignment of the three modalities by minimizing the negative cosine similarity of the modal content and setting a boundary value. The metric of cosine similarity is expressed as:

[0131]

[0132] Where ||·||2 is the L2 norm, n1 and n2 represent modal features, (n1,n2)∈[(I,F),(I,V)], then the goal of trimodal content fusion can be expressed as:

[0133] l rank (I,F,V)=max[0,-(sim(I,F)+sim(I,V))+λ]

[0134] Here, λ is a boundary hyperparameter. During trimodal contrastive learning, even with semantically identical identity, face, and voice features, there can still be discrepancies in semantic alignment due to differences in data representation and the lack of objective entity concepts in the identity modality.

[0135] By utilizing the fused identity modality to bring the face and voice modalities closer, and setting boundaries to allow for a certain degree of error tolerance and relaxation during the internal fusion learning process, the model maintains some fine-grained perception capabilities, further improving the general performance of the model. The trimodal fusion loss function is then expressed as:

[0136]

[0137] Finally, the overall training loss function of the cross-face-speech alignment and fusion model based on trimodal contrastive learning is:

[0138]

[0139] Dataset introduction: To verify the effectiveness of the proposed model, experiments were conducted on the Voxceleb1 and VGGFace datasets. Voxceleb1 is a cross-face-speech public dataset containing more than 100,000 audio data points and more than 20,000 video data points. The Voxceleb1 dataset is derived from celebrity interview videos of 2,622 identities on YouTube. VGGFace is a large-scale face recognition dataset. For associative learning, 1,225 identities with overlapping identities in the two datasets were selected. The model was implemented using the Pytorch framework and Python code, using the Adam optimizer with a learning rate of 1e-5. The queue update method is first-in-first-out, the batch size of each momentum update queue is set to 16, the queue length is set to 1024, the momentum coefficient is set to 0.99, the temperature parameter is set to 0.07, the modal fusion parameter μ is set to 0.4, and the boundary value of the three-modal internal fusion is set to 0.1.

[0140] Cross-modal face-to-voice verification: The cross-modal face-to-voice verification task aims to determine whether a given face and voice belong to the same identity. This task can be considered a binary classification problem, and its evaluation metric is the AUC (Area Under Curve), which is the area under the curve. A larger value indicates better model performance.

[0141] Cross-modal two-way matching task: In cross-modal face-speech recognition, this task is divided into two cases: FV and VF. For FV, the two-way matching task involves, given a face image, selecting a matching voice from two selected speech segments. For VF, given a speech segment, selecting a matching face image from two tested faces. This task uses accuracy (ACC) as the evaluation metric; higher accuracy indicates better model performance.

[0142] In order to verify the effectiveness of the method, the present invention selected several representative methods in this field for comparison:

[0143]

SVHF

[0144] [DIMNet]Wen Y,Ismail MA,Liu W,et al.Disjoint mapping network for cross-modal matching of voices and faces[J].arXiv preprint arXiv:1807.04836,2018.

[0145]

LDJE

[0146] [SSS] Wen P, Xu Q, Jiang Y, et al. Seeking the shape of sound: An adaptive framework for learning voice-face association[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2021:16347-16356.

[0147] The experimental results are shown in Table 1:

[0148] Table 1 Comparison of experimental results of the present invention and other methods

[0149]

[0150] Cross-modal face-voice verification and matching tasks are divided into two cases: face matching voice FV and voice matching face VF. The symbol "U" indicates that the selection of face and voice input during the test is random without any restrictions, and "G" indicates that the face and voice input during the test are of the same gender. From the experimental results, it can be seen that the present invention has achieved an overall improvement over the baseline method in cross-modal face-voice verification and matching tasks, verifying the effectiveness of the method of the present invention. SVHF is the first to fuse the high-dimensional features obtained by the feature extraction network and input the fused features into the Softmax layer, thereby converting it into a binary classification task. Through deep learning training, the performance of the obtained model in matching tasks is comparable to that of humans. However, the model obtained by this method is only suitable for matching tasks, and the network structure needs to be readjusted as the number of matches increases, and the model lacks flexibility. DIMNet adopts a disjoint mapping network and uses identity, gender and nationality as covariates to supervise model learning. However, this method lacks direct distance measurement for cross-modal related data. Under more challenging tasks, the model performs poorly. LDJE uses a bidirectional five-tuple spatial constraint loss and identity as supervisory information to assist model convergence. The model obtained by this method has good performance, but it does not fully utilize the potential relationship between face and voice data. SSS uses a two-level modal alignment method to capture local information while dynamically updating the cross-modal shared weights. Since the weights assigned to each sample vary depending on the difficulty level, the difficulty value of each sample changes after repeated training. The model obtained by this method has an overall high improvement, but the limitation is that the method of sorting difficult categories is not highly transferable. In contrast, the method proposed in the present invention can better associate features between modalities to further enhance learning capabilities, and uses supervised training to optimize feature representation using identity labels, thereby enhancing the fitting efficiency of the model. As can be seen from Table 1, the method proposed in the present invention provides competitive results in cross-modal verification and two-way matching tasks, and the experimental results strongly demonstrate the effectiveness of this method.

[0151] like Figure 4 FIG. 1 is a schematic diagram of a system according to an embodiment of the present invention, comprising a training module 10 and a testing module 20, wherein the training module 10 comprises:

[0152] Preprocessing unit 101, constructs a training sample data set to obtain initial facial features and initial speech features

[0153] The multimodal mixing unit 102 is used to construct the identity mode, i.e., the identity mode; and the feature classification layer is used for classification learning of the identity mode, wherein the feature classification includes an identity modality fusion optimization unit, a trimodal contrast learning alignment unit, and a trimodal internal fusion optimization unit.

[0154] A preloading unit 103 loads pre-trained parameters of the facial feature convolutional network and the speech feature convolutional network to speed up model fitting and improve model training efficiency;

[0155] A setting unit 104 sets parameters required for training the trimodal fusion contrast learning model; the parameters include a learning rate, a number of training rounds, and a batch size, and initializes a queue length, a momentum parameter, a temperature parameter, a modal fusion parameter, and a boundary value for internal fusion of the trimodal model;

[0156] An iterative unit 105 performs iterative training on the trimodal fusion contrastive learning model based on a loss function using the training dataset. During the iteration, gradient backpropagation is performed on the parameters of the face and voice feature convolutional networks and the multimodal hybrid encoder until the model reaches a preset training cycle. The trimodal fusion contrastive learning model with the highest score on the validation set is selected and saved as the trained trimodal fusion contrastive learning model.

[0157] In the testing phase module 20, the parameters of the trained trimodal fusion contrastive learning model are loaded into the randomly initialized trimodal fusion contrastive learning model, and the results of the cross-modal face-voice verification and two-way matching tasks are obtained based on the test set to complete the biometric matching task.

[0158] It can be seen that the cross-face-voice verification method and system based on trimodal fusion contrast learning proposed in the present invention can effectively associate face and voice data and eliminate the semantic gap between deep features.

[0159] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-face-voice verification method based on trimodal fusion contrastive learning includes a training phase and a testing phase, and is characterized by: The training phase consists of the following steps: S1, build a training data set and obtain the initial features of the face and initial speech features Preprocess the input face images and voice clips to form a training data set where f i and v i represents the i-th face-speech sample pair, y i represents the identity label information of the i-th pair of data, and N represents the total number of training samples. According to the division of the cross-modal face-speech field on the Voxceleb1 dataset, paired and matched face-speech data are obtained from the training dataset by random sampling, and the initial facial features are extracted through the face feature convolutional network Fnet(·) Extracting initial speech features through speech feature convolutional network Vnet(·) S2, build a trimodal fusion contrast learning model; the trimodal fusion contrast learning model includes a multimodal hybrid encoder and a feature classification layer; the multimodal hybrid encoder is used to and initial speech features Obtain facial modal features F i , speech modal features V i and identity modal features I i ;The feature classification layer is used for classification learning of identity modality; S3, load pre-training parameters; load pre-training parameters of face feature convolutional network Fnet(·) and voice feature convolutional network Vnet(·); S4, setting parameters required for trimodal fusion contrast learning model training; the parameters include learning rate, number of training rounds and batch size, and initializing queue length, momentum parameter, temperature parameter, modal fusion parameter and boundary value of trimodal internal fusion; S5, using the training dataset The trimodal fusion contrastive learning model is iteratively trained based on the loss function. During the iteration process, the facial feature convolutional network Fnet(·), the speech feature convolutional network Vnet(·), and the multimodal hybrid encoder are updated by gradient backpropagation until the model reaches the preset training cycle. The trimodal fusion contrastive learning model with the highest score on the validation set is selected and saved as the trained trimodal fusion contrastive learning model. The testing phase involves loading the parameters of the trained trimodal fusion contrastive learning model into a randomly initialized trimodal fusion contrastive learning model, obtaining the results of the cross-modal face-voice verification and two-way matching tasks based on the test set, and completing the biometric matching task. In S2, a three-modal fusion contrast learning model is built as follows: S21, construct a multimodal hybrid encoder Fusion(·), fuse the initial face features and the initial speech features, and obtain the face modal features F i , speech modal features V i and identity modal features I i , output to the feature classification layer; S22, the feature classification layer performs the following operations: S221, obtain the identity classification score and optimize the features of the identity modality using cross entropy loss; S222, face modal feature F i , speech modal features V i and identity modal features I i Conduct cross-modal momentum comparative learning to explore the correlation between identity modal features, facial modal features, and voice modal features, promote the fusion and alignment of the three modalities, and bridge the semantic gap between facial data and voice data; S223, the internal fusion loss of the three modalities, associates the features of the three modalities; The multimodal hybrid encoder of S21 is expressed as: Fusion(x)=Drop(W2(ReLUσ(W1x))) Where x represents the input, ReLU represents the activation function ReLU, W1 represents the weight of the first fully connected layer, W2 represents the weight of the second fully connected layer, σ represents the Batch Norm layer, which is mainly used to prevent model collapse, Drop(·) represents the Dropout operation, and the dimension of the fully connected layer changes from 512→256→128; After the multimodal hybrid encoder, the face modality feature F of the i-th person in the training dataset i , speech modal features V i and identity modal features I i The construction process is expressed as: Among them, μ∈(0,1) is a modal fusion parameter, which is mainly used to adjust the input ratio of features during the construction of identity modal features.

2. The cross-face-voice verification method based on trimodal fusion contrast learning according to claim 1 is characterized in that: In S1, the face-speech data is preprocessed as follows: S11, using MTCNN face detection to obtain facial key points on the acquired facial image, cropping the image to the corresponding size, then performing de-meaning and normalization operations, and inputting the pre-processed image into the facial feature convolutional network to obtain the initial facial features; S12, using the Mel-frequency cepstral coefficients of the acquired speech segment to obtain the spectrum of the short-term audio, and inputting the extracted spectrum into the speech feature convolutional network to obtain the initial speech features.

3. The cross-face-voice verification method based on trimodal fusion contrast learning according to claim 1 is characterized in that: The optimization process of S221 is represented by the loss function Expressed as: Among them, CE(I i ,y i ) represents the cross entropy loss function, M represents the total number of training identity categories, It represents the probability of the identity modal feature being mapped to the sample label category c through a single fully connected layer, using the identity label as the category.

4. The cross-face-voice verification method based on trimodal fusion contrast learning according to claim 1 is characterized in that: The overall objective of the trimodal contrastive learning of S222 is expressed as: l con =l(I,F)+l(I,V)+l(F,V)+l(V,F) Among them, m1 and m2 represent different modalities, (m1,m2)∈[(I,F),(I,V),(F,V),(V,F)], F, V and I represent all facial modality features F in the training set respectively. i , speech modal features V i and identity modal features I i l(I,F) represents the comparative learning goal between identity modality and face modality, l(I,V) represents the comparative learning goal between identity modality and voice modality, l(F,V) and l(V,F) represent the comparative learning goals between face modality and voice modality; and Represent samples of another modality with the same identity label as m1 and different identity label from m1 respectively; the positive examples of trimodal contrastive learning adopt the combination of identity modality features, face modality features and voice modality features, and the negative samples of trimodal contrastive learning are constructed in the form of momentum encoder and global queue, where the momentum encoder φ f (·) and φ v (·) is constructed by copying the parameters of the face feature convolutional network Fnet(·), the speech feature convolutional network Vnet(·) and the multimodal hybrid encoder Fusion(·).

5. The cross-face-voice verification method based on trimodal fusion contrast learning according to claim 1, characterized in that: The S223 optimizes the internal alignment of the three modalities by minimizing the negative cosine similarity of the modal content and setting the boundary value, so the goal of the three modal content fusion is l rank (I,F,V) can be expressed as: l rank (I,F,V)=max[0,-(sim(I,F)+sim(I,V))+λ] Where λ is a boundary value hyperparameter and the cosine similarity metric is expressed as: Where ||·||2 is the L2 norm, n1 and n2 represent modal features, (n1,n2)∈[(I,F),(I,V)]; The identity modality is used to bring the face modality and voice modality closer. At the same time, a certain degree of tolerance and relaxation are given by setting boundaries in the content fusion learning process, so that the model can maintain some fine-grained perception capabilities and further improve the general performance of the model. The three-modal fusion loss function Expressed as:

6. The cross-face-voice verification method based on trimodal fusion contrast learning according to claim 1, characterized in that: In S5, the face and voice feature convolutional network and the multimodal hybrid encoder use gradient backpropagation to update network parameters during training, and the cross-modal momentum encoder uses momentum update to update the encoder. The specific update strategy is expressed as follows: given the momentum coefficient η, the update process is expressed as: i f ,i v ←optimizer(θ f ,i v ,r) x f ←ex f +(1-η)θ f x v ←ex v +(1-η)θ v in , the network parameters of the face feature convolution network Fnet(·) and the multimodal hybrid encoder Fusion(·) are θ f , the network parameters of speech feature convolutional network Vnet(·) and Fusion(·) are θ v , then the momentum encoder φ f (·) and φ v The network parameter of (·) is expressed as ξ f and ξ v ; optimizer is the optimizer, r is the learning rate of the optimizer, η∈(0,1] is a momentum coefficient; Trimodal contrastive learning uses a global queue of faces and voices The global negative sample queue is stored in a way that is filled by the momentum encoder. The queue construction process is expressed as: F i mom =φ f (f i ),V i mom =φ v (v i ) Where K is the maximum length of the queue, mom represents the momentum feature generated by the momentum encoder, and the loss function of cross-modal face-speech contrastive learning is expressed as: Where τ is the temperature parameter, the contrastive learning loss function based on identity modality is expressed as: During the trimodal contrastive learning training process, the features in the queue do not participate in the model gradient backpropagation. Stopgrad is used to make the features lose their gradients, as shown below: stopgrad(F mom ,In mom ) In the universal common subspace, the trimodal contrastive learning loss function can be expressed as: Finally, the overall training loss function of the cross-face-speech alignment and fusion model based on trimodal contrastive learning is yes:

7. The cross-face-voice verification method based on trimodal fusion contrastive learning according to claim 1 is characterized in that: In S3, the pre-training parameters of the facial feature convolutional network are obtained by performing face identity classification training based on the MS-1M dataset, and the pre-training parameters of the voice feature convolutional network are obtained by performing voice identity classification training based on the Voxceleb2 dataset.

8. A cross-face-voice verification system based on trimodal fusion contrastive learning, used to implement the cross-face-voice verification method based on trimodal fusion contrastive learning according to any one of claims 1 to 7, characterized in that: It includes a training module and a testing module, wherein the training module includes: Preprocessing unit, builds training sample data set, obtains initial face features and initial speech features A multimodal mixing unit for constructing the identity modality; and a feature classification layer for classification learning of the identity modality, wherein the feature classification includes an identity modality fusion optimization unit, a trimodal contrast learning alignment unit, and a trimodal internal fusion optimization unit. Preloading unit, loading pre-training parameters of face feature convolutional network and speech feature convolutional network to speed up model fitting and improve model training efficiency; A setting unit is used to set parameters required for training the trimodal fusion contrast learning model; the parameters include a learning rate, a number of training rounds, a batch size, and to initialize a queue length, a momentum parameter, a temperature parameter, a modal fusion parameter, and a boundary value of the trimodal internal fusion; The iteration unit uses the training dataset to perform iterative training on the trimodal fusion contrastive learning model based on the loss function. During the iteration process, the parameters of the face and voice feature convolutional networks and the multimodal hybrid encoder are updated by gradient backpropagation until the model reaches the preset training cycle. The trimodal fusion contrastive learning model with the highest score on the validation set is selected and saved as the trained trimodal fusion contrastive learning model. The testing module loads the parameters of the trained trimodal fusion contrastive learning model into a randomly initialized trimodal fusion contrastive learning model, obtains the results of the cross-modal face-voice verification and two-way matching tasks based on the test set, and completes the biometric matching task.

Images