A lightweight identity recognition method combining voiceprint and earprint features

Abstract

The application provides a lightweight identity recognition method combining voiceprint and earprint features. The method comprises the following steps: obtaining voice and ear canal echo signals of a known registered person and a person to be verified; extracting and fusing 13-dimensional mel-frequency cepstral coefficients (MFCC) of the voice and ear canal echo signals, i.e., obtaining voiceprint and earprint fusion features; inputting the fusion features into a lightweight identity recognition model to extract 128-dimensional embedding features of the known registered person and the person to be verified; calculating the similarity of the embedding features of the two types of persons by using a probabilistic linear discriminant analysis (PLDA) method; and determining whether the person to be verified is the known registered person according to the similarity. In the application, the fusion of earprint and voiceprint features improves the recognition performance of the classification model. The lightweight identity recognition model is obtained through a pre-training process, which reduces the equal error rate (EER) and greatly reduces the parameter quantity of the identity recognition model.

Description

Technical Field

[0001] This invention relates to the field of deep learning model technology, and in particular to a lightweight identity recognition method that combines voiceprint and earprint features. Background Technology

[0002] With the rapid development of technologies such as artificial intelligence, various identity recognition technologies have emerged, such as iris recognition, fingerprint recognition, facial recognition, and voiceprint recognition. Voiceprint recognition technology has a relatively low accuracy rate compared to other biometric technologies because voice data collection is easily affected by environmental noise, thus reducing recognition accuracy. Combining voiceprint data with earprint data can reduce interference from environmental noise and improve the accuracy of identity recognition. It should be noted that the earprint features described in the claims of this invention refer to the characteristics of the echo signal in the human ear canal, not to the external texture features of the ear like fingerprints.

[0003] Early patents provided methods for extracting acoustic features of ear prints (e.g. [1] Yu Anbo. An online method and system for identifying acoustic features of ear prints suitable for noise-canceling headphone design [P]. Beijing: CN201610842206.8, 2017-08-22.), but did not involve how to use ear print features for identity recognition.

[0004] In recent years, some patents have provided methods for identity recognition using earprint features (such as patent [2] Huang Qinbo, Guo Quan, Li Jing. Privacy recognition method and device based on earprint [P]. Guangdong: CN201610751856.1, 2018-03-09; patent [3] J.P. Lesso, W.E. Sherwood, P. Bardsley, et al. Biometric process, device and machine-readable medium [P]. UK: CN202080013151.4, 2021-09-17.). For example, patent [2] utilizes the distinctiveness of earprints between people, and identifies whether the person to be verified is a known authorized person by matching the earprint of the person to be verified with the earprint of a known authorized person. However, such patents do not involve methods for using voiceprints and earprints together for identity recognition.

[0005] Furthermore, patent [4] (Dai Zuhua, Zhang Yinong. Target user verification method, device, electronic device and readable storage medium [P]. Jiangsu Province: CN202311661230.8, 2024-03-26.) proposes a method for identity recognition using earprint and other biometric features (including voiceprint), but in this method, earprint and voiceprint features are used for identity recognition through first and second verification models respectively. In contrast, this invention fuses earprint and voiceprint features and inputs them into a common identity recognition model, and focuses on describing the lightweight structure of the identity recognition model to meet the needs of the recognition model used in small smart devices such as headphones.

[0006] Small smart devices such as Bluetooth headsets have limited parameter storage and computational capabilities, while deep learning neural network models generally have a large number of parameters and computational demands. Therefore, to make deep learning models suitable for small devices with limited storage and computational capabilities, it is necessary to reduce the number of model parameters, i.e., model lightweighting. However, the challenge of model lightweighting is how to balance the contradiction between reducing the number of model parameters and improving performance, because reducing the number of parameters will lead to a decrease in model performance. The method proposed in this invention achieves a significant reduction in the number of model parameters compared to existing methods, while maintaining a relatively good error rate (EER). Summary of the Invention

[0007] The purpose of this invention is to solve the contradiction between reducing the number of model parameters and improving performance when using voiceprint and earprint for identity recognition on small smart devices such as Bluetooth headsets. Compared with the method of identity recognition using only voiceprint data, this invention can significantly reduce the number of parameters of the identity recognition model while reducing the equal error rate (EER).

[0008] To achieve the above-mentioned objectives, this invention provides a lightweight identity recognition method that combines voiceprint and earprint features. The technical problem to be solved by this invention is achieved through the following technical solution:

[0009] This invention provides a lightweight identity recognition method that combines voiceprint and earprint features, the identity recognition method comprising:

[0010] Step 1: Obtain the speech and ear canal echo signals of the known registered user;

[0011] Step 2: Extract the 13-dimensional Mel-Frequency Cepstral Coefficients (MFCCs) of the known registered person's speech and ear canal echo signals respectively, and fuse the Mel-Frequency Cepstral Coefficients of the two to obtain the fused features of the known registered person's voiceprint and earprint.

[0012] Step 3: Input the fused features obtained in Step 2 into the lightweight identity recognition model and output a 128-dimensional embedded feature vector;

[0013] Step 4: Obtain the speech and ear canal echo signals of the person to be verified;

[0014] Step 5: Extract the 13-dimensional Mel frequency cepstral coefficients of the speech and ear canal echo signals of the person to be verified, and fuse the Mel cepstral feature coefficients of the two to obtain the fused features of the voiceprint and earprint of the person to be verified.

[0015] Step 6: Input the voiceprint and earprint fusion features obtained in Step 5 into the lightweight identity recognition model, and output a 128-dimensional embedded feature vector;

[0016] Step 7: Compare the 128-dimensional embedded feature vectors output from Step 3 and Step 6 respectively. If the similarity is greater than the threshold, the person to be verified is determined to be a known registrant; otherwise, the person to be verified is determined not to be a known registrant.

[0017] The lightweight identity recognition model is used to extract high-level abstract features of the fusion features of voiceprint and earprint layer by layer to obtain the representation features at the signal frame level. The lightweight identity recognition model includes a multi-layer neural network and a fully connected structure, wherein each layer of the neural network includes multiple fully connected layers, activation functions, and stacking layers in sequence, and the fully connected structure includes a fully connected layer, activation function, statistical pooling layer, and fully connected output layer in sequence.

[0018] In one embodiment of the invention, the lightweight identity recognition model includes a multi-layer neural network comprising:

[0019] Concat layer: used to stack the output features of the multiple fully connected layers, resulting in a 64-dimensional feature output.

[0020] In one embodiment of the invention, the fully connected structure in the lightweight identity recognition model includes:

[0021] Statistical pooling layer: 128×256, where 128 represents 128-dimensional input and 256 represents 256-dimensional output, used to calculate the mean and standard deviation of input features to aggregate frame features.

[0022] Fully connected output layer: 256×128Dense, where 256 represents 256-dimensional input and 128 represents 128-dimensional embedded features at output.

[0023] In one embodiment of the invention, the similarity calculation in step 7 uses probabilistic linear discriminant analysis (PLDA) to calculate the similarity of the embedded layer features between the person to be verified and the registered person.

[0024] In one embodiment of the invention, the lightweight identity recognition model needs to be pre-trained. The pre-training includes two models: one is a delayed neural network using ECAPA (Emphasized Channel Attention, Propagation and Aggregation) as pre-trained model 1, and the other is the lightweight identity recognition model as pre-trained model 2. The training process of the two pre-trained models is as follows:

[0025] Acquire training sample speech and ear canal echo signals, and label the training sample speech and ear canal echo signals to form labeled training samples;

[0026] Extract the 13-dimensional Mel frequency cepstral coefficients of each tagged speech and ear canal echo signal, and fuse the Mel cepstral feature coefficients of the two to obtain the voiceprint and earprint fusion features;

[0027] All the tagged voiceprint and earprint fusion features are input into pre-trained model 1 and pre-trained model 2 respectively;

[0028] The total loss functions for pre-trained model 1 and pre-trained model 2 are defined as follows:

[0029]

[0030] in These are the feature vectors output by pre-trained model 1 and pre-trained model 2, respectively, and their scores are... 1 and score 2 These represent the class probability scores of the output features of pre-trained model 1 and pre-trained model 2, respectively; N represents the number of samples, and M represents the total number of classes; y k It is a sign function; it takes the value 1 if the true class of sample i is equal to c, and 0 otherwise. α is the predicted probability that sample i belongs to category c, and α (0 < α < 1) is the weight used to balance the two loss functions.

[0031] After the pre-trained model is trained, the parameters of pre-trained model 2 are saved, thus obtaining the trained lightweight identity recognition model.

[0032] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0033] (1) In view of the problem that voice signals are easily interfered with and the accuracy of voiceprint identity recognition decreases, this invention proposes a method to fuse voiceprint and earprint data features to improve the anti-interference ability of the identity recognition model.

[0034] (2) This invention addresses the problem that large-parameter identity recognition models cannot be deployed on small smart devices such as Bluetooth headsets by constructing a lightweight identity recognition model. First, the dimensionality of the model input data features is reduced, that is, only the 13-dimensional Mel-frequency cepstral coefficients of voiceprint and earprint data are input. Second, while reducing the number of model parameters, the model is pre-trained from a large-parameter pre-trained model to ensure the recognition accuracy of the lightweight model, thereby effectively balancing the contradiction between the number of model parameters and performance improvement.

[0035] (3) Compared with the method of identity recognition using only voiceprint data, the present invention reduces the error rate (EER) and significantly reduces the number of parameters in the identity recognition model.

[0036] Figure and Table Description

[0037] Figure 1 This is a flowchart of a lightweight identity recognition method that combines voiceprint and earprint features, provided by an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the lightweight identity recognition model provided in an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions in the embodiments of this patent will be further described in detail below with reference to the accompanying drawings and specific examples.

[0040] The specific embodiments of this patent are described in detail below. It should be understood that the embodiments described in the accompanying drawings are only for explaining the implementation method of this patent, and not for limiting this patent.

[0041] The lightweight identity recognition method based on combined voiceprint and earprint features provided in this invention can be applied to terminal devices and can be implemented by combining software and hardware devices, and integrated into terminal devices, such as Bluetooth headsets.

[0042] See Figure 1 The present invention will now describe a lightweight identity recognition method combining voiceprint and earprint features according to an embodiment of the present invention. The specific implementation method of this embodiment includes the following steps:

[0043] Step 1: Obtain the voice and ear canal echo signals of the known registrant.

[0044] Specifically, the registered personnel are provided with a wired headset with playback and recording functions. The other end of the headset is connected to a sound card device, and a two-second Gaussian white noise is played. The echo signals obtained after the Gaussian white noise passes through the left and right ear canals of the registered personnel are collected through the headset recording. The voice of the registered personnel speaking into the microphone connected to the headset is recorded for about 3 seconds.

[0045] The ear canal echo signal is Gaussian white noise and does not require preprocessing. For the speech signal, dual threshold endpoint detection is required. The short-time energy and zero-crossing rate of each frame of speech signal are calculated. Based on the threshold, voiced segments are retained and unvoiced and silent segments are removed.

[0046] Step 2: Extract the 13-dimensional Mel-Frequency Cepstral Coefficients (MFCCs) of the known registered person's speech and ear canal echo signals respectively, and fuse the Mel-Frequency Cepstral Coefficients of the two to obtain the fused features of the known registered person's voiceprint and earprint.

[0047] Specifically, the process of extracting the 13-dimensional Mel frequency cepstral coefficients is as follows: for the 2-second ear canal echo signal and the preprocessed 1-second speech signal, pre-emphasis, framing, windowing, fast Fourier transform, Mel filter bank, logarithmic operation, discrete Fourier transform, and dynamic feature extraction are performed respectively. The selected frame length is 256 points, the frame shift is 160 points, the window length is 512, the pre-emphasis coefficient is 0.97, and 26 Mel filters are used to extract the 13-dimensional Mel cepstral coefficients, which include the first dimension of energy.

[0048] Extract the 13-dimensional Mel-frequency cepstral coefficients of the ear canal echo signal, denoted as A = [A1, A2, A3, ..., A...]. 13 Extract the 13-dimensional Mel-frequency cepstral coefficients of the speech signal, denoted as B = [B1, B2, B3, ..., B]. 13 By concatenating A and B in dimensional order, we obtain a 13-dimensional vector that integrates the ear canal echo and the Mel-frequency cepstral coefficients of the speech signal, denoted as C = [C1, C2, C3, ..., C]. 13 ], where C i ={A i B i}, the symbol {,} indicates concatenation, i∈{1,2,...,13}.

[0049] Step 3: Input the fused features obtained in Step 2 into the lightweight identity recognition model and output a 128-dimensional embedded feature vector.

[0050] Specifically, such as Figure 2 As shown, the lightweight identity recognition model includes a multi-layer neural network and a fully connected structure. Each layer of the neural network includes multiple fully connected layers, activation functions, and stacking layers. The fully connected structure includes fully connected layers, activation functions, statistical pooling layers, and a fully connected output layer.

[0051] In this multi-layer neural network, the first layer consists of five fully connected linear interconnects and one stacked layer: the first four fully connected layers are 13×12Dense, and the fifth is 13×16Dense. 13×12 represents a 13-dimensional input and a 12-dimensional output, and 13×16 represents a 13-dimensional input and a 16-dimensional output. Finally, the outputs of these five fully connected layers are concatenated into the stacked layer to obtain a 64-dimensional output. This neural network setup is equivalent to decomposing a single 65×64Dense model into the aforementioned five fully connected layers, which can significantly reduce the number of model parameters. For example, a 65×64Dense model has 4160 parameters, while using four 13×12Dense layers and one 13×16Dense layer results in a total of 832 parameters. This demonstrates that the number of model parameters is reduced to one-fifth of that of a single 65×64Dense model, making the model more lightweight.

[0052] In a multilayer neural network, the second layer consists of three fully connected linear layers and one concatenated layer: the first two fully connected layers are 64×20 density, and the third is 64×24 density. 64 is the output dimension of the first layer, which should equal the input dimension of the second layer. Finally, the outputs of these three fully connected layers are concatenated into the concatenated layer to obtain a 64-dimensional output. This neural network setup is equivalent to decomposing a single 192×64 density layer into the three fully connected layers described above, as mentioned in the first layer section, which significantly reduces the number of model parameters.

[0053] In a multilayer neural network, the third layer consists of three fully connected linear layers and one concatenated layer: the first two fully connected layers are 64×20 density, and the third is 64×24 density. 64 is the output dimension of the first layer, which should be equal to the input dimension of the second layer. Finally, the outputs of these three fully connected layers are concatenated into the concatenated layer to obtain a 64-dimensional output. This neural network setup is equivalent to decomposing a single 192×64 density layer into the three fully connected layers described above, as mentioned in the first layer section, which can significantly reduce the number of model parameters.

[0054] Furthermore, such as Figure 2 As shown, the fully connected layer in the fully connected structure of the lightweight identity recognition model consists of two fully connected layers: 64×64Dense and 64×128Dense. 64×64 indicates that both the input and output are 64-dimensional, and 64×128 indicates that the input and output are 64 and 128-dimensional respectively. The 64-dimensional features of the previous layer are enhanced to 128-dimensional to extract high-level features.

[0055] The statistical pooling layer in the fully connected structure of the lightweight identity recognition model is: 128×256StatsPooling, where 128×256 indicates that the input and output are 128 and 256 dimensions respectively. The mean and standard deviation of the input features are calculated to obtain a 256-dimensional output, which aggregates frame features.

[0056] The fully connected output layer in the fully connected structure of the lightweight identity recognition model is a linear fully connected layer: 256×128Dense. The 256-dimensional output of the statistical pooling layer is used as the input of this layer, and the output is a 128-dimensional embedded feature vector.

[0057] Step 4: Obtain the voice and ear canal echo signals of the person to be verified.

[0058] Specifically, the person to be verified is given a wired headset with playback and recording functions. The other end of the headset is connected to a sound card device. A two-second Gaussian white noise is played, and the echo signals obtained after the Gaussian white noise passes through the left and right ear canals of the person to be verified are collected through the headset recording. The voice of the person to be verified is recorded into the microphone connected to the headset for about 3 seconds.

[0059] The ear canal echo signal is Gaussian white noise and does not require preprocessing. Dual threshold endpoint detection is performed on the speech signal to calculate the short-time energy and zero-crossing rate of each frame of speech signal. Voiced segments are retained according to the threshold, while unvoiced and silent segments are removed.

[0060] Step 5: Extract the 13-dimensional Mel frequency cepstral coefficients of the speech and ear canal echo signals of the person to be verified, and fuse the Mel cepstral feature coefficients of the two to obtain the fused features of the voiceprint and earprint of the person to be verified.

[0061] Specifically, the process of extracting the 13-dimensional Mel frequency cepstral coefficients and obtaining their fusion features is the same as described in step 2 above.

[0062] Step 6: Input the voiceprint and earprint fusion features obtained in Step 5 into the lightweight identity recognition model, and output a 128-dimensional embedded feature vector;

[0063] Specifically, it is the same as described in step 3 above.

[0064] Step 7: Compare the 128-dimensional embedded feature vectors output from Step 3 and Step 6 respectively. If the similarity is greater than the threshold, the person to be verified is determined to be a known registrant; otherwise, the person to be verified is determined not to be a known registrant.

[0065] Specifically, in step 7, the similarity calculation uses probabilistic linear discriminant analysis (PLDA) to calculate the similarity of the embedded layer features between the person to be verified and the registered person. If the similarity value exceeds a set threshold, the identity verification is successful, meaning the registered person and the person to be verified are the same user; otherwise, the identity verification fails, meaning the person to be verified is not a registered person.

[0066] In the above embodiments, the lightweight identity recognition model needs to be pre-trained. The pre-training includes two models: one is a time-delayed neural network using ECAPA (Emphasized Channel Attention, Propagation and Aggregation) as pre-trained model 1, and the other is the lightweight identity recognition model as pre-trained model 2. The training process of the two pre-trained models is as follows:

[0067] Acquire training sample speech and ear canal echo signals, and label the training sample speech and ear canal echo signals to form labeled training samples.

[0068] Specifically, the process of acquiring training sample speech and ear canal echo signals is as follows: First, the data collectors (hereinafter referred to as the subjects) are numbered sequentially. Second, each subject wears an earphone with playback and recording functions, the other end of which is connected to a sound card. A two-second Gaussian white noise is played, and the echo signals of this Gaussian white noise passing through the subject's left and right ear canals are recorded through the earphone. Each subject needs to wear the earphone 15 times in each ear, removing and re-wearing it each time to ensure a certain degree of randomness. Each person has 30 data entries, each 2 seconds long, and at least 100 people are collected, including both males and females. Speech signal acquisition only requires recording. To increase the robustness of the model, the recording content is different for each person, and each person's recording is 3 seconds long. Finally, based on the subject's number, the speech and ear canal echo signals of each subject are labeled, and each label is unique and non-repeating, forming a tag for each subject's speech and ear canal echo signals.

[0069] Furthermore, the ear canal echo signal is Gaussian white noise and does not require preprocessing, while the speech signal needs to undergo dual-threshold endpoint detection to calculate the short-time energy and zero-crossing rate of each frame of speech signal. Based on the set threshold, voiced segments are retained, while unvoiced and silent segments are removed.

[0070] Extract the 13-dimensional Mel-frequency cepstral coefficients of each tagged speech and ear canal echo signal, and fuse the Mel-frequency cepstral coefficients of the two to obtain the voiceprint and earprint fusion features.

[0071] Specifically, the process of extracting the 13-dimensional Mel frequency cepstral coefficients and obtaining their fusion features is the same as described in step 2 above.

[0072] All the fused voiceprint and earprint features with labels are input into pre-trained model 1 and pre-trained model 2 respectively, and the total loss functions of pre-trained model 1 and pre-trained model 2 are set as follows:

[0073]

[0074] in These are the feature vectors output by pre-trained model 1 and pre-trained model 2, respectively, and their scores are... 1 and score 2 These represent the class probability scores of the output features of pre-trained model 1 and pre-trained model 2, respectively; N represents the number of samples, and M represents the total number of classes; y k It is a sign function; it takes the value 1 if the true class of sample i is equal to c, and 0 otherwise. α is the predicted probability that sample i belongs to category c, and α (0 < α < 1) is the weight used to balance the two loss functions.

[0075] Train the pre-trained model until the total loss function is less than a threshold or the output error rate (EER) no longer decreases. After training, save the parameters of the pre-trained model 2 to obtain the trained lightweight identity recognition model.

[0076] According to the above-described embodiment of a lightweight identity recognition method combining voiceprint and earprint features, the number of parameters and recognition performance of its lightweight model are shown in Table 1. It can be seen that, compared to methods that rely solely on voiceprint data for identity recognition, this invention significantly reduces the number of parameters in the identity recognition model while lowering the equal error rate (EER).

[0077] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0078] Table 1 compares the lightweight performance of a lightweight identity recognition method provided in the embodiments of the present invention.

[0079]

[0080] [1] Casanova, Edresson, et al. "Speech2Phone: a novel and efficient method for training speaker recognition models." Brazilian Conference on IntelligentSystems. Cham: Springer International Publishing, 2021.

[0081] [2]Chung, Joon Son, et al. "In defence ofmetric learning for speakerrecognition." arxiv preprint arxiv:2003.11982,2020.

[0082] [3]Wan, Li, et al. "Generalized end-to-end loss for speaker verification." 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018.

[0083] [4] Zhu Zhen, Wu Jing, Zhu Wenbo, et al. A lightweight single-channel voiceprint recognition method and system based on deep learning. 202310168991, 2024.

Claims

1.A lightweight identity recognition method combining voiceprint and earprint features, characterized in that: After the voiceprint and earprint features of the person to be verified are fused, they are input into a lightweight identity recognition model to determine whether the person to be verified is a registered person, comprising the following steps: Step 1: Obtain the speech and ear canal echo signals of a known registered person; Step 2: Extract the 13-dimensional Mel-Frequency Cepstral Coefficients (MFCC) of the speech and ear canal echo signals of the known registered person, respectively, and fuse the Mel cepstral feature coefficients of the two, i.e., obtain the voiceprint and earprint fusion features of the known registered person; Step 3: Input the fusion features obtained in Step 2 into a lightweight identity recognition model to output a 128-dimensional embedding feature vector; Step 4: Obtain the speech and ear canal echo signals of the person to be verified; Step 5: Extract the 13-dimensional Mel-Frequency Cepstral Coefficients (MFCC) of the speech and ear canal echo signals of the person to be verified, respectively, and fuse the Mel cepstral feature coefficients of the two, i.e., obtain the voiceprint and earprint fusion features of the person to be verified; Step 6: Input the voiceprint and earprint fusion features obtained in Step 5 into a lightweight identity recognition model to output a 128-dimensional embedding feature vector; Step 7: Compare the 128-dimensional embedding feature vectors output in Steps 3 and 6, respectively, in terms of similarity, and if the similarity is greater than a threshold value, determine that the person to be verified is the known registered person, otherwise, determine that the person to be verified is not the known registered person; The lightweight identity recognition model is used to extract high-level abstract features of the voiceprint and earprint fusion features layer by layer to obtain signal frame-level representation features; the lightweight identity recognition model includes a multi-layer neural network and a fully connected structure, wherein each layer of the neural network includes a plurality of fully connected layers, an activation function, and a stacking layer in turn, and the fully connected structure includes a fully connected layer, an activation function, a statistical pooling layer, and a fully connected output layer in turn; In the multi-layer neural network of the lightweight identity recognition model, the stacking layer: concat, is used to stack the output features of the plurality of fully connected layers, and finally outputs a 64-dimensional feature; in the fully connected structure of the lightweight identity recognition model, the statistical pooling layer: 128x256, 128 represents the input of 128 dimensions, and 256 represents the output of 256 dimensions, is used to calculate the mean and standard deviation of the input features to aggregate frame features; the fully connected output layer: 256x128Dense, 256 represents the input of 256 dimensions, and 128 represents the output of 128-dimensional embedding features. 2.The lightweight identity recognition method combining voiceprint and earprint features according to claim 1, characterized in that: In Step 7, the similarity is calculated using the Probabilistic Linear Discriminant Analysis (PLDA) method to calculate the similarity of the embedding layer features of the person to be verified and the registered person. 3.The lightweight identity recognition method combining voiceprint and earprint features according to claim 1, characterized in that: The lightweight identity recognition model needs to be pre-trained, and the pre-training includes two models, one is an ECAPA (Emphasized Channel Attention, Propagation and Aggregation) delay neural network as a pre-training model 1, and the other is the lightweight identity recognition model of claim 1 as a pre-training model 2, and the training process of the two pre-training models is as follows: Obtain training sample voice and ear canal echo signals, and label the training sample voice and ear canal echo signals to form labeled training samples, i.e., labeled voice and ear canal echo signals; Extract 13-dimensional mel frequency cepstral coefficients of each labeled voice and ear canal echo signal, and fuse the mel frequency cepstral feature coefficients of the two, i.e., obtain labeled voiceprint and earprint fusion features; Input all labeled voiceprint and earprint fusion features into pre-training model 1 and pre-training model 2, respectively. The total loss function of the pre-training model 1 and the pre-training model 2 is set as follows: wherein are the feature vectors output by the pre-trained model 1 and the pre-trained model 2 respectively, score 1 and score 2 are the class probability scores of the features output by the pre-trained model 1 and the pre-trained model 2 respectively; N represents the number of samples, and M represents the total number of classes; y ic is a symbol function, which takes 1 if the real class of the sample i is equal to c, and 0 otherwise; is the predicted probability value of the sample i belonging to the class c, and a (0 < a < 1) is a weight used to balance the two loss functions. After the pre-training model is trained, the parameters of the pre-training model 2 are saved, i.e., a trained lightweight identity recognition model is obtained.

Images