An identity authentication method using a microphone of an intelligent ear-wearable device

Abstract

The application discloses an identity authentication method using a microphone of an intelligent ear-wearing device, and belongs to the technical field of mobile computing application.The method records the bone conduction heart sound signals of a wearer through an in-ear microphone, analyzes unique biological characteristics therein, and realizes user identity authentication.First, the noise interference fragments in the binaural bone conduction heart sound signals recorded by the microphone of the intelligent ear-wearing device are detected.Then, the noise interference in the heart sound signals is eliminated, and the biological characteristics in the binaural bone conduction heart sound signals are extracted.Finally, a user identity authentication deep neural network is trained, the network parameters are continuously updated, the natural changes of the biological characteristics of the user over time are overcome, and accurate recognition of the user identity is realized.The application can realize reliable user identity authentication only by using the microphone of the intelligent ear-wearing device, does not require the user to perform special behaviors, does not rely on a loudspeaker to play sound, and does not interfere with the user.

Description

Technical Field

[0001] This invention relates to a user authentication method, specifically an authentication method using a microphone from a smart ear-worn device, belonging to the field of mobile computing application technology. Background Technology

[0002] With the widespread adoption of smart ear-worn devices (such as smart headphones) in daily life, the issue of privacy breaches in these devices has attracted considerable attention. Attackers could steal users' smart headphones and eavesdrop on private conversations, or activate the smart device's voice assistant. In recent years, smart headphones have commonly been equipped with multiple microphones facing different directions. Given the increasing sensitivity of personal data, there is an urgent need for authentication mechanisms for smart ear-worn devices to protect this sensitive data from intrusion.

[0003] Traditional authentication methods primarily include those based on user knowledge and those based on biometrics. User knowledge-based methods, such as password unlocking and swipe gesture unlocking, are easily stolen, and smart earphones cannot capture these input passwords and unlocking gestures. Biometric methods can be divided into those based on biological behavior and those based on physiological signals. Currently, a popular technology based on physiological signals utilizes fingerprint recognition for user identification. This method achieves high recognition accuracy, but it relies on specialized sensors and cannot be applied to existing commercial smart earphones. In recent years, multiple microphones facing different directions have been embedded in smart earphones to continuously record sounds from the user's ear and surrounding environment. Some companies use the microphones in smart earphones to record the user's voice for voice authentication. However, due to the lack of liveness detection, it is impossible to distinguish between the played voice and the user's actual voice.

[0004] Currently, research in the field of identity authentication using smart ear-worn devices is still in its early stages. Some researchers, based on the unique characteristics of the human ear canal's contour, have used microphones and speakers within the ear canal to form an active sonar system, mapping the ear canal contour to identify users. Similarly, research has found that changes in the ear canal contour when different users speak can also uniquely identify them. Therefore, theoretically, using microphones and speakers within the ear canal could capture changes in the ear canal contour and distinguish between legitimate and illegitimate users. However, such methods require users to perform specific actions (such as remaining still or saying specific content), and continuously sending audio signals into the ear canal can interfere with the user, affecting the practical effectiveness and user experience.

[0005] In summary, existing methods have various shortcomings, and new methods are urgently needed to overcome their limitations. Summary of the Invention

[0006] The purpose of this invention is to overcome the technical shortcomings of existing ear-worn device-based authentication methods, such as requiring users to perform special actions and resulting in poor user experience. This invention creatively proposes an authentication method utilizing the microphone of a smart ear-worn device. This method records the wearer's bone conduction heart sound signals through an in-ear microphone, analyzes the unique biometric characteristics contained within, and thereby achieves user authentication.

[0007] The principle of this invention is as follows: Heart sounds are generated by the heartbeat, which depends on the unique geometry and motor function of the human heart. Heart sounds propagate through bones and muscles, undergoing complex absorption and diffusion before finally reaching the ear canal. Propagation characteristics are influenced by factors such as bone structure, bone density, and the bone-to-muscle ratio. Furthermore, because the heart is located slightly to the left of the chest, and individual body structures vary, bone conduction heart sound signals recorded by both ears are also unique. Using the in-ear microphones commonly embedded in smart ear-worn devices, bone conduction heart sounds can be recorded simultaneously in both ears. The unique heartbeat activity information, bone conduction information, and human structural information obtained from these recordings can be used to identify the user.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] An authentication method using a microphone in a smart ear-worn device includes the following steps:

[0010] Step 1: Detect noise-affected segments in the bone conduction heart sound signals recorded by the microphone of the smart ear-worn device.

[0011] The microphones embedded in smart ear-worn devices face into the ear canal and are sensitive to music played through headphones and sounds generated by human movement. This causes noise interference in the recorded bone conduction heart sounds, affecting authentication performance. Therefore, by utilizing the periodicity of heart sound signal segments and the periodic disruption of noise-interfered sound segments, noise-affected segments in the heart sound signals recorded by the ear-worn device's microphone can be detected.

[0012] Step 2: Eliminate noise interference in the heart sound signal.

[0013] Specifically, it includes the following steps:

[0014] Step 2.1: Process the heart sound signal segment affected by noise obtained in Step 1 using a low-pass filter to eliminate high-frequency interference that does not overlap with the frequency of the heart sound signal.

[0015] Step 2.2: Use nonnegative matrix decomposition to further eliminate noise and reconstruct a pure bilateral bone conduction heart sound signal.

[0016] Step 3: Extract biomarkers from the bone conduction heart sound signals from both ears.

[0017] Specifically, it includes the following steps:

[0018] Step 3.1: Analyze the biometric information contained in the bone conduction signals from both ears, and design heart rate activity characteristics, bone conduction characteristics, and human structural characteristics.

[0019] Step 3.2: Combine and recombine the biometric information obtained in Step 3.1 to obtain a feature image for subsequent processing.

[0020] Step 4: Identify the user.

[0021] Specifically, it includes the following steps:

[0022] Step 4.1: Develop a deep neural network for user authentication.

[0023] The feature images obtained in step 3.2 are processed using convolutional neural networks (CNNs) to establish an accurate user authentication model.

[0024] Step 4.2: Using continuous learning techniques, train the user authentication model established in Step 4.1 and continuously update the network parameters. This overcomes the natural changes in user biometrics over time and accurately identifies user identities.

[0025] Thus, from step 1 to step 4, the objective of this invention has been achieved, namely: identity authentication based on the microphone of a smart ear-worn device.

[0026] Beneficial effects

[0027] The method of the present invention has the following advantages compared with existing authentication technologies:

[0028] 1. This invention achieves reliable user authentication using only the microphone of a smart ear-worn device. It requires no special user action, does not rely on speakers to play sound, and does not interfere with the user.

[0029] 2. By using an improved nonnegative matrix decomposition method, this invention can effectively eliminate various noise interferences in bone conduction heart sound signals and effectively improve signal quality.

[0030] 3. This invention designs a novel biometric feature that combines heart rate activity, bone conduction, and human structural features. The proposed feature cannot be stolen from external sources or artificially generated, and thus possesses advantages such as protection against data leakage and attack.

[0031] 4. This invention proposes a deep learning model for user authentication. By using CNN and continuous learning techniques, this method can achieve reliable authentication using only limited registration data and can overcome the natural changes in user biometrics over time. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the identity authentication method using the microphone of a smart ear-worn device according to the present invention.

[0033] Figure 2 The three matrices are composed of features designed for this invention;

[0034] Figure 3 This is a diagram of the user-independent blood pressure measurement hybrid neural network structure developed for this invention;

[0035] Figure 4 This refers to the FAR results for each participant in a static scene according to an embodiment of the present invention;

[0036] Figure 5 This is the FRR result for each participant in a static scenario according to an embodiment of the present invention;

[0037] Figure 6 The average FAR and FRR results for this embodiment of the invention over a continuous 24-hour period are shown.

[0038] Figure 7 The values ​​represent the average FAR and FRR results of the embodiments of the present invention during a long-term experiment lasting 12 weeks. Detailed Implementation

[0039] The method of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] like Figure 1 As shown, an authentication method using a microphone in a smart ear-worn device includes the following steps:

[0041] Step 1: Detect noise-affected segments in the bone conduction heart sound signals recorded by the microphone of the smart ear-worn device.

[0042] The microphones embedded in smart ear-worn devices face into the ear canal and are sensitive to music played through headphones and sounds generated by human movement. This can cause noise interference in the recorded bone conduction heart sounds, affecting authentication performance. By utilizing the periodicity of heart sound signal segments and the periodic disruption of noise-interfered sound segments, this method detects noise-affected segments in the heart sound signals recorded by the ear-worn device's microphones.

[0043] Specifically, the acquired signals can be processed using a sliding window with a size of 60s and a step size of 30s. The autocorrelation function (i.e., the cross-correlation function between the signal itself and the signal at different lag steps) is calculated for each signal segment within the sliding window. If the signal segment within the sliding window exhibits periodicity, the time lag corresponding to the local maximum value of the autocorrelation function is the signal period. Since heart sound signals have the same period as heartbeats, approximately 0.5s to 1s, based on this characteristic, the obtained autocorrelation function is calculated to determine if a maximum value appears between 0.5 and 1s. If a maximum value appears, the segment is considered unaffected by noise; otherwise, the segment is considered affected by noise.

[0044] Step 2: Eliminate noise interference in the heart sound signal.

[0045] Step 2.1: Process the heart sound signal segment affected by noise obtained in Step 1 using a low-pass filter to eliminate high-frequency interference that does not overlap with the frequency of the heart sound signal.

[0046] Specifically, since the frequency of bone conduction heart sound signals is mainly below 160Hz, a fourth-order Butterworth low-pass filter with a cutoff frequency of 160Hz can be used to eliminate high-frequency interference in the acquired ear canal sound signals that do not overlap with the frequency of the heart sound signals.

[0047] Step 2.2: Further eliminate noise using nonnegative matrix factorization technology to reconstruct pure bilateral bone conduction heart sound signals.

[0048] Specifically, the sound in the ear canal collected by the in-ear microphone of the smart ear-wearing device is denoted as x(n), which is mixed with multiple different sound sources, including bone conduction heart sound signal s(n), headphone audio input a(n) and other body sounds b(n), and is expressed as x(n) = s(n) + a(n) + b(n), where n marks the signal sequence.

[0049] Furthermore, this paper proposes a noise reduction method based on nonnegative matrix factorization, taking x(n) as input and aiming to extract s(n). Specifically:

[0050] First, the input audio signal is subjected to a short-time Fourier transform to obtain the time spectrum x(f,t) = s(f,t) + a(f,t) + b(f,t), where the frequency f ∈ [1,F] and the time t ∈ [1,T], and F and T are constants representing the maximum values ​​of the frequency and time, respectively. The time spectrum x(f,t) is then represented as a time spectrum matrix. x = [|x(1,t)|,…,|x(F,t)|] TT represents the matrix transpose operator. Let the time-frequency spectra s(f,t), a(f,t), and b(f,t) be represented as follows: and It represents the set of real numbers.

[0051] Nonnegative matrix factorization is a technique that decomposes the time-spectrum matrix into a nonnegative basis matrix. and a nonnegative coefficient matrix The product separation of multiple sound sources with different local features:

[0052]

[0053] in, It is an estimated matrix. K is a positive integer less than T and F, representing the number of columns in matrix W and the number of rows in matrix H.

[0054] Ideally, by estimating Clustering sub-components with different local features can yield clean bone conduction heart sound signals. Headphone audio input and body sounds Where H = [H S H A H B ] T W = [W S W A W B ], T represents the matrix transpose operator; W S W represents the non-negative basis matrix obtained by decomposing bone conduction heart sound signals. A W represents the non-negative basis matrix obtained by decomposing the headphone audio input signal. B H represents the non-negative basis matrix obtained from the decomposition of the body sound signal. S H represents the non-negative coefficient matrix obtained from the decomposition of bone conduction heart sound signals. A H represents the nonnegative coefficient matrix obtained by decomposing the headphone audio input signal. B This represents the non-negative coefficient matrix obtained from the decomposition of the body sound signal.

[0055] However, in reality, besides bone conduction heart sound signals, headphone audio input, and body sounds, the sounds recorded by the in-ear microphone also contain a small amount of diverse and unpredictable noise interference within the ear canal. Therefore, this method further incorporates unsupervised components. W un H represents the nonnegative basis matrix obtained from unsupervised component decomposition. unThis represents the non-negative coefficient matrix obtained from unsupervised component decomposition, used to handle various interferences not addressed during the training phase, ensuring that the bone conduction heart sound signal after noise reduction has high quality.

[0056] To estimate W and H in Equation 1, the loss function for the training process is defined as follows:

[0057]

[0058] Where, η S η A and η B Let H be the weights of the bone conduction heart sound signal, headphone audio input, body sound, body vibration, and motion noise components during nonnegative matrix factorization; D is the difference between the real signal and the estimated signal, defined as the sum of the Kullback-Leibler divergence and the first normal form of H:

[0059]

[0060] Where μ is the weight of the first normal form of H.

[0061] By minimizing the loss function, i.e., minimizing Equation 2, a pure bone conduction heart sound signal is extracted from the collected sound signal within the ear canal. Specifically, the gradient descent method is used to iteratively update W and H alternately, with the update rule as shown in Equation 4:

[0062]

[0063] Where W and H are initialized as random nonnegative matrices; X represents the time-frequency spectrum matrix; T represents the matrix transpose; The multiplication and division methods represent the multiplication and division of matrix elements, respectively. O is a matrix consisting of F rows and T columns, with all elements being 1.

[0064] Once W and H are obtained, separation is achieved by comparing the spectral similarity between the sub-components and the pure bone conduction heart sound signal saved at registration. Bone conduction heart sound signals, headphone audio input, and body sounds.

[0065] Specifically, calculate separately The cosine distance between sub-components at different frequencies throughout the entire time period and the pure bone conduction heart sound signal saved during user registration is calculated. If the cosine distance exceeds a threshold γ, the sub-component is considered to originate from the bone conduction heart sound signal. All sub-components with cosine distances exceeding the threshold γ are clustered to obtain the time-spectrum matrix of the bone conduction heart sound signal. Finally, the pure bone conduction heart sound signal s(n) is recovered using the inverse short-time Fourier transform.

[0066] Step 3: Extract biomarkers from the bone conduction heart sound signals from both ears.

[0067] Step 3.1: Analyze the biometric information contained in the bone conduction signals from both ears, and design heart rate activity characteristics, bone conduction characteristics, and human structural characteristics.

[0068] Since commonly used statistical features cannot capture the complex biological features in bone conduction heart sound signals, this method designs a novel biological feature representation method to describe the biological feature information in bone conduction heart sound signals from three aspects: user heartbeat activity features, bone conduction features, and human body structural features.

[0069] Heartbeat activity characteristics: Mel-Frequency Cepstral Coefficients (MFCCs) describe the linear and nonlinear characteristics of sound signals across different frequency ranges, and can capture unique biological information in the heart sounds generated by different users' heart movements.

[0070] Specifically, this method extracts 16th-order Mel frequency cepstral coefficient features (a total of 16×2 features) from bone conduction heart sound signals collected from the left ear and the right ear, respectively.

[0071] Bone conduction characteristics: Linear prediction coefficients (LPCs) can extract the propagation characteristics of sound in a specific channel. This method treats the human skeleton and muscles as special propagation channels and uses linear prediction to extract the propagation differences caused by differences in bones and muscles in the bone conduction heart sound signals of different users.

[0072] Specifically, this method extracts 16th-order linear prediction coefficient features (a total of 16×2 features) from bone conduction heart sound signals collected from the left ear and the right ear, respectively.

[0073] Human structural characteristics: Because the heart is located slightly to the left of the center of the chest cavity, heart sounds travel through different pathways to reach the left and right ears, resulting in differences in bone conduction heart sound signals collected from the left and right ears. These differences depend on the individual user's anatomy. Based on the finding that bone conduction heart sound signals collected from the left and right ears show varying degrees of enhancement or attenuation within different frequency ranges, the differences in left and right ear signals at different frequencies were quantified.

[0074] First, the bone conduction heart sound signals collected from the left and right ears were divided into 32 equal sub-bands (each with a bandwidth of 5Hz) in the frequency range of 0Hz to 160Hz.

[0075] Then, the Euclidean distance between the bone conduction heart sound signals collected by the left and right ears within each frequency band was calculated. The Euclidean distances of the 32 frequency bands were recorded as a 32-dimensional feature vector according to the frequency from low to high.

[0076] Step 3.2: Combine and recombine the biometric information obtained in Step 3.1 to obtain a feature image for subsequent processing.

[0077] To avoid building multiple user authentication classifiers for three heterogeneous features, this method merges heart rate activity features, bone conduction features, and human body structure features into a single feature image for subsequent processing.

[0078] Specifically, the bone conduction heart sound signals collected from the left and right ears are first divided into non-overlapping windows of not less than 120ms. A window size of 130ms is preferred because the human heart sound signal cycle is usually no more than 130ms, and other settings within [120ms, ∞) are also within the scope of this invention.

[0079] Then, a Hamming window was applied to divide the data within the window into 32 overlapping frames of 8ms each, and heart rate activity features, bone conduction features, and human body structure features were extracted from each frame.

[0080] Finally, these features are combined into three matrices (such as...). Figure 2 As shown in the diagram: The first matrix consists of the heartbeat activity features (16*32) and bone conduction features (16*32) of the left ear bone conduction heart sound signal within the window, with a size of 32*32; the second matrix consists of the heartbeat activity features (16*32) and bone conduction features (16*32) of the right ear bone conduction heart sound signal within the window, with a size of 32*32; the third matrix consists of human body structural features within the window, with a size of 32*32. The element values ​​of the three matrices are normalized to integers between 0 and 255 using the maximum-minimum normalization method. The mapping rule is shown in Equation 5:

[0081]

[0082] in, and q i,j Let i and j represent the elements in the i-th row and j-th column of the updated and original matrices, respectively. and These represent the minimum and maximum values ​​of the elements in the j-th column of the matrix, respectively. The three matrices are mapped and combined to form an RGB image as the final feature image, with the three matrices corresponding to the red, green, and blue channels, respectively.

[0083] Step 4: Identify the user.

[0084] Step 4.1: Develop a deep neural network for user authentication. Use a convolutional neural network (CNN) to process the feature images obtained in step 3.2 to build an accurate user authentication model.

[0085] Develop a deep neural network model based on CNN, with the network structure as follows: Figure 3 As shown, it consists of three consecutive convolutional blocks, a flattening layer, a fully connected layer, and an output layer. The convolutional blocks extract hidden user identity information from the input feature image. Each convolutional block includes a 2D convolutional layer using the ReLU function and a pooling layer. The three convolutional layers use 32, 64, and 64 kernels respectively, with a kernel size of 3×3. The flattening layer transforms the extracted multidimensional representation into a one-dimensional vector. The fully connected layer is set to 64 neurons to map the extracted features to real numbers. The output layer uses the softmax function to estimate the probability that the current user is a specific user and outputs the authentication result.

[0086] Step 4.2: Use continuous learning to train the user authentication model established in Step 4.1, continuously update the network parameters, overcome the natural changes in user biometrics over time, and accurately identify user identities.

[0087] Furthermore, this method designs an Elastic Weight Consolidation (EWC) approach to continuously calibrate the developed deep neural network for user authentication, enabling it to achieve reliable authentication with only a small amount of training data and adapt to the natural changes in user biometrics over time. EWC protects authentication performance by constraining the range of parameter variations in existing user authentication models and continuously updates the parameters in the user authentication model to adapt to changes in biometrics.

[0088] The method for consolidation based on plasticity weights is as follows:

[0089] First, a threshold τ is set as the minimum probability of identifying legitimate users.

[0090] When a new signal segment is fed into the user authentication model, the probability p that the user is a legitimate user is calculated and compared with τ. If p is greater than τ, the sample is pseudo-labeled as a positive value (legitimate user); otherwise, the sample is pseudo-labeled as a negative value (attacker).

[0091] The samples are sorted according to the probability of being classified as legitimate users. The K heart sound segments with the highest probability of being classified as legitimate users are constructed into a positive class dataset, assuming that all data in this dataset comes from a single legitimate user. The K heart sound segments with the lowest probability of being classified as legitimate users are constructed into a negative class dataset, assuming that all data in this dataset comes from attackers. The value of K is preferably set to 70%, but other values ​​between 50% and 100% are also within the scope of this invention.

[0092] Then, the user authentication model is updated using the newly established positive and negative class datasets. If we denote the user authentication model before the update as M and the updated user authentication model as N, both models M and N achieve their optimal parameters by minimizing the residuals (i.e., the difference between the true and estimated values). The loss function L of model N is defined as:

[0093]

[0094] Where, θ MN It refers to each parameter of model N after updating it based on the parameters of model M; model L N (θ MN ) represents the residual when the user authentication model N reaches its optimal solution, and λ adjusts the weights of M compared to N; Γ is the diagonal element of the Fisher information matrix. During continuous learning, new positive and negative class datasets are built each time to update the model. If all historical data are used to train the user authentication model and reach its optimal solution, a global model M will be obtained. * , Each parameter is represented by 'i', which marks the parameter sequence.

[0095] Considering the limited training data in the initial stages of identity authentication model development, the parameters in the model are often inaccurate, affecting subsequent continuous learning. Therefore, a global loss function L is defined for the identity authentication model. * as follows:

[0096] L * =L+c∑ k Ψ k (θ k-1 -θ k ) 2 (7)

[0097] Where k represents the number of times the model is updated during training, θ k Let be the parameters after the k-th update of the identity authentication model; c is a constant used to adjust the weights during each parameter update, which can be set to 0.6, but other settings within (0,1) are also within the scope of this invention. Regularization strength Ψ k =∑ωk / (Δk) 2 Ψ k Subject to parameter θ k The changes in the loss function (Equation 6) and the parameter θ at the k-th and (k-1)-th updates k The combined effect of the change Δk itself.

[0098] Example

[0099] To verify the performance of this method, a prototype earphone with an in-ear microphone was constructed to collect bone conduction heart sound data. A total of 45 participants (26 men and 19 women, aged 20 to 50 years) were recruited for the experiment. During the experiment, participants wore the prototype earphone naturally and recorded approximately 130 to 200 minutes of binaural bone conduction heart sound data.

[0100] False Accepted Rate (FAR) and False Rejected Rate (FRR) are used to evaluate system performance. The FAR is the percentage of unauthorized users accepted by the system; the FRR is the percentage of legitimate users incorrectly rejected. Lower values ​​for these metrics indicate better performance.

[0101] First, the overall performance of this method is tested. The overall performance is evaluated by examining the FAR and FRR in a static scene. Figure 4 and Figure 5 Detailed FAR and FRR results for each participant are shown. The experiment used five-fold cross-validation, where the data was randomly shuffled, with 20% of the data used for testing and the remaining 80% used for training. During the experiment, each participant was treated as a legitimate user, and the remaining participants as attackers. The mean FAR and FRR for the 45 participants were 1.6% and 1.8%, respectively. The results demonstrate that this method achieves good performance, extracting unique biometrics from bone conduction heart sound data and accurately identifying users.

[0102] Secondly, the reliability of this method was tested under noisy conditions. The experiment required five participants to continuously record bilateral bone conduction heart sound data (including a large amount of daily activity) over 24 hours. Figure 6The average FAR and FRR for all five participants across different time periods are shown. Due to the inherent variability of human heartbeats, bone conduction heart sound signals captured by the in-ear microphone vary throughout the day, resulting in slight fluctuations in FAR and FRR. Overall, the FAR results for all users across all time periods were below 3.2%, with an average of 2.0%. The FRR results for all users across all time periods were below 5.0%, with an average of 3.8%. These results demonstrate that this method can overcome noise to achieve reliable identity authentication and exhibits good performance in continuous authentication.

[0103] Finally, the long-term performance of this method was tested. Because user biometrics constantly change over time, the accuracy of identity authentication decreases after several weeks. Verifying the stability of long-term performance is an important aspect of identity authentication. After developing the deep neural network for user authentication, data from each participant was tested at 1, 2, 4, 8, and 12 weeks. The user authentication model was updated each time using the collected data. Figure 7 The average FAR and FRR of all 45 participants at different registration times are shown. It can be observed that this method maintains both FAR and FRR below 2.5% for 12 weeks after registration. Experimental results indicate that this method has long-term stability.

[0104] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is used to explain the present invention. It is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identity authentication using a microphone in a smart ear-worn device, characterized in that, Includes the following steps: Step 1: Detect noise-affected segments in the binaural bone conduction heart sound signals recorded by the microphone of the smart ear-worn device; Step 2: Eliminate noise interference in the heart sound signal; Step 2.1: Process the heart sound signal segment affected by noise obtained in Step 1 using a low-pass filter to eliminate high-frequency interference that does not overlap with the frequency of the heart sound signal; Step 2.2: Further eliminate noise using nonnegative matrix decomposition to reconstruct pure bilateral bone conduction heart sound signals; The sound inside the ear canal collected by the in-ear microphone of the smart ear-wearing device is recorded as... This includes superimposed signals such as bone conduction heart sounds. Headphone audio input Other body sounds , represented as , The signal sequence was labeled; a noise reduction method based on nonnegative matrix factorization was used to... As input, used for extraction ; First, the input audio signal is subjected to a short-time Fourier transform to obtain the time spectrum. , ,time , and Let be constants, representing the maximum values ​​of frequency and time, respectively; the time spectrum... Represented as a time-frequency spectrum matrix , , This represents the matrix transpose operator; time spectrum , and They are respectively represented as , and , Represents the set of real numbers; Nonnegative matrix factorization is achieved by decomposing the time-spectrum matrix into a nonnegative basis matrix. and a nonnegative coefficient matrix The product separation of multiple sound sources with different local features: （1） in, , Less than and A positive integer representing a matrix. The number of columns, which also represents the matrix. number of rows; By estimation Clustering sub-components with different local features yields a clean bone conduction heart sound signal. Headphone audio input and body sounds ,in , , This represents the matrix transpose operator; This represents the non-negative basis matrix obtained from the decomposition of bone conduction heart sound signals. This represents the non-negative basis matrix obtained by decomposing the headphone audio input signal. This represents the non-negative basis matrix obtained from the decomposition of the body sound signal. This represents the non-negative coefficient matrix obtained from the decomposition of bone conduction heart sound signals. This represents the non-negative coefficient matrix obtained by decomposing the headphone audio input signal. This represents the non-negative coefficient matrix obtained from the decomposition of the body sound signal; By increasing unsupervised components , This represents the non-negative basis matrix obtained from unsupervised component decomposition. This represents the nonnegative coefficient matrix obtained from unsupervised component decomposition, used to handle various disturbances not addressed during the training phase. The loss function for the training process is defined as follows: （2） in, , and The weights of the bone conduction heart sound signal, headphone audio input, body sound, body vibration, and motion noise components during the non-negative matrix decomposition process are: The difference between the true signal and the estimated signal is defined as the Kullback-Leibler divergence. The sum of the first normal forms: （3） in, for The weight of the first normal form; By minimizing the loss function, i.e., minimizing Equation 2, a pure bone conduction heart sound signal is extracted from the collected sound signal in the ear canal. The gradient descent method is used to iteratively update the signal. and The update rule is as shown in equation 4: （4） in, and It is initialized as a random nonnegative matrix; Represents the time-frequency spectrum matrix; Indicates matrix transpose; The multiplication and division methods represent the multiplication and division of matrix elements, respectively. , It is a by lines and A matrix consisting of columns, all of which are 1s; When obtained and Subsequently, separation was achieved by comparing the spectral similarity between the sub-components and the pure bone conduction heart sound signals saved at the time of registration. Bone conduction heart sound signals, headphone audio input, and body sounds; Step 3: Extract biomarkers from the bone conduction heart sound signals from both ears; Step 3.1: Analyze the biometric information contained in the bone conduction signals from both ears, and design heart activity characteristics, bone conduction characteristics, and human structural characteristics; Step 3.2: Combine and recombine the biometric information obtained in Step 3.1 to obtain a feature image for subsequent processing; Step 4: Identify the user; Step 4.1: Develop a deep neural network for user authentication; The feature images obtained in step 3.2 are processed using a convolutional neural network to establish an accurate user authentication model; Step 4.2: Using continuous learning technology, train the user authentication model established in Step 4.1 and continuously update the network parameters; thereby overcoming the natural changes in user biometrics over time and accurately identifying user identities.

2. The identity authentication method using a microphone of a smart ear-worn device as described in claim 1, characterized in that, In step 1, the acquired signal is processed using a sliding window with a size of 60s and a step size of 30s; the autocorrelation function of the signal segment within the sliding window is calculated respectively. If the signal segment within the sliding window has periodicity, the time lag corresponding to the local maximum value of the autocorrelation function is the period of the signal. The autocorrelation function obtained is calculated to determine whether it reaches a maximum value in the time step of 0.5-1s. If it does, the segment is considered to be free from noise interference; otherwise, it is considered to be affected by noise interference.

3. The identity authentication method using a microphone of a smart ear-worn device as described in claim 1, characterized in that, In step 2.1, a fourth-order Butterworth low-pass filter with a cutoff frequency of 160Hz is used to eliminate high-frequency interference in the acquired ear canal sound signal that does not overlap with the frequency of the heart sound signal.

4. The authentication method using a microphone in a smart ear-worn device as described in claim 1, characterized in that, In step 2.2, calculate respectively The cosine distance between the sub-components at different frequencies throughout the entire time period and the pure bone conduction heart sound signal saved during user registration is considered if the cosine distance exceeds a threshold. If the sub-component originates from bone conduction heart sound signals, then all cosine distances exceeding the threshold are considered to originate from bone conduction heart sound signals. Sub-component clustering yields the time-spectrum matrix of the bone conduction heart sound signal. Finally, the pure bone conduction heart sound signal was recovered using the inverse short-time Fourier transform. .

5. The identity authentication method using a microphone of a smart ear-worn device as described in claim 1, characterized in that, In step 3.1, the user's heartbeat activity features are extracted from the bone conduction heart sound signals collected from the left ear and the right ear, respectively, using 16th-order Mel frequency cepstral coefficient features. Bone conduction features: 16th-order linear prediction coefficient features were extracted from bone conduction heart sound signals collected from the left ear and the right ear, respectively. Human structural features: First, the bone conduction heart sound signals collected from the left and right ears were divided into 32 equal sub-bands according to the frequency range of 0Hz to 160Hz, with each band having a bandwidth of 5Hz; then, the Euclidean distance between the bone conduction heart sound signals collected from the left and right ears in each band was calculated, and the Euclidean distances of the 32 bands were recorded as a 32-dimensional feature vector according to the frequency from low to high.

6. The authentication method using a microphone in a smart ear-worn device as described in claim 1, characterized in that, In step 3.2, the bone conduction heart sound signals collected from the left and right ears are first divided into non-overlapping windows of not less than 120ms. Then, the Hamming window was applied to divide the data within the window into 32 overlapping frames of 8ms each, and heart rate activity features, bone conduction features and human body structure features were extracted from each frame. Finally, these features are combined into three matrices: the first matrix consists of 16*32 heartbeat activity features and 16*32 bone conduction features of the left ear bone conduction heart sound signal within the window, with a size of 32*32; the second matrix consists of 16*32 heartbeat activity features and 16*32 bone conduction features of the right ear bone conduction heart sound signal within the window, with a size of 32*32; the third matrix consists of human body structural features within the window, with a size of 32*32. The element values ​​of the three matrices are normalized to integers between 0 and 255 using the maximum-minimum normalization method; the mapping rule is shown in Equation 5: （5） in, and Represent the updated and original matrices respectively. OK Column elements, and Represent the matrix number respectively The minimum and maximum values ​​of the elements in the column; the three matrices are combined into an RGB image after mapping as the final feature image, with the three matrices corresponding to the red, green and blue channels respectively.

7. The identity authentication method using a microphone of a smart ear-worn device as described in claim 6, characterized in that, Bone conduction heart sound signals acquired from the left and right ears were segmented into 130ms windows.

8. The authentication method using a microphone of a smart ear-worn device as described in claim 1, characterized in that, In step 4.1, a deep neural network model based on CNN is developed, consisting of three consecutive convolutional blocks, a flattening layer, a fully connected layer, and an output layer. The convolutional blocks extract hidden user identity information from the input feature image. Each convolutional block includes a two-dimensional convolutional layer using the ReLU function and a pooling layer. The three convolutional layers use 32, 64, and 64 kernels respectively, with a kernel size of 3×3. The flattening layer transforms the extracted multidimensional representation into a one-dimensional vector. The fully connected layer is set to 64 neurons to map the extracted features to real numbers. The output layer uses the softmax function to estimate the probability that the current user is a specific user and outputs the identity authentication result.

9. The authentication method using a microphone in a smart ear-worn device as described in claim 1, characterized in that, In step 4.2, a method based on plastic weight consolidation is adopted to continuously calibrate the developed deep neural network for user identity authentication, so that it can achieve reliable identity authentication with only a small amount of training data and can adapt to the natural changes in user biometrics over time. The identity authentication performance is protected by constraining the range of parameter changes in the existing user identity authentication model, and the parameters in the user identity authentication model are continuously updated to adapt to changes in biometrics. First, set a threshold. As the minimum probability for identifying legitimate users; When a new signal segment is fed into the user authentication model, the probability that the user is a legitimate user is calculated. and with Compare, if Greater than If the value is positive, the sample will be pseudo-marked as a positive value, indicating a legitimate user; otherwise, the sample will be pseudo-marked as a negative value, indicating an attacker. The samples are sorted according to the probability of being classified as legitimate users; those with the highest probability of being classified as legitimate users are... Each heart sound segment is used to construct a positive class dataset, assuming that all data in it comes from a single legitimate user, and the data segment with the lowest probability of being classified as a legitimate user is considered to be positive. Each heart sound segment is constructed into a negative class dataset, and it is assumed that all the data in it comes from the attacker; The selection range is 50% to 100%; Then, the user authentication model is updated using the newly established positive and negative class datasets; let the user authentication model before the update be denoted as... The updated user authentication model is denoted as , and Both models achieve the optimal solution for model parameters by minimizing the residuals; loss function Defined as: （6） in, In the model The updated model based on the parameters Each parameter of the model; It is a user authentication model The residual at which the optimal solution is reached. Adjusted and The weights compared; These are the diagonal elements of the Fisher information matrix. During continuous learning, new positive and negative class datasets are created each time to update the model. If all historical data is used to train the user authentication model and reach the optimal solution, a global model will be obtained. , This represents each parameter. Mark the parameter sequence; Define the global loss function for the identity authentication model. as follows: （7） in, This indicates the number of times the model has been updated during training. For the first Each parameter after updating the identity authentication model; It is a constant that adjusts the weights each time the parameters are updated, and its set range is (0,1); regularization strength , Subject to parameters In the Second and third Changes in the loss function and parameters at the next update Changes in itself The combined effects of these factors.

10. The authentication method using a microphone in a smart ear-worn device as described in claim 9, characterized in that, 70%, 0.6.

Images