Neural network based acoustic user identification and heartbeat monitoring earpiece system and method

Abstract

The application discloses a kind of based on neural network's acoustic user identification and heartbeat monitoring earphone system and method, the system includes the earphone with microphone, acoustic analog-digital conversion equipment, the back-end service equipment at least includes: data acquisition module is used for acoustic signal transmission and reception;Data processing module is used to obtain preprocessed received signal;Human ear canal feature extraction module is used to extract MFCC feature and ear canal feature based on transfer function;User identity identification module is used to obtain trained neural network model;Heartbeat information extraction module is used to obtain processed phase information;Self-interference elimination module obtains phase signal not containing direct path interference;Heartbeat frequency calculation module is used to obtain heartbeat frequency.The application can reduce the cost of identity recognition and heartbeat monitoring, and robustness, high robustness.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of wireless sensing user identity information and monitoring user heartbeat, and particularly relates to a neural network-based acoustic user identification and heartbeat monitoring earphone system and method. BACKGROUND

[0002] Earphones are one of the most popular wearable devices in our daily life. Improving user experience and supporting new functions such as touch control have become an increasingly growing trend for earphones to be smart. In order to improve the recording quality, Huawei Freebuds has a built-in bone vibration sensor. Apple's Airpods pro uses an additional inward-facing microphone to analyze noise from the outside, so that the system can generate an anti-noise waveform to eliminate noise, thereby obtaining better audio quality. In addition to the basic music playing and voice communication functions, earphones can also achieve some new functions, such as the user identity recognition and heartbeat monitoring functions proposed in the present application.

[0003] Currently, there are mainly three ways for user identity recognition, i.e., using user passwords and keys for identity recognition, using user body features for recognition, and using user-held certificates for recognition. The method of using user passwords and keys for user identity recognition has the risk of user passwords or keys being cracked, which poses a security risk. The method of using user body features is generally safe, but requires additional equipment to collect user body features such as fingerprints, faces, and irises, which is relatively high in cost. The method of using user-held certificates for recognition also has security risks, such as loss or theft of the certificate, and the need to carry the certificate at all times.

[0004] Current heartbeat monitoring methods include using specialized medical equipment for monitoring, and using some smart wearable devices such as smart watches and smart bracelets for monitoring user heartbeat. The main disadvantage of using professional medical equipment to monitor user heartbeat is high cost, relatively complex operation, and poor portability. Although the method of using smart wearable devices to monitor user heartbeat is simple to operate and easy to use, some smart wearable devices require the installation of some sensing elements, which results in relatively high cost compared to the method proposed in the present application.

[0005] In summary, the existing user recognition methods mainly have the problems of security risks and high cost. The existing heartbeat monitoring methods mainly have the problems of high cost, relatively complex operation, and poor portability. These problems will limit or inconvenience the use of users. SUMMARY

[0006] The present application aims to provide a neural network-based and acoustic user identification and heartbeat monitoring earphone system and method to solve the technical problems of complex operation and poor portability in the prior art.

[0007] To achieve the above technical task, the present application adopts the following technical solutions:

[0008] A neural network-based and acoustic user identification and heartbeat monitoring system, comprising an in-ear earphone with a microphone, an acoustic analog-to-digital conversion device, and a backend service device connected in sequence; wherein:

[0009] The microphone of the in-ear earphone with a microphone is placed in front of the loudspeaker in the earplug of the earphone.

[0010] The acoustic analog-to-digital conversion device is used to convert the analog acoustic signal received by the in-ear earphone with a microphone into a digital acoustic signal and segment it periodically.

[0011] The backend service device at least comprises:

[0012] A data acquisition module for controlling the loudspeaker and microphone on the earphone to emit and receive acoustic signals.

[0013] A data processing module for cleaning, denoising, and segmenting the data acquired by the data acquisition module to obtain preprocessed received signals.

[0014] A human ear canal feature extraction module for extracting feature information from the preprocessed received signals of the data processing module, extracting MFCC features and ear canal features based on transfer functions.

[0015] A user identity recognition module for concatenating the MFCC features and ear canal features based on transfer functions output by the human ear canal feature extraction module, then inputting them into a neural network model for training to obtain a trained neural network model, which is used for user identity recognition.

[0016] A heartbeat information extraction module for mixing the transmitted signal and the received signal to obtain a mixed signal, extracting the phase information in the mixed signal, and then processing the phase information in the mixed signal using a VMD algorithm to obtain processed phase information as heartbeat information.

[0017] A self-interference cancellation module for detecting the maximum peak position of the processed phase information output by the heartbeat information extraction module, subtracting the maximum peak position of the phase information of the direct path signal without the reflection path from the maximum peak position of the processed phase information in the time domain to obtain a phase signal without direct path interference.

[0018] a heart rate calculation module configured to detect the number of peaks of the phase signal output by the self-interference cancellation module, divide the number of peaks by the signal duration to obtain the heart rate.

[0019] The application also provides a user identity recognition method based on the above-mentioned user identity recognition and heart rate monitoring system based on a neural network and a sound signal.

[0020] Step 1: The data acquisition module in the backend service device controls the loudspeaker and microphone of the earphone to emit and receive sound wave signals, and the acoustic analog-to-digital conversion device converts the received analog sound wave signals into digital sound wave signals and divides them into cycles.

[0021] Step 2: The data processing module cleans, denoises and segments the data obtained in step 1 to obtain the preprocessed received signal.

[0022] Step 3: The human ear canal feature extraction module extracts the user ear canal features from the preprocessed received signal to obtain the MFCC ear canal features and the ear canal features based on the transfer function.

[0023] Step 4: The MFCC ear canal features and the ear canal features based on the transfer function obtained by the human ear canal feature extraction module are expanded into vectors and then spliced into features, and then input into the neural network model for training to obtain the trained neural network model, which is used to realize the identity recognition of the to-be-detected user.

[0024] Further, in step 1, the loudspeaker sends a 1kHz-21kHz FMCW linear frequency modulation sound wave signal with a PN preamble, and the sampling rate of the collected signal is 48kHz.

[0025] Further, step 3 includes the following steps:

[0026] Step 31: The MFCC algorithm is used to extract features from the preprocessed received signal obtained in step 2 to obtain the MFCC ear canal features.

[0027] Step 32: The frequency response of the left and right ear canals of the user is calculated, and the ear canal frequency response H(f) is defined as follows:

[0028]

[0029] where f is the frequency of the transmitted signal, P xy (f) is the cross-power spectral density between the preprocessed received signal and the transmitted signal, P xx (f) is the self-power spectral density of the transmitted signal.

[0030] The ear canal features based on the transfer function Hd (f):

[0031]

[0032] where H r is the right ear canal frequency response, H l is the left ear canal frequency response.

[0033] Further, in the step 4, the neural network model adopts a three-layer MLP network structure.

[0034] The application further provides a heartbeat monitoring method based on the above-mentioned system for user identity recognition and heartbeat monitoring based on a neural network and a sound signal.

[0035] Step 1: control the loudspeaker and microphone of the earphone to emit and receive sound wave signals through the data acquisition module in the backend service device, and convert the received analog sound wave signals into digital sound wave signals through an acoustic analog-to-digital conversion device and segment them periodically.

[0036] Step 2: the data processing module cleans, denoises and segments the data acquired by the data acquisition module to obtain the preprocessed received signal.

[0037] Step 3: the heartbeat information extraction module mixes the emitted signal and the preprocessed received signal to obtain a mixed signal and extracts the phase information in the mixed signal, and then processes the phase information in the mixed signal through a VMD algorithm to obtain processed phase information.

[0038] Step 4: the self-interference elimination module subtracts the maximum peak point of the phase of the pre-stored direct path reflection signal without other reflection paths from the processed phase information output in step 3 after aligning them in the time domain to obtain a phase signal without direct path interference.

[0039] Step 5: the heartbeat frequency calculation module detects the number of phase signal peaks of the phase signal output in step 4 and divides the number by the signal duration corresponding to the phase signal in step 4 to calculate the heartbeat frequency.

[0040] Further, in the step 1, the emitted signal is a 16kHz-21kHz FMCW linear frequency modulation sound wave signal with a PN preamble, and the sampling rate of the collected signal is 48kHz.

[0041] Compared with the prior art, the application has the following beneficial effects:

[0042] 1. The system can be applied to any in-ear earphone integrated with a loudspeaker and a microphone, and can realize user identity recognition and heartbeat monitoring by using low-cost earphones with microphones without additional detection equipment, thereby reducing the cost of identity recognition and heartbeat monitoring.

[0043] 2. The system adopts MFCC features in the field of acoustics and transfer function features based on the structure of human ear canals, realizes user identity recognition through a neural network, and improves the robustness of the system.

[0044] 3. The system adopts a VMD algorithm to remove motion state interference and proposes a method based on phase maximum peak alignment to eliminate hardware self-interference, so that the heartbeat monitoring function is more robust. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The figure is a structural diagram of the system.

[0046] Figure 2 The figure is a hardware modification diagram of the earphone in the system.

[0047] Figure 3 The figure is a structural diagram of the user identity recognition module in the system.

[0048] Figure 4 The figure is a schematic diagram of the heartbeat information extraction module in the system.

[0049] Figure 5 The figure is a schematic diagram of the self-interference elimination module in the system.

[0050] Figure 6 The figure is a schematic diagram of the user heartbeat frequency calculation and extraction module in the system.

[0051] Figure 7 The figure is a flowchart of the user identity recognition method.

[0052] Figure 8 The figure is a time-frequency spectrum of the signal of the user recognition function in the system.

[0053] Figure 9 The figure is a neural network model structure diagram of the user identity recognition module in the system.

[0054] Figure 10 The figure is a flowchart of the user heartbeat monitoring method.

[0055] Figure 11 The figure is a time-frequency spectrum of the signal of the user heartbeat monitoring function in the system.

[0056] Figure 12It is a user identity recognition experimental result graph in the present application, which shows the recognition effect of the user identity recognition method of the present application.

[0057] Figure 13 It is a user identity recognition experimental result graph in the present application, which shows that the user identity recognition method of the present application has strong robustness for different user activities and earphone wearing positions.

[0058] Figure 14 It is a user identity recognition experimental result graph in the present application, which shows that the user identity recognition method of the present application has strong anti-noise ability.

[0059] Figure 15 It is a user identity recognition experimental result graph in the present application, which shows that the user identity recognition method of the present application has stable recognition ability.

[0060] Figure 16 It is a user heartbeat monitoring method experimental result graph in the present application, which shows that the user heartbeat monitoring method of the present application has strong robustness for user daily activities.

[0061] Figure 17 It is a user heartbeat monitoring method experimental result graph in the present application, which shows that the user heartbeat monitoring method of the present application is suitable for most users.

[0062] Figure 18 It is a user heartbeat monitoring method experimental result graph in the present application, which shows that the user heartbeat monitoring method of the present application is not affected by earphone music.

[0063] Figure 19 It is a user heartbeat monitoring method experimental result graph in the present application, which shows that the user heartbeat monitoring method of the present application can remove the influence of earphone music. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and advantages of the present application more clear and obvious, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0065] Referring to Figure 1 The user identity recognition and heartbeat monitoring system based on neural network and sound signal provided by the present application at least includes an in-ear earphone with a microphone, an acoustic analog-to-digital conversion device and a backend service device connected in sequence. Wherein:

[0066] Referring to Figure 2The earphone modification diagram of the earphone. By removing the original microphone module of the common earphone with microphone, connecting the new microphone module with the microphone circuit of the earphone, and encapsulating the circuit, the new microphone module is placed in front of the speaker in the earphone earplug to achieve the purpose of allowing the microphone to receive the reflected signal of the ear canal. Preferably, the new microphone module is a common microphone.

[0067] The acoustic analog-to-digital conversion device is used to convert the analog sound wave signal received by the earphone into a digital sound wave signal and segment it periodically.

[0068] The backend service device is used to process the data processed by the acoustic analog-to-digital conversion device using the acoustic propagation characteristics of the human ear canal, complete user identification, and heartbeat monitoring. The backend service device at least includes a data acquisition module, a data processing module, a human ear canal feature extraction module, a user identity identification module, a heartbeat information extraction module, a self-interference elimination module, and a heartbeat frequency calculation module.

[0069] The data acquisition module is used to control the speaker and microphone of the earphone to transmit and receive sound wave signals. By connecting the earphone with the backend service device, the data acquisition module uses the audio IO of the backend service device to control the speaker and microphone on the earphone to transmit and receive sound wave signals. The data acquisition module collects the data information returned by the user's ear canal and sends it to the data processing module.

[0070] The backend service device adopts any device with audio IO, such as a mobile phone, a computer, etc. The data acquisition module, the data processing module, the human ear canal feature extraction module, the user identity identification module, the heartbeat information extraction module, the self-interference elimination module, and the heartbeat frequency calculation module are loaded in the backend service device as control modules.

[0071] The data processing module is used to clean, denoise, and segment the data acquired by the data acquisition module to obtain the preprocessed received signal, to remove errors and useless data, provide clean and representative data, reduce the influence of noise and multipath effect existing in the data, and facilitate subsequent feature extraction.

[0072] The human ear canal feature extraction module is used to extract feature information from the preprocessed received signal of the data processing module, based on the structure and contour features of the human ear canal, to extract MFCC features and ear canal features based on the transfer function for the use of the user identity identification module.

[0073] Referring to Figure 3The user identity recognition module is used for splicing the ear canal features of the MFCC features and the features based on the transfer function output by the human ear canal feature extraction module, and then inputting the neural network model to perform training, so as to obtain the trained neural network model, and the trained neural network model is used for realizing user identity recognition.

[0074] Referring to Figure 4 The heartbeat information extraction module is used for mixing the transmitted signal and the received signal to obtain a mixed signal, and taking the phase information in the mixed signal m(t), and then processing the phase information in the mixed signal m(t) by using the VMD algorithm to obtain the processed phase information as the heartbeat information. The algorithm can eliminate motion interference.

[0075] Referring to Figure 5 The self-interference cancellation module is used for detecting the maximum peak position of the processed phase information output by the heartbeat information extraction module, and subtracting the maximum peak position of the phase information of the direct path signal without the reflection path which is pre-stored from the maximum peak position in the time domain, so as to obtain the phase signal without the direct path interference. Thus, the direct path self-interference can be eliminated to obtain an accurate phase signal.

[0076] Referring to Figure 6 The heartbeat frequency calculation module is used for detecting the peak number of the phase signal output by the self-interference cancellation module, and dividing the peak number by the signal length to calculate the heartbeat frequency.

[0077] Referring to Figure 7 Based on the system of the present application, the method for user identity recognition provided by the present application comprises the following steps:

[0078] Step 1: The data acquisition module in the back-end service device controls the loudspeaker and the microphone of the earphone to emit and receive sound wave signals, and the acoustic analog-to-digital conversion device converts the received analog sound wave signals into digital sound wave signals and divides them into cycles.

[0079] Specifically, when collecting data, the earphone earplug is inserted into the ear canal of the user. After receiving the control instruction sent by the data acquisition module, the loudspeaker continuously sends the FMCW linear frequency modulation sound wave signal with a PN preamble of 1kHz-21kHz (see the time-frequency spectrum shown in Figure 8 ). The preamble appears only once in one detection, and is used for synchronization signal. Each FMCW linear frequency modulation sound wave signal is a cycle of 0.05 seconds. The sound wave is reflected after passing through the ear canal structure and contour, and then returns to the microphone of the earphone. When collecting signals, the sampling rate is 48kHz, so one cycle is 2400 data points.

[0080] The acoustic analog-digital conversion device converts the microphone received analog sound wave signal into digital sound wave signal, the data acquisition module saves the digital sound wave signal and divides it into cycles. Further save these segmented information in the file in the format of.mat, complete the data collection.

[0081] Step 2: The data processing module cleans, denoises and segments the data obtained by the data acquisition module to obtain the preprocessed received signal. The specific steps are as follows:

[0082] Firstly, the original data obtained by the data acquisition module, such as the slow start of the microphone of the earphone, has not started to work and has not received data. Such data will lag by about half a cycle, and part of the data is not complete, which belongs to error data and has a great influence on identification, so the data should be discarded or completed.

[0083] Secondly, through the observation of each group of data, it is found that the data noise is large in the early stage of collection, and the change is very obvious. Such change is caused by the system and is not suitable for identification. After the data is stable, the difference between adjacent cycles is small, and the characteristic information contained is insufficient. Based on the above two reasons, the direct current component is removed, the band-pass filter processing is selected, the middle data segment with small data noise and relatively stable change is selected as the preprocessed received signal.

[0084] Step 3: The user ear canal feature extraction module extracts the user ear canal feature from the preprocessed received signal, uses the uniqueness of the user ear canal and contour structure, and obtains the ear canal feature of MFCC and the ear canal feature based on the transfer function.

[0085] Since the difference between the data after preprocessing is not obvious, it cannot be directly used for identification, and further processing and feature extraction are needed to enlarge the difference between the data, so as to facilitate the identification module to capture the characteristics and realize correct classification. Therefore, the present application adopts the MFCC feature and the transfer function feature based on the human ear canal structure to further enlarge the difference between the data. The ear canal feature extraction by using the MFCC feature and the transfer function feature includes the following steps:

[0086] 1) The MFCC algorithm is used for feature extraction on the preprocessed received signal obtained in step 2 to obtain the MFCC ear canal feature. The MFCC algorithm is an existing algorithm and will not be described here. It mainly converts the signal frequency into the mel frequency and calculates the cepstrum coefficient of the signal in the mel frequency.

[0087] 2) Transfer function feature extraction. Since the MFCC feature alone is not accurate in user identity recognition, it is not possible to distinguish by using a single MFCC feature, so the transfer function feature is added. First, calculate the frequency response of the left and right ear canals of the user. The ear canal frequency response H(f) is defined as follows:

[0088]

[0089] where f is the frequency of the transmitted signal, P xy (f) is the cross power spectral density between the pre-processed received signal and the transmitted signal, P xx (f) is the auto power spectral density of the transmitted signal.

[0090] Dividing the frequency response of the right ear by the frequency response of the left ear gives the user ear canal characteristics based on transfer function. The ear canal characteristics based on transfer function H d (f) is defined as follows:

[0091]

[0092] where H r is the right ear canal frequency response, H l is the left ear canal frequency response.

[0093] Step 4: The user identity recognition module trains the neural network model using the feature information output by the human ear canal feature extraction module, and obtains a trained neural network model.

[0094] Specifically, the MFCC ear canal features and the ear canal characteristics based on transfer function obtained by the human ear canal feature extraction module are expanded into vectors, then the vectors are spliced, and then the spliced vectors are input into the neural network model for training, so as to obtain a trained neural network model. The trained neural network model is used to realize identity recognition of a to-be-detected user.

[0095] The neural network model structure of the user identity recognition module is shown in Figure 9 The neural network model structure of the user identity recognition module is shown in

[0096] Referring to Figure 10 , based on the system of the present application, the heartbeat monitoring method provided by the present application comprises the following steps:

[0097] Step 1: Except for the transmitted signal, the rest is the same as step 1, which will not be repeated.

[0098] In this method, the transmitted signal is a 16kHz-21kHz FMCW linear frequency modulated sound wave signal with a PN preamble, as shown in the time-frequency spectrum. Figure 11 The preamble appears only once in one detection, and is used for synchronization signal. Each FMCW linear frequency modulated sound wave signal has a period of 0.2 seconds, of which 0.15 seconds is an interval period. When collecting the signal, the sampling rate is 48kHz, so one period has 9600 data points.

[0099] Step 2: The data processing module cleans, denoises and segments the data acquired by the data acquisition module to obtain the preprocessed received signal.

[0100] Step 3: The heartbeat information extraction module mixes the transmitted signal and the preprocessed received signal to obtain a mixed signal m(t) and extracts the phase information in the mixed signal m(t); then the VMD algorithm is used to process the phase information in the mixed signal m(t) to obtain the processed phase information as the heartbeat information.

[0101] When the user moves, the extracted phase information will contain motion information interference, so it is necessary to eliminate the motion interference. When moving, the signal phase change will have a frequency multiplication signal, so the VMD algorithm is used to decompose the phase change in the mixed signal m(t) according to the frequency, remove the motion interference with the frequency multiplication signal, and extract the phase signal containing only the heartbeat information. Since the VMD decomposition is an existing algorithm, its principle will not be described.

[0102] Step 4: The self-interference elimination module subtracts the maximum peak point of the phase of the pre-stored direct path reflection signal without other reflection paths from the processed phase information output by step 3 after aligning in the time domain to obtain a phase signal without direct path interference. Thus, the self-interference can be eliminated to obtain an accurate phase signal.

[0103] Reference Figure 2 It can be seen that the microphone is placed in front of the loudspeaker, so the received signal contains the direct path signal from the microphone to the loudspeaker, which will drown out the phase peak value generated by the heartbeat. Therefore, this step is used to eliminate the direct path effect.

[0104] Step 5: The heartbeat frequency calculation module detects the peak value of the phase signal to obtain the number of phase signal peaks, divides the number of phase signal peaks by the signal duration corresponding to the phase signal obtained in step 4 to calculate the heartbeat frequency. The calculation formula is as follows:

[0105]

[0106] Where peakN is the number of phase signal peaks, t is the duration corresponding to the phase signal obtained in step 4, and Freq is the heartbeat frequency.

[0107] Experimental part:

[0108] For the convenience and universality of the present application, the embodiment uses a PC as the sound wave analog-to-digital conversion device and the backend service device. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. The present application includes a user, a PC, and an in-ear earphone. The microphone and the speaker in the earphone constitute a set of acoustic sensing devices, a set of acoustic signal transceiving nodes. The PC is configured as a sound wave analog-to-digital conversion device and a backend service device, and is configured to receive and convert the signals collected by the earphone for analysis.

[0109] Specific operation process: the server controls the earphone speaker to send acoustic signals, the earphone microphone records the echo signals, and sends the echo signals to the server for processing through the analog-to-digital conversion device, and performs pretreatment operations such as filtering and denoising. For the user identification function, the MFCC and the transfer function characteristics are calculated and spliced to form a feature vector, the user identity is identified through a trained model, and the result is displayed on the PC, thereby realizing the identification of the user identity. For the user heartbeat monitoring function, the signal phase information is extracted to eliminate motion interference, and then the self-interference is eliminated, and finally the number of peak points of the detected signal phase is counted to calculate the user heartbeat frequency and display the result on the PC.

[0110] 1. Performance experiment of user identity recognition method

[0111] The experiment identifies user identity information based on a neural network and a sound signal user identity recognition method. For this purpose, 120 participants were recruited, including 53 male and 67 female participants, and the age range of the participants was between 10 and 90 years old. During the identification process, the participants wore the earphones in a reasonable manner that they felt comfortable, and the number of participants was increased from 10 to 120. Referring to Figure 12 , it is observed that although the overall recognition accuracy is continuously decreasing as the number of users increases, when the number of users is 120, the overall recognition accuracy can still reach more than 96%, and the F1 score can also reach more than 96%. Therefore, the neural network and the sound signal user identity recognition method have high user recognition accuracy.

[0112] 2. Robustness experiment of user identity recognition method for user activity and earphone wearing position

[0113] The user identity recognition method provided by the present application has strong robustness to user activities and earphone wearing positions, can be used in daily life, and can also consider the earphone wearing positions of different users. Therefore, in order to verify that the user can be used normally in daily life, three activities in daily life that have an influence on recognition are selected, and the user identity is recognized under the conditions of head movement, speaking and chewing. In order to verify that the earphone wearing positions of different users can be considered, the user identity is recognized under the conditions of four wearing angles of the earphone and the ear canal, which are 0°, 90°, 180° and 270°, and the user identity is recognized under the conditions of three wearing depths of the earphone, which are shallow, moderate and deep. See Figure 13 It is observed that for the three different activities, the balance accuracy and F1 score of head movement and speaking are both above 98%, and the balance accuracy of the chewing activity which has a greater influence on performance is also above 90%. This is because chewing can cause slight deformation of the user's ear canal, thereby causing a decrease in accuracy. It is observed that different wearing angles have a slight influence on recognition performance, but the F1 score and balance accuracy of the four wearing angles are all above 90%. Finally, it is observed that for different wearing depths, the F1 score and balance accuracy are all above 90%. It is proved that the user identity recognition method provided by the present application has strong robustness to user daily activities and earphone wearing positions.

[0114] 3. Anti-noise ability experiment of user identity recognition method

[0115] The user identity recognition method provided by the present application can be used in most scenarios in daily life and has high anti-noise performance. Therefore, the user identity is recognized under the conditions of four different noise environments, which are 55dB, 63dB, 75dB and 84dB. See Figure 14 It is observed that when the environmental noise is lower than 80dB, the false acceptance rate and the false recognition rate are lower than 3%. When the noise increases to 84dB, the false acceptance rate and the false rejection rate increase sharply, but noise exceeding 80 decibels is rare in daily life. Therefore, it is proved that the user identity recognition method provided by the present application has strong anti-noise performance.

[0116] 4. Stable recognition experiment of user identity recognition method

[0117] The user identity recognition method provided by the present application can realize long-term stable recognition of user identity. Therefore, four time dimensions are selected for verification, and the user identity is recognized under the conditions of one hour, one day, one week and one month. The specific experimental process is to record three participants for one month, and collect data once a day. The user recognition performance over time is recorded. See Figure 15It is observed that the recognition result is slightly reduced in all aspects over time, but the balanced accuracy, F1 score and recognition accuracy of the recognition result are all greater than 95%. It is proved that the user identity recognition method has time stability.

[0118] 5. User heartbeat monitoring method for user activity robustness experiment

[0119] The user heartbeat monitoring method provided by the present application can realize user heartbeat monitoring without affecting the daily activities of the user. Therefore, six activity scenarios are selected, including sleeping, standing, speaking, head moving, walking and running, and 120 participants are used to evaluate the accuracy of the heartbeat monitoring method provided by the present application in the six scenarios. The specific experimental process is as follows: a standard heartbeat measuring instrument is selected as a benchmark, and the heartbeat data of each user in each activity scenario under the standard heartbeat measuring instrument and the user heartbeat measuring method provided by the present application are recorded, and the absolute error of the two is used to measure the accuracy of the heartbeat monitoring method provided by the present application. Figure 16 It is observed that the average absolute error of sleeping, sitting, speaking, head moving, walking and running is 1.04bp, 1.71bpm, 3.42bpm, 4.63bpm, 6.31bpm and 7.28bpm respectively. It can be seen that in the relatively static environment of the user, the average absolute error of the method is low. Although the average absolute error of the method is high in the motion state of the user, the error rate is still very low, because in the motion state, the heartbeat frequency is about 120bpm, and the difference between the actual heartbeat and the measured heartbeat is within 10% of the heartbeat frequency, which is effective data. It is proved that the user heartbeat monitoring method provided by the present application is robust to user activities.

[0120] 6. User heartbeat monitoring method for different user applicability experiment

[0121] The user heartbeat monitoring method provided by the present application is applicable to most users, i.e. accurate user heartbeat monitoring can be realized for most users. Therefore, the number of users is gradually increased from 20 to 120 in the sitting scenario, and the heartbeat data of each user under the standard heartbeat measuring instrument and the user heartbeat measuring method provided by the present application is recorded, and the average absolute error is calculated. Referring to Figure 17 It is observed that the average absolute error of the method is stable at about 1.7 with the increasing number of people. It is proved that the user heartbeat monitoring method provided by the present application is applicable to most users.

[0122] 7. User monitoring method for earphone music playback robustness experiment

[0123] This invention aims to provide a user heart rate monitoring method that can monitor a user's heart rate without interfering with normal headphone use. Therefore, a user's heart rate data was recorded over 20 minutes using a standard heart rate measuring instrument and the proposed method. The data from the standard heart rate measuring instrument was used as a baseline. During the use of the proposed method, the user played music through the same headphones. Figure 18 It was observed that even when music was playing, the method of the present invention closely approximates the benchmark. This is because the frequency of music sounds is typically below 4kHz, far below the frequency of perceived signals (16-21kHz). Figure 19 Therefore, the filter can easily remove music signals from the received signal. This result demonstrates that the proposed user heartbeat monitoring method can monitor a user's heartbeat without interfering with normal headphone use.

[0124] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A headphone system for sound wave user identification and heartbeat monitoring based on neural networks, characterized in that, It includes at least an in-ear headphone with a microphone, an acoustic analog-to-digital converter, and a back-end service device connected in sequence; wherein: The microphone of the in-ear headphones with a microphone is located at the front of the speaker in the earbud. The acoustic analog-to-digital converter is used to convert the analog sound wave signal received by the in-ear headphones with a microphone into a digital sound wave signal and to segment it into periodic segments. The backend service equipment includes at least: The data acquisition module is used to control the speakers and microphones on the headphones to transmit and receive sound wave signals; The data processing module is used to clean, denoise, and segment the data acquired by the data acquisition module to obtain the preprocessed received signal. The human ear canal feature extraction module is used to extract feature information from the received signal after preprocessing by the data processing module, and extract MFCC features and features based on transfer function to obtain ear canal features. The user identification module is used to concatenate the MFCC features output by the human ear canal feature extraction module and the features based on the transfer function, and then input them into the neural network model for training to obtain a trained neural network model. The trained neural network model is used to realize user identification. The heartbeat information extraction module is used to mix the transmitted signal and the received signal to obtain a mixed signal, extract the phase information in the mixed signal, and then use the VMD algorithm to process the phase information in the mixed signal to obtain the processed phase information, which is used as the heartbeat information. The self-interference cancellation module is used to detect the maximum peak position of the processed phase information output by the heartbeat information extraction module, and subtract it from the maximum peak position of the phase information of the pre-saved direct path signal without reflection path in the time domain to obtain the phase signal without direct path interference. The heart rate calculation module is used to detect the number of peaks in the phase signal output by the self-interference cancellation module, and divide it by the signal duration to calculate the heart rate.

2. A method for user identification, the method being based on the neural network-based acoustic user identification and heartbeat monitoring headphone system of claim 1, comprising the following steps: Step 1: The data acquisition module in the backend service device controls the speaker and microphone of the headphones to transmit and receive sound wave signals. The acoustic analog-to-digital converter converts the received analog sound wave signals into digital sound wave signals and segments them periodically. Step 2: The data processing module cleans, denoises, and segments the data obtained in Step 1 to obtain the preprocessed received signal; Step 3: Extract user ear canal features from the preprocessed received signal using the human ear canal feature extraction module to obtain the ear canal features of MFCC and ear canal features based on the transfer function. Step 4: Expand the MFCC ear canal features obtained from the human ear canal feature extraction module and the ear canal features based on the transfer function into vectors, then concatenate the features, and then input them into the neural network model for training to obtain a trained neural network model. The trained neural network model is used to realize the identification of the user to be detected.

3. The user identification method as described in claim 2, characterized in that, In step 1, the loudspeaker sends an FMCW linear frequency modulated sound wave signal with a PN preamble at a frequency of 1kHz-21kHz, and the sampling rate of the acquired signal is 48kHz.

4. The user identification method as described in claim 2, characterized in that, Step 3 includes the following steps: Step 31: Use the MFCC algorithm to extract features from the received signals obtained in step 2 and the preprocessed signals to obtain MFCC ear canal features; Step 32: Calculate the frequency response of the user's left and right ear canals. The ear canal frequency response H(f) is defined as follows: Where f is the frequency of the transmitted signal, P xy (f) represents the cross-power spectral density between the preprocessed received and transmitted signals, P xx (f) represents the self-power spectral density of the transmitted signal; Calculate ear canal features H based on transfer function d (f): Where H r For the frequency response of the right ear canal, H l Frequency response of the left ear canal.

5. The user identification method as described in claim 2, characterized in that, In step 4, the neural network model adopts a three-layer MLP network structure.

6. A heartbeat monitoring method, the method being based on the neural network-based acoustic user identification and heartbeat monitoring headphone system of claim 1, comprising the following steps: Step 1: The data acquisition module in the backend service device controls the speaker and microphone of the headphones to transmit and receive sound wave signals. The acoustic analog-to-digital converter converts the received analog sound wave signals into digital sound wave signals and segments them in a periodic manner. Step 2: The data processing module cleans, denoises, and segments the data acquired by the data acquisition module to obtain the preprocessed received signal; Step 3: The heartbeat information extraction module mixes the transmitted signal and the preprocessed received signal to obtain a mixed signal, and extracts the phase information from the mixed signal; then, the VMD algorithm is used to process the phase information in the mixed signal to obtain the processed phase information. Step 4: The self-interference cancellation module subtracts the processed phase information output from Step 3 from the maximum peak point of the phase of the pre-stored direct path reflection signal without other reflection paths after aligning them in the time domain, to obtain a phase signal without direct path interference. Step 5: The heart rate calculation module detects the number of phase signal peaks in the phase signal output in Step 4, divides it by the signal duration corresponding to the phase signal obtained in Step 4, and calculates the heart rate.

7. The heartbeat monitoring method as described in claim 6, characterized in that, In step 1, the transmitted signal is a 16kHz-21kHz FMCW linear frequency modulated acoustic wave signal with a PN preamble, and the sampling rate of the acquired signal is 48kHz.

Images