A method and system for detecting heart sound abnormalities based on earphones
By reconstructing heart sound signals using a physical information neural network within the earphone, the problem of inaccurate inversion of in-ear sound signals is solved, achieving highly usable and accurate monitoring of heart sound signals, suitable for daily health monitoring and telemedicine.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2026-04-10
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, the heart sound signals retrieved from the ear sound signals obtained through headphones are inaccurate, making it difficult to achieve heart sound monitoring with high ease of use, low interference, and continuous acquisition.
A method based on physical information neural networks, including the source pulse wave inversion network HemoNet and the cross-modal spatiotemporal mapping network MappingNet, is used to invert and map the intra-ear sound signals obtained from headphones, reconstruct the phonocardiogram, and perform anomaly detection and identification.
It improves the accuracy and stability of reconstructing heart sound signals from in-ear sounds, and realizes the precision and convenience of heart rate abnormality monitoring, making it suitable for daily health monitoring and telemedicine.
Smart Images
Figure CN122157705A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of signal processing technology, and more specifically, relates to a method and system for detecting abnormal heart sounds based on headphones. Background Technology
[0002] Heart sounds, as important physiological signals reflecting the motion and hemodynamic characteristics of heart valves, are a crucial foundation for monitoring cardiac health and identifying diseases. Heart sounds contain rich information such as the frequency, duration, energy distribution, and murmur type of the first and second heart sounds, and are widely used to reflect clinical symptoms such as valvular dysfunction, arrhythmias, and murmur origins. They offer advantages such as being non-invasive, low-cost, and having high information density, making them particularly suitable for early screening and long-term follow-up of heart diseases in home settings. In recent years, with the development of digital auscultation and artificial intelligence technologies, automated assisted diagnosis based on heart sounds has gradually become an important direction for intelligent detection of cardiovascular diseases.
[0003] Current methods for acquiring heart sounds still face many limitations. Traditional methods mainly rely on electronic stethoscopes or chest lead microphones, which require precise surface positioning and operation by professionals, limiting their application in everyday life. Furthermore, surface sensors suffer from problems such as clothing obstruction, motion interference, and noise pollution, making it difficult to continuously acquire high-quality signals over long periods, which is detrimental to the dynamic assessment of cardiac status.
[0004] Research has revealed components in the intra-ear sound signal that are highly correlated with heart sounds. Given the widespread adoption of smart wearable devices, there is an urgent need to develop a user-friendly, low-interference, and continuously acquireable non-contact or semi-contact heart sound monitoring technology. While some existing technologies use earphone microphones to acquire intra-ear sound signals and analyze them for heart rate detection, the processing of the heart rate signal obtained from intra-ear sound processing in current methods is relatively coarse. Furthermore, the pressure change information captured by the microphone located inside the ear is weak, making it difficult to accurately reflect heart rate information using heart sound signals calculated from intra-ear sounds. Summary of the Invention
[0005] In view of the shortcomings of related technologies, the purpose of this invention is to provide a method and system for detecting abnormal heart sounds based on headphones, which aims to solve the problem that the heart sound signals obtained by headphones are not accurately reflected from the in-ear sounds in the prior art.
[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for detecting abnormal heart sounds based on headphones, comprising: Acquire discrete intra-ear sound signals at each time stamp collected by the headphones; The intraocular sound signal is inverted and mapped based on a pre-constructed physical information neural network to obtain a reconstructed phonocardiogram; Anomaly detection and identification are performed based on the reconstructed phonocardiogram; The physical information neural network includes a source pulse wave inversion network HemoNet and a cross-modal spatiotemporal mapping network MappingNet; the source pulse wave inversion network HemoNet is used to invert discrete intraocular sound signals to obtain continuous intraocular sound signals. And based on continuous intraocular sound signals The arterial pulse pressure wave signal at the eardrum was obtained by inversion at the eardrum. Then the arterial pulse pressure wave signal The original pulse wave signal was obtained by inversion at the heart. The cross-modal spatiotemporal mapping network MappingNet is used to map the source pulse wave signal. The heart sound signal is generated by mapping, resulting in a reconstructed phonocardiogram; L represents the distance from the heart sound source to the blood vessels in the ear canal.
[0007] Optionally, the source pulse wave inversion network HemoNet includes a first inversion network and a second inversion network; The first inversion network includes a sub-network with a multilayer perceptron structure. Hezi Network The sub-network Used to invert the continuous inner ear sound signal based on each timestamp t corresponding to the discrete inner ear sound signal; The sub-network Used to invert the pulse pressure wave signal at the eardrum based on each time stamp t corresponding to the discrete intraocular sound signal; Based on the continuous intraocular sound signal and the pulse pressure wave signal at the eardrum, a first objective function is constructed for the first inversion network. The first objective function is then optimized and solved to obtain the sub-network. Hezi Network The optimal parameters, and based on the subnetwork Hezi Network The optimal parameters were used to obtain the pulse pressure wave signal at the eardrum. ; The second inversion network includes a backbone fluctuation network. and compensation network The backbone wave network Sampling points at each location within the distance from the heart to the eardrum. The tuple formed by each timestamp t corresponding to the discrete intraocular sound signal The inversion yields the distance from the heart. Continuous pressure at the point ; The value range is [0, L].
[0008] The compensation network Used for binary-based The generation of heart sound signals during propagation Error compensation signal at the location; Based on distance from the heart Continuous pressure at the point The second objective function of the second inversion network is constructed using the error compensation signal. The second objective function is then optimized to obtain the backbone wave network. and compensation network The optimal parameters were determined based on the backbone wave network. and compensation network The optimal parameters are obtained from the distance from the heart. Continuous pressure at the point Thus, the source pulse wave signal is obtained. .
[0009] Optionally, the first objective function used in the first inversion network is expressed as:
[0010] in, and These are the weighting coefficients for the corresponding loss function; Observational loss For sub-networks The output prediction value and the in-ear sound signal The mean square error between them is expressed as: , for Intra-auricular sound signals at any given moment for The length of the discretized sequence, , They refer to the network respectively. Weights and biases; Physical residual loss Used to constrain subnetworks Output satisfies the in-ear sound signal Arterial pulse wave signal The response is expressed as:
[0011] Among them, the equivalent damping coefficient Equivalent natural frequency squared Coupling gain coefficient , The number of sample points. express The Internet Predicted pressure values at any given time express The output is at the eardrum. Predicted pressure values at any given time; The second objective function used in the second inversion network is expressed as follows:
[0012] in, and These are the weight coefficients corresponding to the loss function; boundary condition loss is used. Compared with the residual loss of the modified partial differential equation The training of the backbone fluctuation network is aimed at optimization. ; Boundary condition loss The expression is:
[0013] Where c represents the pulse wave propagation speed. , They refer to the network respectively. Weights and biases; Subnetworks in the first inversion network Output Predicted pressure at the eardrum at all times; Residual loss of partial differential equations The expression is:
[0014] in, In the space-time domain The number of random sampling points; express The output is at a distance from the heart Place, Predicted pressure values at any given time; According to the compensation network Error compensation generated residual loss for partial differential equations Modifications were made, and the residual loss of the corrected partial differential equation was determined. The expression is: .
[0015] Optionally, the physical information neural network further includes: a source heart sound inversion network, CardiNet; The source heart sound inversion network CardiNet includes: a backbone wave network. With compensation network The backbone wave network With compensation network With the backbone fluctuation network in the second inversion network With compensation network Same structure; The optimization process of the CardiNet source heart sound inversion network is the same as that of the second inversion network.
[0016] Optionally, the cross-modal spatiotemporal mapping network MappingNet uses the source pulse wave output by the source pulse wave inversion network HemoNet as input and the source heart sound signal output by the source heart sound inversion network CardiNet as a label to learn cross-modal mapping relationships through a data-driven approach, thereby establishing source pulse waves originating from the heart. The nonlinear mapping relationship between the heart sound signal and the reconstructed heart sound signal; The cross-modal spatiotemporal mapping network MappingNet adopts a symmetric U-shaped encoder-decoder network architecture, including cascaded encoders, Transformer modules, and decoders. The encoder includes N cascaded coding units, each of which includes cascaded multi-scale convolutional modules, residual connection structures, and time-frequency attention modules. The encoder is used to extract features of the heart sound signal and pass them to the attention mechanism of the Transformer module. The Transformer module calculates the global weight correlation of the features extracted by the encoder and automatically identifies the features related to heart sounds and the temporal pattern of feature peaks in the source pulse wave. The decoder is structurally mirror-symmetric to the encoder and includes N cascaded decoding units, used to reconstruct the corresponding heart sound signal by upsampling layer by layer according to the temporal pattern; where N is a positive integer.
[0017] Optionally, the process of training MappingNet includes: The labeled source heart sound signal output by the CardiNet source heart sound inversion network is denoted as The preset source heart sound signal for the cross-modal spatiotemporal mapping network MappingNet is... The composite loss function is constructed as follows:
[0018] in, and For the weight coefficients of the corresponding loss function, the frequency domain loss... The expression is: Temporal loss The expression is: , This is the short-time Fourier transform of the input signal.
[0019] Optionally, the abnormality detection and identification based on the reconstructed phonocardiogram includes: Heart rate feature anomaly detection and heart sound waveform anomaly detection are performed based on the reconstructed phonocardiogram.
[0020] Secondly, the present invention also provides a heart sound abnormality detection system based on headphones, comprising: headphones and a processor; The earphones are connected to the processor via Bluetooth; The earphone has a built-in microphone for acquiring sound signals from inside the user's ear; The processor is used to execute the headphone-based heart sound anomaly detection method as described in any one of the first aspects, and output the anomaly detection result to the user.
[0021] Compared with the prior art, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: 1. This invention provides a method for detecting abnormal heart sounds based on headphones. By introducing a physically constrained physical information neural network, the source pulse wave signal is inverted from the intraocular sound signal acquired by the headphones. Compared with the traditional method of directly estimating heart sounds from distant signals, this method can effectively alleviate the problem of information loss during signal propagation and improve the physical consistency and accuracy of the reconstructed signal. In the inversion process, based on the actual physical model of the intraocular sound signal transmission process, the source pulse wave is introduced as an intermediate physical variable. This allows for explicit modeling of the physiological mechanism of heart sound generation, decomposing the complex mapping process from intraocular sound to heart sound into two sub-processes: "inversion of source pulse wave from intraocular sound signal" and "reconstruction of heart sound from source pulse wave signal." This effectively reduces the complexity of the learning task, improves the stability and trainability of the model, and achieves accurate inversion of the source pulse wave signal and reconstruction of heart sounds.
[0022] 2. The present invention provides a method for detecting abnormal heart sounds based on headphones. By acquiring heart sound signals through headphones and inverting them to obtain accurate source pulse wave signals, the heart sound signals can be obtained, making the task of monitoring abnormal heart rate integrated into daily life. Non-invasive cardiac monitoring based on wearable devices provides a new technical path that can be widely used in daily health monitoring, telemedicine and early screening of heart diseases. Attached Figure Description
[0023] Figure 1 This is a schematic diagram illustrating the relationship between internal sounds and heart sounds in an embodiment of the present invention; Figure 2 This is a schematic diagram of the physical model of intraocular sound signal transmission in an embodiment of the present invention; Figure 3 This is a schematic diagram of the physical information neural network architecture in an embodiment of the present invention; Figure 4 This is a schematic diagram of the HemoNet network architecture in an embodiment of the present invention; Figure 5This is an example diagram illustrating the estimation and reconstruction of heart sound signals in the middle ear using a heart sound anomaly detection method based on headphones, provided in an embodiment of the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0025] The following description, in conjunction with a preferred embodiment, illustrates the content involved in the above embodiments.
[0026] Example 1 This invention provides a method for detecting abnormal heart sounds based on headphones, comprising: Acquire discrete intra-ear sound signals at each time stamp collected by the headphones; The intraocular sound signal is inverted and mapped based on a pre-constructed physical information neural network to obtain a reconstructed phonocardiogram; Anomaly detection and identification are performed based on the reconstructed phonocardiogram; The physical information neural network includes a source pulse wave inversion network HemoNet and a cross-modal spatiotemporal mapping network MappingNet; the source pulse wave inversion network HemoNet is used to invert discrete intraocular sound signals to obtain continuous intraocular sound signals. And based on continuous intraocular sound signals The arterial pulse pressure wave signal at the eardrum was obtained by inversion at the eardrum. Then the arterial pulse pressure wave signal The original pulse wave signal was obtained by inversion at the heart. The cross-modal spatiotemporal mapping network MappingNet is used to map the source pulse wave signal. The heart sound signal is generated by mapping, resulting in a reconstructed phonocardiogram; L represents the distance from the heart sound source to the blood vessels in the ear canal.
[0027] Among them, the physical information neural network is composed of a deep neural network combined with a physical model of the transmission of sound signals in the ear.
[0028] Studies have shown that the pulse wave generated during the heart's pumping process propagates to the head and neck via the aorta, common carotid artery, and external carotid artery system. This pulse wave then acts on the tissues surrounding the ear canal via related arterial branches (including but not limited to the superficial temporal artery), causing pressure disturbances related to the pulse wave within the closed ear canal. These disturbances can be collected by acoustic sensors within the ear canal, forming intraocular sound signals. Preliminary experiments have verified a significant correspondence between intraocular sounds and standard chest lead heart sounds in terms of temporal structure, frequency distribution, and abnormal patterns. Figure 1 The diagram illustrates the correlation between intraocular sound signals and heart sound signals (source pressure waveform). Although intraocular sounds differ from traditional heart sounds in waveform morphology, they exhibit stable and repeatable responses after each first heart sound, second heart sound, and cardiac mechanical event, and their rhythm is highly consistent with heart sounds, indicating a clear physiological correlation between intraocular sounds and cardiac activity. Therefore, intraocular sound signals carry rich cardiac physiological information, and inferring heart sound signals and detecting abnormalities by acquiring intraocular sound signals is a promising technical solution.
[0029] like Figure 2 As shown, a physical model of sound signal transmission in the inner ear is constructed. The specific process is as follows: First, the generation process of sound in the inner ear is modeled into two stages: First, the propagation of arterial pulse waves from the heart to the ear; Second, this pressure wave is transformed into a structure-acoustic coupling process of measurable sound in the closed ear canal.
[0030] An initial propagation model of arterial pulse pressure waves from the heart to the ear was established using the one-dimensional Navier-Stokes equations. The initial propagation model is simplified and a compensation term is introduced. The target propagation model is obtained as follows:
[0031] in, This is the arterial pulse pressure wave. The distance to the pressure source, The arterial pulse pressure wave is formed by the source pulse wave signal generated by the heartbeat propagating along the arterial vascular system as an excitation source over time. It is the pulse wave velocity, and A is the cross-sectional area of the blood vessel. C is the density of blood, and C is the compliance of the blood vessel wall, which characterizes the elastic deformation ability of blood vessels under pressure. As compensation; Among them, such as Figure 2 In process 1, the propagation of the pulse wave in the artery is governed by the one-dimensional Navier-Stokes equations, which describe the arterial pressure wave. The relationship between blood vessel cross-section and volumetric flow rate.
[0032] Because this system of equations contains complex terms such as nonlinear convection (inertial effects) and viscous damping (shear stress), it can be simplified for large artery blood flow: under the low Mach number approximation prevalent in arterial blood flow, the nonlinear convection term can be neglected; the viscous effect is mainly confined to the boundary layer and has minimal impact on the mainstream pulse wave, therefore, the damping term can be omitted in the one-dimensional average theory. By simplifying and introducing a vascular compliance equation to form a closed system of equations, the model simplifies to a standard one-dimensional wave equation. Simultaneously, to compensate for unmodeled physical phenomena, such as arterial wave reflection or slight nonlinear effects, a compensation term is introduced. In the above model, It is considered as a learnable parameter, and its value range is restricted to 4–12 m / s, which conforms to physiological characteristics.
[0033] In the second stage, such as Figure 2 As shown in process 2, the closed ear canal is modeled as a cavity undergoing isothermal compression, and the arterial pulse pressure wave reaches the eardrum. It acts on the eardrum and surrounding tissues, producing changes in sound pressure within the closed ear canal. Ideally modeling this process as a forced harmonic oscillator, the displacement equation of the eardrum... ( This can be represented as:
[0034] in, , and These represent the mass, damping coefficient, and stiffness of the eardrum, respectively. This represents the effective area of the eardrum.
[0035] The resulting sound pressure is the internal sound signal that can be acquired. ( ) and the displacement of the eardrum ( In connection with this, since the closed ear canal is modeled as a cavity undergoing isothermal compression, the change in sound pressure can be linearly approximated to a small displacement as follows:
[0036] in, and These are static atmospheric pressure and ear canal volume, respectively. It is the absolute pressure inside the sealed ear canal; substituting this linear relationship back into the above equation for eardrum displacement. ( In this process, displacement variables can be eliminated. ( After rearrangement, a standard second-order ordinary differential equation is obtained, which directly describes the arterial pulse pressure wave at the eardrum. Changes in sound pressure are generated within the closed ear canal. Through structural-acoustic coupling, it is converted into an intraocular sound signal that can be measured at one end of the ear canal. ( Constructing intraocular sound signals ( Arterial pulse pressure wave Response model:
[0037] Where L represents the distance from the sound source in the heart to the blood vessels in the ear canal; the equivalent damping coefficient Equivalent natural frequency squared Coupling gain coefficient The response model, as the final lumped parameter model, defines the equivalent damping coefficient. Equivalent natural frequency squared Coupling gain coefficient .
[0038] By integrating the target propagation model and the response model, a physical model for the transmission of sound signals in the inner ear is obtained.
[0039] Furthermore, a physical information neural network (PINNs) is constructed using a deep neural network combined with a physical model of intraocular sound signal transmission to process the intraocular sound signal and output a cardiac pressure wave signal. For example... Figure 3 As shown, specifically, the physical information neural network includes the source pulse wave inversion network HemoNet, the source heart sound inversion network CardiNet, and the cross-modal spatiotemporal mapping network MappingNet.
[0040] Under the constraints of physical equations, the source pulse wave inversion network HemoNet and the source heart sound inversion network CardiNet respectively characterize the reverse propagation process of sound waves in the vascular cavity and pleural tissue, inverting peripheral observation signals to a unified cardiac source-end representation space. Based on this, the system backend introduces the cross-modal spatiotemporal mapping network MappingNet to model the nonlinear correlation between the source pulse wave and the source heart sound.
[0041] The HemoNet source pulse wave inversion network comprises a first inversion network and a second inversion network, and its main objective is to invert the source pulse wave based on observed intraocular sound signals. Inferring the source pressure waveform at the origin of the heart This problem can be viewed as a typical inverse spacetime problem driven by the coupling of partial differential equations (PDEs) and ordinary differential equations (ODEs).
[0042] Furthermore, such as Figure 4 As shown, the first inversion network is constructed in conjunction with the response model, and the first inversion network includes sub-networks with the same architecture as multilayer perceptrons. and The second inversion network is constructed in conjunction with the target propagation model, including a backbone fluctuation network. With compensation network .
[0043] The sub-network The subnetwork is used to invert continuous intraocular sound signals based on each timestamp t corresponding to discrete intraocular sound signals; Used to invert the pulse pressure wave signal at the eardrum based on each time stamp t corresponding to the discrete intraocular sound signal; Based on the continuous intraocular sound signal and the pulse pressure wave signal at the eardrum, a first objective function is constructed for the first inversion network. The first objective function is then optimized to obtain the subnetwork corresponding to the minimization of the first objective function. Hezi Network The optimal parameters, and based on the subnetwork Hezi Network The optimal parameters were used to obtain the pulse pressure wave signal at the eardrum. ; The second inversion network includes a backbone fluctuation network. and compensation network The backbone wave network Sampling points at each location within the distance from the heart to the eardrum. The tuple formed by each timestamp t corresponding to the discrete intraocular sound signal The inversion yields the distance from the heart. Continuous pressure at the point ; The value range is [0, L]; the compensation network Used for binary-based The generation of heart sound signals during propagation Error compensation signal at the location; Based on distance from the heart Continuous pressure at the point The second objective function of the second inversion network is constructed using the error compensation signal. The second objective function is then optimized and solved to obtain the backbone fluctuation network corresponding to the minimization of the second objective function. and compensation network The optimal parameters were determined based on the backbone wave network. and compensation network The optimal parameters are obtained from the distance from the heart. Continuous pressure at the point Thus, the source pulse wave signal is obtained. .
[0044] Structurally, the network is functionally divided according to the physical propagation path. The first inversion network is responsible for processing the in-ear sound signals. Inversely mapped to arterial pulse pressure waves at distal blood vessels . and Both employ a multilayer perceptron (MLP) design with the same architecture, and their outputs correspond to the observed intraocular sound signals. and the distal pressure after reversal In this architecture, Its function is to fit the in-ear audio samples, while This forces the network to follow physical laws (response model). This can be easily achieved through automatic differentiation (AD) in deep learning frameworks.
[0045] The first objective function used in the first inversion network is expressed as:
[0046] in, and These are the weight coefficients corresponding to the first objective function; Observational loss For sub-networks The output prediction value and the in-ear sound signal The mean square error between them is expressed as: , for Intra-auricular sound signals at any given moment for The length of the discretized sequence, , They refer to the network respectively. Weights and biases; Physical residual loss Used to constrain subnetworks Output satisfies the in-ear sound signal Arterial pulse wave signal The response is expressed as:
[0047] Among them, the equivalent damping coefficient Equivalent natural frequency squared Coupling gain coefficient , The number of sample points. express The Internet Predicted value at time, express At x=L, The predicted value at any given time; Based on this, the second inversion network consists of the backbone fluctuation network. With compensation network The structure, through wave equation constraints and residual compensation mechanisms, incorporates arterial pulse pressure waves. Further inversion back to the source pressure waveform at the origin of the heart .
[0048] Backbone Fluctuation Network The spatiotemporal evolution process used to approximate the intravascular pressure field, i.e., using subnetworks Derived distal arterial pulse pressure wave Further inversion of source pressure waveform Its main structure also adopts a multi-layer sensing mechanism, based on spatiotemporal coordinates. The input is the estimated pressure field, and the output is the estimated pressure field. .
[0049] The second objective function used in the second inversion network is expressed as follows:
[0050] in, and The weight coefficients correspond to the second objective function; boundary condition loss is used. Compared with the residual loss of the modified partial differential equation The training of the backbone fluctuation network is aimed at optimization. ; Boundary condition loss The expression is:
[0051] Where c represents the pulse wave propagation speed. , They refer to the network respectively. Weights and biases; Subnetworks in the first inversion network Output Predicted pressure at the eardrum at all times; Residual loss of partial differential equations The expression is:
[0052] in, Let Ω = [0, L] × [0, T] be the number of random sampling points in the spatiotemporal domain, where L is the length of the blood vessel from the heart to the ear canal, and T is the duration of the sound signal in the ear.
[0053] Because partial differential equations contain unknown terms Furthermore, the cardiac origin lacks direct supervisory labels, and the subnetwork... Convergence instability may occur during training. On the one hand, the network tends to optimize and reduce the residuals of partial differential equations in the early stages of optimization; on the other hand, since unknown terms are not explicitly modeled, boundary condition errors decrease slowly, leading to inconsistent convergence behavior among different loss terms. Furthermore, standard multilayer perceptrons suffer from spectral bias when approximating physical solutions containing multi-scale features, easily resulting in non-physical high-frequency oscillations in the prediction results. To alleviate these problems, in the backbone network... Introducing Fourier feature mapping layer and compensation network The Fourier feature mapping layer is deployed on the backbone network. The input end is used to enhance the network's ability to express multi-scale frequency components. To solve... To address the convergence imbalance problem, a compensation network is introduced. We then perform parameterized modeling on it. This network also employs a multilayer perceptron structure, and its output is denoted as... The compensation network output will be used to compensate for the network output. The unmodeled terms are explicitly incorporated into the residuals of the differential equation and represented separately.
[0054] According to the compensation network Error compensation generated residual loss for partial differential equations Modifications were made, and the residual loss of the corrected partial differential equation was determined. The expression is: .
[0055] Furthermore, the physical information neural network also includes: the source heart sound inversion network CardiNet; The source heart sound inversion network CardiNet includes: a backbone wave network. With compensation network The backbone wave network With compensation network With the backbone fluctuation network in the second inversion network With compensation network Same structure; The optimization process of the CardiNet source heart sound inversion network is the same as that of the second inversion network.
[0056] CardiNet, a source-end heart sound inversion network, is used to characterize the dynamic process of heart sounds propagating from the heart source through the soft tissues of the chest cavity to the body surface. Heart sounds are essentially sound pressure waves mediated by tissue media, and their propagation behavior can be described by partial differential equations. Therefore, this task can be modeled as the problem of inverting the acoustic field of an unknown source under known observed signals on the body surface. Similar to hemodynamics, the core challenge faced by CardiNet lies in the lack of direct supervision information on the source-end heart sounds, while surface heart sounds can be obtained through non-invasive auscultation. This "unknown source-measurable boundary" problem can be solved through physically constrained inversion modeling. In terms of network structure design, CardiNet reuses the partial differential equation solution framework from HemoNet, including a core wavelet network. With compensation network .
[0057] Furthermore, the cross-modal spatiotemporal mapping network MappingNet is used to establish the spatiotemporal correspondence between the source pulse wave and the reconstructed source heart sound, thereby completing the cross-modal mapping of heart sounds inverted from in-ear sounds. The MappingNet network uses the source pulse wave output from the HemoNet source pulse wave inversion network as input and the source heart sound signal output from the CardiNet source heart sound inversion network as a label. Through a data-driven approach, it learns the cross-modal mapping relationship to establish the source pulse wave originating from the heart. The nonlinear mapping relationship between the heart sound signal and the reconstructed heart sound signal; The cross-modal spatiotemporal mapping network MappingNet adopts a symmetric U-shaped encoder-decoder network architecture, including cascaded encoders, Transformer modules, and decoders. The encoder includes N cascaded coding units, each of which includes cascaded multi-scale convolutional modules, residual connection structures, and time-frequency attention modules. The encoder is used to extract features of the heart sound signal and pass them to the attention mechanism of the Transformer module. The Transformer module calculates the global weight correlation of the features extracted by the encoder and automatically identifies the features related to heart sounds and the temporal pattern of feature peaks in the source pulse wave. The decoder is structurally mirror-symmetric to the encoder and includes N cascaded decoding units, used to reconstruct the corresponding heart sound signal by upsampling layer by layer according to the temporal pattern; where N is a positive integer.
[0058] The symmetrical design preserves low-frequency structural information extracted during the encoding stage during reconstruction, while also providing constraints for the recovery of high-frequency details. The skip connection mechanism further enhances information transfer between shallow and deep features, thus maintaining the consistency of the temporal dynamics structure. Overall, this symmetrical U-shaped structure forms a synergistic mechanism between local feature modeling and global dependency capture, ensuring a stable mapping of the reconstructed heart sounds in terms of frequency distribution and rhythmic structure. This design is not a simple signal transformation, but rather establishes a cross-modal feature alignment framework in a multi-scale representation space.
[0059] To enhance the model's ability to characterize key acoustic events such as the first heart sound, second heart sound, premature beats, and murmurs, this study introduces a physical prior-based attention guidance mechanism into the network, rather than relying solely on implicit learning from large-scale data. This mechanism explicitly embeds known physiological acoustic laws into the network training process based on the structural characteristics of heart sounds in the time and frequency domains.
[0060] Optionally, the process of training MappingNet includes: Labeled source heart sound signals output by CardiNet, a source heart sound inversion network. The preset source heart sound signal for the cross-modal spatiotemporal mapping network MappingNet is... The composite loss function is constructed as follows:
[0061] in, and For the weight coefficients of the corresponding loss function, the frequency domain loss... The expression is: Temporal loss The expression is: , This is the short-time Fourier transform of the input signal.
[0062] The aforementioned cascaded structure integrates physical constraints and feature learning mechanisms within the same framework.
[0063] Based on the above, in a specific embodiment, in order to ensure that each module can fully characterize the flow-acoustic coupling process, and at the same time guarantee the data stability and convergence efficiency of the inversion training, a systematic design was carried out for the data preprocessing process, network structure parameters, and network training strategy.
[0064] Before network training, all acquired acoustic signals underwent a three-stage preprocessing process. First, bandwidth limiting was applied based on the spectral distribution characteristics of different signals. Intra-auricular sounds, primarily reflecting the acoustic response induced by low-frequency pulse waves, had their low-pass cutoff frequency set to 30Hz. Surface heart sounds, containing abundant mid-to-high frequency components, had their cutoff frequency set to 500Hz to preserve their diagnostically relevant frequency band information. Subsequently, a periodic segmentation method based on envelope detection was used to segment the continuous acoustic signals at the cardiac cycle level, thereby creating structurally aligned training sample pairs. Finally, amplitude normalization was applied to all samples, linearly mapping the signals to the [-1,1] interval to improve gradient propagation conditions and enhance the stability of the optimization process.
[0065] Configure network parameters: In HemoNet, subnetworks and Both employ a two-layer fully connected structure with 64 neurons per layer and the Tanh activation function to enhance their ability to express smooth, low-frequency dynamic processes. (Backbone oscillating network) With compensation network It consists of a 5-layer fully connected structure, with 128 neurons in each layer. The sine function is used as the activation function to improve the approximation of periodic and oscillatory solutions. (Backbone oscillating network) The input terminal introduces an FFL with a frequency sampling range of 0-20Hz and a frequency step of 2Hz. This frequency band coincides with the main frequency range of the pulse wave, which helps to improve the expression accuracy of low-frequency dynamic processes.
[0066] In CardiNet, the model's structure was adjusted accordingly because the acoustic response of the heart sound band is significantly higher than that of the pulse wave. First, this module eliminates the need for an ordinary differential equation (ODE) front-end network, directly employing a partial differential equation (PDE) backbone structure. Second, to enhance high-frequency representation capabilities, the number of neurons per layer of the backbone network was expanded to 256. Its free-float frequency range (FFL) was extended to 0-200Hz, with a step size of 2Hz, to cover the main spectral range of heart sounds and improve the resolution of transient high-frequency components.
[0067] In MappingNet, the encoder comprises three cascaded stages, each consisting of a multi-scale convolutional module, a residual structure, and a max-pooling layer. The multi-scale convolutional module uses four parallel one-dimensional convolutional branches with kernel sizes of 3, 7, 15, and 31. The branch outputs are concatenated along the channel dimension, then batch normalized and activated using PReLU. Subsequently, they are downsampled using a max-pooling layer with a stride of 4, employing a channel doubling strategy to progressively expand the feature dimension from 1 to 256. The residual module embeds a dual-branch time-frequency attention structure, one branch constructing the frequency response based on Fast Fourier Transform, and the other characterizing temporal dependencies through temporal convolution. Building upon the encoder, a Transformer module is introduced for global feature association modeling. This module consists of four self-attention layers, each with eight attention heads and a 1024-dimensional feedforward network, and employs Dropout regularization (with a dropout rate of 0.1) to mitigate overfitting. The decoder and encoder are mirrored and stacked together. The time resolution is restored step by step through three upsampling layers with an amplification factor of 4, and the final waveform is output in the channel order of [256,128,64,1].
[0068] Network training strategy: The entire process involves optimizing HemoNet, CardiNet, and MappingNet sequentially.
[0069] HemoNet employs a phased training strategy to reduce the instability caused by multi-physics-constrained coupled optimization. The training process is divided into two phases. The first phase optimizes only the sub-networks. and Backbone fluctuation network With compensation network The parameters remain frozen. This phase involves 2000 training epochs, using the AdamW optimizer, with an initial learning rate set to 10. 3. The corresponding loss weight is =1、 =50. The goal of this stage is to stabilize the front-end feature representation, ensuring it is fully aligned with the boundary supervision signals. The second stage fixes the front-end network parameters and then focuses on solving the core partial differential equations. With compensation network Perform joint optimization. At this point, set... =1、 =5. The optimization process of the core partial differential equation solving network is further divided into two sub-stages: pre-training and fine-tuning. The pre-training stage consists of 500 rounds, using only boundary loss (LBC) with a learning rate set to 10. -3 Based on this, a fine-tuning phase was conducted, consisting of 2000 iterations, jointly optimizing all loss terms and reducing the learning rate to 8×10. -4To mitigate training imbalance caused by differences in gradients among different loss terms, an adaptive weight update mechanism is introduced every 500 epochs to dynamically balance the contributions of various gradients. CardiNet maintains the same core training process as HemoNet.
[0070] The acquired intraocular sound signals can be processed using the trained network. Inverted source pressure waveform to the origin of the heart .
[0071] like Figure 5 As shown, a set of heart sound signal samples reconstructed based on in-ear sounds are presented. It can be seen that the heart sound signals reconstructed based on the method proposed in this scheme have a very high degree of consistency with the heart sound signals directly collected at the heart using an electronic stethoscope, demonstrating high accuracy and reliability.
[0072] Based on the above, after obtaining the reconstructed phonocardiogram, anomaly detection and identification are performed based on the corresponding heart sound signals, including: Heart rate feature anomaly detection and heart sound waveform anomaly detection are performed based on the reconstructed heart sound signal.
[0073] Furthermore, the detection of abnormal heart rate features based on the reconstructed phonocardiogram includes: A heart rate variability analysis method based on heart sound event detection is used to statistically and nonlinearly model the time interval between adjacent cardiac cycles; By evaluating the stability, rhythmic regularity, and abnormal pulsation patterns of the heart rate, it is determined whether there are any abnormalities in the heart rate characteristics corresponding to the reconstructed heart sound signal.
[0074] Furthermore, the detection of abnormal heart sound waveforms based on the reconstructed heart sound signal includes: The duration, energy distribution, and relative relationship of the first and second heart sounds are modeled using a method based on the temporal structure and time-frequency joint feature analysis of heart sounds. By identifying abnormalities in the phase structure and abnormal spectral components of heart sounds, it can be determined whether there are pathological abnormalities in the heart sound waveform corresponding to the reconstructed heart sound signal.
[0075] This invention provides a method for detecting abnormal heart sounds based on headphones. By introducing a physically constrained physical information neural network, it inverts the source pulse wave signal from the intraocular sound signal acquired by the headphones. Compared with the traditional method of directly estimating heart sounds from distant signals, this method effectively alleviates the problem of information loss during signal propagation and improves the physical consistency and accuracy of the reconstructed signal. It solves the problem of inaccurate heart sound signal inversion from intraocular sound acquired by headphones in existing technologies. During the inversion process, based on the actual physical model of the intraocular sound signal transmission process, the source pulse wave is introduced as an intermediate physical variable. This allows for explicit modeling of the physiological mechanism of heart sound generation, decomposing the complex mapping process from intraocular sound to heart sound into two sub-processes: "inversion of source pulse wave from intraocular sound signal" and "reconstruction of heart sound from source pulse wave signal." This effectively reduces the complexity of the learning task and improves the stability and trainability of the model. It achieves simple acquisition of heart sound signals and accurate inversion of source pulse wave signals, improving the accuracy and convenience of heart rate monitoring.
[0076] Example 2 This invention provides a heart sound abnormality detection system based on headphones, comprising: headphones and a processor; The earphones are connected to the processor via Bluetooth; The earphone has a built-in microphone for acquiring sound signals from inside the user's ear; The processor is used to execute the headphone-based heart sound anomaly detection method as described in any one of Embodiment 1, and output the anomaly detection result to the user.
[0077] In one specific embodiment, an upgraded EarAce platform with a sampling rate of 4 kHz is used for data acquisition. A W380NB headset is used, with one end connected to the data acquisition platform and the other end split into two channels: one headset is inserted into the ear canal to collect intraocular sounds, and the other headset is connected to a multi-functional stethoscope to collect surface heart sounds. A 30-second audio pair is acquired. After acquisition, the intraocular sound information and heart sound information are transmitted to the user's mobile phone. On the mobile phone, the received intraocular sound signal is sequentially processed to generate cardiac pressure wave signals and heart sound signals. The mobile phone compares and calculates the generated heart sound signals with the acquired heart sound signals, outputting heart rate feature anomaly detection results and heart sound waveform anomaly detection results, and presenting the results to the user.
[0078] Non-invasive cardiac monitoring based on wearable devices offers a new technological approach that can be widely applied in areas such as daily health monitoring, telemedicine, and early screening for heart disease.
[0079] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for detecting abnormal heart sounds based on headphones, characterized in that, include: Acquire discrete intra-ear sound signals at each time stamp collected by the headphones; The intraocular sound signal is inverted and mapped based on a pre-constructed physical information neural network to obtain a reconstructed phonocardiogram; Anomaly detection and identification are performed based on the reconstructed phonocardiogram; The physical information neural network includes a source pulse wave inversion network HemoNet and a cross-modal spatiotemporal mapping network MappingNet; the source pulse wave inversion network HemoNet is used to invert discrete intraocular sound signals to obtain continuous intraocular sound signals. And based on continuous intraocular sound signals The arterial pulse pressure wave signal at the eardrum was obtained by inversion at the eardrum. Then the arterial pulse pressure wave signal The original pulse wave signal was obtained by inversion at the heart. The cross-modal spatiotemporal mapping network MappingNet is used to map the source pulse wave signal. The heart sound signal is generated by mapping, resulting in a reconstructed phonocardiogram; L represents the distance from the heart sound source to the blood vessels in the ear canal.
2. The method for detecting abnormal heart sounds as described in claim 1, characterized in that, The source pulse wave inversion network HemoNet includes a first inversion network and a second inversion network; The first inversion network includes a sub-network with a multilayer perceptron structure. Hezi Network The sub-network The subnetwork is used to invert continuous intraocular sound signals based on each timestamp t corresponding to discrete intraocular sound signals; Used to invert the pulse pressure wave signal at the eardrum based on each time stamp t corresponding to the discrete intraocular sound signal; Based on the continuous intraocular sound signal and the pulse pressure wave signal at the eardrum, a first objective function is constructed for the first inversion network. The first objective function is then optimized and solved to obtain the sub-network. Hezi Network The optimal parameters, and based on the subnetwork Hezi Network The optimal parameters were used to obtain the pulse pressure wave signal at the eardrum. ; The second inversion network includes a backbone fluctuation network. and compensation network The backbone wave network Sampling points at each location within the distance from the heart to the eardrum. The tuple formed by each timestamp t corresponding to the discrete intraocular sound signal The inversion yields the distance from the heart. Continuous pressure at the point ; The value range is [0, L]; the compensation network Used for binary-based The generation of heart sound signals during propagation Error compensation signal at the location; Based on distance from the heart Continuous pressure at the point The second objective function of the second inversion network is constructed using the error compensation signal. The second objective function is then optimized to obtain the backbone wave network. and compensation network The optimal parameters were determined based on the backbone wave network. and compensation network The optimal parameters are obtained from the distance from the heart. Continuous pressure at the point Thus, the source pulse wave signal is obtained. .
3. The method for detecting abnormal heart sounds as described in claim 2, characterized in that, The first objective function used in the first inversion network is expressed as: in, and These are the weight coefficients corresponding to the first objective function; Observational loss For sub-networks The output prediction value and the in-ear sound signal The mean square error between them is expressed as: , for Intra-auricular sound signals at any given moment for The length of the discretized sequence, , They refer to the network respectively. Weights and biases; Physical residual loss Used to constrain subnetworks Output satisfies the in-ear sound signal Arterial pulse wave signal The response is expressed as: Among them, the equivalent damping coefficient Equivalent natural frequency squared Coupling gain coefficient , The number of sample points. express The Internet Predicted pressure values at any given time express The output is at the eardrum. Predicted pressure values at any given time; The second objective function used in the second inversion network is expressed as follows: in, and The weight coefficients correspond to the second objective function; boundary condition loss is used. Compared with the residual loss of the modified partial differential equation The training of the backbone fluctuation network is aimed at optimization. ; Boundary condition loss The expression is: Where c represents the pulse wave propagation speed. , They refer to the network respectively. Weights and biases; Subnetworks in the first inversion network Output Predicted pressure at the eardrum at all times; Residual loss of partial differential equations The expression is: in, In the space-time domain The number of random sampling points; express The output is at a distance from the heart Place, Predicted pressure values at any given time; According to the compensation network Error compensation generated residual loss for partial differential equations Modifications were made, and the residual loss of the corrected partial differential equation was determined. The expression is: 。 4. The method for detecting abnormal heart sounds as described in claim 3, characterized in that, The physical information neural network also includes: CardiNet, a source heart sound inversion network; The source heart sound inversion network CardiNet includes: a backbone wave network. With compensation network The backbone wave network With compensation network With the backbone fluctuation network in the second inversion network With compensation network Same structure; The optimization process of the CardiNet source heart sound inversion network is the same as that of the second inversion network.
5. The method for detecting abnormal heart sounds as described in claim 4, characterized in that, The cross-modal spatiotemporal mapping network MappingNet uses the source pulse wave output from the source pulse wave inversion network HemoNet as input and the source heart sound signal output from the source heart sound inversion network CardiNet as a label to learn cross-modal mapping relationships through a data-driven approach, thereby establishing source pulse waves originating from the heart. The nonlinear mapping relationship between the heart sound signal and the reconstructed heart sound signal; The cross-modal spatiotemporal mapping network MappingNet adopts a symmetric U-shaped encoder-decoder network architecture, including cascaded encoders, Transformer modules, and decoders. The encoder includes N cascaded coding units, each of which includes cascaded multi-scale convolutional modules, residual connection structures, and time-frequency attention modules. The encoder is used to extract features of the heart sound signal and pass them to the attention mechanism of the Transformer module. The Transformer module calculates the global weight correlation of the features extracted by the encoder and automatically identifies the features related to heart sounds and the temporal pattern of feature peaks in the source pulse wave. The decoder is structurally mirror-symmetric to the encoder and includes N cascaded decoding units, used to reconstruct the corresponding heart sound signal by upsampling layer by layer according to the temporal pattern; where N is a positive integer.
6. The method for detecting abnormal heart sounds as described in claim 5, characterized in that, The process of training MappingNet includes: The labeled source heart sound signal output by the CardiNet source heart sound inversion network is denoted as The preset source heart sound signal for the cross-modal spatiotemporal mapping network MappingNet is... The composite loss function is constructed as follows: in, and For the weight coefficients of the corresponding loss function, the frequency domain loss... The expression is: Temporal loss The expression is: , This is the short-time Fourier transform of the input signal.
7. The method for detecting abnormal heart sounds as described in claim 1, characterized in that, The abnormality detection and identification based on the reconstructed phonocardiogram includes: Heart rate feature anomaly detection and heart sound waveform anomaly detection are performed based on the reconstructed phonocardiogram.
8. A heart sound anomaly detection system based on headphones, characterized in that, include: Headphones and processor; The earphones are connected to the processor via Bluetooth; The earphone has a built-in microphone for acquiring sound signals from inside the user's ear; The processor is used to execute the headphone-based heart sound anomaly detection method as described in any one of claims 1-7, and output the anomaly detection result to the user.