Multimodal perceptual squeal detection and active compensation method

Abstract

The application discloses a multi-modal perception sound leakage detection and active compensation method, relates to the technical field of intelligent audio devices, and fuses multi-modal data collected by external sound sensors, internal sound sensors, wearing sensors and six-axis motion chips, realizes high-precision detection of sound leakage through a sound leakage classification model, and adopts multiple strategies of acoustic compensation, hardware compensation and algorithm compensation for active sound compensation when sound leakage is detected; closed-loop control from detection to compensation is realized, sound leakage detection accuracy and sound leakage suppression effect are improved, multi-scene requirements are adapted to, earphone wearing experience and sound listening privacy are significantly improved, and the technical problems that the existing earphone sound leakage processing has low sound leakage detection accuracy, is easily affected by motion, has wearing sealing performance and cannot perform individualized sound leakage suppression or compensation in multiple scenes are solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent audio device technology, and in particular to a multimodal sensing method for sound leakage detection and active compensation. Background Technology

[0002] With the increasing popularity of wireless headphones, over-ear headphones, and active noise-canceling headphones, users' demands for sound quality, noise isolation, and personalized listening experiences are constantly rising. Especially in noisy environments, sports scenarios, or open spaces, sound leakage not only affects the wearer's listening privacy but can also cause external noise to interfere with the listening experience and even negatively impact sound quality. Therefore, the industry is gradually focusing on the technological research and development of headphone sound leakage detection and compensation.

[0003] Existing sound leakage detection methods primarily rely on a single microphone to collect acoustic signals from the outside or inside of the earphone, and determine the degree of leakage by comparing the difference between the external ambient noise and the sound signal picked up inside the ear canal. However, this method based on single-microphone waveform comparison is easily affected by changes in ambient noise, especially in complex scenarios such as wind noise, traffic noise, or noisy environments, often leading to misjudgments or decreased detection accuracy. Furthermore, since the detection algorithms are mostly based on passive judgment, the system often only provides a prompt or compensates by simple volume adjustments after identifying sound leakage, lacking active suppression or physical intervention methods for the leakage itself.

[0004] Furthermore, while existing headphone products generally have wear detection capabilities to detect whether the headphones are securely worn or removed, these are mostly used for pausing playback or power saving, lacking the ability to deeply integrate wear status information with sound leakage detection and active compensation strategies. In other words, even if a loose fit or decreased seal is detected, the system usually still responds simply with volume control, failing to incorporate hardware-level adaptive sealing or active acoustic compensation structures to dynamically optimize sound leakage suppression.

[0005] More importantly, traditional noise reduction often relies on physical sealing methods, such as earplugs fitting tightly into the ear canal or headbands increasing coverage to block external noise. However, this purely physical noise isolation method suffers from poor wearing comfort, pressure during prolonged wear, and decreased seal during strenuous exercise. Furthermore, the lack of intelligent scene recognition and adaptive adjustment mechanisms makes it difficult to balance noise reduction, comfort, and battery life in changing environments.

[0006] While some high-end headphones have attempted to incorporate active noise cancellation (ANC) technology to counteract some external noise, these technologies primarily focus on suppressing ambient noise and still have significant limitations in terms of the accuracy, response speed, and dynamic adjustment capabilities for sound leakage detection and compensation. Especially in scenarios such as sports, running, and fitness, the headphones are prone to loosening, leading to significant transient changes in sound leakage, and traditional solutions often fail to create a real-time closed loop for detection and compensation.

[0007] The existing technology has the following problems:

[0008] 1. Relying on a single microphone for detection makes it susceptible to environmental noise interference, resulting in limited accuracy in sound leakage detection;

[0009] 2. Most only provide passive prompts such as volume adjustment, lacking a comprehensive compensation mechanism at the active acoustic and physical levels;

[0010] 3. It fails to effectively combine wearing status detection with usage scenarios, and lacks dynamic and adaptive sound leakage detection and compensation strategies;

[0011] 4. Traditional physical sound insulation methods suffer from discomfort when worn, poor sealing that is easily affected by movement, and a lack of intelligent adjustment, failing to meet the higher requirements for sound leakage suppression and user experience in various scenarios.

[0012] In summary, the existing technology has at least the following technical problems:

[0013] Existing headphone leakage handling suffers from technical problems such as low accuracy in leakage detection, susceptibility to wear seal issues due to exercise, and inability to provide personalized leakage suppression or compensation in various scenarios. Summary of the Invention

[0014] The purpose of this invention is to provide a multimodal sensing method for sound leakage detection and active compensation, in order to solve the technical problems of low sound leakage detection accuracy, susceptibility to motion affecting the fit and seal, and inability to perform personalized sound leakage suppression or compensation in multiple scenarios in existing headphone sound leakage processing.

[0015] The preferred technical solutions among the many technical solutions provided by this invention can produce a variety of technical effects, which are described in detail below.

[0016] To address the aforementioned technical problems, the present invention provides the following technical solution:

[0017] This invention provides a multimodal sensing method for sound leakage detection and active compensation, comprising hardware compensation including an external sound sensor, a speaker, an internal sound sensor, a wear sensor, an integrated six-axis chip, and a circuit element storing a sound leakage classification model, all disposed within the housing of an earphone; the external sound sensor, the speaker, the internal sound sensor, the wear sensor, and the six-axis chip are all electrically connected to the circuit element; the six-axis chip is used to acquire data on the three-dimensional motion state of the earphone; and algorithm compensation including step 1 of environmental audio acquisition, step 2 of detecting the wear state, step 3 of sound leakage judgment, and step 4 of selecting to perform active sound compensation or continuous monitoring based on the sound leakage judgment; wherein, in step 1, after the original audio is emitted by the speaker, environmental noise is acquired through the external sound sensor and the internal sound sensor, and the acquired environmental noise data is transmitted to the earphone housing. The circuit elements process and input the data into the sound leakage classification model; during step 2, the wearing sensor detects the headphone wearing status data and transmits it to the circuit elements for processing and input into the sound leakage classification model; after the six-axis chip obtains that the headphone is being worn, it starts detecting the headphone's motion state and transmits the detected headphone's three-dimensional motion data to the circuit elements and inputs it into the sound leakage classification model; step 3 integrates and analyzes the input environmental noise data, headphone wearing status data, and headphone three-dimensional motion data through the sound leakage classification model to determine whether there is sound leakage; when sound leakage is determined, the circuit elements adjust the output audio signal, and the speaker performs audio output to achieve active sound compensation; when sound leakage is determined not to have occurred, the circuit elements coordinate the external sound sensor, the speaker, the internal sound sensor, and the six-axis chip to perform continuous monitoring.

[0018] In one embodiment, the detection end of the external sound sensor faces and covers the external sound channel provided by the housing, and the back of the external sound sensor faces the back of the speaker, for detecting only sounds outside the housing, including external noise and sound leakage from the speaker to the outside of the housing; the sound-emitting end of the speaker faces the sound-emitting chamber of the housing, the sound-emitting chamber for providing a continuous channel for sound transmission to the ear or ear canal; the internal sound sensor is disposed in the sound-emitting chamber, and the detection direction of the detection end of the internal sound sensor is towards the sound-emitting end of the speaker, or offset from the sound-emitting end of the speaker, or parallel to the sound-emitting end of the speaker, for detecting the sound emitted by the speaker and the echo in the ear or ear canal; the detection end of the wearing sensor is disposed in the housing in close contact with the ear canal or towards the ear, for detecting the wearing status of the headphones.

[0019] In one embodiment, steps 1 and 2 are executed in parallel, and steps 3 and 4 are executed sequentially after steps 1 and 2 have been completed.

[0020] In one embodiment, it further includes reverse acoustic compensation, wherein the reverse acoustic compensation involves emitting a sound wave opposite to the detected sound leakage through the speaker to cancel noise and leaked sound during active sound compensation; the hardware compensation also includes a memory seal sleeve with a shape memory alloy wire frame, the memory seal sleeve being disposed at the end of the sound channel outlet of the sound chamber of the housing, the shape memory alloy wire frame being electrically connected to the circuit element, and energizing the shape memory alloy wire frame to control the deformation of the memory seal sleeve to conform to the contour of the target wearing part, thereby improving the seal between the sleeve and the wearing part; the algorithm compensation, when sound leakage is detected, adjusts the high-frequency EQ curve output by the speaker to attenuate the energy of frequencies above 20kHz to reduce air-conducted sound leakage.

[0021] In one embodiment, the first detection end of the wear sensor is disposed on the housing in close contact with the ear canal or facing the ear, for detecting the overall wearing status of the earphone; the second detection end of the wear sensor is disposed on the memory seal sleeve to detect the fit with the ear canal or auricle, outputs fit detection data to the circuit element, and drives the memory seal sleeve to deform and fit based on the fit detection data, so as to optimize the wearing seal.

[0022] In one embodiment, the fit detection data is displayed via a mobile terminal application paired with the headphones, and user prompts are provided through human-computer interaction, including warnings of loose fit and personalized wearing optimization suggestions.

[0023] In one embodiment, the sound leakage classification model employs a dynamic threshold adaptive strategy that integrates multimodal data, including: environmental noise data collected by the external sound sensor and the internal sound sensor; headphone wearing status data collected by the wearing sensor; and headphone three-dimensional motion data collected by the six-axis chip. The dynamic threshold adaptive strategy is used to adjust the sound leakage judgment threshold in real time based on the environmental noise intensity, fit detection data, and user habits to avoid misjudgment in high-noise scenarios.

[0024] In one embodiment, a context-based intelligent strategy is also included, wherein: when the headphone acceleration is detected to be greater than 2 m / s², 2When the device enters sports mode, the wearing detection frequency is increased to 50Hz, and the active compensation priority is increased to reduce sound leakage caused by loose wearing. When the ambient sound pressure exceeds 80dB, the device enters high noise environment mode, lowers the sound leakage judgment threshold, and avoids environmental noise interfering with sound leakage detection. When a preset voice keyword is detected, the sound leakage judgment threshold is lowered by 20%, and the device enters full compensation mode, which includes activating the reverse acoustic compensation, the algorithm compensation, and the deformation of the memory sealing sleeve supplemented by the hardware. Through acoustic cancellation of noise, reduction of sound leakage, and physical isolation of noise, the compensation intensity of the full compensation mode is increased.

[0025] Also provided is an in-ear headphone, which includes the above-mentioned multimodal sensing leakage detection and active compensation method. The hardware compensation also includes a memory sealing sleeve with a shape memory alloy wire frame, the memory sealing sleeve being earbud-shaped. The shape memory alloy wire frame is electrically connected to the circuit element. When the shape memory alloy wire frame is energized, the shape memory sealing sleeve is deformed to fit the shape of the ear canal, thereby improving the sealing performance between the sound outlet of the sound chamber of the shell and the ear canal.

[0026] Also provided is a pair of headphones, including the above-mentioned multimodal sensing leakage detection and active compensation method. The hardware compensation includes a memory sealing sleeve with a shape memory alloy wire frame, the memory sealing sleeve being ear-shaped. The shape memory alloy wire frame is electrically connected to the circuit element. When the shape memory alloy wire frame is energized, the shape memory alloy wire frame is energized to control the deformation of the memory sealing sleeve to fit the cheek area outside the ear, thereby improving the sealing between the sound outlet of the sound chamber of the shell and the cheek area.

[0027] The beneficial effects of this invention are as follows:

[0028] First, this invention overcomes the problems of traditional single-microphone solutions being susceptible to environmental noise interference and having a high false alarm rate by combining multi-modal fusion detection technology with multi-dimensional data collected from external sound sensors, internal sound sensors, wearable sensors, and a six-axis motion chip. This improves the accuracy of sound leakage detection to 97.2% and reduces the false alarm rate to below 2%, significantly improving detection reliability, especially in high-noise scenarios where it can still maintain accurate detection.

[0029] Secondly, this invention goes beyond simply detecting sound leakage; it breaks through the limitations of traditional passive compensation methods that rely solely on volume adjustment. Through a hardware-algorithm collaborative active compensation system, it achieves multiple linkages between acoustic compensation, hardware compensation, and algorithm compensation: acoustic compensation utilizes the principle of reverse sound wave cancellation to reduce sound leakage by 15-20dB; hardware compensation uses shape memory alloy-driven adaptive air pressure earplugs to achieve dynamic adjustment of the seal, improving sealing performance by approximately 30% and significantly reducing sound leakage caused by loose fit; algorithm compensation further enhances the compensation effect by dynamically adjusting the high-frequency EQ curve to reduce air conduction sound leakage.

[0030] Furthermore, this invention introduces a scenario-based intelligent strategy to achieve intelligent processing that is closer to the actual user experience; the system in the circuit components can adjust according to the motion state (e.g., acceleration greater than 2m / s²). 2 Based on ambient noise levels (e.g., greater than 80dB) or detected user voice keywords, the system automatically switches between different leakage detection and compensation modes to ensure that the headphones maintain a stable leakage suppression effect in complex scenarios such as sports and noisy environments, avoiding misjudgment and overcompensation, and improving user wearing comfort and listening experience.

[0031] More importantly, the "detection-compensation" closed-loop design proposed in this invention has real-time performance and high sensitivity, with a compensation response time of less than 200ms and an additional power consumption increase of only about 8%. It fully balances the needs of performance, battery life and user experience, and can greatly enhance the product competitiveness of smart headphones.

[0032] In summary, the multimodal sensing leakage detection and active compensation method of the present invention not only solves the problems of low accuracy of leakage detection and susceptibility to loose wearing and motion interference in the prior art, but also provides a systematic, accurate and efficient solution for leakage control of smart headphones in multiple scenarios through innovative multimodal detection and active compensation technology, which has extremely high practical value and prospects. Attached Figure Description

[0033] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating the multimodal sensing leakage detection and active compensation method of the present invention;

[0035] Figure 2 This is a schematic diagram of the algorithm flow of the multimodal sensing leakage detection and active compensation method of the present invention;

[0036] Figure 3 This is a block diagram of the leakage sound classification model of the multimodal sensing leakage sound detection and active compensation method of the present invention;

[0037] Figure 4 This is one of the partial cross-sectional structural schematic diagrams of the in-ear headphone of the present invention;

[0038] Figure 5 This is the second partial cross-sectional structural schematic diagram of the in-ear headphone of the present invention;

[0039] Figure 6 This is a partial cross-sectional structural diagram of the headset of the present invention.

[0040] The reference numerals in the attached figures are as follows:

[0041] 1. External sound sensor;

[0042] 2. Loudspeaker;

[0043] 3. Internal sound sensor;

[0044] 4. Wear sensors;

[0045] 5. Six-axis chip;

[0046] 6. Circuit components;

[0047] 7. Shell;

[0048] 8. Memory sealing sleeve;

[0049] 9. Battery. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0051] This specific implementation provides a multimodal sensing method for sound leakage detection and active compensation. This method integrates multimodal data collected from external sound sensors, internal sound sensors, wear sensors, and a six-axis motion chip. It achieves high-precision sound leakage detection through a sound leakage classification model, and actively compensates for sound leakage using multiple strategies including acoustic compensation, hardware compensation, and algorithmic compensation when leakage is detected. This achieves closed-loop control from detection to compensation, improving sound leakage detection accuracy and suppression effectiveness, adapting to various scenarios, and significantly improving the headphone wearing experience and listening privacy. It effectively solves the technical problems of low sound leakage detection accuracy, susceptibility to motion affecting fit, and inability to perform personalized sound leakage suppression or compensation in various scenarios in existing headphone sound leakage processing.

[0052] The first implementation of the multimodal sensing leakage detection and active compensation method is as follows: Figures 1 to 6The hardware compensation includes an external sound sensor 1, a speaker 2, an internal sound sensor 3, a wear sensor 4, an integrated six-axis chip 5, and a circuit element 6 storing a sound leakage classification model, all housed within the earphone housing 7. The external sound sensor 1, speaker 2, internal sound sensor 3, wear sensor 4, and six-axis chip 5 are all electrically connected to the circuit element 6. The six-axis chip 5 is used to acquire data on the three-dimensional motion state of the earphone. The algorithm compensation includes steps 1 (ambient audio acquisition), 2 (wearing status detection), 3 (sound leakage judgment), and 4 (selecting between active sound compensation and continuous monitoring based on the sound leakage judgment). In step 1, after the original audio is emitted by the speaker 2, ambient noise is acquired through the external sound sensor 1 and internal sound sensor 3, and the acquired ambient noise data is transmitted to the circuit element. Step 6 processes and inputs the sound leakage classification model; during execution, the wearing sensor 4 detects the headphone wearing status data and transmits it to the circuit element 6 for processing and inputting the sound leakage classification model; after the six-axis chip 5 obtains that the headphone is being worn, it starts detecting the headphone's motion state and transmits the detected headphone's three-dimensional motion data to the circuit element 6 and inputting it into the sound leakage classification model; step 3 integrates and analyzes the input environmental noise data, headphone wearing status data, and headphone three-dimensional motion data through the sound leakage classification model to determine whether there is sound leakage; step 4, when sound leakage is determined, the circuit element 6 adjusts the output audio signal, and the speaker 2 performs audio output to realize active sound compensation; when sound leakage is determined not to have occurred, the circuit element 6 coordinates the external sound sensor 1, the speaker 2, the internal sound sensor 3, and the six-axis chip 5 to perform continuous monitoring.

[0053] Specifically, such as Figure 1 As shown, in the algorithm, steps 1 and 2 are executed in parallel, and steps 3 and 4 are executed sequentially after steps 1 and 2 have been completed.

[0054] The multimodal sensing leakage detection and active compensation method of this invention not only solves the problems of low accuracy and susceptibility to loose wearing and motion interference in existing technologies, but also provides a systematic, accurate, and efficient solution for leakage control of smart headphones in multiple scenarios through innovative multimodal detection and active compensation technology. It has extremely high practical value and promising prospects. Specifically:

[0055] First, this invention overcomes the problems of traditional single-microphone solutions being susceptible to environmental noise interference and having a high false alarm rate by using multimodal fusion detection technology, which combines multi-dimensional data collected by external sound sensor 1, internal sound sensor 3, wearable sensor 4 and six-axis motion chip. This increases the accuracy of sound leakage detection to 97.2% and reduces the false alarm rate to below 2%, significantly improving detection reliability. It can maintain accurate detection even in noisy environments such as subways, streets, and offices.

[0056] Secondly, this invention goes beyond simply detecting sound leakage; it breaks through the limitations of traditional passive compensation methods that rely solely on volume adjustment. Through a hardware-algorithm collaborative active compensation system, it achieves multiple linkages between acoustic compensation, hardware compensation, and algorithm compensation: acoustic compensation utilizes the principle of reverse sound wave cancellation to reduce sound leakage by 15-20dB; hardware compensation uses shape memory alloy-driven adaptive air pressure earplugs to achieve dynamic adjustment of the seal, improving sealing performance by approximately 30% and significantly reducing sound leakage caused by loose fit; algorithm compensation further enhances the compensation effect by dynamically adjusting the high-frequency EQ curve to reduce air conduction sound leakage.

[0057] Furthermore, this invention introduces a scenario-based intelligent strategy to achieve intelligent processing that is closer to the actual user experience; the system in circuit element 6 can adjust according to the motion state (e.g., acceleration greater than 2m / s²). 2 Based on ambient noise levels (e.g., greater than 80dB) or detected user voice keywords, the system automatically switches between different leakage detection and compensation modes to ensure that the headphones maintain a stable leakage suppression effect in complex scenarios such as sports and noisy environments, avoiding misjudgment and overcompensation, and improving user wearing comfort and listening experience.

[0058] More importantly, the "detection-compensation" closed-loop design proposed in this invention has real-time performance and high sensitivity, with a compensation response time of less than 200ms and an additional power consumption increase of only about 8%. It fully balances the needs of performance, battery life and user experience, and can greatly enhance the product competitiveness of smart headphones.

[0059] As one alternative implementation method:

[0060] Regarding the specific arrangement and structure of the aforementioned external sound sensor 1, speaker 2, internal sound sensor 3, and wearable sensor 4, this embodiment is, for example... Figures 4 to 6 As shown, the detection end of the external sound sensor 1 faces and covers the external sound channel provided in the housing 7, and the back of the external sound sensor 1 faces the back of the speaker 2. It is used to detect only the sound outside the housing 7, including external noise and sound leakage from the speaker 2 to the outside of the housing 7. The sound-emitting end of the speaker 2 faces the sound-emitting chamber of the housing 7. The sound-emitting chamber is used to provide a continuous channel for sound transmission to the ear or ear canal. The internal sound sensor 3 is disposed in the sound-emitting chamber. The detection direction of the detection end of the internal sound sensor 3 is towards the sound-emitting end of the speaker 2, or offset from the sound-emitting end of the speaker 2, or parallel to the sound-emitting end of the speaker 2. It is used to detect the sound emitted by the speaker 2 and the echo in the ear or ear canal. The detection end of the wearing sensor 4 is disposed in the housing 7 in close contact with the ear canal or towards the ear. It is used to detect the wearing status of the headphones.

[0061] In application, the external sound sensor 1 is located at the external sound channel of the earphone shell 7, with the detection end facing outward and the back facing the back of the speaker 2; the internal sound sensor 3 is located inside the sound cavity and is oriented towards, offset from, or parallel to the sound-emitting end of the speaker 2; the wearing sensor 4 is placed close to the ear canal or facing the ear; the external sound sensor 1 monitors the ambient noise and leakage signal outside the shell 7, the internal sound sensor 3 acquires the original audio and echo inside the ear canal, and the wearing sensor 4 detects the fit in real time; this achieves the separation and acquisition of external ambient sound and internal ear canal sound, improving the accuracy of leakage detection; and overcomes the problem that traditional single microphones are easily interfered with by external ambient noise, leading to misjudgment of leakage.

[0062] Regarding the specific settings for the aforementioned active sound compensation, this implementation is as follows: Figures 4 to 6 As shown, it also includes reverse acoustic compensation: the sound wave emitted by the speaker 2 is opposite to the detected sound leakage, which is used to cancel noise and sound leakage when performing active sound compensation; the hardware compensation also includes a memory seal sleeve 8 with a shape memory alloy wire frame. The memory seal sleeve 8 is set at the end of the sound channel outlet of the sound chamber of the housing 7. The shape memory alloy wire frame is electrically connected to the circuit element 6. Powering the shape memory alloy wire frame controls the deformation of the memory seal sleeve 8 to fit the contour of the target part of the wear, which is used to improve the sealing between the wear part and the wear part; the algorithm compensation adjusts the high-frequency EQ curve of the speaker 2 when sound leakage is detected, and attenuates the energy of the frequency band above 20kHz to reduce air conduction sound leakage.

[0063] In application, the active sound compensation is implemented through reverse acoustic compensation, hardware compensation, and algorithm compensation. A sound wave with the opposite phase to the leakage signal is emitted by the speaker 2 to cancel out leakage noise. The shape memory alloy wire frame driven by circuit element 6 controls the deformation of the memory sealing sleeve 8, ensuring the earbud or earmuff contour precisely fits the ear canal or auricle, enhancing the seal. The high-frequency EQ curve is adjusted using an algorithm to attenuate energy above 20kHz when leakage is detected, reducing leakage caused by air conduction paths. Acoustic compensation effectively reduces leakage by 15-20dB. Hardware compensation through the memory sealing sleeve 8 improves the headphone seal by approximately 30%, preventing leakage caused by movement or loosening. Algorithm compensation further reduces high-frequency leakage, improving user privacy in noisy environments. This solves the problem of existing technologies relying solely on volume adjustment and lacking an active compensation mechanism, meeting the need for leakage suppression in various scenarios.

[0064] Specifically, sound leakage is mainly caused by the original audio signal being output through the speaker and then leaking into the external environment along the headphone cavity structure.

[0065] In this embodiment, the external sound sensor is used to collect the total sound signal in the environment. Combined with the original audio signal collected by the internal sound sensor and the data on wearing status and movement status, a sound leakage estimation model is constructed to generate an estimated sound leakage signal n(t).

[0066] The sound leakage estimation model is set in the circuit element, and the generated estimated sound leakage signal is transmitted to the estimation chip in the circuit element. Based on the estimation result output by the estimation chip, the speaker is driven to output an inverted compensation signal a(t). This compensation signal is superimposed on the sound leakage waveform in space to reduce the total sound energy.

[0067] In engineering modeling, the leakage signal is considered as the original audio signal s(t) being transmitted through the headphone cavity, i.e.: n(t) = s(t) * h n (t);

[0068] Among them, h n (t) represents the impulse response of the headphone leakage path. The compensation signal is transmitted to the environment via another path, denoted as h. a (t). To achieve the ideal compensation effect, the compensated sound satisfies: a(t)*h a (t)≈﹣n(t);

[0069] Thus, the total external sound y(t) = n(t) + a(t) * h a (t) is the smallest.

[0070] Due to the impulse response h of the compensation path a The error signal y(t) is difficult to solve precisely. In practical systems, adaptive filtering algorithms (such as LMS or FxLMS) are used to dynamically adjust the compensation signal. The circuit elements use the inputs of the external sound sensor 1 and the internal sound sensor 3 as reference signals, and iteratively update the compensation filter parameters in real time to minimize the external sound error signal y(t) detected by the sensors, thereby achieving closed-loop compensation control.

[0071] Furthermore, this invention combines a CNN sound leakage classification model to dynamically adjust the frequency range, phase parameters, and intensity of reverse compensation based on different sound leakage levels and usage scenarios, thereby enhancing the personalization and response speed of the compensation. Especially in high-noise scenarios or during exercise, this compensation mechanism can adapt to changes in headphone sealing in real time, effectively reducing sound leakage by 15–20 dB and improving headphone sound quality and privacy.

[0072] Optionally, the leakage estimation model can be combined with frequency band selective control to perform reverse sound compensation only on the main leakage frequency band of 20Hz to 1kHz, avoiding negative impacts on the main frequency of human voice or high-frequency perception. In addition, the compensation response time can be controlled within 200ms, balancing response speed and energy consumption optimization.

[0073] In addition, the structure controlling the deformation of the memory seal can be replaced from the memory alloy wire frame to a micro airbag structure to adapt to different ear shapes or wearing needs, while improving wearing comfort.

[0074] To address the low accuracy of sound leakage detection in existing headphone technology, the specific settings of the aforementioned sound leakage classification model are as follows: Figure 3 The block diagram shown is of the sound leakage classification model. The sound leakage classification model adopts a dynamic threshold adaptive strategy that integrates multimodal data. The multimodal data includes: environmental noise data, collected by external sound sensor 1 and internal sound sensor 3; headphone wearing status data, collected by wearing sensor 4; and headphone three-dimensional motion data, collected by six-axis chip 5. The dynamic threshold adaptive strategy is used to adjust the sound leakage judgment threshold in real time according to the environmental noise intensity, fit detection data and user habits, so as to avoid misjudgment in high noise scenarios.

[0075] High-noise environments include, for example, heavy industrial manufacturing sites, heavy machinery construction sites, and firing ranges.

[0076] Specifically, the leaky sound classification model is a CNN (Convolutional Neural Network) leaky sound classification model.

[0077] In application, a CNN (Convolutional Neural Network) sound leakage classification model is used, inputting multimodal data and dynamically adjusting the sound leakage judgment threshold. Specifically, environmental noise and ear canal acoustic signals are collected through external sound sensor 1 and internal sound sensor 3; fit data is detected through wearing sensor 4; and three-dimensional motion data of the earphone is collected through a six-axis IMU chip. Then, all the collected data is input into the CNN sound leakage classification model for real-time analysis and sound leakage judgment. After the sound leakage is judged, the sound leakage classification result is output. Finally, the chip in the circuit components analyzes and determines whether to adopt active sound compensation or continuous monitoring. Through the comprehensive judgment of the CNN sound leakage classification model, the sound leakage detection accuracy reaches 97.2%, and the sound leakage false alarm rate is less than 2%. This sound leakage judgment system can flexibly adapt to different sound leakage scenarios in dynamic environments, overcoming the misjudgment problem caused by the fixed threshold of traditional algorithms in noisy environments such as subways and streets.

[0078] To address the technical challenges of personalized optimization for sound leakage suppression and compensation in various headphone usage scenarios, a scenario-based intelligent strategy was implemented. This strategy utilizes a six-axis chip to detect headphone acceleration greater than 2 m / s². 2When the device enters sports mode, the wear detection frequency is increased to 50Hz, and the active compensation priority is increased to reduce sound leakage caused by loose wear. When the ambient sound pressure exceeds 80dB, the device enters high noise environment mode, lowers the sound leakage judgment threshold, and avoids environmental noise interfering with sound leakage detection. When a preset voice keyword is detected, the sound leakage judgment threshold is lowered by 20%, and the device enters full compensation mode, which includes activating reverse acoustic compensation, algorithm compensation, and hardware supplementation of the deformation of the memory sealing sleeve 8. This enhances the compensation intensity of the full compensation mode by acoustically canceling noise, reducing sound leakage, and physically isolating noise.

[0079] Specifically, the six-axis chip 5 is a six-axis IMU (motion) chip.

[0080] When applying the application, set the motion mode, high-noise environment mode, and full compensation mode triggered by voice keywords; during execution, when the six-axis IMU chip detects an acceleration > 2m / s², 2 When entering sports mode, the wear detection frequency is increased to 50Hz, improving compensation priority and preventing sound leakage due to earphone loosening during exercise. When the ambient sound pressure level is >80dB(A), a high-noise environment mode is entered, lowering the sound leakage threshold and improving detection sensitivity. When a set voice keyword (such as "Xiao Mo") is detected, a full compensation mode is entered, simultaneously activating acoustic compensation, algorithm compensation, and hardware compensation to maximize sound leakage suppression. It can implement personalized compensation strategies to improve the earphone's user experience in different scenarios and avoid excessive power consumption or sound quality loss due to over-compensation. It also solves the problem of the inability to link scene recognition and sound leakage detection in existing technologies.

[0081] In summary, this invention utilizes the fusion analysis of multimodal data and a proactive compensation mechanism that combines software and hardware to not only improve the accuracy of sound leakage detection, but also intelligently adjust the compensation strategy according to different usage scenarios, comprehensively solving the problems of low accuracy of sound leakage detection, susceptibility to motion interference, and lack of personalized sound leakage suppression in existing headphones.

[0082] A second implementation of the multimodal sensing leakage detection and active compensation method is as follows: Figures 4 to 6 As shown, the difference between this embodiment and the first embodiment is that the first detection end of the wearing sensor 4 is set in the position of the housing 7 close to the ear canal or facing the ear, and is used to detect the overall wearing status of the earphone; the second detection end of the wearing sensor 4 is set on the memory sealing sleeve 8 to detect the fit with the ear canal or auricle, outputs fit detection data to the circuit element 6, and drives the memory sealing sleeve 8 to deform and fit based on the fit detection data, so as to optimize the wearing seal.

[0083] Among them, the wear sensor 4 can be based on a pressure sensor, a capacitive sensor or an infrared sensor to adapt to different headphone structures, while the fit can be detected by a pressure sensor or a capacitive sensor arrayed on the memory sealing sleeve 8.

[0084] When the user wears the headphones, the first detection end detects in real time whether the headphones are worn correctly to avoid sound leakage due to improper wearing; the second detection end continuously monitors the fit of the memory seal sleeve 8 to the ear canal or auricle; when the fit is detected to drop below a set threshold, the circuit element 6 immediately powers on to drive the shape memory alloy wire frame to heat up, causing the memory seal sleeve 8 to deform and fit the ear canal or auricle contour better, thereby improving the seal; this can improve the sealing reliability of the headphones, effectively reduce sound leakage caused by loose wearing, achieve real-time and automated physical compensation, improve wearing comfort and listening privacy, avoid users frequently manually adjusting the earbud position, and optimize the user experience; it solves the problem that traditional headphones cannot detect fit and the seal is difficult to guarantee, and overcomes the defect of sound leakage caused by loose wearing due to movement, facial expressions or external force.

[0085] Users can also customize the sensitivity or threshold for triggering fit compensation through software based on their preferences.

[0086] The third embodiment of the multimodal sensing sound leakage detection and active compensation method differs from the first embodiment in that the fit detection data is displayed through a mobile terminal application that comes with the headphones, and user prompts are provided in a human-computer interaction manner, including warnings of loose fit and personalized fit optimization suggestions.

[0087] When applied, fit detection data is transmitted to the mobile terminal application (APP) via a wireless communication module (e.g., Bluetooth, Wi-Fi); the APP interface displays the fit data and provides human-computer interaction prompts; the APP provides users with warnings of loosening and personalized wearing optimization suggestions.

[0088] After the user puts on the headphones, the fit detection data is uploaded to the APP in real time. The APP analyzes the fit fluctuation trend. If a risk of loosening is detected, a warning prompt will pop up. The user can adjust the wearing position or activate the automatic compensation function according to the prompt. The APP can also combine the user's historical wearing habits to provide personalized wearing suggestions for different users, such as "It is recommended to replace with larger ear tips" or "It is recommended to adjust the wearing angle".

[0089] This allows users to intuitively understand the fit and correct any issues promptly; it also provides personalized wearing guidance, improving wearing comfort and product satisfaction; it reduces the probability of sound leakage, thus ensuring listening privacy and sound quality; and it solves the problem of existing headphones lacking user-perceptible feedback on fit, overcoming the inability of users to accurately judge the fit of headphones, leading to sound leakage or discomfort.

[0090] Based on the above embodiments of the multimodal sensing leakage detection and active compensation method, an in-ear headphone is provided, such as... Figure 4 and Figure 5 As shown, the hardware compensation includes the application of the above-mentioned multimodal sensing leakage detection and active compensation methods. The hardware compensation also includes a memory sealing sleeve 8 with a shape memory alloy wire frame. The memory sealing sleeve 8 is in the shape of an earplug. The shape memory alloy wire frame is electrically connected to the circuit element 6. Powering the shape memory alloy wire frame controls the deformation of the memory sealing sleeve 8 to fit the shape of the ear canal, thereby improving the sealing between the sound outlet of the sound chamber of the shell 7 and the ear canal.

[0091] When in use, the in-ear headphone housing 7 also contains a battery 9 that is electrically connected to the circuit element 6.

[0092] Based on the above embodiments of the multimodal sensing leakage detection and active compensation method, a headset is provided, such as... Figure 6 As shown, the hardware compensation includes the application of the above-mentioned multimodal sensing leakage detection and active compensation methods. The hardware compensation also includes a memory sealing sleeve 8 with a shape memory alloy wire frame. The memory sealing sleeve 8 is ear-shaped. The shape memory alloy wire frame is electrically connected to the circuit element 6. Powering the shape memory alloy wire frame controls the deformation of the memory sealing sleeve 8 to fit the cheek area outside the auricle, thereby improving the sealing between the sound outlet of the sound chamber of the shell 7 and the cheek area.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described.

Claims

1. A multimodal sensing method for sound leakage detection and active compensation, characterized in that, The device includes an external sound sensor, a speaker, an internal sound sensor, a wear sensor, an integrated six-axis chip, and hardware compensation for circuit elements storing a sound leakage classification model, all housed within the earphone housing. The external sound sensor, the speaker, the internal sound sensor, the wear sensor, and the six-axis chip are all electrically connected to the circuit elements. The six-axis chip is used to acquire data on the three-dimensional motion state of the earphone. And algorithm compensation, including step 1 after the original audio is emitted by the speaker, environmental noise is collected through the external sound sensor and the internal sound sensor, and the collected environmental noise data is transmitted to the circuit element for processing and input into the sound leakage classification model; During step 2, the wearing sensor detects the earphone wearing status data and transmits it to the circuit element for processing and input into the sound leakage classification model; After the six-axis chip detects that the earphone is being worn, it starts detecting the earphone's motion state and transmits the detected three-dimensional motion data of the earphone to the circuit element and inputs it into the sound leakage classification model. Step 3 uses the sound leakage classification model to integrate and analyze the input environmental noise data, headphone wearing status data, and headphone three-dimensional motion data to determine whether there is sound leakage; Step 4: When sound leakage is detected, the output audio signal is adjusted through the circuit components, and the speaker outputs the audio signal to achieve active sound compensation. When it is determined that no sound leakage has occurred, the circuit elements coordinate the external sound sensor, the speaker, the internal sound sensor, and the six-axis chip to perform continuous monitoring.

2. The multimodal sensing leakage detection and active compensation method according to claim 1, characterized in that, The detection end of the external sound sensor faces and covers the external sound channel provided in the housing, and the back of the external sound sensor faces the back of the speaker, for detecting only the sound outside the housing, including external noise and sound leakage from the speaker to the outside of the housing; The sound-emitting end of the speaker faces the sound-emitting chamber of the housing, which provides a continuous channel for sound transmission to the ear or ear canal. The internal sound sensor is disposed in the sound-emitting chamber. The detection direction of the detection end of the internal sound sensor is towards the sound-emitting end of the sound-emitting horn, or displaced from the sound-emitting end of the sound-emitting horn, or parallel to the sound-emitting end of the sound-emitting horn, for detecting the sound emitted by the sound-emitting horn and the echo in the ear or ear canal. The detection end of the wear sensor is positioned in close contact with the ear canal or towards the ear on the housing, and is used to detect the wearing status of the headphones.

3. The multimodal sensing leakage detection and active compensation method according to claim 2, characterized in that, In the algorithm, steps 1 and 2 are executed in parallel, and steps 3 and 4 are executed sequentially after steps 1 and 2 are completed.

4. The multimodal sensing leakage detection and active compensation method according to claim 3, characterized in that, It also includes reverse acoustic compensation, which involves emitting a sound wave that is opposite to the detected sound leakage through the speaker, in order to cancel out noise and leaked sound when performing active sound compensation; The hardware compensation also includes a memory sealing sleeve with a shape memory alloy wire frame. The memory sealing sleeve is located at the end of the sound channel outlet of the sound chamber of the housing. The shape memory alloy wire frame is electrically connected to the circuit element. Powering the shape memory alloy wire frame controls the deformation of the memory sealing sleeve to conform to the contour of the target part of the wearer, thereby improving the seal between the sleeve and the wearer. When sound leakage is detected, the algorithm compensation adjusts the high-frequency EQ curve of the speaker output to attenuate the energy of frequencies above 20kHz to reduce air-conducted sound leakage.

5. The multimodal sensing leakage detection and active compensation method according to claim 4, characterized in that, The first detection end of the wear sensor is located in the position where the shell is close to the ear canal or facing the ear, and is used to detect the overall wearing status of the earphone; The second detection end of the wear sensor is disposed on the memory seal sleeve to detect the fit with the ear canal or auricle, outputs fit detection data to the circuit element, and drives the memory seal sleeve to deform and fit based on the fit detection data, so as to optimize the wearing seal.

6. The multimodal sensing leakage detection and active compensation method according to claim 5, characterized in that, The fit detection data is displayed through a mobile terminal application that comes with the headphones, and user prompts are provided in a human-computer interaction manner, including warnings of loose fit and personalized fitting optimization suggestions.

7. The multimodal sensing leakage detection and active compensation method according to claim 5, characterized in that, The sound leakage classification model employs a dynamic threshold adaptive strategy that integrates multimodal data, including: Environmental noise data is collected by the external sound sensor and the internal sound sensor; Headphone wearing status data is collected by the wearing sensor; The three-dimensional motion data of the headphones is acquired by the six-axis chip. The dynamic threshold adaptive strategy is used to adjust the sound leakage judgment threshold in real time based on the ambient noise intensity, fit detection data and user habits, so as to avoid misjudgment in high noise scenarios.

8. The multimodal sensing leakage detection and active compensation method according to claim 7, characterized in that, It also includes a scenario-based intelligent strategy, which includes: when the headphone acceleration is detected to be greater than 2m / s², it enters sports mode, increases the wearing detection frequency to 50Hz, and increases the priority of active compensation to reduce sound leakage caused by loose wearing. When the ambient sound pressure exceeds 80dB, the system enters a high-noise environment mode to lower the sound leakage detection threshold and avoid environmental noise interfering with sound leakage detection. When a preset voice keyword is detected, the sound leakage judgment threshold is lowered by 20%, and the full compensation mode is entered. This includes activating the deformation of the memory sealing sleeve of the reverse acoustic compensation, the algorithm compensation, and the hardware compensation. The compensation intensity of the full compensation mode is increased by acoustically canceling noise, reducing sound leakage, and physically isolating noise.

9. An in-ear headphone, characterized in that, The method includes the multimodal sensing leakage detection and active compensation method according to any one of claims 1 to 8, wherein the hardware compensation further includes a memory sealing sleeve provided with a shape memory alloy wire frame, and the memory sealing sleeve is in the shape of an earplug; The shape memory alloy wire frame is electrically connected to the circuit element. Powering the shape memory alloy wire frame controls the deformation of the memory sealing sleeve to fit the shape of the ear canal, thereby improving the sealing performance between the sound outlet of the sound chamber of the housing and the ear canal.

10. A type of over-ear headphone, characterized in that, The method includes the multimodal sensing leakage detection and active compensation method according to any one of claims 1 to 5, wherein the hardware compensation further includes a memory sealing sleeve provided with a shape memory alloy wire frame, and the memory sealing sleeve is in the shape of an ear cover; The shape memory alloy wire frame is electrically connected to the circuit element. Powering the shape memory alloy wire frame controls the deformation of the memory sealing sleeve to fit the cheek area outside the ear, thereby improving the sealing between the sound outlet of the sound chamber of the housing and the cheek area.

Images