Earphone-based snoring sound processing method, earphone and storage medium

Abstract

The application discloses a snoring sound processing method based on earphones, earphones and a storage medium, relates to the technical field of earphones, and discloses a snoring sound processing method based on earphones, which comprises the following steps: acquiring snoring sound collected by an audio acquisition unit; extracting convolution features of the snoring sound, and extracting time sequence features corresponding to the convolution features based on a long short-term memory network; extracting a frequency spectrum diagram of the snoring sound, and modeling spectrum features of the frequency spectrum diagram based on a sliding window transformer; fusing the time sequence features and the spectrum features, and outputting an anti-phase signal control parameter corresponding to fused features based on a full connection network; and outputting an anti-phase signal according to the anti-phase signal control parameter. Based on this, the frequency domain and time domain features of the snoring sound signal are cooperatively extracted and fused, the dynamic capturing capability for snoring sound changes is improved, the reliability of the anti-phase signal control parameter is improved, and the noise reduction effect of the snoring sound is improved.

Description

Technical Field

[0001] The present application relates to the field of earphone technology, and in particular to an earphone-based snoring sound processing method, earphone, and storage medium. Background Art

[0002] Related active noise reduction technologies typically use fixed filters to eliminate ambient noise. However, noise levels vary across different environments. When ambient noise fluctuates significantly, such as when snoring occurs irregularly, fixed filters are unable to effectively reduce snoring, disrupting the sleep quality of others.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a snoring sound processing method based on headphones, headphones and storage medium, aiming to solve the technical problem of poor noise reduction effect.

[0005] To achieve the above objectives, the present application proposes a method for processing snoring sounds using headphones, the method comprising:

[0006] Obtaining snoring audio collected by the audio collection unit;

[0007] Extracting convolution features of the snoring audio, and extracting temporal features corresponding to the convolution features based on a long short-term memory network;

[0008] Extracting a spectrogram of the snoring audio, and modeling spectral features of the spectrogram based on a sliding window transformer;

[0009] Fusing the time series features and the spectrum features, and outputting an inverted signal control parameter corresponding to the fused features based on a fully connected network;

[0010] An inverted signal is output according to the inverted signal control parameter.

[0011] In one embodiment, the spectrogram is a short-time Fourier spectrogram, and the steps of extracting the spectrogram of the snoring audio and modeling the spectrogram based on a sliding window transformer to obtain spectral features include:

[0012] Extracting a short-time Fourier spectrum of the snoring audio, and segmenting the short-time Fourier spectrum to obtain local blocks of the same size;

[0013] determining, according to the window multi-head self-attention module of the sliding window transformer, a local frequency band energy distribution of a plurality of the local blocks;

[0014] According to the sliding window self-attention module of the sliding window transformer, modeling the global frequency band correlation information between the plurality of local blocks, wherein the window multi-head self-attention module and the sliding window self-attention module are alternately stacked based on the sliding window transformer module;

[0015] A pyramid feature is constructed according to the downsampling and multi-scale fusion module of the sliding window transformer, and the spectrum feature is output based on the pyramid feature.

[0016] In one embodiment, the step of extracting the convolutional features of the snoring audio and extracting the temporal features corresponding to the convolutional features based on a long short-term memory network includes:

[0017] Extracting convolution features of the snoring audio based on a convolutional neural network;

[0018] The rhythmic structure and long-term dependency of the convolutional features are modeled based on a bidirectional long short-term memory network to obtain the temporal features.

[0019] In one embodiment, the step of outputting the inverted signal according to the inverted signal control parameter includes:

[0020] Determine a noise reduction coefficient according to the inverted signal control parameter, wherein the noise reduction coefficient includes a time domain noise reduction coefficient and a frequency domain noise reduction coefficient;

[0021] generating a time-domain inverse phase signal according to the time-domain noise reduction coefficient and the time-domain signal of the snoring audio;

[0022] A frequency domain inverse phase signal is generated according to the frequency domain noise reduction coefficient and the frequency domain signal of the snoring audio.

[0023] In one embodiment, the noise reduction coefficient includes a time domain noise reduction coefficient and a frequency domain noise reduction coefficient, and the step of determining the noise reduction coefficient according to the inverted signal control parameter includes:

[0024] Determining the time-domain noise reduction coefficient according to the snoring intensity and snoring category confidence corresponding to the inverted signal control parameter, and an activation function;

[0025] A normalization function processing result of the inverted signal control parameter is determined, and the processing result is set as the frequency domain noise reduction coefficient.

[0026] In one embodiment, after the steps of extracting the spectrogram of the snoring audio and modeling the spectrogram based on a sliding window transformer to obtain spectral features, the headphone-based snoring processing method further includes:

[0027] The temporal features and the spectral features are fused, and the snoring type corresponding to the fused features is output based on a fully connected network.

[0028] In one embodiment, before the step of obtaining the snoring audio collected by the audio collection unit, the headphone-based snoring processing method further includes:

[0029] Acquire a snore test audio of a detection area, and generate and output a test inverted phase signal of the snore test audio based on the snore test audio;

[0030] determining a residual signal between the test inverted signal and the snoring test audio;

[0031] The audio optimization parameters of the audio acquisition unit are updated according to the residual signal.

[0032] In one embodiment, the step of obtaining the snoring audio collected by the audio collection unit is performed when a snoring noise reduction condition is met, and the snoring noise reduction condition at least includes:

[0033] The ambient noise decibel level is higher than the preset decibel level;

[0034] The image features detected in the sleep image are consistent with the snoring characteristics;

[0035] The preset detection period has been reached.

[0036] In addition, to achieve the above objectives, the present application also proposes an earphone, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the earphone-based snoring processing method as described above.

[0037] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the steps of the headphone-based snoring processing method as described above are implemented.

[0038] One or more technical solutions proposed in this application have at least the following technical effects:

[0039] After obtaining the current snoring audio, the frequency domain and time domain features of the snoring audio are synchronously extracted to obtain time series features and spectrum features. The extracted time series features and spectrum features are then fused, and the inverted signal control parameters corresponding to the fused features are output based on the fully connected network. Finally, the inverted signal is output through the inverted signal control parameters. Based on this, the collaborative extraction and fusion of frequency domain and time domain features are achieved, the ability to capture the dynamic changes of snoring is improved, thereby improving the reliability of the inverted signal control parameters and further improving the noise reduction effect of snoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 A flowchart of the first embodiment of the headphone-based snoring processing method of this application is provided;

[0043] Figure 2 A flowchart of the second embodiment of the headphone-based snoring processing method of this application is provided;

[0044] Figure 3 This is a brief flowchart of a method for processing snoring using an earphone obtained by combining various embodiments of the present application;

[0045] Figure 4 Schematic diagram of the device structure of the hardware operating environment involved in the headphone-based snoring sound processing method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0048] Related active noise reduction technologies typically use fixed filters to eliminate ambient noise. However, noise in different environments varies. When ambient noise fluctuates significantly, such as when snoring occurs irregularly, fixed filters are unable to effectively reduce snoring, resulting in poor noise reduction results.

[0049] The main solution of the embodiment of the present application is to: obtain the snoring audio collected by the audio collection unit;

[0050] Extracting convolution features of the snoring audio, and extracting temporal features corresponding to the convolution features based on a long short-term memory network;

[0051] Extracting a spectrogram of the snoring audio, and modeling spectral features of the spectrogram based on a sliding window transformer;

[0052] Fusing the time series features and the spectrum features, and outputting an inverted signal control parameter corresponding to the fused features based on a fully connected network;

[0053] An inverted signal is output according to the inverted signal control parameter.

[0054] Specifically, after collecting the snoring signal, the time series features and frequency domain features of the snoring signal are extracted, and the features of the snoring signal in the time domain and frequency domain are fused to improve the accuracy of snoring feature recognition, thereby improving the reliability and accuracy of the inverted signal control parameters generated based on the fused features. Finally, the inverted signal is output through the inverted signal control parameters to complete the noise reduction processing of the snoring audio and improve the audio noise reduction effect.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program execution functions, or a headset capable of implementing the above functions. The following uses the headset as an example to illustrate this embodiment and the following embodiments.

[0056] At the same time, in order to better understand the technical solution of this application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The embodiment of the present application provides a method for processing snoring based on earphones, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the headphone-based snoring processing method of the present application.

[0058] In this embodiment, the earphone-based snoring processing method includes steps S10 to S50:

[0059] Step S10: Acquire the snoring audio collected by the audio collection unit.

[0060] In this embodiment, the audio collection unit is an audio collection function module of the headset, which may be a microphone array of the headset, and its sampling rate must meet the requirements of snoring analysis, for example, a sampling rate ≥ 16 kHz.

[0061] After receiving the raw snoring signal captured by the microphone array, the headset performs pre-emphasis processing (for example, using a first-order high-pass filter H(z) = 1-0.97z⁻¹) to enhance high-frequency components and frames the signal (frame length 20-40ms, frame shift 50%). For example, the microphone array samples snoring at a 24kHz sampling rate, with each frame being 30ms (720 samples). This signal is then windowed using a Hamming window to generate the signal sequence to be processed.

[0062] Optionally, when the snoring noise reduction conditions are met, the headset performs the action of step S10. The snoring noise reduction conditions include at least that the ambient noise decibel is greater than a preset decibel, that snoring movements are present in the image features of the detected sleep image, or that a preset detection period has been reached. For example, when the current ambient noise decibel is at a decibel level corresponding to normal snoring, snoring audio is collected. In addition, when the headset is provided with a camera, it can also capture images of the user while they are sleeping, and analyze whether there are movements corresponding to the user's snoring in the sleep image. If so, snoring audio is collected. Optionally, during a preset detection period, such as from 11 p.m. to 8 a.m. the next day, snoring audio can be collected in real time.

[0063] Step S20: extracting convolution features of the snoring audio, and extracting temporal features corresponding to the convolution features based on a long short-term memory network.

[0064] The above-mentioned convolutional features refer to local time-frequency features extracted by convolutional neural networks (CNNs). The long short-term memory (LSTM) network is a recurrent neural network with a gating mechanism (input gate, forget gate, and output gate) that is used to capture long-range temporal dependencies.

[0065] In this embodiment, after collecting snoring audio, convolutional features of the snoring audio can be extracted based on a convolutional neural network. A 1D convolutional layer (1D-CNN) can be used to perform local waveform perception and extract short-term structural features, such as peaks, bursts, and periodic segments. The calculation process is as follows:

[0066] ,

[0067] in, is the time domain waveform of the original snoring audio, is the convolution kernel weight matrix, is the bias term of 1D-CNN, is the output feature sequence of 1D-CNN, namely the convolution feature.

[0068] After obtaining the convolutional features, the long-term dependency and rhythmic structure of the convolutional features can be modeled based on the bidirectional long short-term memory network (BiLSTM), thereby obtaining the repetitiveness and pattern changes of snoring on the time axis, that is, obtaining the temporal characteristics of the snoring signal. The calculation process includes:

[0069]

[0070] in, is the output of 1D-CNN, which serves as the input sequence of BiLSTM; is the weight matrix of BiLSTM, including the weights of the input gate, forget gate, output gate, and candidate memory units; is the bias term of BiLSTM, corresponding to the bias of each gate; It is the hidden state output of BiLSTM at time step t, which can provide high-resolution temporal expression capabilities for subsequent recognition models to distinguish pathological snoring (obstructive sleep apnea) from normal snoring.

[0071] This embodiment complements the local features of 1D-CNN with the global temporal features of BiLSTM to form multi-scale expression capabilities and improve the ability to recognize and analyze snoring signals.

[0072] Step S30: extracting a spectrogram of the snoring audio, and modeling the spectrum features of the spectrogram based on a sliding window transformer.

[0073] In this embodiment, the spectrogram refers to the time-frequency matrix generated by the short-time Fourier transform (STFT) of the snoring audio. The sliding window transformer (hereinafter referred to as the Swin-Transformer) uses a local window self-attention mechanism and a sliding window self-attention mechanism to process the spectral sequence. It should be noted that before performing STFT processing on the audio signal, windowing processing using a Hamming window is required. The formula is as follows:

[0074] ,

[0075] in Number the sampling points within the frame; The Hamming window is used to reduce spectrum leakage and alleviate boundary effects, thereby improving the stability of spectrum modeling.

[0076] When modeling the spectral features of the spectrogram, the snoring audio is converted into a complex spectrum through short-time Fourier transform. The spectrogram is then divided into multiple local blocks of different sizes, such as 4x4 local blocks. Finally, the spectrogram is modeled and processed using the Swin-Transformer hierarchical structure to obtain the spectral features of the spectrogram.

[0077] It should be noted that step S20 and step S30 are usually performed simultaneously.

[0078] Step S40: fusing the time series features and the spectrum features, and outputting an inverted signal control parameter corresponding to the fused features based on a fully connected network.

[0079] The aforementioned feature fusion refers to the joint representation of features from different modalities. The fully connected network (FCN) is a multilayer perceptron used for nonlinear mapping. The inverted signal control parameters include snoring characteristics such as energy, spectral shape, and confidence. It can be understood that the inverted signal of the noise signal is a waveform with the same amplitude as the original noise signal and a phase difference of 180°. Its core function is to achieve noise cancellation through acoustic interference.

[0080] In this embodiment, the process of fusing time series features and spectral features includes at least feature alignment, fusion calculation, and parameter generation. The feature alignment stage involves aligning the time axis of the BiLSTM output time series features with the Swin-Transformer output frequency domain features. The fusion stage involves feature fusion through weighted concatenation (with the weights controlled by a learnable parameter α). The parameter generation stage involves inputting the fused features into a fully connected network, ultimately generating the inverted signal amplitude and phase parameters through the output layer based on a preset function, such as the Tanh activation function.

[0081] Specifically, the temporal features output by BiLSTM And the frequency domain features of Swin-Transformer output After output splicing, the inverted signal control parameter y corresponding to the snoring audio is obtained inv .

[0082] Optionally, in addition to outputting the inverted signal control parameter through the fully connected network, a snoring type corresponding to the snoring signal may also be generated, so as to enhance the subsequent processing effect of the inverted signal based on the snoring type.

[0083] Step S50: outputting an inverted signal according to the inverted signal control parameter.

[0084] In this embodiment, the inverted signal control parameter y is obtained inv After that, you can use y inv Generate an equal-amplitude, inverted waveform:

[0085] ,

[0086] in, is the noise reduction coefficient, based on y inv OK, y inv Contains snoring intensity and snoring category confidence, For the original snoring audio, is an inverting signal.

[0087] It should be noted that this embodiment uses CNN-LSTM to capture the fine-grained temporal variation patterns of snoring, combined with the global modeling capabilities of Swin-Transformer for spectral features, to achieve multimodal recognition and noise reduction of snoring audio. This results in snoring recognition accuracy exceeding 95%, and an average improvement of 6-8 dB in signal-to-noise ratio after noise reduction. Furthermore, in terms of latency, the end-to-end active noise cancellation (ANC) response time is less than 15ms, meeting the real-time requirements of sleep aid devices. Furthermore, the system performs stably in various background environments, adapts to different acoustic environments and individual differences, and demonstrates strong detection robustness.

[0088] This embodiment provides a headphone-based snoring processing method. Using CNN-LSTM to capture the fine-grained temporal variation patterns of snoring, combined with the global modeling capabilities of Swin-Transformer for spectral features, this method achieves multimodal recognition and noise reduction of snoring audio, effectively balancing real-time and accurate noise reduction. Furthermore, through an active inverse signal synthesis mechanism combined with deep recognition feedback, snoring audio noise reduction is achieved. This multi-dimensional analysis and processing of snoring audio improves the noise reduction effect.

[0089] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction, and no further details will be given later. Figure 2 , step S30 includes steps S31 to S34:

[0090] Step S31, extracting the short-time Fourier spectrogram of the snoring audio, and segmenting the short-time Fourier spectrogram to obtain local blocks of the same size.

[0091] In this embodiment, the snoring audio Converted to complex spectrum by short-time Fourier transform:

[0092] ,

[0093] in: is the real part; is the imaginary part; is frame shift, The spectrogram is expanded into a two-channel input tensor with frequency index:

[0094] .

[0095] After obtaining the short-time Fourier spectrum, the spectrum is embedded into Divided into The block patch (such as ), each block is flattened and linearly mapped:

[0096] ,

[0097] So that each patch is mapped to a dimension of (like ) token.

[0098] Step S32: determining the local frequency band energy distribution of the plurality of local blocks according to the window multi-head self-attention module of the sliding window transformer.

[0099] In this embodiment, Swin-Transformer includes W-MSA (Window-based Multi-head Self-Attention, window multi-head self-attention module) and SW-MSA (Shifted Window Multi-head Self-Attention, sliding window self-attention module). In the process of determining the local frequency band energy distribution of the local block, the local non-overlapping window can be used to Internally execute the multi-head self-attention mechanism:

[0100] ,

[0101] This is used to model the local energy distribution and local resonance modes in the frequency band.

[0102] Step S33: Modeling global frequency band correlation information between the plurality of local blocks according to the sliding window self-attention module of the sliding window transformer.

[0103] In this embodiment, when modeling the global frequency band association relationship between multiple local blocks, the window can be slid by a certain step (such as 2 frequency bands) and the attention operation can be performed again to model the contextual relationship across the window boundary. The contextual information includes the cross-band information flow and the coupling of high-frequency harmonics and low-frequency trunks.

[0104] It should be noted that the windowed multi-head self-attention module and the sliding window self-attention module are stacked alternately based on the sliding window transformer module. The two work together based on the modular combination structure (Transformer Block). Each layer includes two LN (Layer Normalization) → Attention (attention operation) → MLP (Multilayer Perceptron):

[0105] ,

[0106] ,

[0107] in, is the layer number of the current Transformer Block, is the output of the previous layer and serves as the input of the current layer; For intermediate outputs (W-MSA and SW-MSA outputs) Layer normalization is performed again. The MLP uses GELU (Gaussian Error Linear Unit) activation.

[0108] Therefore, W-MSA and SW-MSA are executed alternately in Swin-Transformer, that is, W-MSA and SW-MSA are used alternately in the Stage. For example, Stage1 contains two Blocks: Block1: W-MSA→MLPBlock2: SW-MSA→MLPBlock1: W-MSA→MLPBlock2: SW-MSA→MLP.

[0109] Step S34 , constructing a pyramid feature according to the downsampling and multi-scale fusion module of the sliding window transformer, and outputting the spectrum feature based on the pyramid feature.

[0110] In this embodiment, the downsampling and multi-scale fusion module is a Patch Merging module, which concatenates two adjacent patches along the channel dimension, performing dimensional compression and downsampling via linear layers to form a four-layer pyramid structure. Stages 1 / 2 include fine-grained local modeling, while Stages 3 / 4 include long-range global spectral dependency modeling. Therefore, after alternating W-MSA and SW-MSA on the data within the Swin-Transformer, the pyramid features of these features are obtained through the downsampling and multi-scale fusion modules. The final output spectral features are:

[0111] ,in, 、 is the number of time frames and frequency bands after downsampling, The number of channels is usually between 64 and 384.

[0112] The obtained spectral features can be fed into different subsequent modules, such as the mask generation network (for speech enhancement / noise reduction), the classification network (for audio event recognition), and the active noise reduction controller (for active noise reduction ANC feedback control).

[0113] It should be noted that the above parameters are for explanation only and are not intended to limit the present application.

[0114] This embodiment provides a headphone-based snoring processing method. It applies the Swin-Transformer to snoring modeling and implements local and global spectrum fusion through a window shift mechanism. This significantly improves the spectrum modeling capability, thereby improving the effectiveness of the control parameters of the inverted signal generated by the fusion of spectral features and time series features, and the noise reduction effect of the snoring audio.

[0115] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction and will not be repeated hereafter. On this basis, step S50 includes S51 to S53:

[0116] Step S51: determining a noise reduction coefficient according to the inverted signal control parameter.

[0117] In this embodiment, the formula for generating the inverted signal is: , is the noise reduction coefficient, For the original snoring audio, The inverted signal includes a signal in the time domain and a signal in the frequency domain. Therefore, the noise reduction coefficient includes a time domain noise reduction coefficient and a frequency domain noise reduction coefficient.

[0118] It should be noted that in traditional designs, the active noise reduction inverting signal is usually expressed as:

[0119] But if There is no inverting signal control parameter with the identification network output If the system does not establish an association, it will not be able to achieve "recognition-driven adaptive active noise reduction". Therefore, in this embodiment, Represents the characteristic energy, spectrum, confidence, etc. of snoring. Through a mapping network, Dynamically generated control coefficients , and is used to adjust the inverted signal. Parameters such as noise intensity and noise category confidence are provided by the aforementioned recognition network. For example, classification confidence is the maximum probability in the Softmax output; snoring intensity is predicted by the regression head, such as loudness, duration, amplitude, etc.; and overall confidence is measured by output entropy. These values ​​are combined into the inverted signal control parameters. , which can then automatically generate a noise reduction coefficient.

[0120] Specifically, the time domain noise reduction coefficient, i.e., the single-channel noise reduction parameter or the time domain noise reduction parameter calculation formula can be determined based on the snoring intensity and snoring category confidence corresponding to the inverted signal control parameter and the activation function as follows:

[0121] ,

[0122] Among them, here Refers to the time domain noise reduction coefficient, is Sigmoid activation, and is a trainable parameter.

[0123] During the frequency domain active noise reduction control process, the normalized function processing result of the inverted signal control parameter can be determined, and the processing result is set as the frequency domain noise reduction coefficient:

[0124] .

[0125] Step S52 : generating a time-domain inverse phase signal according to the time-domain noise reduction coefficient and the time-domain signal of the snoring audio.

[0126] In this embodiment, the time domain inverse signal formula in the time domain form is as follows:

[0127] , is the time domain inverted signal; is the time domain noise reduction coefficient, The snore audio is in the time domain.

[0128] Step S53 : generating a frequency domain inverted signal according to the frequency domain noise reduction coefficient and the frequency domain signal of the snoring audio.

[0129] In this embodiment, the frequency domain inverted signal formula in the frequency domain is as follows:

[0130] ,in, is the frequency domain inverted signal, is the frequency domain noise reduction coefficient, The snore audio is in the time domain.

[0131] This embodiment provides a headphone-based snoring processing method, which calculates the corresponding inverse signals in the time domain and frequency domain respectively through the inverse signal control parameter, and implements closed-loop control logic between the inverse signal and the inverse signal control parameter data, physics and model.

[0132] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description and will not be repeated hereafter. On this basis, before performing the snoring audio noise reduction process, the device can also be tested to determine whether its audio noise reduction effect meets the usage requirements. Specifically, before step S10, steps S01 to S03 are also included:

[0133] Step S01 : acquiring a snore test audio in a detection area, and generating and outputting a test inverted phase signal of the snore test audio based on the snore test audio.

[0134] In this embodiment, the snoring test audio is a test snoring signal containing background noise, recorded by the audio acquisition unit in the actual operating environment of the noise reduction system. The test inverted signal is an acoustic wave signal generated based on the current test snoring signal, used to simulate the active noise reduction effect. During device use, snoring noise reduction testing is performed on the device so that device parameters can be adjusted based on the test results to improve the actual snoring noise reduction effect.

[0135] Step S02: determining a residual signal between the test inverted signal and the snoring test audio.

[0136] In this embodiment, the residual signal refers to the signal after the snoring test signal and the test inverted signal are superimposed. If the signals can be accurately replicated in inverse phase, the noise energy of the residual signal of the two is reduced to below the ambient noise level, thereby achieving a perfect noise reduction effect.

[0137] Step S03: updating the audio optimization parameters of the audio acquisition unit according to the noise reduction coefficient.

[0138] In this embodiment, the audio optimization parameters may be a weight vector, a phase compensation amount, a gain coefficient, etc. of the adaptive filter, which directly affects the effect of generating the anti-phase signal.

[0139] If the noise energy of the residual signal is greater than the preset ambient noise, it means that the current snoring noise reduction effect does not meet the actual usage requirements. At this time, the audio optimization parameters can be adjusted to achieve a more accurate snoring denoising effect. If the noise energy of the residual signal is less than the preset ambient noise, it means that the current snoring noise reduction effect meets the requirements. At this time, the audio optimization parameters can be set to the default value. It is understandable that different residual signals have different noise energy values, and their corresponding audio optimization parameters are also different. The noise energy value is usually bound to or associated with an audio optimization parameter.

[0140] This embodiment provides a headphone-based snoring processing method. Before collecting snoring audio, a snoring noise reduction test is performed on the current detection area. Based on the test results, the audio optimization parameters are adjusted to provide more accurate audio optimization parameters for actual snoring denoising, thereby improving the snoring denoising effect.

[0141] For example, to help understand the implementation process of the earphone-based snoring sound processing method obtained by combining the various first embodiments, please refer to Figure 3 , Figure 3This paper provides a simplified flowchart of a headphone-based snoring processing method. Specifically, after acquiring raw audio x_t, convolutional features are extracted using a 1D-CNN. BiLSTM performs rhythm modeling to obtain time-domain features. Simultaneously, the raw audio is processed using a short-time Fourier transform (STFT) to obtain a spectrogram X_f,t. Subsequently, a Swin-transformer performs local and global spectral modeling to obtain frequency-domain features. These features are then fused using a time-frequency fusion module. A fully connected network (FC) is then used to classify and output the snoring type y_label and the inverted signal control parameter y_inv. An inverted waveform synthesizer then generates the inverted signal x_inv_t and the amplitude modulator a_, the noise reduction coefficient. Finally, these signals are fed into a buffer controller and a digital-to-analog converter (DAC), where they are then played back by a speaker in real time to achieve noise reduction of the snoring signal.

[0142] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the headphone-based snoring sound processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0143] The present application provides an earphone, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the earphone-based snoring processing method of the first embodiment described above.

[0144] Reference below Figure 4 , which shows a structural schematic diagram of headphones suitable for implementing the embodiments of the present application. Figure 4 The earphones shown are merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0145] like Figure 4As shown, the headset may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for headset operation. The processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008, such as a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003, such as a magnetic tape or hard disk; and communication device 1009. The communication device 1009 can allow the headset to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a headset with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0146] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0147] The headphones provided in this application utilize the headphone-based snoring processing method described in the aforementioned embodiment to address the technical issue of poor noise reduction. Compared to the prior art, the headphones provided in this application achieve the same beneficial effects as the headphone-based snoring processing method described in the aforementioned embodiment. Other technical features of the headphones are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0148] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0149] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0150] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the earphone-based snoring sound processing method in the above embodiment.

[0151] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0152] The computer-readable storage medium may be contained in the earphone, or may exist independently without being assembled into the earphone.

[0153] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the headset, the headset:

[0154] Obtaining snoring audio collected by the audio collection unit;

[0155] Extracting convolution features of the snoring audio, and extracting temporal features corresponding to the convolution features based on a long short-term memory network;

[0156] Extracting a spectrogram of the snoring audio, and modeling spectral features of the spectrogram based on a sliding window transformer;

[0157] Fusing the time series features and the spectrum features, and outputting an inverted signal control parameter corresponding to the fused features based on a fully connected network;

[0158] An inverted signal is output according to the inverted signal control parameter.

[0159] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0161] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0162] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned headphone-based snoring processing method, thereby resolving the technical issue of poor noise reduction. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the headphone-based snoring processing method provided in the aforementioned embodiment, and are not further elaborated here.

[0163] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for processing snoring based on headphones, characterized in that: The earphone-based snoring processing method includes: Acquire a snore test audio of a detection area, and generate and output a test inverted phase signal of the snore test audio based on the snore test audio; determining a residual signal between the test inverted signal and the snoring test audio; updating audio optimization parameters of the audio acquisition unit according to the residual signal; Acquiring the snoring audio collected by the audio collection unit; Extracting convolution features of the snoring audio, and extracting temporal features corresponding to the convolution features based on a long short-term memory network; Extracting a spectrogram of the snoring audio, and modeling the spectrogram based on a sliding window transformer to obtain a spectrum feature; Fusing the time series features and the spectrum features, and outputting an inverted signal control parameter corresponding to the fused features based on a fully connected network; Determining a time-domain noise reduction coefficient according to the snoring intensity and snoring category confidence corresponding to the inverted signal control parameter, and an activation function; Determine a normalization function processing result of the inverted signal control parameter, and set the processing result as a frequency domain noise reduction coefficient; generating a time-domain inverse phase signal according to the time-domain noise reduction coefficient and the time-domain signal of the snoring audio; A frequency domain inverse phase signal is generated according to the frequency domain noise reduction coefficient and the frequency domain signal of the snoring audio, wherein the noise reduction coefficient is dynamically generated based on a control parameter of the inverse phase signal.

2. The method for processing snoring using earphones according to claim 1, wherein: The spectrogram is a short-time Fourier spectrogram. The steps of extracting the spectrogram of the snoring audio and modeling the spectrogram based on a sliding window transformer to obtain spectrum features include: Extracting a short-time Fourier spectrum of the snoring audio, and segmenting the short-time Fourier spectrum to obtain local blocks of the same size; determining, according to the window multi-head self-attention module of the sliding window transformer, a local frequency band energy distribution of a plurality of the local blocks; According to the sliding window self-attention module of the sliding window transformer, modeling the global frequency band correlation information between the plurality of local blocks, wherein the window multi-head self-attention module and the sliding window self-attention module are alternately stacked based on the sliding window transformer module; A pyramid feature is constructed according to the downsampling and multi-scale fusion module of the sliding window transformer, and the spectrum feature is output based on the pyramid feature.

3. The method for processing snoring using earphones according to claim 1, wherein: The step of extracting the convolution features of the snoring audio and extracting the temporal features corresponding to the convolution features based on the long short-term memory network includes: Extracting convolution features of the snoring audio based on a convolutional neural network; The rhythmic structure and long-term dependency of the convolutional features are modeled based on a bidirectional long short-term memory network to obtain the temporal features.

4. The method for processing snoring using earphones according to claim 1, wherein: After the steps of extracting the spectrogram of the snoring audio and modeling the spectrogram based on a sliding window transformer to obtain spectral features, the headphone-based snoring processing method further includes: The temporal features and the spectral features are fused, and the snoring type corresponding to the fused features is output based on a fully connected network.

5. The method for processing snoring using earphones according to claim 1, wherein: When a snoring noise reduction condition is met, the step of acquiring the snoring audio collected by the audio collection unit is performed, and the snoring noise reduction condition at least includes: The ambient noise decibel level is higher than the preset decibel level; The image features detected in the sleep image are consistent with the snoring characteristics; The preset detection period has been reached.

6. A headset, characterized in that: The headset comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the headset-based snoring processing method according to any one of claims 1 to 5.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the earphone-based snoring processing method according to any one of claims 1 to 5 are implemented.

Images