Sound signal processing method, device, system, equipment and storage medium

Abstract

The present application provides a sound signal processing method, device, system, equipment and storage medium, the method comprising: obtaining the ambient sound information around the current user, the current user wearing a sound channel transmission device; performing snoring feature recognition on the ambient sound information to obtain the snoring feature information in the ambient sound information; generating a cancellation signal of the snoring feature information according to the snoring feature information and the sound channel transmission function of the sound channel transmission device; converting the cancellation signal into a cancellation sound signal, and playing the cancellation sound signal. The present application achieves targeted cancellation of snoring information in ambient sound, which can help users improve their sleeping environment.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a sound signal processing method, device, system, equipment and storage medium. Background Art

[0002] With the rapid promotion of true wireless TWS (True Wireless Stereo) headphones, more and more consumers choose to wear Bluetooth headphones for voice calls and music playback. At the same time, the functions of headphones are becoming increasingly rich, including headphones that help users sleep. This type of headset detects the ambient noise around the sleeper, and when the noise exceeds a certain threshold, it activates the active noise reduction function to achieve noise elimination and help the user sleep. This method has obvious shortcomings. First of all, the intensity of external environmental noise cannot be simply used as a trigger for noise elimination. There are many comfortable white noises that are conducive to sleep, such as the sound of wind, rain and running water in nature. These sounds can help people fall asleep peacefully; however, people are very sensitive to certain special sounds. Even if the signal strength is not high, it will disrupt normal sleep and make it difficult for people to fall asleep. A typical example is snoring, which is a continuous, continuous, and short sound. Secondly, the current active noise reduction headphones have a poor effect on snoring noise reduction. Usually, the noise filtering parameters of active noise reduction headphones are set at the factory stage. This filtering is only for common low-frequency white noise, so it cannot effectively filter out snoring. Moreover, this kind of filtering will also filter out the comfortable white noise, which is not conducive to helping sleep. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a sound signal processing method, apparatus, system, device and storage medium, which can achieve targeted cancellation of snoring information in environmental sounds, and can help users improve their sleeping environment.

[0004] A first aspect of an embodiment of the present application provides a sound signal processing method, comprising: obtaining ambient sound information around a current user, the current user wearing a sound channel transmission device; performing snoring feature recognition on the ambient sound information to obtain snoring feature information in the ambient sound information; generating a cancellation signal for the snoring feature information based on the snoring feature information and a sound channel transmission function of the sound channel transmission device; converting the cancellation signal into a cancellation sound signal, and playing the cancellation sound signal.

[0005] In one embodiment, after converting the cancellation signal into a sound signal and playing the sound signal, the method further includes: collecting a residual sound signal after the cancellation sound signal and the ambient sound signal are superimposed; extracting a residual snoring feature from the residual sound signal, and adjusting the cancellation signal of the snoring feature information according to the residual snoring feature, the snoring feature information and the vocal tract transfer function, converting the adjusted cancellation signal into an adjusted sound signal, and playing the adjusted sound signal.

[0006] In one embodiment, adjusting the cancellation signal of the snoring feature information according to the remaining snoring feature, the snoring feature information and the vocal tract transfer function includes: performing correction processing on the vocal tract transfer function according to the remaining snoring feature to obtain a corrected vocal tract transfer function; and adjusting the cancellation signal of the snoring feature information according to the remaining snoring feature and the corrected vocal tract transfer function.

[0007] In one embodiment, generating a cancellation signal for the snoring characteristic information according to the snoring characteristic information and a vocal channel transfer function of the vocal channel transmission device includes: performing noise filtering processing on the ambient sound information according to the snoring characteristic information to obtain a filtered snoring signal; performing frequency domain processing on the snoring signal to obtain a frequency spectrum feature of the snoring signal; and generating a cancellation signal for the snoring characteristic information according to the frequency spectrum feature of the snoring signal and the vocal channel transfer function.

[0008] In one embodiment, the performing noise filtering processing on the environmental sound information according to the snoring feature information to obtain a filtered snoring signal includes: comparing the snoring feature information with a snoring database to determine whether the snoring feature information is a known snoring feature existing in the snoring database; if the snoring feature information is a known snoring feature existing in the snoring database, performing noise filtering processing on the environmental sound information according to a filtering parameter corresponding to the known snoring feature to obtain the filtered snoring signal.

[0009] In one embodiment, the performing noise filtering processing on the environmental sound information according to the snoring feature information to obtain a filtered snoring signal also includes: if the snoring feature information is not a known snoring feature existing in the snoring database, performing noise filtering processing on the environmental sound information according to a preset default filtering parameter to obtain a filtered snoring signal.

[0010] In one embodiment, before performing noise filtering processing on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal, the method further includes: obtaining current breathing vibration information and movement information of the current user; judging whether the snoring characteristic information comes from the current user according to the breathing vibration information, the movement information and the snoring characteristic information; if the snoring characteristic information does not come from the current user, performing the step of performing noise filtering processing on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal.

[0011] In one embodiment, before performing noise filtering on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal, the method further includes: if the snoring characteristic information is from the current user, counting the duration of the snoring characteristic information and the frequency of respiratory pauses of the current user, and generating a snoring index of the current user based on the duration and the frequency of respiratory pauses.

[0012] According to a second aspect of an embodiment of the present application, there is provided a sound signal processing device, comprising: an acquisition module, configured to acquire environmental sound information around a current user, the current user being equipped with a sound channel transmission device; an identification module, configured to perform snoring feature identification on the environmental sound information to obtain snoring feature information in the environmental sound information; a generation module, configured to generate a cancellation signal of the snoring feature information according to the snoring feature information and a sound channel transmission function of the sound channel transmission device; and a playback module, configured to convert the cancellation signal into a cancellation sound signal and play the cancellation sound signal.

[0013] In one embodiment, it also includes: a collection module, which is used to collect the remaining sound signal after the cancellation signal is converted into a sound signal and the sound signal is played; an adjustment module, which is used to extract the remaining snoring characteristics in the remaining sound signal, and adjust the cancellation signal of the snoring characteristic information according to the remaining snoring characteristics, the snoring characteristic information and the vocal tract transfer function, convert the adjusted cancellation signal into an adjusted sound signal, and play the adjusted sound signal.

[0014] In one embodiment, adjusting the cancellation signal of the snoring feature information according to the remaining snoring feature, the snoring feature information and the vocal tract transfer function includes: performing correction processing on the vocal tract transfer function according to the remaining snoring feature to obtain a corrected vocal tract transfer function; and adjusting the cancellation signal of the snoring feature information according to the remaining snoring feature and the corrected vocal tract transfer function.

[0015] In one embodiment, the generating module is used to: perform noise filtering on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal; perform frequency domain processing on the snoring signal to obtain a spectrum feature of the snoring signal; generate a cancellation signal of the snoring characteristic information according to the spectrum feature of the snoring signal and the vocal tract transfer function

[0016] In one embodiment, the performing noise filtering processing on the environmental sound information according to the snoring feature information to obtain a filtered snoring signal includes: comparing the snoring feature information with a snoring database to determine whether the snoring feature information is a known snoring feature existing in the snoring database; if the snoring feature information is a known snoring feature existing in the snoring database, performing noise filtering processing on the environmental sound information according to a filtering parameter corresponding to the known snoring feature to obtain the filtered snoring signal.

[0017] In one embodiment, the performing noise filtering processing on the environmental sound information according to the snoring feature information to obtain a filtered snoring signal also includes: if the snoring feature information is not a known snoring feature existing in the snoring database, performing noise filtering processing on the environmental sound information according to a preset default filtering parameter to obtain a filtered snoring signal.

[0018] In one embodiment, a detection module is further included, which is used to: before performing noise filtering processing on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal, obtain the current breathing vibration information and movement information of the current user; determine whether the snoring characteristic information comes from the current user according to the breathing vibration information, the movement information and the snoring characteristic information; if the snoring characteristic information does not come from the current user, perform the step of performing noise filtering processing on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal.

[0019] In one embodiment, the detection module is further used to, before performing noise filtering on the ambient sound information according to the snoring characteristic information to obtain a filtered snoring signal, if the snoring characteristic information is from the current user, count the duration of the snoring characteristic information and the frequency of respiratory pauses of the current user, and generate a snoring index of the current user based on the duration and the frequency of respiratory pauses.

[0020] According to a third aspect of an embodiment of the present application, a sound signal processing system is provided, comprising: a first audio collector, for collecting environmental sound information; a snoring identifier, connected to the first audio collector, for performing snoring feature recognition on the environmental sound information to obtain snoring feature information in the environmental sound information; an acoustic channel, for transmitting the environmental sound information; a signal generator, connected to the snoring identifier, for generating a cancellation signal of the snoring feature information according to the snoring feature information and an acoustic channel transfer function of the acoustic channel; and a speaker, connected to the signal generator, for converting the cancellation signal into a cancellation sound signal and playing the cancellation sound signal.

[0021] In one embodiment, it also includes: a second audio collector, used to collect the remaining sound signal after the cancellation sound signal and the environmental sound signal are superimposed; a feedback module, used to extract the remaining snoring features in the remaining sound signal; and the snoring identifier is also used to adjust the parameters of the signal generator according to the remaining snoring features, the snoring feature information and the vocal tract transfer function; the signal generator is also used to adjust the cancellation signal of the snoring feature information according to the adjusted parameters; the speaker is also used to convert the adjusted cancellation signal into an adjusted sound signal and play the adjusted sound signal.

[0022] A fourth aspect of an embodiment of the present application provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the method of the first aspect of an embodiment of the present application and any one of its embodiments.

[0023] The fifth aspect of the embodiments of the present application provides a non-transitory electronic device readable storage medium, including: a program, which, when run by an electronic device, enables the electronic device to execute the method of the first aspect of the embodiments of the present application and any one of its embodiments.

[0024] The sound signal processing method, apparatus, system, device and storage medium provided in the present application obtain the snoring characteristic information in the environment by performing snoring characteristic recognition on the ambient sound information around the current user, and then generate a cancellation signal of the snoring characteristic information based on the snoring characteristic information in the environment and the channel transmission function of the channel transmission device worn by the current user, and then convert the cancellation signal into sound and play it out. In this way, when the cancellation signal and the ambient sound information enter the ears of the current user at the same time, the cancellation signal is superimposed on the ambient sound information, thereby canceling out the snoring in the ambient sound information. The sound actually heard by the current user is the ambient sound after the snoring is canceled out, thereby achieving targeted cancellation of the snoring information in the ambient sound, and can help the user improve the sleeping environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0027] Figure 2 A schematic diagram of the structure of a sound signal processing system according to an embodiment of the present application;

[0028] Figure 3 A flowchart of a sound signal processing method according to an embodiment of the present application;

[0029] Figure 4 A flowchart of a sound signal processing method according to an embodiment of the present application;

[0030] Figure 5 A schematic diagram of the structure of a sound signal processing device according to an embodiment of the present application.

[0031] Reference numerals:

[0032] 200 - sound information processing system, 21 - first audio collector, 22 - snoring identifier, 23 - acoustic channel, 24 - signal generator, 25 - speaker, 26 - second audio collector, 27 - feedback module, 28 - accelerometer, 29 - vibration sensor. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the description of the present application, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0034] like Figure 1 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 1 A processor is taken as an example. The processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11, and the instructions are executed by the processor 11, so that the electronic device 1 can execute all or part of the process of the method in the following embodiment, so as to cancel the snoring information in the environmental sound in a targeted manner, and can help the user improve the sleeping environment.

[0035] In one embodiment, the electronic device 1 may be a smart headset, a mobile phone, a tablet computer, a laptop computer, a desktop computer, or a large computing system composed of multiple computers.

[0036] Please see Figure 2 , which is a sound signal processing system 200 of an embodiment of the present application, comprising: a first audio collector 21, a snoring identifier 22, an acoustic channel 23, a signal generator 24 and a speaker 25, wherein:

[0037] The first audio collector 21 is used to collect environmental sound information. The first audio collector 21 can be a microphone or other sound receiving device. For example, the first audio collector 21 can be implemented by a feedforward microphone, which collects external environmental sounds, and the collected signals are simultaneously sent to the snoring recognizer 22 and the signal generator 24.

[0038] The snoring sound recognizer 22 is connected to the first audio collector 21 and is used to perform snoring sound feature recognition on the environmental sound information to obtain snoring sound feature information in the environmental sound information. The snoring sound recognizer 22 can be a neural network unit with deep learning capability, which can distinguish snoring, normal breathing and other noises in the environmental sound by learning the snoring sound in the current user's sleeping environment.

[0039] The acoustic channel 23 is used to transmit the ambient sound information to the ears of the current user.

[0040] The signal generator 24 is connected to the snoring identifier 22 and is used to generate a cancellation signal of the snoring characteristic information according to the snoring characteristic information and the vocal channel transfer function of the acoustic channel 23 .

[0041] In one embodiment, the signal generator 24 may include: an adaptive filter H(t), a frequency domain processor FFT and a generation module, wherein the snoring identifier 22 provides filtering parameters for the adaptive filter by extracting the spectrum information of the snoring. The adaptive filter H(t) performs noise reduction processing on the ambient sound signal input from the feedforward microphone to remove the remaining noise except the snoring. The frequency domain processor FFT calculates the amplitude and phase information of each frequency point of the snoring. The generation module outputs a cancellation signal waveform for canceling the snoring according to the amplitude and phase information of each frequency point of the snoring and the channel transfer function G(f) of the acoustic channel 23.

[0042] The speaker 25 is connected to the signal generator 24 and is used to convert the cancellation signal into a cancellation sound signal and play the cancellation sound signal. That is, the cancellation signal is finally sent out through the speaker 25 and superimposed with the snoring sound in the environmental sound information transmitted to the ear through the acoustic channel 23, so as to achieve the effect of eliminating the snoring sound.

[0043] In one embodiment, the system may further include: a second audio collector 26, which is used to collect the residual sound signal after the cancellation sound signal and the environmental sound signal are superimposed, so the second audio collector 26 can be a feedback microphone. A feedback module 27 is used to extract the residual snoring features in the residual sound signal. And, the snoring identifier 22 is also used to adjust the parameters of the signal generator 24 according to the residual snoring features, snoring feature information and the vocal tract transfer function. The signal generator 24 is also used to adjust the cancellation signal of the snoring feature information according to the adjusted parameters. The speaker 25 is also used to convert the adjusted cancellation signal into an adjusted sound signal and play the adjusted sound signal.

[0044] That is, the feedback microphone collects the residual sound signal after the snoring in the environment is cancelled and sends it to the feedback module 27. At the same time, the generation module for generating the cancellation signal provides the current cancellation signal amplitude. The feedback module 27 calculates the residual amplitude of each frequency point of the snoring, provides the amplitude adjustment information to the generation module, and dynamically adjusts the cancellation amplitude of each frequency point. In addition, the feedback module 27 transmits the calculated frequency adjustment information to the snoring identifier 22 for updating the snoring spectrum information, thereby optimizing the parameters of the adaptive filter.

[0045] In one embodiment, the sound signal processing system 200 may also be integrated with an accelerometer 28 and a vibration sensor 29 to detect vibrations caused by the user's own movement and breathing, and further identify whether the current user is snoring.

[0046] The above-mentioned sound signal processing system 200 can be fully integrated in a single earphone device, and the current user can wear the earphone device to eliminate snoring. The above-mentioned sound signal processing system 200 can also be an assembled system composed of related component devices.

[0047] In one embodiment, some functions of the snoring identifier 22 can be completed in a mobile phone or a cloud server. For example, for a mobile phone that supports deep neural network learning, the earphone can transmit the sound signal collected during sleep to the mobile phone, and the neural network processor integrated in the mobile phone completes the analysis and calculation, realizes the extraction and memory of the snoring characteristics, and transmits the characteristic value to the earphone for storage for subsequent snoring cancellation. It is also possible to use the powerful computing power of the cloud server to store the collected sleep audio signal as a file through the mobile phone, send it to the cloud server for processing, and obtain the characteristic value required for snoring recognition.

[0048] For the user's own snoring, the headset monitors during sleep to obtain the duration of snoring, the number of apnea and the length of the pause, and calculates the snoring index. This sleep information can be displayed to the user through the mobile phone, and can also be sent to the network cloud platform for health monitoring. The cloud platform tracks the development trend of snoring. If severe symptoms or deterioration trends occur, the user will be warned through the mobile phone display or headset prompts to notify him to seek medical treatment.

[0049] Please see Figure 3 , which is a sound signal processing method of an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown in the figure may be used to perform the operation, or the electronic device 1 may be used to perform the operation. Figure 2 The sound signal processing system 200 shown in the figure can be used to cancel the snoring information in the environmental sound in a targeted manner, which can help the user improve the sleeping environment. The method includes the following steps:

[0050] Step 301: Acquire the ambient sound information around the current user, the current user is wearing a sound channel transmission device.

[0051] In this step, the sound channel transmission device can be the above-mentioned electronic device 1 or Figure 2 The sound signal processing system 200 shown in the figure is implemented, for example, the sound channel transmission device can be an earphone device integrated with the sound signal processing system 200. The current user is the user wearing the earphone. When the user wears the earphone device to sleep, the ambient sound information around the current user can be obtained through the earphone device.

[0052] Step 302: Perform snoring feature recognition on the environmental sound information to obtain snoring feature information in the environmental sound information.

[0053] In this step, the snoring feature recognition can adopt the common neural network recognition method based on deep learning. Before snoring recognition, the earphone device needs to go through the snoring learning stage. In this stage, the earphone learns and trains a certain snoring sound that occurs repeatedly during the current user's sleep. The learning process usually lasts for 3 to 5 days to complete the analysis and extraction of the snoring characteristics. When each person snores, the vibration form of the pharyngeal tissue is different, the source of the sound is also different, and the physiological structure is also different, so the snoring has different characteristics from others. After this stage, the earphone has the ability to recognize specific snoring sounds.

[0054] In one embodiment, the snoring characteristic information may include: peak frequency, center frequency (the area under the frequency domain curve on both sides of the frequency point is equal), fundamental frequency, harmonic frequency, formant (the frequency and amplitude of the first formant F1, the first formant F2 and the first formant F3) and crest factor ratio (the ratio of the sound amplitude peak to the sound amplitude root mean square) and other parameters.

[0055] Step 303: Generate a cancellation signal of the snoring characteristic information according to the snoring characteristic information and the vocal channel transfer function of the vocal channel transmission device.

[0056] In this step, the channel transmission function G(f) of the channel transmission device represents the signal attenuation frequency response curve of the external sound entering the user's ear canal through the earphone. This transmission function is obtained through the production test calibration process when the earphone leaves the factory. Since the ambient sound information enters the ear of the current user through the acoustic channel 23 of the earphone, the cancellation signal is related to the channel transmission function of the earphone. Considering the snoring feature information in the ambient sound information and the channel transmission function of the channel transmission device at the same time, the cancellation signal of the snoring feature information can be made more accurate.

[0057] Step 304: convert the cancellation signal into a cancellation sound signal, and play the cancellation sound signal.

[0058] In this step, the cancellation signal is a signal that is inversely proportional to the snoring characteristic information. The ambient sound information enters the human ear through the acoustic channel 23 of the earphone, and the cancellation signal is converted into a sound signal and played out. In this way, the cancellation sound signal and the ambient sound information are superimposed, thereby canceling out the snoring in the ambient sound information. The sound actually heard by the current user is the ambient sound after the snoring has been canceled out, thereby achieving targeted cancellation of the snoring information in the ambient sound, which can help the user improve the sleeping environment.

[0059] The above-mentioned sound signal processing method obtains the snoring characteristic information in the environment by performing snoring feature recognition on the ambient sound information around the current user, and then generates a cancellation signal of the snoring characteristic information based on the snoring characteristic information in the environment and the channel transmission function of the channel transmission device worn by the current user, and then converts the cancellation signal into sound and plays it out. In this way, when the cancellation signal and the ambient sound information enter the ears of the current user at the same time, the cancellation signal is superimposed on the ambient sound information, thereby canceling out the snoring in the ambient sound information. The sound actually heard by the current user is the ambient sound after the snoring is canceled out, thereby achieving targeted cancellation of the snoring information in the ambient sound, and can help the user improve the sleeping environment.

[0060] Please see Figure 4 , which is a sound signal processing method of an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown in the figure may be used to perform the operation, or the electronic device 1 may be used to perform the operation. Figure 2 The sound signal processing system 200 shown in the figure is used to cancel the snoring information in the environmental sound in a targeted manner, which can help the user improve the sleeping environment. Taking the headphone device integrated with the above-mentioned sound signal processing system 200 as an example, the method includes the following steps:

[0061] Step 401: Acquire the ambient sound information around the current user, the current user is wearing a sound channel transmission device. For details, refer to the description of step 301 in the above embodiment.

[0062] In one embodiment, before step 402, it may also include: the current user wears an earphone device integrated with the above-mentioned sound signal processing system 200, and during the current user's sleep, the earphone detects the external environment volume through the first audio collector 21, and when the volume is greater than the monitoring threshold Sth, the snoring recognition process is triggered. The snoring identifier 22 analyzes the environmental sound collected by the first audio collector 21, distinguishes the amplitude difference between the sound segment and the silent segment, separates the snoring and non-snoring segments, and determines whether there is snoring in the environmental sound information. If there is snoring in the environmental sound information, execute step 402, otherwise it means that there is no snoring of other users in the sleeping environment where the current user is located, and no snoring cancellation processing is required. It can return to step 401 and continue to detect the external environment volume through the first audio collector 21.

[0063] Step 402: Perform snoring feature recognition on the environmental sound information to obtain snoring feature information in the environmental sound information. For details, refer to the description of step 302 in the above embodiment.

[0064] Step 403: Obtain the current breathing vibration information and movement information of the current user.

[0065] In this step, when there is snoring in the environmental sound information, further, the breathing vibration information and motion information of the current user collected by the accelerometer 28 and the vibration sensor 29 can be used. The accelerometer 28 is used to collect the motion signal M(t) generated when the current user wearing the headset turns over during sleep, and the vibration sensor 29 collects the vibration signal S(t) on the surface of the human body. The vibration signal S(t) includes the vibration signal A(t) when the human body turns over and the vibration signal V(t) generated by sleep breathing. Let A(t) = k*M(t), k is a proportional constant, k is based on the actual scene, and t is time.

[0066] Step 404: Based on the respiratory vibration information, the motion information and the snoring characteristic information, determine whether the snoring characteristic information comes from the current user. If yes, proceed to step 405; otherwise, proceed to step 406.

[0067] In this step, the measurement results of the accelerometer 28 and the vibration sensor 29 can be used to obtain the current user's own breathing vibration signal V(t) as follows:

[0068] V(t)=S(t)-k*M(t)

[0069] The human body's own breathing vibration signal V(t) and the snoring signal W(t) in the environmental sound information identified by the snoring identifier 22 are correlated. If the correlation between the two reaches a certain correlation threshold, it is determined that the snoring sound is generated by the current user, and the process goes to step 405. If the correlation between the two does not reach the above correlation threshold, it means that the snoring sound is not generated by the current user, and the process goes to step 406.

[0070] Step 405: Count the duration of the snoring characteristic information and the frequency of the current user's breathing pauses, and generate a snoring index for the current user based on the duration and the frequency of the breathing pauses.

[0071] In this step, if the snoring feature information comes from the current user, there is no need to offset the snoring. Instead, the number of respiratory obstructions and the duration of each obstruction of the current user can be counted, and respiratory behavior information such as the frequency of respiratory pauses in the sleep state can be obtained. This respiratory behavior information can be used to calculate the snoring index of the current user to help determine whether the current user is simply snoring or has obstructive sleep apnea hypopnea syndrome (OSAHS). The user's snoring data can be collected without the current user having to go to the hospital's snoring intensive care unit to use special equipment. This is used to track the development trend of snoring and provide support data for the treatment of snoring.

[0072] In actual scenarios, there will be noise in the ambient sound information. In order to obtain a more accurate cancellation signal, the snoring signal to be cancelled must be relatively clean. Therefore, it is necessary to filter the ambient sound information according to the snoring feature information to obtain the filtered snoring signal. Therefore, the method may also include:

[0073] Step 406: Compare the snoring feature information with the snoring database to determine whether the snoring feature information is a known snoring feature existing in the snoring database. If yes, proceed to step 407; otherwise, proceed to step 408.

[0074] In this step, the snoring database stores one or more known snoring features that often appear in the sleeping environment of the current user. The known snoring features can be obtained through the earphone during the snoring learning stage. For example, the known snoring features can be the snoring features of the current user's family. If the snoring feature information is a known snoring feature existing in the snoring database, the process proceeds to step 407, otherwise, the process proceeds to step 408.

[0075] Step 407 : noise filtering is performed on the environmental sound information according to the filtering parameters corresponding to the known snoring characteristics to obtain a filtered snoring signal, and then the process proceeds to step 409 .

[0076] In this step, if the snoring feature information is a known snoring feature existing in the snoring database, the frequency spectrum of the known snoring feature can be used to obtain the filtering parameters of the adaptive filter. The filtering parameters may include: filter center frequency, filter envelope, etc. Based on the above filtering parameters, the ambient sound information is subjected to noise filtering processing to obtain a filtered snoring signal.

[0077] Step 408 : Perform noise filtering on the environmental sound information according to the preset default filtering parameters to obtain a filtered snoring signal, and then proceed to step 409 .

[0078] In this step, if the snoring feature information is not a known snoring feature existing in the snoring database, the default filtering parameters may be used to configure an adaptive filter to perform noise reduction filtering on the ambient sound information to obtain a filtered snoring signal with a high signal-to-noise ratio.

[0079] In one embodiment, the algorithm implementation process of performing noise reduction filtering on the ambient sound information may be as follows:

[0080] The adaptive filter H(t) is a set of bandpass filters, which filters multiple main peaks of the snoring spectrum. The snoring peak frequency and filtering envelope are provided by the snoring identifier 22, that is, the center frequency (f i ) and the filter envelope h i (t). Assume that the time domain signal sequence of the ambient sound information collected by the first audio collector 21 of the headset is expressed as: N ={x 0 ,x 1 ,...x N}, N is the number of sampling points (N is a positive integer), then the adaptive filter filtering process is expressed as:

[0081]

[0082] In the above formula, τ is the discrete convolution delay parameter, h(τ) is the discrete function of the filter impulse response, y(i) is the filter output of the adaptive filter for the i-th snoring peak, and M is the total number of filters, which is the same as the number of snoring spectrum energy peaks that need to be filtered. j is the order of the jth filter, K j .

[0083] Step 409: Perform frequency domain processing on the snoring signal to obtain frequency spectrum characteristics of the snoring signal.

[0084] In this step, the snoring signal after filtering and noise reduction is processed in the frequency domain. For example, the snoring signal after filtering and noise reduction can be processed by FFT (Fast Fourier Transform) to obtain the spectrum characteristics of the snoring signal. The spectrum characteristics can be the frequency domain amplitude information and phase information of the main energy in the snoring signal.

[0085] In one embodiment, taking the filtering process in step 409 as an example, the frequency domain processing stage performs FFT transformation on the snoring signal by the following formula (2) to obtain the amplitude and phase of the spectrum energy peak of the snoring signal:

[0086]

[0087] Where M is the number of snoring spectrum energy peaks that need to be filtered, f i is the frequency of the ith snoring peak, A i is the amplitude of the ith snoring peak, is the phase of the i-th snoring peak, and e is a natural constant.

[0088] Step 410: Generate a cancellation signal of snoring characteristic information according to the frequency spectrum characteristics and vocal tract transfer function of the snoring signal.

[0089] In this step, the transmission function G(f) of the transmissive ear acoustic channel 23 represents the signal attenuation frequency response curve of the external environment sound entering the user's ear canal through the earphone. This transmission function is obtained through the production test and calibration process when the earphone leaves the factory. The generation of the snoring cancellation signal can be obtained by combining the known channel transfer function G(f) of the earphone with the spectral characteristics of the snoring signal. Therefore, based on the above formula (2), the cancellation signal y(t) of the snoring characteristic information that changes with time t can be expressed by the following formula:

[0090]

[0091] Step 411: Convert the cancellation signal into a cancellation sound signal, and play the cancellation sound signal.

[0092] In this step, the cancellation signal is a signal that is inversely proportional to the snoring signal. The cancellation signal can be converted into a cancellation sound signal through the speaker 25 and played out. The sound cancellation signal is superimposed on the ambient sound information transmitted into the ear, which can specifically eliminate most of the snoring sounds, thereby ensuring the user's sleep quality.

[0093] Step 412: Collect the remaining sound signal after the cancellation sound signal and the ambient sound signal are superimposed.

[0094] In this step, in order to adapt to different environments and adaptively improve the accuracy of the cancellation signal, the residual sound signal after cancellation can be analyzed, and the analysis result is fed back to the signal generator 24 as a feedback signal to dynamically adjust the cancellation signal. The residual sound signal after the cancellation sound signal and the ambient sound signal are superimposed can be collected by the second audio collector 26 of the headset.

[0095] Step 413: extract the remaining snoring features from the remaining sound signal, and adjust the cancellation signal of the snoring feature information according to the remaining snoring features, the snoring feature information and the vocal tract transfer function, convert the adjusted cancellation signal into an adjusted sound signal, and play the adjusted sound signal.

[0096] In this step, the remaining sound signal is sent to the feedback module 27 for processing, and the amplitude value of each frequency point of the snoring sound in the remaining sound signal (i.e., the remaining snoring sound feature) can be obtained, which is provided to the signal generator 24 and can be used to correct the transfer function G(f) of the acoustic channel 23 and the amplitude of each frequency point of the cancellation signal. If the feedback module 27 identifies that there are snoring sound energy peaks that are not cancelled in the remaining sound signal, or some snoring sound frequency points do not need to be cancelled, the frequency point adjustment information is generated and sent to the snoring sound identifier 22 to update the filter parameters, thereby improving the final snoring sound cancellation effect.

[0097] In one embodiment, step 413 may specifically include: modifying the vocal tract transfer function according to the remaining snoring characteristics to obtain a modified vocal tract transfer function, and adjusting the cancellation signal of the snoring characteristic information according to the remaining snoring characteristics and the modified vocal tract transfer function.

[0098] In the above steps, the residual sound signal dY(t) after cancellation collected by the second audio collector 26 is sent to the feedback module 27 for processing. Considering that the channel transfer function G(f) is obtained by calibration of the earphone product at the factory stage, and the actual channel transfer function G′(f) and the factory G(f) may deviate due to differences in wearing and temperature changes during the use of the earphone, the residual amplitude and residual phase of each energy peak of snoring in the residual sound signal dY(t) can be calculated. For example, for the i-th snoring energy peak, its frequency is f i , the frequency f can be calculated according to the following formula i The corresponding amplitude phase function within a period T

[0099]

[0100] Among them, ΔA i represents the residual amplitude of the i-th snoring energy peak in the residual sound signal dY(t), represents the residual phase of the i-th snoring energy peak in the residual sound signal dY(t). Thus, ΔAi yes The maximum value of yes The phase value corresponding to the maximum value is taken. The above method can be used to obtain the residual amplitude set of all snoring energy peaks in the residual sound signal dY(t) ΔA = {ΔA 0 ,ΔA 1 ,…,ΔA M-1} and the remaining phase set

[0101] ΔA and The generation module provided to the signal generator 24 can modify the channel transfer function, and the modified result can be expressed as:

[0102]

[0103] Among them, G′(f i ) is the modified vocal tract transfer function, which can be used for subsequent snoring cancellation signal generation.

[0104] Furthermore, the feedback module 27 can obtain a peak set with larger energy except for snoring on the spectrum by analyzing the spectrum of the residual sound signal dY(t), which is called a non-snoring energy peak set and can be expressed as follows:

[0105] Fext = {(fe 0 ,a 0 ),(fe 1 ,a 1 ),…,(fe n ,a n )}

[0106] Among them, n represents the frequency of the nth non-snoring energy peak in the residual sound signal dY(t), a n is the amplitude of the nth non-snoring energy peak in the residual sound signal dY(t).

[0107] Furthermore, according to the modified channel transfer function G′(f i ) and the residual amplitude set ΔA={ΔA 0 ,ΔA 1 ,…,ΔA M-1}, the amplitude set GA of each peak value of external snoring sound transmitted through the headphone acoustic channel 23 is obtained, which is expressed as:

[0108] GA={G′(f 0 )*A 0 ,G′(f 1 )*A 1 ,…,G′(f N )*AN}

[0109] The non-snoring energy peak set Fext and GA are sent to the snoring identifier 22 as frequency adjustment information. The snoring identifier 22 determines whether there are snoring frequencies that need to be offset in the frequency set Fext. If so, these snoring frequencies are updated to the filter parameters. It is also possible to determine whether there are small snoring signals with amplitudes less than the ignore threshold in the snoring peak frequency points currently offset according to the amplitudes of each signal in the set GA. These small snoring signals can be ignored and do not need to be offset. The filter parameters of these small snoring signals can be deleted from the original filter parameters, and the filter parameters can be updated, thereby improving the actual snoring offset effect.

[0110] The above-mentioned sound signal processing method is based on a neural network algorithm to train the snoring sounds that occur repeatedly during the user's sleep, extract the acoustic feature values ​​of the snoring sounds, and enable it to have the ability to recognize the snoring sounds. Later, when the snoring sounds are recognized during the user's sleep, adaptive filtering and noise reduction are started, and frequency domain and time domain signal processing are performed to generate a snoring cancellation signal, which is superimposed and eliminated with the snoring sounds that enter the user's ear canal. In addition, in combination with the acceleration sensor and the vibration sensor 29, it is confirmed whether the snoring sounds are emitted by the user. If so, the snoring sounds are counted, the number of respiratory pauses and the length of the pauses during the snoring sounds are obtained, and the snoring index is calculated. The recorded data is used for long-term medical observation to provide data support for treatment.

[0111] Please see Figure 5 , which is a sound signal processing device 500 of an embodiment of the present application, the device can be applied to Figure 1 The electronic device 1 shown may be applied to Figure 2 The sound signal processing system 200 shown can help users improve their sleeping environment by specifically canceling out snoring information in environmental sounds. The device includes: an acquisition module 501, an identification module 502, a generation module 503 and a playback module 504. The principle relationship between each module is as follows:

[0112] The acquisition module 501 is used to acquire the ambient sound information around the current user, and the current user is wearing a sound channel transmission device. The recognition module 502 is used to perform snoring feature recognition on the ambient sound information to obtain the snoring feature information in the ambient sound information. The generation module 503 is used to generate a cancellation signal of the snoring feature information according to the snoring feature information and the sound channel transmission function of the sound channel transmission device. The playback module 504 is used to convert the cancellation signal into a cancellation sound signal and play the cancellation sound signal.

[0113] In one embodiment, the method further includes: a collection module 505, which is used to collect the residual sound signal after the cancellation signal is converted into a sound signal and the sound signal is played. An adjustment module 506 is used to extract the residual snoring characteristics in the residual sound signal, and adjust the cancellation signal of the snoring characteristic information according to the residual snoring characteristics, the snoring characteristic information and the vocal tract transfer function, convert the adjusted cancellation signal into an adjusted sound signal, and play the adjusted sound signal.

[0114] In one embodiment, adjusting the cancellation signal of the snoring feature information according to the remaining snoring feature, the snoring feature information and the vocal tract transfer function includes: performing correction processing on the vocal tract transfer function according to the remaining snoring feature to obtain a corrected vocal tract transfer function. Adjusting the cancellation signal of the snoring feature information according to the remaining snoring feature and the corrected vocal tract transfer function.

[0115] In one embodiment, the generating module 503 is used to: perform noise filtering on the ambient sound information according to the snoring characteristic information to obtain a filtered snoring signal; perform frequency domain processing on the snoring signal to obtain the spectrum characteristics of the snoring signal; generate a cancellation signal of the snoring characteristic information according to the spectrum characteristics of the snoring signal and the vocal tract transfer function;

[0116] In one embodiment, noise filtering is performed on environmental sound information according to snoring feature information to obtain a filtered snoring signal, including: comparing the snoring feature information with a snoring database to determine whether the snoring feature information is a known snoring feature existing in the snoring database. If the snoring feature information is a known snoring feature existing in the snoring database, noise filtering is performed on the environmental sound information according to a filtering parameter corresponding to the known snoring feature to obtain a filtered snoring signal.

[0117] In one embodiment, noise filtering is performed on environmental sound information according to snoring feature information to obtain a filtered snoring signal, and the method further includes: if the snoring feature information is not a known snoring feature existing in a snoring database, noise filtering is performed on the environmental sound information according to a preset default filtering parameter to obtain a filtered snoring signal.

[0118] In one embodiment, a detection module 507 is further included, which is used to: before performing noise filtering on the environmental sound information according to the snoring characteristic information to obtain the filtered snoring signal, obtain the current breathing vibration information and movement information of the current user. According to the breathing vibration information, movement information and snoring characteristic information, it is determined whether the snoring characteristic information comes from the current user. If the snoring characteristic information does not come from the current user, the step of performing noise filtering on the environmental sound information according to the snoring characteristic information to obtain the filtered snoring signal is executed.

[0119] In one embodiment, the detection module 507 is further used to perform noise filtering on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal. If the snoring characteristic information is from the current user, the duration of the snoring characteristic information and the frequency of respiratory pauses of the current user are counted, and a snoring index of the current user is generated based on the duration and the frequency of respiratory pauses.

[0120] For a detailed description of the above-mentioned sound signal processing device 500, please refer to the description of the relevant method steps in the above-mentioned embodiment.

[0121] The embodiment of the present invention also provides a non-transitory electronic device readable storage medium, including: a program, when it is run on an electronic device, the electronic device can execute all or part of the process of the method in the above embodiment. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc. The storage medium can also include a combination of the above types of memory.

[0122] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A sound signal processing method, applied to a sound signal processing system, It is characterized in that include: Acquiring ambient sound information around a current user, wherein the current user wears a sound channel transmission device, wherein the sound channel transmission device is a headphone device integrated with a sound signal processing system; the current user wears an accelerometer and a vibration sensor, wherein the accelerometer and the vibration sensor are used to collect breathing vibration information and motion information of the current user; Performing snoring feature recognition on the environmental sound information to obtain snoring feature information in the environmental sound information; Acquire the current breathing vibration information and movement information of the current user; determining, according to the respiratory vibration information, the movement information and the snoring characteristic information, whether the snoring characteristic information comes from the current user; If the snoring characteristic information does not come from the current user, generating a cancellation signal for the snoring characteristic information according to the snoring characteristic information and a vocal channel transmission function of the vocal channel transmission device; The cancellation signal is converted into a cancellation sound signal, and the cancellation sound signal is played.

2. The method according to claim 1, It is characterized in that After converting the cancellation signal into a cancellation sound signal and playing the cancellation sound signal, the method further includes: Collecting a residual sound signal after the cancellation sound signal and the ambient sound signal are superimposed; Extracting the residual snoring features from the residual sound signal, adjusting the cancellation signal of the snoring feature information according to the residual snoring features, the snoring feature information and the vocal tract transfer function, converting the adjusted cancellation signal into an adjusted sound signal, and playing the adjusted sound signal.

3. The method according to claim 2, It is characterized in that The step of adjusting the cancellation signal of the snoring feature information according to the remaining snoring feature, the snoring feature information and the vocal tract transfer function comprises: According to the residual snoring sound characteristics, the vocal tract transfer function is modified to obtain a modified vocal tract transfer function; The cancellation signal of the snoring feature information is adjusted according to the remaining snoring feature and the modified vocal tract transfer function.

4. The method according to claim 1, It is characterized in that The step of generating a cancellation signal of the snoring characteristic information according to the snoring characteristic information and a vocal channel transfer function of the vocal channel transmission device comprises: Performing noise filtering on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal; Performing frequency domain processing on the snoring signal to obtain frequency spectrum characteristics of the snoring signal; A cancellation signal of the snoring characteristic information is generated according to the frequency spectrum characteristics of the snoring signal and the vocal tract transfer function.

5. The method according to claim 4, It is characterized in that The performing noise filtering on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal includes: Comparing the snoring feature information with a snoring database to determine whether the snoring feature information is a known snoring feature existing in the snoring database; If the snoring feature information is a known snoring feature existing in the snoring database, noise filtering is performed on the environmental sound information according to filtering parameters corresponding to the known snoring feature to obtain the filtered snoring signal.

6. The method according to claim 5, It is characterized in that The performing noise filtering on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal also includes: If the snoring feature information is not a known snoring feature existing in the snoring database, noise filtering is performed on the environmental sound information according to preset default filtering parameters to obtain a filtered snoring signal.

7. The method according to claim 1, It is characterized in that Before generating a cancellation signal of the snoring characteristic information according to the snoring characteristic information and the vocal channel transfer function of the vocal channel transmission device, the method further includes: If the snoring characteristic information is from the current user, the duration of the snoring characteristic information and the frequency of respiratory pauses of the current user are counted, and a snoring index of the current user is generated based on the duration and the frequency of respiratory pauses.

8. A sound signal processing device, applied to a sound signal processing system, It is characterized in that include: An acquisition module is used to acquire the ambient sound information around the current user, the current user is wearing a sound channel transmission device, the sound channel transmission device is a headphone device integrated with a sound signal processing system; the current user is wearing an accelerometer and a vibration sensor, the accelerometer and the vibration sensor are used to collect the breathing vibration information and movement information of the current user; an identification module, configured to perform snoring feature identification on the environmental sound information to obtain snoring feature information in the environmental sound information; A detection module, used to obtain the current breathing vibration information and movement information of the current user; and determine whether the snoring characteristic information comes from the current user according to the breathing vibration information, the movement information and the snoring characteristic information; a generating module, configured to generate a cancellation signal of the snoring characteristic information according to the snoring characteristic information and a vocal channel transmission function of the vocal channel transmission device if the snoring characteristic information does not come from the current user; The playing module is used to convert the cancellation signal into a cancellation sound signal and play the cancellation sound signal.

9. The device according to claim 8, It is characterized in that Also includes: A collection module, configured to collect a residual sound signal after the cancellation signal is converted into a cancellation sound signal and the cancellation sound signal is played; The adjustment module is used to extract the remaining snoring features in the remaining sound signal, and adjust the cancellation signal of the snoring feature information according to the remaining snoring features, the snoring feature information and the vocal tract transfer function, convert the adjusted cancellation signal into an adjusted sound signal, and play the adjusted sound signal.

10. The device according to claim 9, It is characterized in that The step of adjusting the cancellation signal of the snoring feature information according to the remaining snoring feature, the snoring feature information and the vocal tract transfer function comprises: According to the residual snoring sound characteristics, the vocal tract transfer function is modified to obtain a modified vocal tract transfer function; The cancellation signal of the snoring feature information is adjusted according to the remaining snoring feature and the modified vocal tract transfer function.

11. The device according to claim 8, It is characterized in that The generation module is used for: Performing noise filtering on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal; Performing frequency domain processing on the snoring signal to obtain frequency spectrum characteristics of the snoring signal; A cancellation signal of the snoring characteristic information is generated according to the frequency spectrum characteristics of the snoring signal and the vocal tract transfer function.

12. The device according to claim 11, It is characterized in that The performing noise filtering on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal includes: Comparing the snoring feature information with a snoring database to determine whether the snoring feature information is a known snoring feature existing in the snoring database; If the snoring feature information is a known snoring feature existing in the snoring database, noise filtering is performed on the environmental sound information according to filtering parameters corresponding to the known snoring feature to obtain the filtered snoring signal.

13. The device according to claim 12, It is characterized in that The performing noise filtering on the environmental sound information according to the snoring characteristic information to obtain a filtered snoring signal also includes: If the snoring feature information is not a known snoring feature existing in the snoring database, noise filtering is performed on the environmental sound information according to preset default filtering parameters to obtain a filtered snoring signal.

14. The device according to claim 8, It is characterized in that The detection module is further configured to count the duration of the snoring characteristic information and the frequency of respiratory pauses of the current user if the snoring characteristic information comes from the current user, and generate a snoring index of the current user based on the duration and the frequency of respiratory pauses.

15. A sound signal processing system, It is characterized in that The sound signal processing system is integrated into the headphone device, and includes: A first audio collector, used to collect environmental sound information; a snoring sound recognizer, connected to the first audio collector, and configured to perform snoring sound feature recognition on the environmental sound information to obtain snoring sound feature information in the environmental sound information; An acoustic channel, used for transmitting the environmental sound information; a signal generator, connected to the snoring identifier, and configured to generate a cancellation signal of the snoring characteristic information according to the snoring characteristic information and a vocal channel transfer function of the acoustic channel; A speaker, connected to the signal generator, configured to convert the cancellation signal into a cancellation sound signal and play the cancellation sound signal; An accelerometer and a vibration sensor are connected to the snoring identifier and are used to collect breathing vibration information and movement information of the current user wearing the headphone device; the breathing vibration information, the movement information and the snoring feature information are used to determine whether the snoring feature information comes from the current user.

16. The system according to claim 15, It is characterized in that Also includes: A second audio collector is used to collect the remaining sound signal after the cancellation sound signal and the ambient sound signal are superimposed; A feedback module, used for extracting the remaining snoring features in the remaining sound signal; And, the snoring sound identifier is further used to adjust the parameters of the signal generator according to the remaining snoring sound features, the snoring sound feature information and the vocal tract transfer function; The signal generator is further used to adjust the cancellation signal of the snoring characteristic information according to the adjusted parameters; The speaker is also used to convert the adjusted cancellation signal into an adjusted sound signal and play the adjusted sound signal.

17. An electronic device, It is characterized in that include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 7.

18. A non-transitory electronic device readable storage medium, It is characterized in that The invention comprises: a program, which, when executed by an electronic device, causes the electronic device to execute the method according to any one of claims 1 to 7.

Images