Ear-worn device and playback method

The ear-worn device addresses the challenge of reproducing surrounding voices by using a microphone and signal processing to enhance specific frequency components, ensuring clear reception of announcement sounds amidst noise.

JP7576780B2Active Publication Date: 2024-11-01PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2022552401
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-06-08
Filing Date
2022-01-12
Publication Date
2024-11-01
Estimated Expiration
2042-01-12

AI Technical Summary

Technical Problem

Existing ear-worn devices fail to effectively reproduce surrounding voices, such as announcement sounds, due to environmental noise interference.

Method used

An ear-worn device equipped with a microphone, signal processing circuit, and speaker that selectively outputs a second sound signal based on the detection of human voices in ambient noise, using filters and machine learning to enhance specific frequency components.

Benefits of technology

The device enhances the reproduction of surrounding voices, such as announcement sounds, by automatically switching modes to emphasize or attenuate sounds based on noise conditions, ensuring users do not miss important announcements while using the device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007576780000003
    Figure 0007576780000003
  • Figure 0007576780000004
    Figure 0007576780000004
  • Figure 0007576780000005
    Figure 0007576780000005
Patent Text Reader

Abstract

This ear-mounted device (20) comprises: a microphone (21) which acquires sound and outputs a first sound signal of the acquired sound; a DSP (22) which outputs a second sound signal based on a first sound signal, when it is determined that the sound satisfies predetermined requirements related to noise components contained in the sound and that the sound contains a human voice; a speaker (28) which outputs a reproduced sound on the basis of the output second sound signal; and a housing which houses the microphone (21), the DSP (22), and the speaker (28).
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an ear-worn device and a playback method. [Background technology]

[0002] Various techniques have been proposed for ear-mounted devices such as earphones and headphones. Patent Document 1 discloses a technique for audio playback headphones. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-093792 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an ear-worn device that can reproduce nearby human voices in accordance with the surrounding noise environment. [Means for solving the problem]

[0005] An ear-worn device according to one embodiment of the present disclosure includes a microphone that acquires sound and outputs a first sound signal of the acquired sound, a signal processing circuit that outputs a second sound signal based on the first sound signal when the sound satisfies certain requirements related to noise components contained in the sound and is determined to include a human voice, a speaker that outputs a reproduced sound based on the output second sound signal, and a housing that contains the microphone, the signal processing circuit, and the speaker. Effect of the Invention

[0006] An ear-worn device according to one aspect of the present disclosure can reproduce the voices of people heard nearby in accordance with the surrounding noise environment. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is an external view of devices constituting a sound signal processing system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the sound signal processing system according to the embodiment. [Diagram 3] FIG. 3 is a flowchart of Example 1 of the ear-worn device according to the embodiment. [Figure 4] FIG. 4 is a first flowchart of the operation of the ear-worn device according to the embodiment in the external sound capture mode. [Diagram 5] FIG. 5 is a second flowchart of the operation of the ear worn device in the external sound capture mode according to the embodiment. [Figure 6] FIG. 6 is a flowchart of the operation of the ear-worn device in the noise cancellation mode according to the embodiment. [Figure 7] FIG. 7 is a flowchart of Example 2 of the ear-worn device according to the embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a selection screen for an operation mode. [Figure 9] FIG. 9 is a flowchart of Example 3 of the ear-worn device according to the embodiment. [Figure 10] FIG. 10 is a first diagram showing the change in spectral flatness over time. [Figure 11] FIG. 11 is a second diagram showing the change in spectral flatness over time. [Figure 12] FIG. 12 is a third diagram showing the change in spectral flatness over time. [Figure 13] FIG. 13 is a fourth diagram showing the change in spectral flatness over time. [Figure 14] FIG. 14 is a first diagram showing a spectrogram of the first sound signal. [Figure 15] FIG. 15 is a second diagram showing a spectrogram of the first sound signal. [Figure 16] FIG. 16 is a third diagram showing a spectrogram of the first sound signal. [Figure 17] FIG. 17 is a fourth diagram showing a spectrogram of the first sound signal. [Figure 18] FIG. 18 is a block diagram showing the functional configuration of a noise removal filter that functions as an adaptive filter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims will be described as optional components.

[0009] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In each drawing, the same reference numerals are used for substantially the same configurations, and duplicated explanations may be omitted or simplified.

[0010] (Embodiment) [composition] First, the configuration of a sound signal processing system according to an embodiment will be described. Fig. 1 is an external view of devices constituting the sound signal processing system according to an embodiment. Fig. 2 is a block diagram showing the functional configuration of the sound signal processing system according to an embodiment.

[0011] 1 and 2, a sound signal processing system 10 according to an embodiment includes an ear-worn device 20 and a mobile terminal 30. First, the ear-worn device 20 will be described.

[0012] The ear-worn device 20 is an earphone-type device that reproduces a fourth sound signal provided from the mobile terminal 30. The fourth sound signal is, for example, a sound signal of music content. The ear-worn device 20 has an external sound capture function (also referred to as an external sound capture mode) that captures sounds around the user while the fourth sound signal is being reproduced.

[0013] The surrounding sound here is, for example, an announcement sound. The announcement sound is, for example, a sound output from a speaker provided inside a moving body such as a train, a bus, or an airplane. The announcement sound includes a human voice.

[0014] The ear-worn device 20 operates in a normal mode in which the fourth sound signal provided by the mobile terminal 30 is reproduced, and in an external sound capture mode in which sounds around the user are captured and reproduced. For example, when a user wearing the ear-worn device 20 is on a moving vehicle and listening to music content in the normal mode, if an announcement sound is output within the moving vehicle and the output announcement sound contains a human voice, the ear-worn device 20 automatically transitions from the normal mode to the external sound capture mode. This prevents the user from missing the announcement sound.

[0015] Specifically, the ear-worn device 20 includes a microphone 21, a DSP 22, a communication circuit 27a, a mixing circuit 27b, and a speaker 28. The communication circuit 27a and the mixing circuit 27b may be included in the DSP 22. The microphone 21, the DSP 22, the communication circuit 27a, the mixing circuit 27b, and the speaker 28 are accommodated in a housing 29 (shown in FIG. 1).

[0016] The microphone 21 is a sound collection device that acquires sounds around the ear-worn device 20 and outputs a first sound signal based on the acquired sounds. Specifically, the microphone 21 is a condenser microphone, a dynamic microphone, or a MEMS (Micro Electro Mechanical Systems) microphone, but is not limited thereto. The microphone 21 may be omnidirectional or directional.

[0017] The DSP 22 performs signal processing on the first sound signal output from the microphone 21, thereby realizing an external sound capture function. The DSP 22, for example, outputs a second sound signal based on the first sound signal to the speaker 28, thereby realizing the external sound capture function. The DSP 22 also has a noise cancellation function, and can also output a third sound signal obtained by performing phase inversion processing on the first sound signal. The DSP 22 is an example of a signal processing circuit. Specifically, the DSP 22 has a filter circuit 23, a CPU (Central Processing Unit) 24, and a memory 26.

[0018] The filter circuit 23 includes a noise elimination filter 23a, a high-pass filter 23b, and a low-pass filter 23c. The noise elimination filter 23a is a filter for eliminating noise contained in the first sound signal output from the microphone 21. The noise elimination filter 23a is, for example, a nonlinear digital filter, but may be a filter using a spectral subtraction method that eliminates noise in the frequency domain. The noise elimination filter unit 23a may also be a Wiener filter.

[0019] The high-pass filter 23b attenuates components in a band of 512 Hz or less that are included in the first sound signal after noise removal output from the noise removal filter 23a. The low-pass filter 23c attenuates components in a band of 512 Hz or more that are included in the first sound signal output from the microphone 21. Note that these cutoff frequencies are merely examples, and the cutoff frequency may be empirically or experimentally determined. The cutoff frequency is determined, for example, according to the type of moving object in which the ear-worn device 20 is expected to be used.

[0020] The CPU 24 includes, as functional components, a speech feature calculation unit 24a, a noise feature calculation unit 24b, a determination unit 24c, and a switching unit 24d. The functions of the speech feature calculation unit 24a, the noise feature calculation unit 24b, the determination unit 24c, and the switching unit 24d are realized, for example, by the CPU 24 executing a computer program stored in the memory 26. The functions of the speech feature calculation unit 24a, the noise feature calculation unit 24b, the determination unit 24c, and the switching unit 24d will be described in detail later.

[0021] The memory 26 is a storage device that stores the computer program executed by the CPU 24 and various information necessary for implementing the external sound capture function. The memory 26 is implemented by a semiconductor memory or the like. Note that the memory 26 may be implemented as an external memory of the DSP 22 instead of being an internal memory of the DSP 22.

[0022] The communication circuit 27a receives the fourth sound signal from the mobile terminal 30. The communication circuit 27a is, for example, a wireless communication circuit, and communicates with the mobile terminal 30 based on a communication standard such as Bluetooth (registered trademark) or BLE (Bluetooth (registered trademark) Low Energy).

[0023] The mixing circuit 27b mixes one of the second sound signal and the third sound signal output by the DSP 22 with the fourth sound signal received by the communication circuit 27a, and outputs the result to the speaker 28. The communication circuit 27a and the mixing circuit 27b may be realized as one SoC (System-on-a-Chip).

[0024] The speaker 28 outputs a reproduced sound based on the mixed sound signal obtained from the mixing circuit 27b. The speaker 28 is a speaker that emits sound waves toward the ear canal (eardrum) of the user wearing the ear-worn device 20, but may be a bone conduction speaker.

[0025] Next, the mobile terminal 30 will be described. The mobile terminal 30 is an information terminal that functions as a user interface device in the sound signal processing system 10 by installing a specific application program. The mobile terminal 30 also functions as a sound source that provides a fourth sound signal (music content) to the ear worn device 20. Specifically, the user can operate the mobile terminal 30 to select music content to be played by the speaker 28 and to switch the operation mode of the ear worn device 20. The mobile terminal 30 includes a UI (User Interface) 31, a communication circuit 32, a CPU 33, and a memory 34.

[0026] The UI 31 is a user interface device that receives operations from a user and presents images to the user, and is realized by an operation receiving unit such as a touch panel and a display unit such as a display panel.

[0027] The communication circuit 32 transmits a fourth sound signal, which is a sound signal of the music content selected by the user, to the ear-worn device 20. The communication circuit 32 is, for example, a wireless communication circuit, and communicates with the ear-worn device 20 based on a communication standard such as Bluetooth (registered trademark) or BLE (Bluetooth (registered trademark) Low Energy).

[0028] The CPU 33 processes information related to displaying an image on the display unit and transmitting the fourth sound signal using the communication circuit 32. The CPU 33 is realized by, for example, a microcomputer, but may also be realized by a processor. The image display function and the fourth sound signal transmission function are realized by the CPU 33 executing a computer program stored in the memory 34.

[0029] The memory 34 is a storage device that stores various information necessary for the CPU 33 to perform information processing, computer programs executed by the CPU 33, the fourth sound signal (music content), etc. The memory 34 is realized by, for example, a semiconductor memory.

[0030] [Example 1] As described above, the ear-worn device 20 can automatically operate in the external sound capture mode when the moving body on which the user is riding is moving and an announcement sound is output inside the moving body. Below, several embodiments of the ear-worn device 20 will be described while giving specific examples of situations. First, an embodiment 1 of the ear-worn device 20 will be described. FIG. 3 is a flowchart of the embodiment 1 of the ear-worn device 20. Note that the embodiment 1 shows an operation assuming use when a user wearing the ear-worn device 20 is riding on a moving body.

[0031] The microphone 21 acquires a sound and outputs a first sound signal of the acquired sound (S11). The noise feature amount calculation unit 24b calculates a spectral flatness by performing signal processing on the first sound signal output from the microphone 21 and to which the low-pass filter 23c has been applied (S12). The spectral flatness is an example of a feature amount of noise contained in the first sound signal, and specifically, is a feature amount indicating the flatness of the signal. The spectral flatness indicates, for example, how close the first sound signal is to noise such as white noise, pink noise, or brown noise. When the cutoff frequency of the low-pass filter 23c is 512 Hz, the spectral flatness calculated in step S12 can be said to indicate the flatness of noise below 512 Hz.

[0032] The complex spectrum of the first sound signal to which the low-pass filter 23c has been applied is denoted by S k , the number of frequency bins of the Fourier transform (in other words, the number of FFT calculation points, the number of sampling points) is N FFT Then, the spectral flatness S F is calculated by the following formula. Note that exp[x] means e to the power x, and ln(x) is log e (x). The numerator on the right side of the following equation corresponds to the calculation of entropy, and the denominator corresponds to the calculation for normalizing the entropy.

[0033]

number

[0034] Next, the voice feature calculation unit 24a performs signal processing on the first sound signal output from the microphone 21, which is the first sound signal to which the noise removal filter 23a and the high-pass filter 23b have been applied, to calculate MFCCs (Mel-Frequency Cepstral Coefficients) (S13). MFCCs are cepstrum coefficients used as features in voice recognition and the like, and are obtained by converting the power spectrum compressed using a Mel filter bank into a logarithmic power spectrum and applying an inverse discrete cosine transform to the logarithmic power spectrum. The calculated MFCCs are output to the determination unit 24c.

[0035] Next, the determination unit 24c determines whether or not the sound acquired by the microphone 21 satisfies a predetermined requirement related to a noise component contained in the sound (S14). F It is determined whether the value of is equal to or greater than a threshold value.

[0036] Spectral Flatness S F takes a value between 0 and 1, and it is considered that the closer the value is to 1, the closer the noise captured by the microphone 21 is to white noise. In other words, the spectral flatness S F When the value of is equal to or greater than the threshold, it is considered that the moving object on which the user is riding is moving. Step S14 can be rephrased as a step of determining whether or not the moving object is moving.

[0037] In step S14, the determination unit 24c determines the spectral flatness S F If the value is equal to or greater than the threshold, it is determined that the sound acquired by the microphone 21 satisfies the predetermined requirement (Yes in S14), and the process of step S15 is performed. The determination unit 24c determines whether or not the sound acquired by the microphone 21 includes a human voice, based on the MFCC output from the voice feature calculation unit 24a (S15).

[0038] The determination unit 24c includes, for example, a machine learning model (neural network) that receives MFCC as input and outputs a determination result as to whether or not the sound contains a human voice, and uses such a machine learning model to determine whether or not a human voice is contained in the sound acquired by the microphone 21. The human voice here is assumed to be a human voice contained in an announcement sound.

[0039] When it is determined that the sound acquired by the microphone 21 includes a human voice (Yes in S15), the switching unit 24d operates in the external sound capture mode (S16). That is, the ear-worn device 20 (switching unit 24d) operates in the external sound capture mode (S16) when the moving object is moving (Yes in S14) and a human voice is being output (Yes in S15).

[0040] 4 is a first flowchart of the operation in the external sound capture mode. In the external sound capture mode, the switching unit 24d generates a second sound signal by performing equalization processing on the first sound signal output by the microphone 21 to emphasize a specific frequency component, and outputs the generated second sound signal (S16a). The specific frequency component is, for example, a frequency component between 100 Hz and 2 kHz. By emphasizing the frequency band corresponding to the human voice in this way, the human voice is emphasized, and the announcement sound (more specifically, the human voice included in the announcement sound) is emphasized.

[0041] The mixing circuit 27b mixes the second sound signal with the fourth sound signal (music content) received by the communication circuit 27a and outputs the mixed sound to the speaker 28 (S16b), and the speaker 28 outputs a playback sound based on the second sound signal mixed with the fourth sound signal (S16c). As a result of the process of step S16a, the announcement sound is emphasized, making it easier for the user of the ear worn device 20 to hear the announcement sound.

[0042] On the other hand, the sound captured by the microphone 21 does not meet the specified requirements (spectral flatness S Fis less than the threshold value (No in S14 in FIG. 3) and if it is determined that the sound does not include a human voice (Yes in S14 and No in S15), the switching unit 24d operates in the normal mode (S17). The reproduced sound (music content) of the fourth sound signal received by the communication circuit 27a is output from the speaker 28, and the reproduced sound based on the second sound signal is not output. In other words, the switching unit 24d does not cause the speaker 28 to output the reproduced sound based on the second sound signal.

[0043] The above process shown in the flowchart of Fig. 3 is repeated at predetermined time intervals. That is, it is determined whether the normal mode or the ambient sound capture mode is to be operated at predetermined time intervals. The predetermined time interval is, for example, 1 / 60 seconds. The ear-worn device 20 operates in the ambient sound capture mode only when the conditions that the moving object is moving and a human voice is being output are satisfied (that is, only when Yes is determined in step S14 and Yes is determined in step S15), and otherwise operates in the normal mode.

[0044] As described above, when the DSP 22 determines that the noise contained in the sound acquired by the microphone 21 satisfies the predetermined requirements and that the sound includes a human voice, it outputs a second sound signal based on the first sound signal. When the DSP 22 determines that the sound acquired by the microphone 21 satisfies the predetermined requirements related to the noise components contained in the sound and that the sound includes a human voice, it outputs a second sound signal obtained by performing signal processing on the first sound signal. This signal processing includes an equalizing process for emphasizing a specific frequency component of the sound. Furthermore, when the DSP 22 determines that the sound acquired by the microphone 21 does not satisfy the predetermined requirements and that the sound does not include a human voice, it does not cause the speaker 28 to output a reproduced sound based on the second sound signal.

[0045] In this way, the ear-worn device 20 can assist a user on a moving object to hear the announcement sound while the moving object is moving. The user is less likely to miss the announcement sound even if he or she is engrossed in music content.

[0046] Note that the operation in the external sound capture mode is not limited to the operation shown in Fig. 4. For example, it is not essential that an equalizing process is performed in step S16a, and the second sound signal may be generated by signal processing that increases the gain (amplitude) of the first sound signal. Also, in the external sound capture mode, it is not essential that signal processing is performed on the first sound signal.

[0047] Fig. 5 is a second flowchart of the operation in the external sound capture mode. In the example of Fig. 5, the switching unit 24d outputs the first sound signal output by the microphone 21 as the second sound signal (S16d). That is, the switching unit 24d outputs the first sound signal substantially as it is as the second sound signal. The switching unit 24d also instructs the mixing circuit 27b to attenuate (gain down, attenuate the amplitude) the fourth sound signal during mixing.

[0048] The mixing circuit 27b mixes the second sound signal with a fourth sound signal (music content) whose amplitude is attenuated compared to the normal mode and outputs the result to the speaker 28 (S16e), and the speaker 28 outputs a reproduced sound based on the second sound signal mixed with the fourth sound signal whose amplitude is attenuated (S16f).

[0049] In this way, during operation in the external sound capture mode after the output of the second sound signal is started by the DSP 22, the fourth sound signal having an attenuated amplitude compared to during operation in the normal mode before the output of the second sound signal is started may be mixed into the second sound signal. As a result, the announcement sound is emphasized, making it easier for the user of the ear worn device 20 to hear the announcement sound.

[0050] Note that the operation in the external sound capture mode is not limited to the operation shown in Fig. 4 and Fig. 5. For example, in the operation in the external sound capture mode of Fig. 4, the second sound signal generated by equalizing the first sound signal may be mixed with the attenuated fourth sound signal as in step S16e of Fig. 5. Also, in the operation in the external sound capture mode of Fig. 5, the process of attenuating the fourth sound signal may be omitted, and the unattenuated fourth sound signal may be mixed with the second sound signal.

[0051] [Example 2] The ear-worn device 20 may have a noise cancellation function (hereinafter also referred to as noise cancellation mode) that reduces environmental noise around the user wearing the ear-worn device 20 while the fourth sound signal (music content) is being played.

[0052] First, the noise cancellation mode will be described. When a user operates the UI 31 of the mobile terminal 30 to instruct the noise cancellation mode, the CPU 33 transmits a setting command to the ear worn device 20 using the communication circuit 32 to set the noise cancellation mode to the ear worn device 20. When the setting command is received by the communication circuit 27a of the ear worn device 20, the switching unit 24d operates in the noise cancellation mode.

[0053] 6 is a flowchart of the operation in the noise cancellation mode. In the noise cancellation mode, the switching unit 24d performs phase inversion processing on the first sound signal output by the microphone 21 and outputs the result as a third sound signal (S18a). 。

[0054] The mixing circuit 27b mixes the third sound signal with the fourth sound signal (music content) received by the communication circuit 27a and outputs the mixed sound to the speaker 28 (S18b), and the speaker 28 outputs a reproduced sound based on the third sound signal mixed with the fourth sound signal (S18c). As a result of the processing of steps S18a and S18b, the user of the ear worn device 20 feels that the sound around the ear worn device 20 is attenuated, and the user can clearly listen to the music content.

[0055] Hereinafter, a second embodiment will be described in which the ear-worn device 20 operates in the noise cancellation mode instead of the normal mode. Fig. 7 is a flowchart of the second embodiment of the ear-worn device 20. Note that the second embodiment shows the operation of the ear-worn device 20 when a user wearing the ear-worn device 20 is riding on a moving object.

[0056] The process from step S11 to step S13 in FIG. 7 is similar to the process from step S11 to step S13 in the first embodiment (FIG. 3).

[0057] After step S13, the determination unit 24c determines whether or not the sound acquired by the microphone 21 satisfies a predetermined requirement related to a noise component contained in the sound (S14). The details of the process of step S14 are the same as those of step S14 in the first embodiment (FIG. 3). Specifically, the determination unit 24c determines the spectral flatness S F It is determined whether the value of is equal to or greater than a threshold value.

[0058] In step S14, the determination unit 24c determines the spectral flatness S F If the value is equal to or greater than the threshold, it is determined that the sound acquired by the microphone 21 satisfies a predetermined requirement (Yes in S14), and the process of step S15 is performed. The determination unit 24c determines whether or not the sound acquired by the microphone 21 includes a human voice based on the MFCC output from the voice feature calculation unit 24a (S15). The details of the process of step S15 are the same as step S15 in the first embodiment (FIG. 3).

[0059] When it is determined that the sound acquired by the microphone 21 includes a human voice (Yes in S15), the switching unit 24d operates in the external sound capture mode (S16). That is, when the moving object is moving (Yes in S14) and a human voice is being output (Yes in S15), the ear worn device 20 (switching unit 24d) operates in the external sound capture mode (S16). The operation in the external sound capture mode is as described with reference to Figures 4 and 5. According to the operation in the external sound capture mode, the announcement sound is emphasized, so that the user of the ear worn device 20 can easily hear the announcement sound.

[0060] On the other hand, the sound captured by the microphone 21 does not meet the specified requirements (spectral flatness S F is less than the threshold value (No in S14) and if it is determined that the sound does not include a human voice (Yes in S14 and No in S15), the switching unit 24d performs the operation in the noise cancellation mode (S18). The operation in the noise cancellation mode is as described with reference to FIG. 6.

[0061] The above process shown in the flowchart of Fig. 7 is repeated at predetermined time intervals. That is, it is determined whether the noise cancellation mode or the ambient sound capture mode is to be operated at predetermined time intervals. The predetermined time interval is, for example, 1 / 60 seconds. The ear-worn device 20 operates in the ambient sound capture mode only when the conditions that the moving object is moving and a human voice is being output are satisfied (that is, only when Yes is determined in step S14 and Yes is determined in step S15), and operates in the noise cancellation mode otherwise.

[0062] In this way, when the DSP 22 determines that the sound acquired by the microphone 21 does not satisfy the predetermined requirements related to the noise components contained in the sound, and when the DSP 22 determines that the sound does not include a human voice, it outputs a third sound signal obtained by performing phase inversion processing on the first sound signal. The speaker 28 outputs a reproduced sound based on the output third sound signal.

[0063] This allows the ear-worn device 20 to assist a user riding on a moving object to clearly listen to music content while the moving object is moving.

[0064] When the user instructs the noise cancellation mode, for example, a selection screen as shown in FIG. 8 is displayed on the UI 31 of the mobile terminal 30. FIG. 8 is a diagram showing an example of a selection screen for an operation mode. As shown in FIG. 8, the operation modes selectable by the user include, for example, three modes: a normal mode, a noise cancellation mode, and an external sound capture mode. That is, the ear-worn device 20 may operate in the external sound capture mode based on an operation on the mobile terminal 30 by the user.

[0065] [Example 3] The ear-worn device 20 calculates a spectral flatness S F Based on the above, it may be determined whether the noise satisfies a predetermined requirement (whether the moving object is moving or not). FIG. 9 is a flowchart of the third embodiment of the ear-worn device 20.

[0066] In addition, the third embodiment shows the operation when a user wearing the ear-worn device 20 is riding on a moving object. In the third embodiment, the first sound signal includes a portion corresponding to a first period and a portion corresponding to a second period following the first period, the first period corresponds to a first partial signal indicating the first sound (i.e., a part of the signal of the first sound signal), and the second period corresponds to a second partial signal indicating the second sound (i.e., another part of the signal of the first sound signal). The second period is, for example, a certain period immediately after the first period.

[0067] The process from step S11 to step S13 is similar to the process from step S11 to step S13 in the first embodiment (FIG. 3).

[0068] After step S34, the determination unit 24c determines whether or not the first sound acquired by the microphone 21 includes a human voice, based on the MFCC output from the voice feature calculation unit 24a (S19).

[0069] When it is determined that the first sound acquired by the microphone 21 does not include a human voice (No in S19), the determination unit 24c determines whether or not the first sound acquired by the microphone 21 satisfies a predetermined requirement related to a noise component included in the first sound (S20). F It is determined whether the value of is equal to or greater than a threshold value.

[0070] In step S20, the determination unit 24c determines the spectral flatness S F If the value is equal to or greater than the threshold, it is determined that the first sound acquired by the microphone 21 satisfies the predetermined requirement (Yes in S20), and the process of step S21 is performed. The determination unit 24c determines whether or not the second sound acquired by the microphone 21 includes a human voice based on the MFCC output from the voice feature calculation unit 24a (S21).

[0071] When it is determined that the second sound acquired by the microphone 21 includes a human voice (Yes in S21), the switching unit 24d performs an operation in the external sound capture mode (S16). The operation in the external sound capture mode is as described with reference to Fig. 4 and Fig. 5. According to the operation in the external sound capture mode, the announcement sound is emphasized, so that the user of the ear worn device 20 can easily hear the announcement sound.

[0072] On the other hand, if it is determined that the first sound includes a human voice (Yes in S19), the first sound does not satisfy the predetermined requirements (spectral flatness S Fis less than the threshold value (No in S20), and if it is determined that the second sound does not include a human voice (Yes in S20 and No in S21), the switching unit 24d performs an operation in the normal mode (S17). Note that instead of step S17, the operation in the noise cancellation mode of the above-mentioned step S18 may be performed. The operation in the noise cancellation mode is as described with reference to FIG. 6.

[0073] The above process shown in the flowchart of Fig. 9 is repeated at predetermined time intervals. That is, it is determined whether the normal mode or the ambient sound capture mode is to be operated at predetermined time intervals. The predetermined time interval is, for example, 1 / 60 seconds. The ear-worn device 20 operates in the ambient sound capture mode only when the conditions that the moving object is moving and a human voice is being output are satisfied (that is, only when Yes is determined in step S20 and Yes is determined in step S21), and otherwise operates in the normal mode.

[0074] In this way, when DSP22 determines that the first sound satisfies predetermined requirements related to the noise components contained in the first sound, the first sound does not contain a human voice, and the second sound contains a human voice, it outputs the second sound signal.

[0075] As a result, the ear-worn device 20 determines whether the noise satisfies the specified requirements using the part of the first sound signal that does not include human voice, thereby improving the accuracy of the determination.

[0076] [Supplementary information on determining whether noise meets specified requirements] In the above embodiment, the determination unit 24c determines whether the noise satisfies a predetermined requirement (spectral flatness S F The appropriateness of the application method of the low-pass filter 23c will be discussed below with respect to the spectral flatness S F This will be supplemented with reference to the waveform.

[0077] FIG. 10 shows the spectral flatness S of the first sound signal obtained by the microphone 21 when the moving object is moving and an announcement sound is being output from within the moving object. F When the spectrum flatness S is calculated, F FIG.

[0078] FIG. 11 shows the spectral flatness S of the first sound signal obtained by the microphone 21 when the moving object is moving and an announcement sound is being output from within the moving object. F When the spectrum flatness S is calculated, F FIG.

[0079] FIG. 12 shows the spectral flatness S of the first sound signal obtained by the microphone 21 when the moving object is stopped and an announcement sound is being output from within the moving object. F When the spectrum flatness S is calculated, F FIG.

[0080] FIG. 13 shows the spectral flatness S of the first sound signal obtained by the microphone 21 when the moving object is stopped and an announcement sound is being output from within the moving object. F When the spectrum flatness S is calculated, F FIG.

[0081] As shown in FIG. 10 and FIG. 12, the spectral flatness S calculated based on the components of the first sound signal above 512 Hz is F The fluctuation is large, and it depends on whether the moving object is moving or not (spectral flatness S F is not suitable for judging whether a

[0082] On the other hand, as shown in FIG. 11 and FIG. 13, the spectral flatness S calculated based on the components of the first sound signal less than 512 Hz isF The fluctuation is relatively small, and it is not clear whether the moving object is moving or not (spectral flatness S F is equal to or greater than a threshold value). In other words, based on the first sound signal to which the low-pass filter 23c has been applied, it is suitable for determining whether or not a moving object is moving (whether or not the spectrum flatness S F is equal to or greater than a threshold value), the accuracy of the determination can be improved.

[0083] In the examples of Figures 11 and 13, the threshold is set to 10 -8 If the threshold value is set near the target position, the determining unit 24c can determine whether the moving object is moving or stopped. Such a threshold value is an example, and the threshold value may be appropriately determined empirically or experimentally by a designer.

[0084] The determination unit 24c determines the spectral flatness S F Whether the noise satisfies a predetermined requirement may be determined based on whether the moving average or moving median of is equal to or greater than a threshold. In this case, the threshold is also set to a value corresponding to the moving average or moving median.

[0085] [Additional information on determining whether a video contains a human voice] In the above embodiment, the determination unit 24c determines whether or not the sound acquired by the microphone 21 includes a human voice based on the first sound signal to which the high-pass filter 23b has been applied. The appropriateness of the method of applying the high-pass filter 23b will be supplemented below with reference to a spectrogram.

[0086] Fig. 14 is a diagram showing a spectrogram of the first sound signal acquired by microphone 21 when the moving object is moving and an announcement sound is being output inside the moving object. Fig. 15 is a diagram showing a spectrogram of the first sound signal acquired by microphone 21 when the moving object is moving and an announcement sound is not being output inside the moving object.

[0087] Fig. 16 is a diagram showing a spectrogram of the first sound signal acquired by the microphone 21 when the moving object is stopped and an announcement sound is being output inside the moving object. Fig. 17 is a diagram showing a spectrogram of the first sound signal acquired by the microphone 21 when the moving object is stopped and an announcement sound is not being output inside the moving object.

[0088] 14 to 17, the whiter the part, the higher the power value, and the darker the part, the lower the power value. Crab Regardless, when the announcement sound is being output (FIGS. 14 and 16), ripples corresponding to a human voice appear in the band of 512 Hz or more. Therefore, the determination unit 24c can determine whether or not a human voice is included in the sound acquired by the microphone 21 based on at least the component of the first sound signal of 512 Hz or more. In addition, the determination unit 24c can improve the accuracy of the determination by determining whether or not a human voice is included in the sound acquired by the microphone 21 based on the first sound signal to which the high-pass filter 23b has been applied.

[0089] [Modification of noise removal filter] The noise removal filter 23a may be an adaptive filter. Specifically, as shown by the dashed arrow from the noise feature amount calculation unit 24b to the noise removal filter 23a in FIG. 2, the noise removal filter 23a adjusts the spectral flatness S F The filter coefficients may be updated using the value of Fig. 18. Fig. 18 is a block diagram showing the functional configuration of the noise removal filter 23a that functions as an adaptive filter.

[0090] As shown in FIG. 18, the noise removal filter 23a realized as an adaptive filter includes a filter coefficient update unit 23a1 and an adaptive filter unit 23a2.

[0091] The filter coefficient update unit 23a1 sequentially updates the coefficients of the adaptive filter based on the following update equation. In the following equation, w is the filter coefficient, x is the first sound signal, and e is the error signal. The error signal is a signal equivalent to the difference between the first sound signal after the filter coefficient is applied and the target signal. μ is a parameter (hereinafter also referred to as step size parameter) indicating the update amount (step size) of the filter coefficient, and is a positive coefficient.

[0092]

number

[0093] The adaptive filter unit 23a2 applies a filter constituted by the filter coefficients calculated by the filter coefficient update unit 23a1 to the first sound signal, and outputs the first sound signal after the filter coefficients have been applied (i.e., the first sound signal after noise has been removed) to the high-pass filter 23b.

[0094] Here, the filter coefficient update unit 23a1 updates the step size parameter with the spectral flatness S F The filter coefficient update unit 23a1 may change the filter coefficient by using, for example, the value of the spectral flatness S F The larger the value of step-size parameter, the larger the value of the step-size parameter is changed to. Specifically, the filter coefficient update unit 23a1 changes the value of the step-size parameter as follows, using a first threshold and a second threshold that is larger than the first threshold.

[0095] The filter coefficient update unit 23a1 updates the spectral flatness S F If the value of is less than the first threshold, the step size parameter is changed to a value smaller than the current value, and the spectral flatness S F If the value of the spectral flatness S is equal to or greater than the first threshold and less than the second threshold, the step-size parameter is maintained at the current value. F If the value of is equal to or greater than the second threshold, the step size parameter is changed to a value larger than the current value.

[0096] As a result, the noise removal filter 23a (the filter coefficient update unit 23a1) can promote adaptive learning as the noise becomes closer to white noise. Note that the filter coefficient update unit 23a1 does not need to change the step size parameter in the external sound capture mode. In other words, the filter coefficient update unit 23a1 may fix the step size parameter to a constant value in the external sound capture mode.

[0097] In the example of FIG. 18, the noise removal filter 23a is realized as an adaptive filter of a feedforward control type using the first sound signal output by the microphone 21, but the noise removal filter 23a may be realized as an adaptive filter of a feedback control type.

[0098] The noise removal filter 23a is not limited to a filter with fixed coefficients or an adaptive filter. The noise removal filter 23a has a plurality of filters of different types, and the spectral flatness S F It may be a filter that can switch between multiple filters based on the value of

[0099] [Effects, etc.] As described above, the ear-worn device 20 includes the microphone 21 that acquires sound and outputs a first sound signal of the acquired sound, the DSP 22 that outputs a second sound signal based on the first sound signal when the sound satisfies predetermined requirements related to noise components contained in the sound and is determined to include a human voice, the speaker 28 that outputs a reproduced sound based on the output second sound signal, and a housing 29 that accommodates the microphone 21, the DSP 22, and the speaker 28. The DSP 22 is an example of a signal processing circuit.

[0100] Such an ear-worn device 20 can reproduce the voices of people heard nearby in accordance with the surrounding noise environment. For example, when an announcement sound is output inside the moving body while the moving body is moving, the ear-worn device 20 can output a reproduction sound including the announcement sound from the speaker 28.

[0101] Furthermore, for example, when the DSP 22 determines that the sound satisfies a predetermined requirement and that the sound includes a human voice, it outputs the first sound signal as the second sound signal.

[0102] Such an ear-worn device 20 can reproduce the voices of people heard in the surroundings based on the first sound signal.

[0103] Furthermore, for example, when the DSP 22 determines that the sound satisfies a predetermined requirement and that the sound includes a human voice, it outputs a second sound signal obtained by performing signal processing on the first sound signal.

[0104] Such an ear-worn device 20 can reproduce the voices of people heard in the surroundings based on the signal-processed first sound signal.

[0105] Furthermore, for example, the signal processing includes equalizing processing for emphasizing specific frequency components of the sound.

[0106] Such an ear-worn device 20 can emphasize and reproduce the voices of people heard around you.

[0107] Also, for example, if DSP22 determines that the sound does not satisfy predetermined requirements and that the sound does not include a human voice, it does not cause speaker 28 to output a reproduced sound based on the second sound signal.

[0108] Such an ear-worn device 20 can stop outputting the reproduced sound based on the second sound signal when, for example, surrounding voices cannot be heard.

[0109] Also, for example, if DSP22 determines that the sound does not satisfy predetermined requirements and that the sound does not include a human voice, it outputs a third sound signal that has been subjected to phase inversion processing on the first sound signal, and speaker 28 outputs a reproduced sound based on the output third sound signal.

[0110] Such an ear-worn device 20 can make surrounding sounds less audible, for example, when surrounding people's voices cannot be heard.

[0111] For example, the ear-worn device 20 further includes a mixing circuit 27b that mixes the output second sound signal with a fourth sound signal provided from a sound source. When the DSP 22 starts outputting the second sound signal, the fourth sound signal, the amplitude of which is attenuated compared to before the output of the second sound signal starts, is mixed into the second sound signal.

[0112] Such an ear-worn device 20 can emphasize and reproduce the voices of people heard around you.

[0113] Also, for example, the DSP 22 determines whether the sound satisfies certain requirements based on the first sound signal to which the low-pass filter 23c has been applied, and determines whether the sound includes a human voice based on the first sound signal to which the high-pass filter 23b has been applied.

[0114] Such an ear-worn device 20 can improve the accuracy of the determination by applying a filter to the first sound signal.

[0115] Also, for example, the DSP 22 determines whether the sound includes a human voice based on the first sound signal to which the adaptive filter has been applied, and changes the update amount of the filter coefficient of the adaptive filter based on the noise included in the sound.

[0116] Such an ear-worn device 20 can vary the effect of the adaptive filter depending on the surrounding noise environment.

[0117] Also, for example, the above sounds include a first sound acquired in a first period and a second sound acquired in a second period after the first period. When the DSP 22 determines that the first sound satisfies a predetermined requirement, that the first sound does not include a human voice, and that the second sound includes a human voice, it outputs a second sound signal.

[0118] Such an ear-worn device 20 can improve the accuracy of determining whether or not the sound satisfies a predetermined requirement.

[0119] In addition, the playback method executed by a computer such as the DSP 22 includes an output step S16a (or S16d) of outputting a second sound signal based on a first sound signal of the sound output by the microphone 21 that acquires the sound, when it is determined that the sound satisfies certain requirements related to the noise components contained in the sound and contains a human voice, and a playback step S16c (or S16f) of outputting a reproduced sound from the speaker 28 based on the output second sound signal.

[0120] This reproduction method makes it possible to reproduce the voices of people heard nearby in accordance with the surrounding noise environment.

[0121] (Other embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0122] For example, in the above embodiment, the ear-worn device is described as an earphone-type device, but it may be a headphone-type device. Also, in the above embodiment, the ear-worn device has a function of playing music content, but it does not have to have the function of playing music content (communication circuit and mixing circuit). For example, the ear-worn device may be earplugs or hearing aids with a noise cancellation function and an external sound capture function.

[0123] In the above embodiment, the determination of whether the sound captured by the microphone contains a human voice is performed using a machine learning model, but may be performed based on other algorithms that do not use a machine learning model, such as pattern matching of voice features. In addition, the determination of whether the sound captured by the microphone satisfies a predetermined requirement related to a noise component contained in the sound is performed using spectral flatness, but may be performed using a machine learning model.

[0124] In the above embodiment, the predetermined requirement related to the noise component is a requirement corresponding to whether or not the moving object is moving. However, the predetermined requirement related to the noise component may be another requirement, such as a requirement corresponding to whether or not the surrounding noise level is greater than a predetermined value.

[0125] The configuration of the ear-worn device according to the above embodiment is an example. For example, the ear-worn device may include components not shown, such as a D / A converter, a filter, a power amplifier, or an A / D converter.

[0126] In the above embodiment, the sound signal processing system is realized by a plurality of devices, but may be realized as a single device. When the sound signal processing system is realized by a plurality of devices, the functional components of the sound signal processing system may be distributed in any manner among the plurality of devices. For example, in the above embodiment, some or all of the functional components of the ear-worn device may be included in a mobile terminal.

[0127] Furthermore, the method of communication between the devices in the above-described embodiments is not particularly limited. When two devices communicate with each other in the above-described embodiments, a relay device (not shown) may be interposed between the two devices.

[0128] The order of the processes described in the above embodiment is merely an example. The order of the processes may be changed, or the processes may be executed in parallel. A process executed by a specific processing unit may be executed by another processing unit. A part of the digital signal processing described in the above embodiment may be realized by analog signal processing.

[0129] In the above embodiment, each component may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0130] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or an integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit, or a dedicated circuit.

[0131] In addition, the general or specific aspects of the present disclosure may be realized in a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. In addition, the present disclosure may be realized in any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. For example, the present disclosure may be implemented as a playback method executed by a computer such as an ear-worn device or a mobile terminal, or may be realized as a program for causing a computer to execute such a playback method. In addition, the present disclosure may be realized as a computer-readable non-transitory recording medium on which such a program is recorded. Note that the program here includes an application program for causing a general-purpose mobile terminal to function as the mobile terminal of the above embodiment.

[0132] In addition, the present disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art may conceive, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the spirit of the present disclosure. [Industrial Applicability]

[0133] The ear-worn device of the present disclosure can output reproduced sound including the voices of people in the surroundings according to the surrounding noise environment. [Explanation of symbols]

[0134] 10 Sound signal processing system 20 Ear-worn devices 21 Microphone 22 DSP 23 Filter section 23a Noise Reduction Filter 23a1 Filter coefficient update unit 23a2 Adaptive filter section 23b High-pass filter 23c Low-pass filter 24 Signal Processing Section 24a Speech feature calculation unit 24b Noise feature calculation unit 24c Judgment section 24d Switching part 26 Memory 27a Communication circuit 27b Mixing circuit 28 Speaker 29 Housing 30 Mobile Devices 31 UI 32 Communication Circuits 33 CPU 34 Memory

Claims

1. a microphone for capturing a sound and outputting a first sound signal of the captured sound; a signal processing circuit that determines whether or not a user is riding on a moving vehicle based on a determination of whether or not a feature amount related to a noise component included in the sound satisfies a predetermined requirement, and outputs a second sound signal based on the first sound signal when it is determined that the user is riding on the moving vehicle and when it is determined that the sound includes a human voice; a speaker that outputs a reproduced sound based on the output second sound signal; a housing for accommodating the microphone, the signal processing circuit, and the speaker; Ear-worn device.

2. the feature amount is a spectral flatness calculated from the first sound signal to which a low-pass filter has been applied, The predetermined requirement is that the feature amount is equal to or greater than a predetermined threshold value. The ear-worn device of claim 1 .

3. the noise components are components of a frequency band lower than a predetermined frequency band corresponding to a human voice among the sound components, The low-pass filter corresponds to a frequency band lower than the predetermined frequency band. The ear-worn device of claim 2 .

4. The signal processing circuit outputs the first sound signal as the second sound signal when it is determined that the feature amount satisfies the predetermined requirement and that the sound includes a human voice. An ear-worn device according to any one of claims 1 to 3.

5. The signal processing circuit outputs the second sound signal obtained by performing signal processing on the first sound signal when it is determined that the feature amount satisfies the predetermined requirement and the sound includes a human voice. An ear-worn device according to any one of claims 1 to 3.

6. The signal processing includes equalization processing for emphasizing specific frequency components of the sound. The ear-worn device of claim 5 .

7. When the signal processing circuit determines that the feature does not satisfy the predetermined requirement and when the signal processing circuit determines that the sound does not include a human voice, the signal processing circuit does not cause the speaker to output a reproduced sound based on the second sound signal. An ear-worn device according to any one of claims 1 to 6.

8. when it is determined that the feature does not satisfy the predetermined requirement and when it is determined that the sound does not include a human voice, the signal processing circuit outputs a third sound signal obtained by performing a phase inversion process on the first sound signal; The speaker outputs a reproduced sound based on the outputted third sound signal. An ear-worn device according to any one of claims 1 to 6.

9. Further, a mixing circuit is provided for mixing the output second sound signal with a fourth sound signal provided from a sound source, When the signal processing circuit starts outputting the second sound signal, the fourth sound signal, the amplitude of which is attenuated compared to before the output of the second sound signal started, is mixed with the second sound signal. An ear-worn device according to any one of claims 1 to 8.

10. The signal processing circuit determines whether the sound includes a human voice based on the first sound signal to which a high-pass filter has been applied. An ear-worn device according to any one of claims 1 to 9.

11. The signal processing circuit includes: determining whether the sound includes a human voice based on the first sound signal to which the adaptive filter has been applied; The update amount of the filter coefficient of the adaptive filter is changed based on noise contained in the sound. An ear-worn device according to any one of claims 1 to 10.

12. the sounds include a first sound captured during a first time period and a second sound captured during a second time period after the first time period; The signal processing circuit outputs the second sound signal when it is determined that the feature amount calculated from the first sound signal of the first sound satisfies the predetermined requirement, the first sound does not include a human voice, and the second sound includes a human voice. An ear-worn device according to any one of claims 1 to 11.

13. an output step of determining whether or not a user is riding on a moving vehicle based on a determination of whether or not a feature amount related to a noise component included in a sound output by a microphone acquiring the sound satisfies a predetermined requirement, and outputting a second sound signal based on a first sound signal of the sound when it is determined that the user is riding on the moving vehicle and when it is determined that the sound includes a human voice; and a reproduction step of outputting a reproduced sound from a speaker based on the output second sound signal. How to play.

14. A program for causing a computer to execute the reproducing method according to claim 13.

Citation Information

Patent Citations

  • Noise canceling method and noise canceling device

    JP1999345000A

  • Particular sound reproducing apparatus and headphone

    JP2006093792A

  • Audio signal processing apparatus, audio signal processing method, and communication terminal

    JP2010062663A

  • Headphone

    JP2011199699A

  • Speech section correction frame number acquisition method, speech section detection method and device

    JP2018523155A