Sound collection device, sound collection method, and sound collection program

Abstract

To provide a technique for suppressing influence of sound other than target sound or a noise component.SOLUTION: A sound collection device 100 includes an adaptive filter 18, a coefficient updating section, and an adaptive control section 34. The adaptive filter 18 executes calculation on a vibration signal 202 acquired in a vibration sensor 14 by using a filter coefficient, and outputs a conversion sound signal 212. The coefficient updating section updates the filter coefficient based on a residual signal 210 being a difference between the sound signal 200 acquired in a microphone 10 and the conversion sound signal 212. The adaptive control section 34 adjusts a frequency of updating of the filter coefficient in the coefficient updating section based on the vibration signal 202 and the sound signal 200.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a sound collection technology, and more particularly to a sound collection device, a sound collection method, and a sound collection program that use a microphone and a vibration sensor. [Background technology]

[0002] The sound collection device is equipped with a microphone that generates an audio signal based on air vibrations and a vibration sensor that generates a vibration signal equivalent to the audio signal based on bone vibrations, thereby capturing clear sound even in a noisy environment (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-251354 A Summary of the Invention [Problem to be solved by the invention]

[0004] The sound collection device converts the vibration signal generated by the vibration sensor into an audio signal in the filtering unit, approximating the vibration signal to the original audio signal propagating through the air, thereby providing easy-to-listen audio. However, when the target speaker speaks, if the conversational voice of people other than the speaker overlaps, the conversion device will mistakenly adapt to components other than the target sound. Therefore, it is necessary to accurately grasp cases in which voices other than the speaker are mixed into the microphone. If the device does not have a function to detect voice components and noise components other than the speaker, there is a risk that the influence of voices or noise components other than the target sound will be mistakenly added to the converted audio when increasing the approximation to the microphone voice in the speech section.

[0005] The present invention has been made in view of the above circumstances, and has an object to provide a technique for suppressing the influence of sounds or noise components other than a target sound. [Means for solving the problem]

[0006] In order to solve the above problems, a sound collection device of one embodiment of the present invention comprises an adaptive filter that performs calculations using filter coefficients on a vibration signal acquired by a vibration sensor and outputs a converted audio signal, a coefficient update unit that updates the filter coefficients based on a residual signal that is the difference between the audio signal acquired by a microphone and the converted audio signal, and an adaptive control unit that adjusts the frequency of updating the filter coefficients in the coefficient update unit based on the vibration signal and the audio signal.

[0007] Another aspect of the present invention is a sound collection method, comprising the steps of: performing an operation using a filter coefficient on a vibration signal acquired by a vibration sensor and outputting a converted sound signal; updating the filter coefficient based on a residual signal that is a difference between a sound signal acquired by a microphone and the converted sound signal; and adjusting the frequency of updating the filter coefficient based on the vibration signal and the sound signal.

[0008] Another aspect of the present invention is a sound collection program that causes a computer to execute a step of performing an operation using a filter coefficient on a vibration signal acquired by a vibration sensor and outputting a converted audio signal, a step of updating the filter coefficient based on a residual signal that is a difference between an audio signal acquired by a microphone and the converted audio signal, and a step of adjusting the frequency of updating the filter coefficient based on the vibration signal and the audio signal.

[0009] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention. Effect of the Invention

[0010] According to the present invention, the influence of sounds or noise components other than the target sound can be suppressed. [Brief description of the drawings]

[0011] [Figure 1]1 is a diagram illustrating a configuration of a sound collecting device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a configuration of an adaptive filter according to the first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating a configuration of a correlation determining unit according to the first embodiment. [Figure 4] 4A and 4B are diagrams illustrating data structures of tables held in the state determination unit according to the first embodiment. [Diagram 5] 5 is a diagram showing time variations in sound pressure level and correlation value in the microphone and the vibration sensor according to the first embodiment. FIG. [Figure 6] 5 is a flowchart showing a procedure for updating a first threshold value and a second threshold value by the sound collection device according to the first embodiment. [Figure 7] 4 is a flowchart showing a processing procedure performed by the sound collection device according to the first embodiment. [Figure 8] 8A and 8B are diagrams illustrating data structures of tables held in a state determining unit according to the second embodiment. [Figure 9] 11 is a flowchart showing a processing procedure performed by the sound collection device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Example 1 Before describing the present invention in detail, an overview will be given first. This embodiment relates to a sound collection device. The sound collection device includes a microphone that generates a sound signal based on air vibrations, a vibration sensor that generates a vibration signal based on vibrations transmitted to the human body, and an adaptive filter that multiplies the vibration signal by a coefficient to generate a converted sound signal in order to correct the vibration signal so as to approximate the sound signal. In addition, when the sound collection device detects the intrusion of sound components other than the target sound into the microphone, it stops updating (learning) the filter coefficients in the adaptive filter, thereby suppressing the approximation of the converted sound signal to sound components other than the target sound.

[0013] The inclusion of audio components other than the target sound is judged using a threshold value that is a reference for the correlation value between the vibration signal and the audio signal, and the reference threshold value is variable in conjunction with the correlation between the vibration signal and the audio signal in non-audio sections, i.e., under background noise. This improves the audio quality of the converted audio signal by detecting correlation with higher accuracy. The converted audio signal is a digital signal related to audio that has been obtained by performing analog-to-digital conversion on the vibration signal acquired by the vibration sensor and then performing calculations using a specified filter coefficient, i.e., it is an audio signal converted based on the vibration signal.

[0014] 1 shows a configuration of a sound collection device 100 according to Example 1. The sound collection device 100 includes a microphone 10, a first AD converter (Analog to Digital converter) 12, a vibration sensor 14, a second AD converter 16, an adaptive filter 18, a subtractor 20, a DA converter (Digital to Analog converter) 22, a detection unit 30, a correlation determination unit 32, and an adaptive control unit 34.

[0015] The microphone 10 converts air vibrations into an audio signal 200. Since the microphone 10 can accurately acquire the audio signal 200, the audio signal 200 acquired by the microphone 10 is relatively close to the audio signal 200 perceived by a human being through the ear. Therefore, by setting the audio signal 200 acquired by the microphone 10 as a target value of a vibration signal 202 described below, it is possible to maintain the audio quality of the converted audio signal 212 output by the adaptive filter 18 at a high level.

[0016] The first AD converter 12 performs analog-to-digital conversion on the audio signal 200 acquired by the microphone 10, and outputs the audio signal 200 converted into a digital signal (hereinafter also referred to as the “audio signal 200”) to the subtractor 20 and the correlation determination unit 32.

[0017] The vibration sensor 14 acquires speech signals emitted by a person as vibration signals 202 that are mainly transmitted to the human body. The vibration sensor 14 may be a microphone that detects sound or vibration, or a camera (visual microphone) that detects vibrations from pixel variations or color fluctuations between video frames to acquire sound signals. Instead of the vibration sensor 14, a receiving device embedded in the body or a microphone that is in direct contact with the human body may be used, or a device that measures vibrations on the surface of the human body without contacting the human body, such as a laser displacement meter.

[0018] The second AD converter 16 performs analog-to-digital conversion on the vibration signal 202 acquired by the vibration sensor 14, and outputs the vibration signal 202 converted into a digital signal (hereinafter also referred to as the "vibration signal 202") to the adaptive filter 18, the detection unit 30, and the correlation determination unit 32.

[0019] The vibration signal 202 has a narrower frequency band and a large bias in frequency band compared to the audio signal 200. Due to this characteristic, the vibration signal 202 sounds very muffled and significantly different from the original audio signal 200. To improve this, in this embodiment, the audio signal 200 acquired at the same time is used as a reference signal, and the adaptive filter 18 brings the vibration signal 202 closer to the audio signal 200.

[0020] The adaptive filter 18 receives the vibration signal 202 from the second AD converter 16 and the residual signal 210 from the subtractor 20. The adaptive filter 18 performs a calculation on the vibration signal 202 using a filter coefficient based on the residual signal 210, and outputs the calculation result as a converted voice signal 212.

[0021] 2 shows a configuration of the adaptive filter 18 according to the first embodiment. The adaptive filter 18 includes a first delay unit 50a to an (N-1)-th delay unit 50n-1 collectively referred to as delay units 50, a first multiplier unit 52a to an N-th multiplier unit 52n collectively referred to as multipliers 52, a first adder unit 54a to an (N-1)-th adder unit 54n-1 collectively referred to as adders 54, and a coefficient update unit 60.

[0022] The delay unit 50 delays the vibration signal 202 and sequentially outputs it. The multiplier 52 multiplies the vibration signal 202 by a filter coefficient. The adder 54 sequentially adds up the multiplication results of the multiplier 52. The addition result of the (N-1)th adder 54n-1 corresponds to the above-mentioned calculation result. The subtractor 20 calculates a residual signal 210 which is a difference value between the audio signal 200 which is a reference signal and the converted audio signal 212 which is the calculation result, and outputs it to the adaptive filter 18.

[0023] The coefficient update unit 60 updates the filter coefficients based on the audio signal 200 and the calculation results by executing, for example, an LMS (Least Mean Square) algorithm as follows.

number

[0024] The DA converter 22 performs digital-to-analog conversion on the converted audio signal 212 output from the adaptive filter 18, and outputs the converted audio signal 212 converted into an analog signal (hereinafter referred to as the "output signal 214").

[0025] In order to improve the accuracy of the process for bringing the vibration signal 202 closer to the voice signal 200 in the adaptive filter 18, it is required to update (learn) the filter coefficient when the voice signal 200 in the speech section does not contain any voice components other than the target voice. For this process, the detection unit 30 detects whether the section is a speech section or a non-speech section. The detection unit 30 is called a VAD (Voice Activity Detection) and detects the speech section based on the vibration signal 202. For example, the detection unit 30 detects the speech section when the sound pressure level of the vibration signal 202 exceeds a predetermined level. The detection of the speech section is not limited to this. The detection unit 30 outputs VAD information 204 indicating whether the section is a speech section or a non-speech section to the correlation determination unit 32. The speech section is a time period in which the speech of the target person is detected.

[0026] Here, if voice components other than the target voice are mixed into the voice signal 200 acquired by the microphone 10, the adaptive filter 18 may mistakenly act to approximate the voice of other people. Voice components other than the target voice include, for example, the voice of a person other than the target person, and the voice reproduced by a speaker or the like at the other party's hand (not shown).

[0027] The seeds of sound signal 200 are generated by the vocal cords of the human body, and various sound signals 200 are generated by passing through the throat and mouth, which are sound tubes. However, there is a relationship between the vibration signal 202 associated with speech acquired at a specific part of the human body and the sound signal 200 acquired by air vibrations at a specific position by microphone 10, which involves a spatial delay when propagating through space and the transmission characteristics of the components of the human body (bones, flesh, skin, etc.).

number

[0028] When a predetermined spatial delay (z1) and gain correction amount are compensated for with respect to the reference vibration signal V(t) and the time axis and sound pressure level of the vibration signal 202 and the voice signal 200 are aligned, the signal values ​​are expected to be similar, although this depends on the type of speech. If there is only a speech signal from the subject, a point of high correlation can be observed near the time difference z1+z2 by performing a cross-correlation calculation between the vibration signal 202 and the voice signal 200. The general formula for calculating cross-correlation is as follows:

number

[0029] In equation (3), the point of highest correlation is observed as the maximum value Cmax of correlation value C. Here, τ corresponds to the time difference (z1+z2) between the audio signal series S(t) and the vibration signal series V(t), which have the same signal source. The maximum value Cmax, which is the correlation value C at the point of highest correlation between the vibration signal 202 and the audio signal 200, calculated using equation (3) is observed as the maximum product-sum value by determining the number of samples T according to the sampling frequency from a predetermined analysis time width, shifting several to a dozen samples before and after τ as the center, and performing a convolution operation.

[0030] The cross-correlation calculation may be done as follows:

number

[0031] On the other hand, when a voice of a person other than the target speaker or a noise component is mixed in, the correlation between the vibration signal 202 and the voice signal 200 decreases, and a low correlation value is observed in contrast. Therefore, the correlation determination unit 32 determines whether or not there is a multiple speech state during the time period when the target speaker speaks, depending on whether or not the correlation value is equal to or greater than a threshold value.

[0032] The correlation determination unit 32 detects a multiple speech state or a state in which the noise environment has deteriorated based on at least one of the maximum value Cmax of the correlation value C calculated by the above-mentioned formula (3) or the minimum value Cmin of the correlation value C calculated by the formula (4). FIG. 3 shows a configuration of the correlation determination unit 32 according to the first embodiment. The correlation determination unit 32 includes a threshold setting unit 70, a correlation value calculation unit 72, and a state determination unit 74. The threshold setting unit 70 receives VAD information 204 indicating whether the section is a speech section or a non-speech section from the detection unit 30. The threshold setting unit 70 uses a correlation value in a case where there is background noise without an external sound source and the section is a non-speech section as a reference, and sets a correlation value that is larger by a predetermined value than the reference correlation value as a first threshold 216, and sets a correlation value that is smaller by a predetermined value than the reference correlation value as a second threshold 218.

[0033] The correlation value calculation unit 72 calculates the correlation value 220 between the voice signal 200 and the vibration signal 202 by executing the above-mentioned formula (3) or formula (4). At this time, the sound pressure levels of the vibration signal 202 and the voice signal 200 may be aligned in advance in order to improve the accuracy of the correlation value 220. For example, a gain adjustment unit may be provided downstream of the first AD converter 12 in FIG. 2 so that the sound pressure levels of the voice signal 200 during speech are approximately the same, and adjustment may be performed in the correlation value calculation unit 72.

[0034] The state determination unit 74 receives the first threshold value 216 or the second threshold value 218 from the threshold value setting unit 70, and receives the correlation value 220 from the correlation value calculation unit 72. The state determination unit 74 compares the first threshold value 216 or the second threshold value 218 with the correlation value 220 to determine the state of target person speech, multiple speech, other person speech, or silence. The state determination unit 74 also determines the control content of the coefficient update unit 60 according to the determined state. Target person speech is a state in which only the target person speaks, multiple speech is a double talk state or a state in which external noise is mixed in, other person speech is a state in which only people other than the target person speak, and silence is a state in which there is no speech.

[0035] 4(a) and 4(b) show data structures of tables held in the state determination unit 74 according to the first embodiment. When the correlation value 220 is equal to or greater than the first threshold value 216 in an utterance section, the state determination unit 74 determines that the subject is speaking and determines the adaptive active of the coefficient update unit 60. When the correlation value 220 is less than the first threshold value 216 in an utterance section, the state determination unit 74 determines that the subject is speaking multiple times and determines the adaptive stop of the coefficient update unit 60. When the correlation value 220 is equal to or greater than the second threshold value 218 in a non-utterance section, the state determination unit 74 determines that the subject is silent and determines the adaptive save of the coefficient update unit 60. When the correlation value 220 is less than the second threshold value 218 in a non-utterance section, the state determination unit 74 determines that the subject is speaking from another person and determines the adaptive stop of the coefficient update unit 60.

[0036] Here, in adaptive active, "α" in equation (1) is set to a first value. The first value is determined in advance. In adaptive save, "α" in equation (1) is set to a second value. The second value is smaller than the first value. In adaptive stop, updating of the filter coefficients in coefficient update unit 60 is stopped. Return to FIG. 3.

[0037] The vibration signal 202 indicates the vibration of the human body during speech, but tends to have a higher noise level during background noise than the voice signal 200 picked up by the microphone 10. When determining the correlation between the vibration signal 202 and the voice signal 200, the first threshold value 216 or the second threshold value 218 may be set based on the correlation value 220 during background noise and non-speech periods when no external sound source exists. The transfer characteristic shown in formula (2) has a range of fluctuation depending on the type of speech, and even when only the speaker is speaking, the correlation value 220 may fluctuate accordingly. In particular, the correlation value 220 during speech may decrease depending on the positional relationship between the part of the human body that is used as the measurement point to pick up the vibration signal 202 during speech and the vocal cords, throat, and mouth that cause sound wave generation.

[0038] However, since the correlation value 220 obtained during background noise and in a non-speech section is calculated using noise components that have no correlation between the vibration signal 202 and the audio signal 200, during speech, the correlation value will not fall below this correlation value 220. Therefore, the threshold setting unit 70 detects that it is a non-speech section based on a signal indicating whether it is a speech section or a non-speech section, and further determines that the background noise state is one in which the correlation value 220 changes over a constant width for a predetermined period of time.

[0039] The threshold setting unit 70 acquires from the correlation value calculation unit 72 a predetermined number of correlation values ​​224 calculated during background noise and non-speech sections, and if the fluctuation range of the acquired predetermined number of correlation values ​​224 is within an acceptable range (within several times a predetermined value), calculates a reference correlation value by, for example, averaging the correlation values ​​224 acquired from the correlation value calculation unit 72 a predetermined value larger than the reference correlation value, and determines that the first threshold value 216 is a predetermined value larger than the reference correlation value, and that the second threshold value 218 is a predetermined value smaller than the reference correlation value.

[0040] FIG. 5 is a diagram showing the time change of the sound pressure level and the correlation value 220 in the microphone 10 and the vibration sensor 14 according to the first embodiment. FIG. 5(a) shows the time change of the vibration signal 202 and the voice signal 200, where the horizontal axis of FIG. 5(a) indicates time and the vertical axis indicates the sound pressure level. FIG. 5(b) shows the time change of the correlation value 220 calculated by the correlation value calculation unit 72, where the horizontal axis of FIG. 5(b) indicates time and the vertical axis indicates the correlation value 220. Since the subject is speaking during the period L1, the sound pressure levels of both the vibration signal 202 and the voice signal 200 fluctuate drastically during the period L1. Since the vibration signal 202 and the voice signal 200 have similar tendencies during the subject's speaking state, the correlation value 220 also increases. Since the period L2 is silent, the vibration signal 202 has an average background noise level and the voice signal 200 has an average background noise level during the period L2. Although these noise components have a certain correlation, they are uncorrelated with each other, so the correlation value 220 is small.

[0041] The period L3 is a state in which the subject is speaking, similar to the period L1. Therefore, the vibration signal 202, the voice signal 200, and the correlation value 220 in the period L3 have the same tendency as those in the period L1. The period L4 is a silent state, similar to the period L2. Therefore, the vibration signal 202, the voice signal 200, and the correlation value 220 in the period L4 have the same tendency as those in the period L2. The period L5 is a state in which someone else is speaking, so in the period L5, the sound pressure level of the vibration signal 202 fluctuates little, but the sound pressure level of the voice signal 200 fluctuates drastically. Therefore, the correlation value 220 in the period L5 drops significantly. The state in which someone else is speaking can also be said to be a state in which external noise is mixed in.

[0042] The period L6 is a state of multiple speech. Since the vibration signal 202 in the period L5 contains only the speech information of the target person himself, the part with rapid movement on the vertical axis is the speech signal of the target speaker. On the other hand, the audio signal 200 in the period L5 contains the speech information of the target person himself and the speech information of others. Therefore, the correlation value 220 in the period L5 decreases. Since the speech information of the target person is included during multiple speech, the correlation value 220 becomes higher than when others are speaking. The period L7 is a state of others' speech, similar to the period L5. Therefore, the vibration signal 202, audio signal 200, and correlation value 220 in the period L7 have the same tendency as those in the period L5.

[0043] The speech sections are periods L1, L3, and L6. In the speech sections, a first threshold value 216 shown in FIG. 5(b) is set to distinguish between a state in which the target person is speaking and a state in which multiple speeches are occurring. On the other hand, the non-speech sections are periods L2, L4, L5, and L7. In the non-speech sections, a second threshold value 218 shown in FIG. 5(b) is set to distinguish between a silent state and a state in which someone else is speaking. Return to FIG. 1.

[0044] The adaptive control unit 34 receives the judgment result (correlation information 206) in the correlation judgment unit 32. If the judgment result is adaptive active, the adaptive control unit 34 outputs an adaptive control signal 208 instructing the coefficient update unit 60 to set "α" to a first value, and if the judgment result is adaptive save, the adaptive control unit 34 outputs an adaptive control signal 208 instructing the coefficient update unit 60 to set "α" to a second value. If the judgment result is stop, the adaptive control unit 34 outputs an adaptive control signal 208 instructing the coefficient update unit 60 to stop updating the filter coefficient. In other words, the adaptive control unit 34 adjusts the degree of update of the filter coefficient in the coefficient update unit 60 based on the vibration signal 202 and the audio signal 200.

[0045] The above configuration can be realized in hardware terms by the CPU, memory, and other LSI of any computer, and in software terms by programs loaded into memory, but here we have depicted functional blocks realized by the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination of both.

[0046] The operation of the sound collecting device 100 having the above configuration will be described. Fig. 6 is a flowchart showing the procedure for updating the first threshold value 216 and the second threshold value 218 by the sound collecting device 100 according to the first embodiment. This shows the operation of setting the first threshold value 216 and the second threshold value 218 for the correlation value 220 of the vibration signal 202 and the audio signal 200, which is a criterion for determining whether or not there is a multiple speech state. This operation is performed in a state where there is no speech and no environmental noise, and the first threshold value 216 and the second threshold value 218 may be set by performing a series of operations before speech in advance, or the above state may be determined during speech and the first threshold value 216 and the second threshold value 218 may be set.

[0047] The vibration sensor 14 acquires the vibration signal 202 (S10). The detection unit 30 acquires the vibration signal 202 that has been analog-to-digital converted by the second AD converter 16 and determines whether or not it is a non-speech section (S12). If it is a non-speech section (Y in S12), the correlation determination unit 32 acquires the audio signal 200 acquired by the microphone 10, the audio signal 200 that has been digital-to-analog converted by the first AD converter 12, and the vibration signal 202 that has been analog-to-digital converted by the second AD converter 16 (S14), and the correlation value calculation unit 72 of the correlation determination unit 32 calculates the correlation value 220 based on the audio signal 200 and the vibration signal 202 a predetermined number of times according to a predetermined analysis time width and a sampling frequency (S16).

[0048] The threshold setting unit 70 of the correlation determination unit 32 acquires the correlation value 224 for a predetermined number of times calculated in step 16 from the correlation value calculation unit 72, and determines whether the fluctuation range of the correlation value is within an allowable range (S18). If the fluctuation range of the correlation value is within the allowable range (Y of S18), that is, if the background noise continues, the threshold setting unit 70 updates the first threshold value 216 and the second threshold value 218 based on the correlation value 224 for a predetermined number of times acquired from the correlation value calculation unit 72 (S20). If the fluctuation range of the correlation value is not within the allowable range (N of S18), step 20 is skipped. If it is not a non-speech section (N of S12), steps 14 to 20 are skipped. If the operation is not finished (N of S22), the process returns to step 10. If the operation is finished (Y of S22), the process is finished.

[0049] Fig. 7 is a flowchart showing a processing procedure by the sound collection device 100 according to the first embodiment. Step 50 is the same as the operation shown in Fig. 6, and here shows a case where a section in which background noise continues during use is detected and the first threshold value 216 and the second threshold value 218 are constantly updated. When the first threshold value 216 and the second threshold value 218 are set in advance in a situation in which background noise continues before an actual conversation starts, step 50 may be omitted.

[0050] The vibration sensor 14 acquires the vibration signal 202 (S52). The detection unit 30 acquires the vibration signal 202 that has been analog-to-digital converted by the second AD converter 16 and determines whether or not it is a speech section (S54). If it is a speech section (Y in S54), the correlation determination unit 32 acquires the audio signal 200 acquired by the microphone 10, the audio signal 200 that has been digital-to-analog converted by the first AD converter 12, and the vibration signal 202 that has been analog-to-digital converted by the second AD converter 16 (S56), and the correlation value calculation unit 72 of the correlation determination unit 32 calculates the correlation value 220 based on the audio signal 200 and the vibration signal 202 (S58).

[0051] The state determination unit 74 of the correlation determination unit 32 obtains the correlation value 220 calculated by the correlation value calculation unit 72 of the correlation determination unit 32 and the first threshold value 216 set by the threshold value setting unit 70 of the correlation determination unit 32, and compares the correlation value 220 with the first threshold value 216 (S60). If the correlation value 220 is not equal to or greater than the first threshold value 216 (N in S60), the state determination unit 74 of the correlation determination unit 32 estimates multiple utterances (S62) and determines adaptation stop (S64). If the correlation value 220 is equal to or greater than the first threshold value 216 (Y in S60), the state determination unit 74 of the correlation determination unit 32 estimates target utterances (S66) and determines adaptation active (S68).

[0052] If it is not a speech section (N in S54), the correlation determination unit 32 acquires the audio signal 200 by the microphone 10, acquires the audio signal 200 that has been digital-to-analog converted by the first AD converter 12, and acquires the vibration signal 202 that has been analog-to-digital converted by the second AD converter 16 (S69), and the correlation value calculation unit 72 of the correlation determination unit 32 calculates the correlation value 220 based on the audio signal 200 and the vibration signal 202 (S70).

[0053] The state determination unit 74 of the correlation determination unit 32 obtains the correlation value 220 calculated by the correlation value calculation unit 72 of the correlation determination unit 32 and the second threshold value 218 set by the threshold value setting unit 70 of the correlation determination unit 32, and compares the correlation value 220 with the second threshold value 218 (S72). If the correlation value 220 is not equal to or greater than the second threshold value 218 (N of S72), the state determination unit 74 of the correlation determination unit 32 estimates that the speech is from another person (S74) and determines to stop adaptation (S76). If the correlation value 220 is equal to or greater than the second threshold value 218 (Y of S72), the state determination unit 74 of the correlation determination unit 32 estimates that the speech is from another person (S78) and determines to stop adaptation (S80).

[0054] The adaptive filter 18 executes adaptive filter processing according to the control content determined by the state determination unit 74 of the correlation determination unit 32 (S82). If the operation has not ended (N in S84), the process returns to step 50. If the operation has ended (Y in S84), the process ends.

[0055] According to this embodiment, the influence of ambient noise components can be suppressed by approximating the characteristics of the vibration signal to those of the voice signal. Also, the presence of voices other than the speaker and sudden noise is detected sequentially from the correlation between the vibration signal and the voice signal, and the degree of approximation to the voice signal is controlled from the detection result, so that the influence of voices or noise components other than the target sound can be suppressed. Also, since the influence of voices or noise components other than the target sound is suppressed, the mixing of voice components other than the speaker can be avoided, and the voice quality of the speaker can be improved.

[0056] Example 2 Next, a second embodiment will be described. The second embodiment relates to a sound collection device 100, as in the first embodiment. The sound collection device 100 according to the first embodiment adjusts the degree of updating of the filter coefficient, that is, adjusts one of adaptive active, adaptive stop, and adaptive save, based on whether the section is a speech section or a non-speech section and on the correlation value 220 between the vibration signal 202 and the audio signal 200. On the other hand, the sound collection device 100 according to the second embodiment adjusts the frequency of updating the filter coefficient, based on whether the section is a speech section or a non-speech section and on the correlation value 220 between the vibration signal 202 and the audio signal 200. The sound collection device 100, adaptive filter 18, and detection unit 30 according to the second embodiment are of the same type as those shown in Figs. 1 to 3. Here, the differences from the first embodiment will be mainly described.

[0057] The state determination unit 74 receives the first threshold value 216 or the second threshold value 218 from the threshold value setting unit 70, and receives the correlation value 220 from the correlation value calculation unit 72. The state determination unit 74 determines the state of target person speech, multiple speech, other person speech, or silence by comparing the first threshold value 216 or the second threshold value 218 with the correlation value 220. The state determination unit 74 also determines the control content of the coefficient update unit 60 according to the determined state.

[0058] 8(a) and 8(b) show data structures of tables held in the state determination unit 74 according to the second embodiment. When the correlation value 220 is equal to or greater than the first threshold value 216 in an utterance period, the state determination unit 74 determines that the utterance is a target person's utterance, and determines that the coefficient update unit 60 updates with the first frequency. When the correlation value 220 is less than the first threshold value 216 in an utterance period, the state determination unit 74 determines that the utterance is multiple utterances, and determines that the coefficient update unit 60 updates with the second frequency. When the correlation value 220 is equal to or greater than the second threshold value 218 in a non-utterance period, the state determination unit 74 determines that the utterance is silence, and determines that the coefficient update unit 60 updates with the third frequency. When the correlation value 220 is less than the second threshold value 218 in a non-utterance period, the state determination unit 74 determines that the utterance is another person's utterance, and determines that the coefficient update unit 60 updates with the second frequency.

[0059] Here, the second frequency is smaller than the first frequency, and the third frequency is smaller than the first frequency and larger than the second frequency. For example, the first frequency is every time, the second frequency is once in 128 times, and the third frequency is between once in two times and once in 64 times. The values ​​of each frequency are not limited to these. Return to FIG. 3.

[0060] The adaptive control unit 34 receives the determination result from the correlation determination unit 32. If the determination result is update at the first frequency, the adaptive control unit 34 instructs the coefficient update unit 60 to update the filter coefficients at the first frequency, and if the determination result is update at the second frequency, the adaptive control unit 34 instructs the coefficient update unit 60 to update the filter coefficients at the second frequency. If the determination result is update at the third frequency, the adaptive control unit 34 instructs the coefficient update unit 60 to update the filter coefficients at the third frequency. In other words, the adaptive control unit 34 adjusts the frequency of update of the filter coefficients in the coefficient update unit 60 based on the vibration signal 202 and the audio signal 200.

[0061] Fig. 9 is a flowchart showing a processing procedure by the sound collection device 100 according to the second embodiment. Step 100 is the same as the operation shown in Fig. 6, and here shows a case where a section in which background noise continues during use is detected and the first threshold value 216 and the second threshold value 218 are constantly updated. When the first threshold value 216 and the second threshold value 218 are set in advance in a situation in which background noise continues before an actual conversation starts, step 100 may be omitted.

[0062] The vibration sensor 14 acquires the vibration signal 202 (S102). The detection unit 30 acquires the vibration signal 202 that has been analog-to-digital converted by the second AD converter 16 and determines whether or not it is a non-speech section (S104). If it is a speech section (Y in S104), the correlation determination unit 32 acquires the audio signal 200 acquired by the microphone 10, the audio signal 200 that has been digital-to-analog converted by the first AD converter 12, and the vibration signal 202 that has been analog-to-digital converted by the second AD converter 16 (S106), and the correlation value calculation unit 72 of the correlation determination unit 32 calculates the correlation value 220 based on the audio signal 200 and the vibration signal 202 (S108).

[0063] The state determination unit 74 of the correlation determination unit 32 obtains the correlation value 220 calculated by the correlation value calculation unit 72 of the correlation determination unit 32 and the first threshold value 216 set by the threshold value setting unit 70 of the correlation determination unit 32, and compares the correlation value 220 with the first threshold value 216 (S110). If the correlation value 220 is not equal to or greater than the first threshold value 216 (N in S110), the state determination unit 74 of the correlation determination unit 32 estimates that there are multiple utterances (S112) and determines to update at the second frequency (S114). If the correlation value 220 is equal to or greater than the first threshold value 216 (Y in S110), the state determination unit 74 of the correlation determination unit 32 estimates that there are target utterances (S116) and determines to update at the first frequency (S118).

[0064] If it is not a speech section (N in S104), the correlation determination unit 32 acquires the audio signal 200 by the microphone 10, acquires the audio signal 200 that has been digital-to-analog converted by the first AD converter 12, and acquires the vibration signal 202 that has been analog-to-digital converted by the second AD converter 16 (S119), and the correlation value calculation unit 72 of the correlation determination unit 32 calculates the correlation value 220 (S120).

[0065] The state determination unit 74 of the correlation determination unit 32 obtains the correlation value 220 calculated by the correlation value calculation unit 72 of the correlation determination unit 32 and the second threshold value 218 set by the threshold value setting unit 70 of the correlation determination unit 32, and compares the correlation value 220 with the second threshold value 218 (S122). If the correlation value 220 is not equal to or greater than the second threshold value 218 (N in S122), the state determination unit 74 of the correlation determination unit 32 estimates that the speech is from another person (S124) and determines to update at the second frequency (S126). If the correlation value 220 is equal to or greater than the second threshold value 218 (Y in S122), the state determination unit 74 of the correlation determination unit 32 estimates that the speech is from another person (S128) and determines to update at the third frequency (S130).

[0066] The adaptive filter 18 executes adaptive filter processing according to the control content determined by the state determination unit 74 of the correlation determination unit 32 (S132). If the operation has not ended (N in S134), the process returns to step 100. If the operation has ended (Y in S134), the process ends.

[0067] According to this embodiment, the frequency of updating the filter coefficient is adjusted based on the correlation value between the vibration signal and the audio signal, so that the influence of audio or noise components other than the target sound can be suppressed. Furthermore, when the correlation value is equal to or greater than the first threshold value in the speech section, the filter coefficient is updated at a first frequency equal to or greater than the second frequency, so that the filter coefficient can be updated to be closer to the audio signal. Furthermore, when the correlation value is smaller than the first threshold value in the speech section, the filter coefficient is updated at a second frequency smaller than the first frequency, so that the influence of audio or noise components other than the target sound can be suppressed. Furthermore, when the correlation value is equal to or greater than the second threshold value in the non-speech section, the filter coefficient is updated at a third frequency smaller than the first frequency, so that the influence of noise components can be suppressed. Furthermore, when the correlation value is smaller than the second threshold value in the non-speech section, the filter coefficient is updated at a second frequency smaller than the first frequency, so that the influence of audio or noise components other than the target sound can be suppressed.

[0068] The present invention has been described above based on the embodiments. These embodiments are merely illustrative, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0069] 10 microphone, 12 first AD converter, 14 vibration sensor, 16 second AD converter, 18 adaptive filter, 20 subtractor, 22 DA converter, 30 detection unit, 32 correlation determination unit, 34 adaptive control unit, 50 delay unit, 52 multiplier, 54 adder, 60 coefficient update unit, 70 threshold setting unit, 72 correlation value calculation unit, 74 state determination unit, 100 sound collection device, 200 audio signal, 202 vibration signal, 204 VAD information, 206 correlation information, 208 adaptive control signal, 210 residual signal, 212 converted audio signal, 214 output signal, 216 first threshold, 218 second threshold, 220 correlation value, 224 correlation value for a predetermined number of times.

Claims

1. an adaptive filter that performs a calculation using a filter coefficient on a vibration signal acquired by a vibration sensor and outputs a converted audio signal; a coefficient update unit that updates the filter coefficients based on a residual signal that is a difference between a voice signal acquired by a microphone and the converted voice signal; an adaptive control unit that adjusts a frequency of updating the filter coefficients in the coefficient updating unit based on the vibration signal and the audio signal; A sound recording device comprising:

2. A detection unit that detects a speech period based on the vibration signal; a correlation value calculation unit that calculates a correlation value between the audio signal and the vibration signal, the adaptive control unit detects the speech period by the detection unit and, when the correlation value calculated by the correlation value calculation unit is equal to or greater than a first threshold value, updates the filter coefficient with a first frequency; the adaptive control unit detects the speech period by the detection unit and, when the correlation value calculated by the correlation value calculation unit is smaller than a first threshold value, updates the filter coefficient with a second frequency; The sound pickup device according to claim 1 , wherein the second frequency is smaller than the first frequency.

3. the adaptive control unit updates the filter coefficients at a third frequency when the detection unit does not detect the speech period and the correlation value calculated by the correlation value calculation unit is equal to or greater than a second threshold value; and the adaptive control unit updates the filter coefficients with a second frequency when the detection unit does not detect the speech period and the correlation value calculated by the correlation value calculation unit is smaller than a second threshold value; the second threshold is less than the first threshold, The sound collection device according to claim 2 , wherein the third frequency is smaller than the first frequency and larger than the second frequency.

4. A step of performing a calculation using a filter coefficient on a vibration signal acquired by the vibration sensor and outputting a converted voice signal; updating the filter coefficients based on a residual signal that is a difference between a voice signal acquired by a microphone and the converted voice signal; adjusting a frequency of updating the filter coefficient based on the vibration signal and the audio signal; A sound collection method comprising:

5. On the computer, A step of performing a calculation using a filter coefficient on a vibration signal acquired by the vibration sensor and outputting a converted voice signal; updating the filter coefficients based on a residual signal that is a difference between a voice signal acquired by a microphone and the converted voice signal; and a step of adjusting an update frequency of the filter coefficient based on the vibration signal and the audio signal.

Images