Audio processing device, audio processing program, and audio processing method

Abstract

There is a demand for a voice processing device that can prevent the inclusion of echoes while supporting voice output processing to a user and voice collection processing of the user. [Solution] The audio processing device of the present invention has a signal mixing means that adjusts a mixing signal including at least an input signal to a predetermined volume level, performs signal mixing processing to mix the signal with a target area sound extraction sound, and generates and outputs a mixed target area sound generated by the signal mixing processing, wherein the signal mixing means is means for limiting the signal mixing processing at least in a specific section where the output audio signal is a sound section, the input signal is a signal input from a plurality of microphone arrays, the output audio signal is a signal related to audio output from a speaker, and the target area sound extraction sound is sound whose sound source is the target area in the input signal.

Description

[Technical Field]

[0001] The present invention relates to an audio processing device, an audio processing program, and an audio processing method, and can be applied to, for example, processing that emphasizes sound in a specific area and suppresses sound in other areas. [Background technology]

[0002] A beamformer (hereinafter referred to as "BF") using a microphone array is a technology for separating and collecting only sounds from a specific direction in an environment where multiple sound sources are present. BF is a technology for forming directivity by utilizing the time difference between signals arriving at each microphone (see Non-Patent Document 1). BF is broadly divided into two types: additive and subtractive. Subtractive BF, in particular, has the advantage of being able to form directivity with fewer microphones than additive BF. Figure 8 is a block diagram showing the configuration of a sound collection device PS that employs a conventional subtractive BF with two microphones. The sound collection device PS that employs the conventional subtractive BF first calculates the time difference between the signals of a sound present in a target direction (hereinafter referred to as "target sound") arriving at microphones M1 and M2 using a delay device, and then aligns the phase of the target sound by adding a delay.

[0003] In the sound collection device PS, the time difference is calculated by the following formula (1): In formula (1), d is the distance between the microphones, c is the speed of sound, and τ L is the amount of delay. Also, in equation (1), θ L is the angle from the perpendicular direction to the line connecting each microphone to the target direction. τ L =(dsinθ L ) / c …(1)

[0004] Here, when a blind spot exists in the direction of microphone M1 with respect to the center of microphones M1 and M2, sound collection device PS performs delay processing on the input signal χ1(t) of microphone M1. After that, sound collection device PS performs signal processing using a subtractor according to equation (2). m(t)=x2(t)-x1(t-τ L )…(2)

[0005] In the sound collection device PS, the subtraction process can be performed in the frequency domain as well, in which case equation (2) is changed to the following equation (3).

number

[0006] θ L When θ = ±π / 2, the directivity formed by the sound collection device PS becomes a cardioid type unidirectional pattern as shown in FIG. L = 0, π, the directivity formed by the sound collection device PS is a figure-8 shaped bidirectional pattern as shown in Figure 9(B). Note that hereinafter, a filter that forms a unidirectional pattern from an input signal will be called a "unidirectional filter," and a filter that forms a bidirectional pattern will be called a "bidirectional filter."

[0007] In addition, the sound collection device PS can use spectral subtraction (SS) to create directivity that is resistant to the blind spots of the bidirectional signal. The SS directivity is created across all frequencies or a specified frequency band according to equation (4). While equation (4) uses the input signal X1 from microphone M1, a similar effect can be achieved with the input signal X2 from microphone M2. In equation (4), β is a coefficient for adjusting the strength of the SS. If the SS processing (subtraction process) results in a negative value, the sound collection device PS performs a flooring process, replacing it with 0 or a smaller value. By applying SS processing, the sound collection device PS can extract sounds that exist outside the target direction (hereinafter referred to as "non-target sounds") using a bidirectional filter, and then subtract the amplitude spectrum of the extracted non-target sounds from the amplitude spectrum of the input signal to emphasize the target sounds. Y(n)=X1(n)-βM(n) …(4)

[0008] In a conventional sound collection device PS, if you want to collect only sounds that exist within a specific area (hereinafter referred to as "target area sounds"), simply using a subtractive BF may result in collecting sound sources that exist around that area (hereinafter referred to as non-target area sounds).

[0009] Therefore, Patent Document 1 proposes an area sound collection device that uses multiple microphone arrays, each directing its directivity toward a target area from a different direction and having the directivities intersect at the target area to collect target area sound. The area sound collection device described in Patent Document 1 first estimates the ratio of the target area sound power contained in the BF output of each microphone array, and uses this as a correction coefficient. In the area sound collection device described in Patent Document 1, when two microphone arrays are used as an example, the correction coefficient for the target area sound power is calculated using the following equations (5) and (6), or the following equations (7) and (8).

number

[0010] In equations (5) to (8), Y 1κ (n), Y 2κ (n) represents the amplitude spectrum of the BF output of the first and second microphone arrays, N represents the total number of frequency bins, κ represents frequency, and α1(n) and α2(n) represent power correction coefficients for each BF output. In equations (5) to (8), mode represents the mode, and median represents the median.

[0011] The sound collection device described in Patent Document 1 then corrects each BF output using a correction coefficient and performs SS to extract non-target area sounds present in the direction of the target area. The sound collection device described in Patent Document 1 can further extract the target area sound by SS of the extracted non-target area sounds from the output of each BF. When extracting non-target area sounds N1(n) present in the direction of the target area as seen from the first microphone array, the sound collection device described in Patent Document 1 performs SS by multiplying the BF output Y1(n) of the first microphone array by the BF output Y2(n) of the second microphone array by a power correction coefficient α2, as shown in the following equation (9). Furthermore, when extracting non-target area sounds N2(n) present in the direction of the target area as seen from the second microphone array, the sound collection device described in Patent Document 1 performs calculation according to equation (10) to extract the non-target area sounds N2(n). N1(n)=Y1(n)-α2(n)Y2(n) …(9) N2(n)=Y2(n)-α1(n)Y1(n) …(10)

[0012] Thereafter, the sound collection device described in Patent Document 1 extracts the target area sound by SS of the non-target area sound from each BF output in accordance with equations (11) and (12). In equations (11) and (12), γ1(n) and γ2(n) are coefficients for changing the intensity during SS. Z1(n)=Y1(n)-γ1(n)N1(n) …(11) Z2(n)=Y2(n)-γ2(n)N2(n) …(12)

[0013] However, in the method of Patent Document 1, when the volume level of background noise or non-target area sounds is high, the SS performed when extracting the target area sound may distort the target area sound or generate an unpleasant noise called musical noise. With the method of Patent Document 1, these effects may make it difficult to hear the sound, which may hinder smooth communication via sound.

[0014] Therefore, Patent Document 2 discloses a sound collection method in which an input signal from a microphone array (an input signal from one of the microphones) and an estimated noise component are mixed with the target area sound extracted by the area sound collection process in order to suppress the distortion and musical noise of the target area sound as described above in area sound collection processing using the BF output of a microphone array. The method described in Patent Document 2 reduces distortion and musical noise of the target area sound by adjusting the volume levels of the microphone input signal and the estimated noise, respectively, and mixing them with the extracted target area sound.

[0015] Furthermore, a conventional device that applies area sound collection processing using the BF output of a microphone array is described in Patent Document 3. Patent Document 3 describes a sound processing device that supports sound output processing (sound output function) that outputs sound (for example, voice guidance or the sound of a far-end talker during a call) from a speaker to a user, and sound collection processing (sound collection function) that collects sound spoken by a user (near-end talker). The sound processing device described in Patent Document 3 performs processing to collect the user's spoken sound without collecting sound from the speaker as a disturbing sound (non-target area sound) based on the relative positions of two microphone arrays, speakers, and users.

[0016] Furthermore, Patent Document 4 discloses a technique that uses the amplitude spectrum ratio of the area sound collection output and the input sound as a method for determining whether or not a target area sound (voice) exists within the sound collection area. [Prior art documents] [Patent documents]

[0017] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-72708 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-183902 [Patent Document 3] Patent Publication No. 2021-033030 [Patent Document 4] Japanese Patent Application Laid-Open No. 2016-127457 [Patent Document 5] Patent Publication No. 2021-136528 [Non-patent literature]

[0018] [Non-Patent Document 1] Futoshi Asano, "Acoustic Technology Series 16: Array Signal Processing of Sound - Localization, Tracking and Separation of Sound Sources", edited by the Acoustical Society of Japan, Corona Publishing, published February 25, 2011 Summary of the Invention [Problem to be solved by the invention]

[0019] However, in the audio processing device (device capable of audio output processing and audio collection processing) described in Patent Document 3, even if area collection processing is performed on the user's speech using the method described in Patent Document 4, it is difficult to collect the target sound without interruption and to prevent echo contamination.

[0020] In other words, in the technology described in Patent Document 4, if the threshold for determining whether or not an area sound is present is set as a parameter so that the sound is not interrupted in sections where the target sound is small, the echo may be erroneously determined to be present as the target sound, and the echo may be mixed into the sound processed by the area sound pickup microphone.

[0021] Furthermore, when the technology described in Patent Document 2 is used in the audio processing device described in Patent Document 3 to improve the sound quality of the picked-up target area sound, if the area sound determination unit erroneously determines that the echo is the target sound, the echo will become even louder as a result of the mixing process.

[0022] In view of the above problems, there is a need for an audio processing device, an audio processing program, and an audio processing method that can prevent the inclusion of echoes while supporting audio output processing to a user and audio collection processing of the user. [Means for solving the problem]

[0023] The first invention of the present invention comprises a signal mixing means that adjusts a mixing signal including at least an input signal to a predetermined volume level, performs signal mixing processing to mix the signal with the target area sound extraction sound, and generates and outputs a mixed target area sound generated by the signal mixing processing, wherein the signal mixing means is means for limiting the signal mixing processing at least in a specific section where an output audio signal is a sound section, the input signal is a signal input from a plurality of microphone arrays, the output audio signal is a signal related to audio output from a speaker, and the target area sound extraction sound is sound whose sound source is the target area in the input signal.

[0024] A second audio processing program of the present invention causes a computer to function as signal mixing means that adjusts a mixing signal including at least an input signal to a predetermined volume level, performs signal mixing processing to mix the signal with a target area sound extraction sound, and generates and outputs a mixed target area sound generated by the signal mixing processing, and causes the signal mixing means to function as means for limiting the signal mixing processing at least in a specific section where the output audio signal is a sound section, wherein the input signal is a signal input from a plurality of microphone arrays, the output audio signal is a signal related to audio output from a speaker, and the target area sound extraction sound is sound whose sound source is the target area in the input signal.

[0025] The third aspect of the present invention is an audio processing method for use in an audio processing device, the audio processing device having a signal mixing means, the signal mixing means adjusting a mixing signal including at least an input signal to a predetermined volume level and performing signal mixing processing to mix the signal with a target area sound extraction sound, generating and outputting a mixed target area sound generated by the signal mixing processing, the signal mixing means being means for limiting the signal mixing processing at least in a specific section where the output audio signal is a sound-containing section, the input signal being a signal input from a plurality of microphone arrays, the output audio signal being a signal related to audio output from a speaker, and the target area sound extraction sound being sound whose sound source is a target area in the input signal. [Effects of the Invention]

[0026] According to the present invention, it is possible to prevent the inclusion of echoes while supporting the process of outputting audio to a user and the process of collecting audio from the user. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a block diagram showing a functional configuration of a voice processing device according to an embodiment. [Figure 2] 1 is a perspective view of the appearance of a sound processing device according to an embodiment; [Figure 3] 3 is a diagram showing processing rules for each parameter related to hangover processing in the voice processing device according to the embodiment. FIG. [Figure 4] 10 is a timing chart showing an example of a hangover process in the audio processing device according to the embodiment. [Figure 5] FIG. 2 is an explanatory diagram showing a specific example (part 1) of the operation of the audio processing device according to the embodiment. [Figure 6] FIG. 10 is an explanatory diagram showing a specific example (part 2) of the operation of the audio processing device according to the embodiment. [Figure 7] FIG. 10 is an explanatory diagram showing a specific example (part 3) of the operation of the audio processing device according to the embodiment. [Figure 8] FIG. 1 is a block diagram showing the configuration of a conventional sound collection device. [Figure 9] FIG. 1 is an explanatory diagram illustrating an example of a directional characteristic formed by a conventional directional filter. DETAILED DESCRIPTION OF THE INVENTION

[0028] (A) Main embodiment Hereinafter, an embodiment of a voice processing device, a voice processing program, and a voice processing method according to the present invention will be described in detail with reference to the drawings.

[0029] (A-1) Configuration of the embodiment FIG. 2 is a perspective view of the appearance of the voice processing device 100 of this embodiment.

[0030] As shown in FIG. 2, the voice processing device 100 is a device including two microphone arrays MA (MA1, MA2), a speaker SP, and a touch panel display D. The voice processing device 100 is a device that supports voice output processing (voice output function) for outputting voice (e.g., voice guidance or the voice of a far-end talker during a call) from the speaker SP to a user, and voice collection processing (voice collection function) for collecting the user's voice using the microphone arrays MA1, MA2. The voice processing device 100 of this embodiment is also equipped with a touch panel display D that can output information to a user and accept information input from the user. For example, a device such as the user-operated terminal described in Patent Document 3 can be applied as the voice processing device 100 of this embodiment. In other words, the voice processing device 100 of this embodiment can have a hardware configuration similar to that of the user-operated terminal of Patent Document 3. Note that the touch panel display D is not an essential component in the voice processing device of this embodiment, and may be omitted.

[0031] For example, the voice processing device 100 may be configured as a device that inputs and outputs information only by voice. The use of the voice processing device 100 is not limited, but it may be configured as, for example, a transaction device (vending machine) that performs transactions such as the sale of various products. In this case, the voice processing device 100 may be configured to be able to perform transactions by receiving voice input from a user (voice collection processing) and outputting voice guidance (voice output processing).

[0032] The microphone arrays MA1 and MA2 are placed at any location (in this embodiment, the surface of the audio processing device 100 facing the user) in a space where a target area (in this embodiment, the position where the user is present) exists. The positions of the microphone arrays MA1 and MA2 relative to the target area may be anywhere as long as their directivities overlap only in the target area. Each microphone array MA is composed of two or more microphones M, and each microphone M collects an acoustic signal. In this embodiment, it is described that each microphone array MA is provided with two microphones M (M1, M2) that collect acoustic signals. In other words, each microphone array MA constitutes a 2-channel microphone array. Note that the number of microphone arrays MA is not limited to two; if there are multiple target areas, it is necessary to place a number of microphone arrays MA that can cover all of the areas.

[0033] Furthermore, although there are no limitations on the relative positions of the microphone arrays MA1, MA2 and the speaker SP, this embodiment will be described assuming that they are arranged as shown in FIG. 2. This embodiment will be described assuming that a user using the sound processing device 100 is located facing the touch panel display D. In FIG. 2, the microphone arrays MA1, MA2 are arranged on the left and right sides of the touch panel display D, respectively, as seen from the user facing the touch panel display D. Also, as shown in FIG. 2, the microphone arrays MA1, MA2 are arranged at the same vertical position (height). Also, in FIG. 2, the speaker SP is arranged on the right side as seen from the user. Note that, in the sound processing device 100 of this embodiment, the arrangement positions and orientations of the microphone arrays MA1, MA2 may be similar to those of Patent Document 3 (for example, the same arrangement as that of FIG. 5 of Patent Document 3).

[0034] As described above, the audio processing device 100 uses two microphone arrays MA (MA1, MA2) to perform audio collection processing (target area audio collection processing) for collecting target area audio from a sound source in the target area, and audio output processing from the speaker SP.

[0035] FIG. 1 is a block diagram showing the functional configuration of a voice processing device 100 according to this embodiment.

[0036] The audio processing device 100 has a signal input unit 101, a noise suppression unit 102, a directivity forming unit 103, a delay correction unit 104, spatial coordinate data 105, a correction coefficient calculation unit 106, a target area sound extraction unit 107, a mixture level calculation unit 108, a mixture level adjustment unit 109, a signal mixing unit 110, an amplitude spectrum ratio calculation unit 111, an area sound determination unit 112, a selection unit 113, a control unit 114, an audio output unit 115, a sound activity period detection unit 116, and a parameter control unit 117. Detailed processing of each functional block constituting the audio processing device 100 will be described later.

[0037] The audio processing device 100 may be configured entirely of hardware (for example, a dedicated chip, etc.), or may be configured partially or entirely as software (a program). The audio processing device 100 may be configured, for example, by installing a program (including the audio processing program of the embodiment) in a computer having a processor and memory in addition to input / output devices (microphone arrays MA1, MA2, a display D, a speaker SP, etc.).

[0038] For signal input section 101, noise suppression section 102, directivity forming section 103, delay correction section 104, spatial coordinate data 105, correction coefficient calculation section 106, and target area sound extraction section 107, it is possible to apply configurations similar to the corresponding configurations in Patent Document 5, for example. Furthermore, for mixture level calculation section 108, mixture level adjustment section 109, and signal mixer 110, it is possible to apply configurations similar to the corresponding configurations in Patent Document 2. Furthermore, for amplitude spectrum ratio calculation section 111 and area sound determination section 112, it is possible to apply configurations similar to the corresponding configurations in Patent Document 4.

[0039] (A-2) Operation of the embodiment Next, the operation of the voice processing device 100 of this embodiment having the above configuration will be described.

[0040] The signal input unit 101 converts the acoustic signals collected by the microphone arrays MA1 and MA2 from analog signals to digital signals and inputs them, then converts them from the time domain to the frequency domain using, for example, fast Fourier transform.

[0041] The noise suppression unit 102 estimates and suppresses background noise components contained in the signal acquired by the signal input unit 101. For the noise suppression processing in the noise suppression unit 102, for example, SS or Wiener filtering can be used.

[0042] The directivity forming unit 103 extracts (e.g., extracts using a bidirectional filter) non-target area sound present in a direction other than the target direction for each microphone array MA, and subtracts the amplitude spectrum of the extracted non-target area sound from the amplitude spectrum of the input signal to obtain sound (BF output) with directivity formed in the target area direction. Specifically, the directivity forming unit 103 obtains, as a BF output, sound with directivity formed in the target area direction by BF according to equation (4) for a signal in which background noise has been suppressed by the noise suppression unit 102 for each microphone array MA. Therefore, in this embodiment, the directivity forming unit 103 obtains, for each microphone array MA, a BF output with directivity formed in the target area direction, and also holds non-target area sound with directivity formed in a direction other than the target area direction, which is obtained in the process of obtaining the BF output. Note that the specific calculation processing method used by the directivity forming unit 103 to obtain the BF output and the non-target area sound with directivity formed in a direction other than the target area direction is not limited.

[0043] The delay correction unit 104 calculates and corrects delays that occur due to differences in distance between the target area and each microphone array. First, the position of the target area and the position of each microphone array MA are obtained from the spatial coordinate data 105, and the difference in arrival time of the target area sound to each microphone array MA is calculated. Next, using the microphone array MA located farthest from the target area as a reference, a delay is added so that the target area sound arrives at all microphone arrays MA at the same time.

[0044] The spatial coordinate data 105 holds the position information of all target areas and the position information of each microphone array MA.

[0045] The correction coefficient calculation unit 106 calculates correction coefficients for equalizing the power of the target area sound components included in each BF output according to equations (5) and (6) or equations (7) and (8).

[0046] The target area sound extraction unit 107 performs SS on each BF output data corrected by the correction coefficient calculated by the correction coefficient calculation unit 106 according to equation (9) or (10) to extract non-target area sounds present in the direction of the target area. Furthermore, the target area sound extraction unit 107 extracts the target area sound by SSing the extracted non-target area sounds from the output of each BF according to equation (11) or (12). Hereinafter, the signal extracted by the target area sound extraction unit 107 will be referred to as the "target area sound extraction signal."

[0047] The mixed level calculation unit 108 calculates the power of the estimated noise estimated by the noise suppression unit 102, the non-target area sound other than the target area direction extracted by the directivity formation unit 103, and the non-target area sound in the target area sound direction extracted by the target area sound extraction unit 107, and determines the volume level (total volume level) of the mixing signal (in this embodiment, the input signal and background noise) to be mixed into the target area sound extraction signal from the magnitude of the total value of these. Note that the processing of the mixed level calculation unit 108 can be similar to that of the mixed level calculation unit of Patent Document 2, for example.

[0048] The mixing level adjustment unit 109 adjusts the volume levels of the input signal and estimated noise to be mixed into the target area sound extraction signal, based on the volume level of the mixing signal calculated by the mixing level calculation unit 108 and the ratio of the power of the estimated noise to the non-target area sound. Note that the volume level adjustment process in the mixing level adjustment unit 109 can be performed, for example, by a process similar to that of the mixing level adjustment unit of Patent Document 2. In this embodiment, like the mixing level adjustment unit of Patent Document 2, the mixing level adjustment unit 109 calculates a variable λ1 that determines the ratio of the input signal and estimated noise to be mixed into the target area sound extraction signal, and a variable μ1 (hereinafter referred to as the “mixing level adjustment variable”) that satisfies the volume level calculated by the mixing level calculation unit 108, and supplies these to the signal mixing unit 110.

[0049] The signal mixing unit 110 performs a process of mixing (adding) the target area sound extraction signal with a mixing signal. Note that the signal mixing process in the signal mixing unit 110 can be, for example, the same process as that of the signal mixing unit of Patent Document 2. As described above, in this embodiment, the mixing signal includes the input signal acquired by the signal input unit 101 and the noise estimated by the noise suppression unit 102, similar to the signal mixing unit of Patent Document 2. Specifically, the signal mixing unit 110 acquires a volume-adjusted mixing signal by multiplying a signal obtained by mixing (adding) the input signal acquired by the signal input unit 101 with the noise estimated by the noise suppression unit 102 at a ratio λ1 by a mixing level adjustment variable μ1 that satisfies the volume level calculated by the mixing level adjustment unit 109. The signal mixing unit 110 then mixes (adds) the acquired volume-adjusted mixing signal with the target area sound extraction signal.

[0050] The signal mixing unit 110 supplies the signal resulting from the mixing as a signal resulting from the sound collection (hereinafter referred to as a "collected sound signal") to the selection unit 113. Furthermore, the format and output means of the output signal of the collected sound signal by the signal mixing unit 110 are not limited, and various configurations can be applied.

[0051] As described above, in the audio processing device 100, the mixing level calculation unit 108, the mixing level adjustment unit 109, and the signal mixing unit 110 constitute a signal mixing means that performs signal mixing processing to mix a mixing signal (a signal including an input signal and an estimated noise) with a target area sound extraction sound.

[0052] On the other hand, the amplitude spectral ratio calculation unit 111 acquires the input signal from the signal input unit 101 and the area sound outputs Z1 and Z2 from the target area sound extraction unit 107, and calculates the amplitude spectral ratio. For example, the amplitude spectral ratio calculation unit 111 calculates the amplitude spectral ratio of the area sound outputs Z1 and Z2 and the input signal for each frequency using the following equations (13) and (14). Then, using the following equations (15) and (16), the amplitude spectral ratios of all frequencies are added together to find the amplitude spectral ratio countable value. Here, in equations (13) and (14), W x1 is the amplitude spectrum of the input signal of the microphone array MA1, and W x2 is the amplitude spectrum of the input signal of the microphone array MA2. Also, Z1 is the amplitude spectrum of the area sound output when area sound collection processing is performed mainly using the microphone array MA1 (when the above-mentioned equation (11) is used), and Z2 is the amplitude spectrum of the area sound output when area sound collection processing is performed mainly using the microphone array MA2 (when the above-mentioned equation (12) is used). Furthermore, U1 obtained in the processing performed using equation (15) is the amplitude spectrum ratio R 1i is summed over the frequency band from the lower limit m to the upper limit n, and U2, which is performed using the processing of equation (16), is the amplitude spectrum ratio R 2i are summed over a frequency band from the lower limit m to the upper limit n. Here, the frequency band to be calculated in the amplitude spectral ratio calculation unit 111 may be limited. For example, the calculation may be performed by limiting the range to be calculated to 100 Hz to 6 kHz, which is a range that sufficiently contains audio information.

[0053] The amplitude spectral ratio is calculated using equation (13) or equation (14) depending on the microphone array MA used as the main in the area sound collection processing. Furthermore, the amplitude spectral ratio is added using equation (15) or equation (16) depending on the microphone array MA used as the main in the area sound collection processing. Specifically, when microphone array MA1 is used as the main in the area sound collection processing, equations (13) and (15) are used, and when microphone array MA2 is used, equations (14) and (16) are used.

number

[0054] Then, the amplitude spectrum ratio calculation unit 111 supplies the calculation result (U1 or U2) to the area sound determination unit 112.

[0055] The area sound determination unit 112 compares the amplitude spectral ratio sum value (U1 or U2) supplied from the amplitude spectral ratio calculation unit 111 with a preset threshold TH (threshold TH1 or threshold TH2) to determine whether or not a target area sound (sound) is present within the target area. Then, the area sound determination unit 112 supplies the determination result to the selection unit 113.

[0056] For example, if the area sound determination unit 112 detects that a target area sound is present within the target area, it outputs "true" as the determination result, and if it detects that a target area sound is not present within the target area, it outputs "false" as the determination result.

[0057] When “true” is output from the area sound determination unit 112, the selection unit 113 selects the collected signal from the signal mixing unit 110, and when “false” is output from the area sound determination unit 112, the selection unit 113 selects silence and supplies it to the control unit 114.

[0058] The control unit 114 is responsible for the overall control of the voice processing device 100. In the voice processing device 100, the control unit 114 is assumed to perform control processing related to voice collection processing and voice output processing. The control unit 114 may also perform information output processing from the touch panel display D and information input reception processing from the touch panel display D. For example, when the voice processing device 100 is configured as a transaction device, the control unit 114 is assumed to have the function of managing transactions (i.e., transactions using voice input and output) using voice collection processing (voice collection processing based on input signals from the microphone arrays MA1 and MA2) and voice output processing (voice output processing from the speaker SP). In this case, the control unit 114 acquires the contents of a user's input by performing voice recognition processing on the collected signal acquired by the voice collection processing, and outputs voice guidance by the voice output processing to guide the user regarding the transaction (e.g., guidance processing for voice input to the user).

[0059] In this embodiment, the control unit 114 will be described as supplying a digital signal of a sound (e.g., a voice guidance) to the sound output unit 115 to be output from the speaker SP. The specific content of the sound signal (digital signal) supplied by the control unit 114 to the sound output unit 115 is not limited. For example, the control unit 114 may synthesize a sound signal based on text, or may obtain pre-recorded data. Hereinafter, the sound signal supplied from the control unit 114 to the sound output unit 115 will be referred to as an "output sound signal."

[0060] The audio output unit 115 converts the output audio signal (digital acoustic signal) supplied from the control unit 114 into an analog signal and supplies it to the speaker SP, which then outputs (emits) audio based on the output audio signal.

[0061] The sound interval detection unit 116 performs a process of determining whether or not the output audio signal is in a sound interval containing speech (hereinafter referred to as "output sound interval determination process"), and supplies the result of the output sound interval determination process to the parameter control unit 117. The specific method of the output sound interval determination process performed by the sound interval detection unit 116 is not limited, and various processes can be applied. For example, the sound interval detection unit 116 may calculate the power (level converted into electric power) of the output audio signal at regular intervals and determine a sound interval by comparing the calculated power with a threshold. The sound interval detection unit 116 may perform the output sound interval determination process using, for example, the method described in Reference 1 below. Reference 1: Japanese Patent Application Laid-Open No. 2011-070084

[0062] There are no limitations on the format of the output sound segment determination processing result output by the sound segment detection unit 116. Hereinafter, a variable (flag) related to the output sound segment determination processing result output by the sound segment detection unit 116 will be represented as "SPK." SPK is a variable (flag) that takes on either a value of "1" indicating a sound segment or a value of "0" indicating a non-sound segment (a silent segment).

[0063] The parameter control unit 117 performs a process (hereinafter referred to as a "specific state determination process") to determine whether the state of audio input / output in the audio processing device 100 is a specific state (hereinafter referred to as a "specific state"), and controls the area sound determination process by the area sound determination unit 112 and the signal mixing process by the signal mixer 110 based on the determination result. In this embodiment, the parameter control unit 117 performs a process to recognize that the state is a specific state at least during the sound interval (SPK=1) as the result of the output sound interval determination process. In other words, the specific state can also be said to be a state in which the state of audio input / output in the audio processing device 100 is a double-talk state or a state in which there is a risk of a double-talk state. Hereinafter, a variable (flag) related to the specific state determination process will be represented as "DT." DT is a variable (flag) that takes either a value of "1" indicating that the state of audio input / output in the audio processing device 100 is a specific state, or a value of "0" indicating that the state is not a specific state (a non-specific state).

[0064] While the result of the specific state determination process indicates a specific state (DT=1), the parameter control unit 117 controls the signal mixing process by the signal mixer 110 to limit the signal mixing process of mixing a mixing signal into a target area sound extraction signal. For example, the parameter control unit 117 may limit the signal mixing process by stopping the signal mixing process by the signal mixer 110 or by reducing the amount of mixing signals (volume level) mixed in the signal mixing process. For example, the parameter control unit 117 can limit the signal mixing process in the signal mixer 110 by adjusting a mixing level adjustment variable μ1 in the signal mixer 110. Specifically, for example, the signal mixer 110 can control the signal mixing process to stop by setting the mixing level adjustment variable μ1 to 0 (or multiplying it by 0). Furthermore, for example, the signal mixer 110 can reduce the amount of mixing signals (volume level) mixed in the signal mixing process by reducing the mixing level adjustment variable μ1 (for example, by multiplying it by a predetermined coefficient less than 1). For example, the parameter control unit 117 may perform control such that μ1=μ1*A during the specific state (DT=1). A is a coefficient for limiting μ1, and can be in the range of 0≦A<1. For example, in the specific state (DT=1), A may be set to approximately 0 to 0.2. When A=0 is set in the specific state (DT=1), the signal mixing process in the signal mixer 110 is stopped.

[0065] However, in reality, an audio output signal is supplied to the audio output unit 115, audio based on the audio output signal is emitted from the speaker SP, the emitted audio (sound) is captured by the microphone arrays MA1 and MA2, and the captured audio is reflected in the picked-up signal (or mixing signal), and there is a delay (hereinafter this delay will be referred to as the "echo path delay") along the path (hereinafter this path will be referred to as the "echo path"), so if the restriction on the signal mixing process is released immediately after the result of the output sound interval determination process switches from a sound interval (SPK=1) to a silent interval (SPK=0), there is a risk that an echo (echo based on the output audio) that occurs during the echo path delay will be mixed into the picked-up signal. In other words, in the audio processing device 100, after the result of the output sound section determination process switches from a sound section (SPK=1) to a silent section (SPK=0), by waiting for a period of time equal to or longer than the above-mentioned echo path delay and then releasing the restriction on the signal mixing process, it is possible to avoid echoes from being mixed into the collected signal during the echo path delay period.

[0066] Therefore, the parameter control unit 117 of this embodiment performs a process (hereinafter referred to as a "hangover process") to extend the specific state (DT=1 state) for a predetermined period (a period equal to or greater than the echo path delay, hereinafter referred to as a "hangover period") after the result of the output sound interval determination process has switched from a sound interval (SPK=1) to a silent interval (SPK=0). In other words, the voice processing device 100 performs a process to release the restriction on the signal mixing process after a period equal to or greater than the echo path delay has elapsed after the result of the output sound interval determination process has switched from a sound interval (SPK=1) to a silent interval (SPK=0). In other words, the hangover process can be said to be a process to extend the resumption of the signal mixing process by the hangover period.

[0067] Furthermore, the parameter control unit 117 controls the area sound determination process by the area sound determination unit 112 while the result of the specific state determination process is the specific state (DT=1). Specifically, with regard to the threshold value TH, which is a threshold value for determining whether or not there is an area sound, a threshold value TH1 for a non-specific state (normal state) and a threshold value TH2 for a specific state (a value higher than the threshold value TH1) are set in advance in the area sound determination unit 112, and the parameter control unit 117 instructs the area sound determination unit 112 to switch the threshold value TH from the threshold value TH1 to the threshold value TH2 during the specific state period. This makes it possible to avoid a situation in which, in the specific state, the area sound determination unit 112 erroneously determines that an echo is present as the presence of a target sound. For example, when a user speaks while a speaker is outputting, it is empirically recognized that the user speaks louder so as not to be drowned out by the speaker output. Therefore, applying a high threshold value TH2 in determining whether or not there is an area sound (presence of user speech) does not cause any problems (it is far better than applying a low threshold value TH1 and erroneously determining that an echo is present as the target sound).

[0068] FIG. 3 is a diagram showing processing rules for each parameter (DT, TIM) related to the hangover processing of parameter control unit 117.

[0069] Here, it is assumed that the parameter control unit 117 has a timer variable TIM for measuring the hangover period. In addition, hereinafter, the frame period (frame interval) when the audio processing device 100 processes an audio signal will be represented as PRD. In addition, hereinafter, the period set as the hangover period in the parameter control unit 117 will be represented as HT. In this case, it is assumed that the frame period PRD=16 ms. In addition, it is assumed that the hangover period HT=160 ms.

[0070] As shown in FIG. 3, the parameter control unit 117 sets the timer variable TIM to HT while the output sound section determination process result in the sound section detection unit 116 is a sound section (SPK=1), and sets the specific state determination process result to a specific state (DT=1).

[0071] Also, as shown in Figure 3, if the output sound section determination process result in the sound section detection unit 116 is a silent section (SPK = 0) and the timer variable TIM is greater than 0 (TIM > 0), the parameter control unit 117 continues the process of repeatedly subtracting the timer variable TIM according to the elapsed time (the process of measuring the hangover period HT using the timer variable TIM), and keeps the specific state determination process result as a specific state (DT = 1).

[0072] Furthermore, as shown in FIG. 3, when the output sound section determination process result in the sound section detection unit 116 is a silent section (SPK=0) and the timer variable TIM is less than 0 (TIM≦0), the parameter control unit 117 determines the specific state determination process result as a non-specific state (DT=0).

[0073] The hangover period HT set in the parameter control unit 117 may be a pre-designed value (a value taking into account the echo path delay), or may be a value based on a value obtained by learning the echo path delay due to the above echo path. In other words, the voice processing device 100 (parameter control unit 117) may be provided with a learning function that measures the echo path delay using an adaptive filter and sets the hangover period HT. As a means for measuring (learning) the echo path delay using an adaptive filter, a technique similar to that used in echo cancellers in various telephone communications, etc., can be applied. Specifically, for example, the technique described in Reference 2 below can be applied.

[0074] The echo canceller described in Reference 2 is equipped with an adaptive filter to determine the echo path delay. When adaptive filter training is performed in an echo canceller in a single-talk state (a state in which only audio is output from the speaker on the near-end side), ideally the filter coefficients will converge to the impulse response of the echo path. In an echo canceller, the filter coefficients after training (the filter coefficients of the adaptive filter) are assumed to have converged to the impulse response of the echo path, and the echo path delay can be obtained from this. It is desirable to train the adaptive filter of the echo canceller, for example, by outputting a training signal such as white noise from the speaker and performing it in a state in which no speech is being spoken into the microphone (i.e., a single-talk state), as this allows for more accurate determination. When training the adaptive filter, it is also desirable to do so in an environment as quiet as possible. Reference 2: Japanese Patent Application Laid-Open No. 2008-153993

[0075] Similarly to the above, the control unit 114 also supplies an output audio signal that serves as a learning signal to the audio output unit 115, causing it to be output from the speaker SP, and supplies the output audio signal and the target area sound extraction sound (or a mixed signal including the input signal) to the adaptive filter, causing it to execute learning processing (filter coefficient learning processing), thereby obtaining an echo path delay (hereinafter referred to as "Ed"). The control unit 114 may obtain the period of the echo path delay Ed obtained by training the adaptive filter described above, or the period obtained by adding a margin α (α is, for example, about 10 to 20 ms) to the echo path delay Ed, as the hangover period HT (i.e., "HT=Ed" or "HT=Ed+α").

[0076] FIG. 4 is a timing chart showing an example of the hangover process performed by the parameter control unit 117. In FIG.

[0077] When the parameter control unit 117 performs the hangover process according to the rules shown in FIG. 3, the parameters (SPK, TIM, DT) transition as shown in FIG.

[0078] In FIG. 4, timings T101 to T103 represent timings in time series.

[0079] [Timing T101] In FIG. 4, it is assumed that at timing T101, the result of the output sound segment determination process in the sound segment detection unit 116 switches from a silent segment (SPK=0) to a sound segment (SPK=1). At this time, the parameter control unit 117 assumes that a specific state has been reached and switches the result of the specific state determination process from a non-specific state (DT=0) to a specific state (DT=1). Also, at this time, the parameter control unit 117 sets the timer variable to HT because the result of the specific state determination process has become the specific state (DT=1). At this time, because the result of the specific state determination process has become the specific state (DT=1), the parameter control unit 117 controls the area sound determination unit 112 and the signal mixing unit 110 to switch the threshold value TH (applying threshold value TH2), which is the determination threshold value in the area sound determination process, and to start limiting the signal mixing process.

[0080] [Timing T102] Assume that after timing T101, the output sound section determination process result continues to be in the sound section (SPK=1) state (i.e., specific state), and at timing T102, the output sound section determination process result switches from the sound section (SPK=1) to the silent section (SPK=0). Then, the parameter control unit 117 starts subtracting the timer variable TIM (measuring the hangover period HT). At this time, the parameter control unit 117 repeats the process of subtracting (decrementing) the value of the frame period PRD from the timer variable TIM (TIM=TIM-PRD) for each frame period PRD.

[0081] [Timing T103] In this case, since the hangover period HT=160 ms, parameter control unit 117 sets timer variable TIM to 0 (TIM≦0) when 160 ms have elapsed since timing T102. In FIG. 4, the timing when timer variable TIM becomes 0 (the timing when the hangover period HT has elapsed since timing T102) is set to timing T103. At timing T103, parameter control unit 117 switches the result of the specific state determination processing from a specific state (DT=1) to a non-specific state (DT=0). Also, at this time, since the result of the specific state determination processing becomes the non-specific state (DT=0), parameter control unit 117 controls area sound determination unit 112 and signal mixing unit 110 to switch threshold value TH, which is the determination threshold value in the area sound determination processing (applying threshold value TH1), and to release the restriction on signal mixing processing.

[0082] FIG. 5 is an explanatory diagram showing a specific example of area sound determination and signal mixing processing when only the target sound is collected, and the processed sound (processed sound selected by the selection unit 113) that is the final output result of the sound processing device 100.

[0083] 5 shows an example in which the target sound is collected between timing T201 and timing T202. During this time, the area sound determination unit 112 determines that the target area sound is present, and the signal mixing means performs signal mixing processing to mix the target area sound (target area sound extraction sound) with a mixing signal. Then, the selection unit 113 outputs, as processed sound, a processed sound obtained by mixing the target area sound with the mixing signal.

[0084] FIG. 6 is an explanatory diagram showing a specific example of each state, parameter, audio processing, etc. when an echo is output from a speaker.

[0085] 6 shows an example in which an echo is output from the speaker between timing T301 and timing T302. During this period, the parameter control unit 117 determines that this is a specific state period (double talk period), and the area sound determination unit 112 applies threshold value T2, which is higher than threshold value T1, to determine that there is no target area sound (determines that there is no target area sound). Also, since this is a specific state period, the signal mixing means performs processing to reduce mixing of the mixing signal (in this example, mixing is turned OFF). Then, since the area sound determination unit 112 has determined that there is no target area sound, the selection unit 113 outputs silence as the processing sound. Then, between timing T302 and timing T303, the specific state period is extended by hangover processing, and the same processing as between timing T301 and timing T302 is performed.

[0086] FIG. 7 is an explanatory diagram showing a specific example of each state, parameter, sound processing, etc. when an echo is output from a speaker and a target sound is picked up.

[0087] When an echo is output from the speaker between timing T401 and timing T402, the situation is the same as between timing T301 and timing T302 in FIG. 6, and therefore a description thereof will be omitted.

[0088] The period from timing T402 to timing T403 is an example in which a target sound is picked up at the same time as an echo is being output from the speaker. During this period, the parameter control unit 117 determines that this is a specific state period (double talk period), and the area sound determination unit 112 applies threshold value T2, which is higher than threshold value T1, to determine that a target area sound is present (determining that a target area sound is present). Also, since this is a specific state period, the signal mixing means performs processing to reduce mixing of the mixing signal (in this example, mixing is turned off). Then, since the selection unit 113 determines that a target area sound is present, it outputs only the target area sound as a processed sound. Then, between timing T403 and timing T404, the specific state period is extended by hangover processing, and the same processing as between timing T402 and timing T403 is performed.

[0089] When only the target sound is collected between timing T404 and timing T405, the situation is the same as between timing T201 and timing T202 in FIG. 5, and therefore a description thereof will be omitted.

[0090] (A-3) Effects of the embodiment According to this embodiment, the following effects can be achieved.

[0091] In the voice processing device 100 of this embodiment, in a specific state, the threshold value for area sound determination is increased to prevent erroneous determination of the presence of an echo as the presence of a target sound, and the signal mixing process by the signal mixer 110 is restricted (the signal mixing process is stopped or the volume level of the signal for mixing is reduced). This makes it possible for the voice processing device 100 to prevent echo (components of the output sound from the speaker SP) from being mixed into the collected signal.

[0092] Furthermore, in the voice processing device 100 of this embodiment, when the restriction on the signal mixing process by the signal mixing unit 110 is lifted, echo path delay is taken into consideration and hangover processing (processing to extend a specific period by the hangover period) is performed, thereby enabling stable echo prevention.

[0093] (B) Other embodiments The present invention is not limited to the above-described embodiment, and modified embodiments such as those exemplified below can also be mentioned.

[0094] (B-1) In the above embodiment, the parameter control unit 117 performs a process in which, in the specific state determination process, it recognizes that it is in a specific state at least while the result of the output sound section determination process is a sound section (SPK=1). However, it may also determine whether the target area sound extraction sound (i.e., the user's speech) is also in a sound section, and recognize a state in which both the speaker output and the target area sound extraction sound are in a sound section (i.e., a state in which it can be determined that it is a complete double talk state) as a specific section.

[0095] (B-2) In the above embodiment, the parameter control unit 117 performs specific state determination processing and, based on the determination results, controls the area sound determination processing by the area sound determination unit 112 and the signal mixing processing by the signal mixing unit 110. However, as a modified example, it may control only the area sound determination processing by the area sound determination unit 112 (it may also dynamically change parameters). [Explanation of symbols]

[0096] 100...audio processing device, 101...signal input unit, 102...noise suppression unit, 103...directivity forming unit, 104...delay correction unit, 105...spatial coordinate data, 106...correction coefficient calculation unit, 107...target area sound extraction unit, 108...mixing level calculation unit, 109...mixing level adjustment unit, 110...signal mixing unit, 111...amplitude spectral ratio calculation unit, 112...area sound determination unit, 113...selection unit, 114...control unit, 115...audio output unit, 116...sound activity period detection unit, 117...parameter control unit, D...touch panel display, MA (MA1, MA2)...microphone array, SP...speaker, TH (TH1, TH2)...threshold value.

Claims

1. adjusting a mixing signal including at least the input signal to a predetermined volume level; Signal mixing processing is performed to mix the target area sound with the extracted sound, a signal mixing means for generating and outputting a mixed target area sound generated by the signal mixing process; The signal mixing means At least in a specific section where the output audio signal is a sound section, means for limiting the signal mixing process; the input signal is a signal input from a multiple microphone array, the output audio signal is a signal related to audio output from a speaker, The target area sound extraction sound is a sound whose sound source is the target area in the input signal.

1. A voice processing device comprising:

2. a target area sound presence / absence determination means for performing a target area sound presence / absence determination process for determining whether or not a target area sound is present in the target area sound extracted sound; and a parameter control means for controlling the specific section so as to change the criteria for the target area sound presence / absence determination process and to limit the signal mixing process.

2. The audio processing device according to claim 1, wherein:

3. The audio processing device according to claim 2, characterized in that the parameter control means controls the target area sound presence / absence determination process so as to apply a second threshold value, which is higher than the first threshold value used under normal circumstances, to the threshold value that is the determination criterion for the target area sound presence / absence determination process in the specific section.

4. 3. The audio processing device according to claim 2, wherein the parameter control means performs hangover processing to extend the state of the specific section for a predetermined hangover period after the output audio signal changes from a sound section to a silent section.

5. a target area sound presence / absence determination means for performing a target area sound presence / absence determination process for determining whether or not a target area sound is present in the target area sound extracted sound; In the destination area sound presence / absence determination means, If it is determined that there is a target area sound, the mixed target area sound is selected; The apparatus further includes a selection means for selecting silence when it is determined that there is no sound in the target area.

2. The audio processing device according to claim 1, wherein:

6. The device further includes a target area sound extraction means for extracting a second non-target area sound from the target area direction using the target area direction sound acquired from the input signal, and for acquiring a target area sound extraction sound having the target area as a sound source using the second non-target area sound and the target area direction sound.

6. The audio processing device according to claim 5,

7. further comprising an amplitude spectrum ratio calculation means for calculating an amplitude spectrum from the output of the target area sound extraction means, calculating an amplitude spectrum ratio for each frequency using the amplitude spectrum and the amplitude spectrum of the input signal of the microphone array, and adding up the amplitude spectrum ratios of each frequency to calculate an amplitude spectrum ratio countable value; The target area sound presence / absence determining means compares the amplitude spectrum ratio countable value with a threshold value, and determines the presence or absence of the target area sound in the target area sound extracted sound based on the comparison result.

7. The audio processing device according to claim 6.

8. In the input signal, a first non-target area sound having directivity formed in a direction other than the target area direction; and a directivity forming means for acquiring the target area directional sound having formed directivity in the target area direction.

7. The audio processing device according to claim 6.

9. 2. The audio processing apparatus according to claim 1, further comprising sound segment detection means for detecting the sound segment.

10. 2. The audio processing apparatus according to claim 1, further comprising: specific section detection means for detecting the specific section.

11. Computer, adjusting a mixing signal including at least the input signal to a predetermined volume level; Signal mixing processing is performed to mix the target area sound with the extracted sound, functioning as a signal mixing means for generating and outputting a mixed target area sound generated by the signal mixing process; The signal mixing means At least in a specific section where the output audio signal is a sound section, functioning as a means for limiting the signal mixing process; the input signal is a signal input from a multiple microphone array, the output audio signal is a signal related to audio output from a speaker, The target area sound extraction sound is a sound whose sound source is the target area in the input signal. A speech processing program characterized by:

12. 1. A voice processing method for use in a voice processing device, comprising: The audio processing device comprises a signal mixing means, the signal mixing means adjusts a mixing signal including at least the input signal to a predetermined volume level, performs signal mixing processing to mix the signal with the target area sound extraction sound, and generates and outputs a mixed target area sound generated by the signal mixing processing; the signal mixing means is means for restricting the signal mixing process at least in a specific section where the output audio signal is a sound section, the input signal is a signal input from a multiple microphone array, the output audio signal is a signal related to audio output from a speaker, The target area sound extraction sound is a sound whose sound source is the target area in the input signal.

1. A sound processing method comprising:

Images