Signal processing device, learning device, signal processing method, learning method, signal processing program, and learning program

Abstract

A signal processing device (10) has: an instantaneous spatial covariance matrix (ISCM) calculation unit (11) for estimating a time frequency mask for an observation signal and calculating, for all observation times for an observation signal, ISCMs calculated only from an observation frame from a certain time; a time-variant spatial covariance matrix (SCM) calculation unit (12) for estimating respective time variation weights for the ISCMs calculated by the ISCM calculation unit (11) using a model that takes the ISCMs for all observation times as input and estimates respective time variation weights of the ISCMs, and calculating a time-variant SCM which is a weighted sum of the ISCMs for all observation times; and a voice enhancement unit (13) for constructing a time-variant beam former on the basis of the time-variant SCM and generating an enhanced signal in which the voice of a speaker is enhanced from an inputted observation signal.

Description

[Technical Field]

[0001] The present invention relates to a signal processing device, a learning device, a signal processing method, a learning method, a signal processing program, and a learning program. [Background technology]

[0002] Mask-based beamforming, which combines neural networks and array signal processing techniques, has attracted much attention in recent years due to its high performance. It has been proven to be particularly effective in reducing noise and interfering talkers.

[0003] The beamformer uses spatial information about the desired and interfering sound sources obtained from spatial covariance matrices (SCMs) to emphasize the signal from the desired sound source direction while suppressing the interfering signals. The mask-based beamformer calculates the SCM using time-frequency masks obtained from other sound source models, such as neural networks (NNs) or complex Gaussian mixture models (cGMMs). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] J. Malek, Z. Koldovsky, and M. Bohac, “Block-Online Multi-Channel Speech Enhancement Using DNN-Supported Relative Transfer Function Estimates”, IET Signal Processing, vol. 14, pp. 124-133, 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] Because SCM acquires spatial information, it is highly sensitive to the movement of sound sources. However, conventional methods generally assume that the target sound source and the interfering sound source do not move during speech, and construct a time-invariant beamforming filter. As a result, conventional methods cannot respond to changes in the sound source environment, such as speaker movement.

[0006] Here, in consideration of practical applications, the estimation of time-varying beamforming filters using online processing or block processing has been proposed to respond to environmental changes such as the movement of sound sources. However, since this depends on the speed of environmental changes and the stability of statistics, it is not always possible to construct an optimal beamforming filter.

[0007] The present invention has been made in consideration of the above, and aims to provide a signal processing device, a learning device, a signal processing method, a learning method, a signal processing program, and a learning program that can process audio signals with high accuracy even when there is a change in the environment of the sound source. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the object, a signal processing device according to the present invention is characterized by having: a first calculation unit that estimates a time-frequency mask for an observed signal and calculates ISCMs (Instantaneous Spatial Covariance Matrices) calculated using only observation frames at a certain time over all observation times of the observed signal; a second calculation unit that uses a model that estimates time-varying weights for each ISCM calculated by the first calculation unit using as input each ISCM at all observation times; and calculates time-varying SCMs (Spatial Covariance Matrices) that are weighted sums of each ISCM at all observation times; and a speech enhancement unit that constructs a time-varying beamformer based on the time-varying SCMs and generates an enhancement signal that enhances the speaker's voice from the input observed signal.

[0009] Furthermore, the learning device according to the present invention is characterized by comprising: a first calculation unit that estimates a time-frequency mask for a noisy observed signal that undergoes environmental changes of a sound source, and calculates ISCMs (Instantaneous Spatial Covariance Matrices) calculated using only observation frames at a certain time, across all observation times of the noisy observed signal; a second calculation unit that uses a model that estimates time-varying weights for each ISCM using the ISCMs at all observation times as input, to estimate time-varying weights for each ISCM calculated by the first calculation unit, and calculates time-varying Spatial Covariance Matrices (SCMs), which are the weighted sum of the ISCMs at all observation times; a speech enhancement unit that constructs a time-varying beamformer based on the time-varying SCMs, and generates an enhancement signal that enhances the speaker's voice from the input noisy observed signal; and an update unit that updates parameters of the model so that the enhancement signal approaches a target speaker's clean signal. [Effects of the Invention]

[0010] According to the present invention, audio signals can be processed with high accuracy even when there is a change in the environment of the sound source. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a signal processing device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating the processing of the time-varying SCM calculation unit shown in FIG. [Figure 3] FIG. 3 is a flowchart showing the processing procedure of the signal processing method according to the embodiment. [Figure 4] FIG. 4 is a flowchart showing the processing procedure of the time-varying SCM calculation processing shown in FIG. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of a learning device according to an embodiment. [Figure 6] FIG. 6 is a flowchart showing the processing procedure of the learning method according to the embodiment. [Figure 7]FIG. 7 shows the microphone array arrangement based on the CHiME-3 corpus. [Figure 8] FIG. 8 is a diagram showing an example of a layout. [Figure 9] FIG. 9 is a diagram showing an example of a beam pattern of a moving sound source. [Figure 10] FIG. 10 is a diagram illustrating an example of a computer that implements a signal processing device by executing a program. DETAILED DESCRIPTION OF THE INVENTION

[0012] An embodiment of the present invention will be described in detail below with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the drawings, identical parts are denoted by the same reference numerals. Note that, hereinafter, when "^A" is written for A, which is a vector or matrix, it is assumed to be the same as "a symbol with "^" written immediately above "A". When "￣A" is written for A, which is a vector or matrix, it is assumed to be the same as "a symbol with "￣" written immediately above "A".

[0013] [Embodiment Mode] In this embodiment, as an example, a signal processing method for observation signals recorded by a plurality of microphones (multi-channel) will be described.

[0014] [Signal processing device] Next, a signal processing device according to an embodiment will be described. Fig. 1 is a diagram schematically illustrating an example of the configuration of a signal processing device according to an embodiment.

[0015] The signal processing device 10 according to the embodiment is realized by loading a predetermined program into a computer or the like including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The signal processing device 10 also has a communication interface for transmitting and receiving various information to and from other devices connected via a network, etc.

[0016] As shown in Fig. 1, the signal processing device 10 includes an ISCM calculation unit 11 (first calculation unit), a time-varying SCM calculation unit 12 (second calculation unit), and a speech enhancement unit 13. The signal processing device 10 processes an observed signal Y recorded in multi-channels. t,f ∈C C is input, where t is the time frame, f is the frequency bin, and c is the channel.

[0017] The ISCM calculation unit 11 calculates the input observed signal Y t,f For the signal Y, we estimate the time-frequency mask and use the ISCM (Instantaneous Spatial Covariance Matrices) calculated only for the observation frame at a certain time. t,f The ISCM calculation unit 11 calculates the ISCM using equation (1).

[0018]

number

[0019] where m v t,f ∈[0, 1] is the time-frequency mask, estimated by a pre-trained neural network. v∈{S, N} is the index of the speech signal and noise signal, respectively. T represents a transposition. v t,f is called the ISCM at time-frequency bin (t, f).

[0020] The time-varying SCM calculation unit 12 calculates the time-varying SCM Ψ for each ISCM at all observation times. v t,f Using the weight estimation model 1221 (model), the time-varying weights c v t ={c v t,t’=1 ,···, c v t,t’=T}∈R T c v t,t’ is the time-varying SCM Φ corresponding to the t-th time frame v t,f The weight estimation model 1221 represents the weight for the t'th ISCM when calculating the ISCM Ψ v t,f The time-varying SCM calculation unit 12 uses the equation (2) to calculate the time-varying SCM (Spatial Covariance Matrices) Φ v t,f Calculate.

[0021]

number

[0022] time-varying weight c v t represents the contribution of each frame in the computation of the Spatial Covariance Matrices (SCM). The contribution determines which frame to use. Therefore, the time-varying weight c v t It can be said that the time frame to be focused on when calculating the SCM at a specific time (i.e., t) among all time frames (t = 1 to T) is determined. In other words, the time-varying weight c v t It can be said that the above indicates which time frame's ISCM should be used to calculate the SCM at a certain time (time frame) t.

[0023] And the time-varying SCM Φ v t,f is the corresponding time-varying weight c for each ISCM at all observation times. v t It can be said that the SCM is calculated by collecting only the contributing ISCMs.

[0024] The speech enhancement unit 13 uses a time-varying SCM Φ v t,f Based on this, a time-varying beamformer is constructed, and the input observation signal Y t,f From the signal, the speaker's voice is emphasized and the enhanced signal ^X t,f The speech enhancement unit 13 generates a time-varying beamformer w based on equation (3). t,f and construct the enhancement signal ^x based on Eq. (4). t,f Get.

[0025]

number

[0026]

number

[0027] where w t,f ∈C C denotes a vector containing the beamforming filter coefficients. S t,f ∈C C×C and Φ N t,f ∈C C×C are the SCMs of the speech signal and noise signal at time-frequency bin (t, f), respectively. C is a one-hot vector representing the index of the reference microphone. H denotes the conjugate transpose.

[0028] [Time-varying SCM calculation section] Next, a description will be given of the time-varying SCM calculation unit 12. Fig. 2 is a diagram for explaining the processing of the time-varying SCM calculation unit 12 shown in Fig. 1.

[0029] As shown in FIG. 1, the time-varying SCM calculation unit 12 includes a feature transformation unit 121, a time-varying weight estimation unit 122 (estimation unit) having a weight estimation model 1221, and an SCM calculation unit 123 (third calculation unit).

[0030] The feature conversion unit 121 uses the formula (5) to convert the ISCM {Ψ v t,f} f=1 F ∈C F×C×C Let us denote the real-valued vector ψ v t ∈R 2FC2 (Step S1 in FIG. 2). The feature conversion unit 121 converts the feature into a predetermined time frame t{Ψ v t,f} f=1 F ∈C F×C×C The ISCM of all frequency bins in is expressed as a real-valued vector ψ v t ∈R 2FC^2 Convert to a real-valued vector ψ v t are input to the weight estimation model 1221, which is a NN.

[0031]

number

[0032] where Vectorize(·) is the vector of the complex tensor {Ψ v t,f} f=1 F ∈C F×C×C Let ψ be a real-valued vector containing the real and imaginary parts of all the elements of the tensor. v t Represents the expansion process that converts

[0033] The time-varying weight estimation unit 122 estimates the real-valued vector ψ of the ISCM. v t is used as the input of the weight estimation model 1221, and the time-varying weights of each ISCM {c v t} t=1 T The weight estimation model 1221 estimates c v t ={c v t,t’=1 ,···, c v t,t’=T}∈R T is estimated (step S2 in FIG. 2).

[0034]

number

[0035] NN V (·) is the nonlinear transformation of the NN, and Λ V is NN V (·) denotes the learnable parameters.

[0036] The weight estimation model 1221 estimates the time-varying weights {c v t} t=1 T The time-varying weight estimation unit 122 outputs {c v t} t=1 T and outputs it to the SCM calculation unit 123.

[0037] The SCM calculation unit 123 calculates the time-varying SCM Φ using equation (2). v t,f (step S3 in FIG. 2) and outputs it to the speech enhancement unit 13. To calculate the time-varying SCM of a certain frame, the SCM calculation unit 123 weights all frames and calculates the sum as the time-varying SCM.

[0038] Here, we will explain how to set the weights for the SCM calculation process in the conventional method. In the conventional method, which treats SCM as time-invariant, the same weight is assigned to all values ​​at any time to calculate the SCM, as shown in equation (7).

[0039]

number

[0040] Furthermore, in the case of conventional online SCM calculation methods, only an exponentially smaller weight is assigned to older ISCMs, as shown in equation (8).

[0041]

number

[0042] In addition, in the case of the conventional block-based calculation method, the signal is divided into continuous time blocks, and weights are assigned only to the frames of the block that includes the time frame t, as shown in equation (9).

[0043]

number

[0044] In contrast, in the signal processing device 10, for each time frame, the time-varying weights c v t Then, the signal processing device 10 calculates time-varying weights c v t and only the contributing ISCMs are collected to calculate the time-varying SCM.

[0045] In other words, in the signal processing device 10, the frame used to calculate the SCM of a certain time frame t is determined by the time-varying weight c v t The time-varying weight c v t has a function to control what weighting should be applied to the ISCMs of the time frames to be added together to obtain a good statistic in order to calculate the SCM of a certain time frame t.

[0046] In the signal processing device 10, this time-varying weight c v tThe weight estimation model 1221 executes the process of obtaining the above. In the embodiment, the weight estimation model 1221 learns how much each ISCM contributes based on data accompanying environmental changes of the sound source (for example, speaker movement).

[0047] Therefore, the signal processing device 10 appropriately collects only the contributing ISCMs to calculate the time-varying SCM Φ v t,f Based on this, a time-varying beamformer is constructed, and the input observation signal Y t,f From the speaker's voice, the enhanced signal ^x t,f By appropriately weighting the ISCM that contributes in response to speaker movement, the signal processing device 10 is believed to be able to process speech signals with high accuracy even when there is a change in the environment of the sound source, such as speaker movement.

[0048] The processing of the time-varying weight estimation unit 122 and the SCM calculation unit 123 may be realized by employing an attention network.

[0049] [Signal processing method] FIG. 3 is a flowchart showing the processing procedure of the signal processing method according to the embodiment.

[0050] As shown in FIG. 3, the signal processing device 10 receives an observed signal Y t,f The ISCM calculation unit 11 receives the input observed signal Y t,f For the signal Y, we estimate the time-frequency mask and use the ISCM calculated only for the observation frame at a certain time. t,f is calculated over all observation times (step S12).

[0051] The time-varying SCM calculation unit 12 calculates the time-varying SCM Ψ for each ISCM at all observation times. v t,f is used as input, and the time-varying weights c of each ISCM are calculated using the weight estimation model 1221. v t ={c v t,t’=1 ,···, c vt,t’=T}∈R T and estimate the time-varying SCM Φ, which is the weighted sum of each ISCM for all observation times. v t,f A time-varying SCM calculation process is performed to calculate (step S13).

[0052] The speech enhancement unit 13 uses a time-varying SCM Φ v t,f Based on this, a time-varying beamformer is constructed, and the input observation signal Y t,f From the speaker's voice, the enhanced signal ^x t,f is generated (step S14).

[0053] [Time-varying SCM calculation processing] FIG. 4 is a flowchart showing the processing procedure of the time-varying SCM calculation processing shown in FIG.

[0054] As shown in FIG. 4, the feature conversion unit 121 converts each ISCM {Ψ v t,f} f=1 F ∈C F×C×C Let us denote the real-valued vector ψ v t ∈R 2FC2 (Step S21). The time-varying weight estimation unit 122 performs feature transformation to convert each ISCM Ψ v t,f is input to the weight estimation model 1221, and the time-varying weights c v t ={c v t,t’=1 ,···, c v t,t’=T}∈R T The SCM calculation unit 123 estimates the time-varying SCM Φ, which is a weighted sum of the ISCMs at all observation times (step S22). v t,f is calculated (step S23).

[0055] [Learning device] Next, a learning device that learns a time-varying weight model will be described. Fig. 5 is a diagram schematically illustrating an example of the configuration of a learning device according to an embodiment.

[0056] The learning device 20 according to the embodiment is realized, for example, by loading a predetermined program into a computer or the like including a ROM, RAM, CPU, etc., and having the CPU execute the predetermined program. The learning device 20 also has a communication interface for transmitting and receiving various information to and from other devices connected via a network, etc.

[0057] 5, learning device 20 has an ISCM calculation unit 21 (first calculation unit), a time-varying SCM calculation unit 22 (second calculation unit), a speech enhancement unit 23, and a parameter update unit 24 (update unit). Time-varying SCM calculation unit 22 has a feature conversion unit 221, a time-varying weight estimation unit 222 having a weight estimation model 2221, and an SCM calculation unit 223. ISCM calculation unit 21, time-varying SCM calculation unit 22, and speech enhancement unit 23 have the same functions as ISCM calculation unit 11, time-varying SCM calculation unit 12, and speech enhancement unit 13 shown in FIG. 1, respectively.

[0058] The training device 20 performs training of the weight estimation model 2221 using a training set of noisy observed signals accompanied by changes in the sound source environment and target speaker clean signals {y, x}. T is the T long-term waveform of the observed signal with noise accompanying the environmental changes of the sound source, and x∈R T is the corresponding target speaker clean signal. Here, we added various moving source conditions to the training set to train robust tracking performance.

[0059] The learning device 20 uses the emphasis signal ^x generated by the speech emphasis unit 23 t,f and the target speaker clean signal x t,f As a loss function to calculate the similarity between and, we used the scale-dependent signal-to-noise ratio (SNR). The loss function L1 is shown in Equation (10).

[0060]

number

[0061] The parameter update unit 24 updates the parameters of the weight estimation model 2221 so that the emphasized signal generated by the speech enhancement unit 23 approaches the target speaker clean signal. The parameter update unit 24 updates the parameters of the weight estimation model 2221 based on the loss function L1 of equation (10).

[0062] [Learning process] FIG. 6 is a flowchart showing the processing procedure of the learning method according to the embodiment.

[0063] As shown in FIG. 6, the learning device 20 receives a noisy observed signal y t When the input is accepted (step S31), ISCM calculation (step S32), time-varying SCM calculation processing (step S33), and speech enhancement (step S34) are performed. Steps S32, S33, and S34 are the same processing as steps S12, S13, and S14, respectively.

[0064] The parameter update unit 24 updates the parameters of the weight estimation model 2221 using the loss function L1 (step S35). The learning device 20 determines whether a predetermined termination condition is met (step S36). The termination condition may be, for example, when the loss is equal to or less than a predetermined threshold, when the number of parameter updates reaches a predetermined number, or when the parameter update amount is equal to or less than a predetermined threshold.

[0065] If the predetermined termination condition is not satisfied (step S36: No), the learning device 20 returns to step S31. The signal processing device 10 repeats the speech enhancement process and the parameter update process until the predetermined termination condition is satisfied. If the predetermined termination condition is satisfied (step S36: Yes), the learning device 20 terminates the learning process. The learning device 20 outputs the weight estimation model 2221 (including the model parameters) to the signal processing device 10.

[0066] [Modification of weight estimation model] The weight estimation model 1221 may smooth the time-varying weights using equation (11).

[0067]

number

[0068] Here, ￣c v t is the smoothed weighting coefficient and L' is the number of frames used for weight smoothing. v t is c v t Similarly, it is estimated over all observation times.

[0069] The weight estimation model 1221 smooths the time-varying weights using equation (11), thereby improving the ASR performance, as will be shown in the evaluation experiment described later.

[0070] [Evaluation experiment] An evaluation experiment was carried out to evaluate the effects of the signal processing method and learning method according to this embodiment.

[0071] [Experimental conditions] For the evaluation experiments, we generated a new dataset of moving sound sources in noisy environments. The source signals were extracted from the WSJ0 corpus (Reference 1), and the noise signals were extracted from the CHiME-3 corpus (Reference 2). Reference 1: DB Paul and J. Baker, “The Design for the Wall Street Journal-based CSR Corpus,” in Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992, 1992. Reference 2: J. Barker, R. Marxer, E. Vincent, and S. Watanabe, “The third 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines”, in IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2015, pp. 504-511.

[0072] Figure 7 shows the microphone array configuration for the CHiME-3 corpus. The CHiME-3 corpus contains noise signals recorded using a tablet device with a rectangular microphone array having six channels, as shown in Figure 7. From this six-channel microphone, the second channel signal captured by the microphone facing the rear of the tablet was removed, and the remaining five channels were used in the following multi-channel experiments (i.e., C = 5).

[0073] Pairs of speech and noise signals were randomly selected from the WSJ0 and CHiME-3 corpora and mixed at various signal-to-noise ratios (SNRs) between 2 and 8 dB. The Room Impulse Response (RIR) of a moving sound source was generated using the gpuRIR simulation toolkit (Ref. 4) based on the Image source method (Ref. 3). Various conditions in the RIR simulation (i.e., room shape, array position, and source trajectory) were randomly set. Reference 3: JB Allen and DA Berkley, “Image method for efficiently simulating small-room acoustics,” The Journal of the Acoustical Society of America, vol. 65, no. 4, pp. 943-950, 1979. Reference 4: D. Diaz-Guerra, A. Miguel, and JR Beltran, “gpuRIR: A Python Library for Room Impulse Response Simulation with GPU Acceleration”, Multimedia Tools and Applications, vol. 80, no. 4, pp. 5653-5671, 2021.

[0074] Figure 8 shows an example of the layout. In the evaluation experiments, the room was assumed to be square and the source speaker moved in a straight line within the room. As shown in Figure 8, the start and end positions of the sound source trajectory were randomly sampled from region R1, and the array position was randomly sampled from region R2. The simulation was set up so that each moving speaker started speaking at the start position and finished speaking at the end position. The source speaker's speed of movement during one utterance was constant, but varied across multiple utterances. The reverberation time (T60) ranged from 0.1 to 0.3 seconds. Table 1 summarizes the configuration of the moving sound source simulation.

[0075] [Table 1]

[0076] 30,000, 2,000, and 2,000 noisy speech signals were generated for the training set, development set, and evaluation set, respectively. The speech sources for the training set were selected from the training set "si_tr_s" of WSJ0. The speech sources for the development set and evaluation set were selected from the development set "si_dt_05" and evaluation set "si_et_05" of WSJ0, respectively. The noisy speech sources in the CHiME-3 corpus were divided into three subsets: the training set, the development set, and the evaluation set, and used to generate the noisy speech signals.

[0077] In addition to the moving source dataset described above, we generated a non-moving source dataset as an additional evaluation set, which has exactly the same configuration as the moving source dataset (i.e., speech-noise source pairs and RIR configuration), except that the source speaker's position is fixed at the starting position.

[0078] The following three speech enhancement indices were used as evaluation indices: 1) Signal-to-Distortion Ratio (SDR) (Reference 5) enables time-invariant filters to generate distortion. 2) Perceptual Evaluation of Speech Quality (PESQ) (Reference 6) 3) Short-time Objective Intelligibility (STOI) (Reference 7) Reference 5: E. Vincent, R. Gribonval, and C. Fevotte, “Performance Measurement in Blind Audio Source Separation,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, no. 4, pp. 1462-1469, 2006. Reference 6: AW Rix, JG Beerends, MP Hollier, and AP Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2001, pp. 749-752. Reference 7: CH Taal, RC Hendriks, R. Heusdens, and J. Jensen, “An Algorithm for Intelligibility Prediction of Time-Frequency Weighted Noisy Speech,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 7, pp. 2125-2136, 2011.

[0079] The word error rate (WER) was used as the speech recognition index.To calculate the speech enhancement index, the clean reverberant signal of the fifth channel with a moving source speaker was used as the reference.

[0080] [Experimental Results] The signal processing device 10 ((e) embodiment) with the weight estimation model 1221 is compared with the (b) time-invariant (tiv_mvdr) (using Equation (7)), (c) online (onl_mvdr) (using Equation (8)), and (d) block-wise (blk_mvdr) (using Equation (9)) MVDR beamformers on a moving / stationary sound source dataset. For comparison, we also provide results obtained by applying a time-frequency mask to the observed signal without beamforming (i.e., masking ((a) masking)). To adjust the forgetting factor α and block size L in the online / block-wise MVDR implementation, we pre-evaluated the enhancement performance for α = {0.999, 0.99, 0.9, 0.7, 0.5} and L = {5, 10, 20, 30, 40, 50}.

[0081] We also evaluated a modified example ((f) weight smoothing) of the weight estimation model 1221. The number of frames for the weight smoothing L' in equation (11) was set to 7 for the moving sound source dataset and 9 for the stationary sound source dataset.

[0082] Table 2 shows the performance metrics for speech enhancement (i.e., SDR, PESQ, STOI) and ASR (i.e., WER) for the stationary and moving source datasets.

[0083] [Table 2]

[0084] The left side of Table 2 shows the results for the stationary source dataset. (a) and conventional methods (b) to (d) were observed to improve the speech enhancement metrics (i.e., SDR, PESQ, and STOI). All of the beamformers for conventional methods (b) to (d) improved the ASR, with conventional method (b) achieving the highest performance.

[0085] The signal processing device 10 ((e)) showed higher SDR, PESQ, and STOI scores than the conventional method (b). Furthermore, when weight smoothing was performed ((f)), the WER score was also higher. The signal processing device 10 can improve SCM calculations even when the subject is stationary.

[0086] The evaluation results for moving sound sources are shown on the right side of Table 2. It was observed that the performance of the conventional method (b) was significantly lower than when the sound source was stationary. This result confirmed the importance of taking the movement of the sound source into consideration when designing the beamformer. Conventional methods (c) and (d) are capable of time-varying beamforming, but their speech enhancement performance and ASR score were not as high as conventional method (b). This indicates the difficulty of setting appropriate hyperparameters to effectively track moving sound sources.

[0087] In contrast, the signal processing device 10 (see (e)) showed higher scores for SDR, PESQ, and STOI compared to the conventional method (b). Furthermore, by applying weight smoothing (see (f)), the signal processing device 10 also showed higher WER performance compared to (e).

[0088] These results confirm that the signal processing device 10 according to the embodiment can mitigate performance degradation caused by movement of the sound source.

[0089] [Beam pattern visualization] By visualizing the beam pattern of the beamformer configured by the signal processing device 10, the behavior of the time-varying beamforming filter actually estimated by the signal processing device 10 during bidirectional speaker movement is analyzed.

[0090] To emphasize the movement of the sound source, we imagine that the sound source is moving in a circle around the room.

[0091] Figure 9 shows an example of a beam pattern for a moving sound source. Figure 9 shows beam patterns for eight time frames. The black lines indicate the actual direction of the source speaker in each frame (i.e., 30°, 60°, 70°, 90°, 120°, 130°, 210°, and 270°). The dashed and solid lines correspond to the beam patterns for 1 kHz and 2 kHz, respectively.

[0092] The 30° beam pattern corresponds to the region where the source is inactive. In this case, the beamformer does not show a clear directional pattern. It was subsequently confirmed that when the source is active (e.g., 60° and 90°), the beamformer has a main lobe pointing toward the source. It was also observed that the beam pattern changes over time to follow the source position (e.g., 210° and 270°).

[0093] The pattern in FIG. 9 suggests that the beamforming filter estimated by the signal processing method according to the embodiment can track the position of a moving sound source speaker.

[0094] [Effects of the embodiment] The signal processing device 10 according to the embodiment adjusts the time-varying weight c v tThe weighted sum of ISCMs weighted by is calculated as a time-varying SCM, and a time-varying beamformer is constructed based on this time-varying SCM.

[0095] In an embodiment, the time-varying weight c v t The weight estimation model 1221, which is a neural network, is used to calculate the ISCM. This weight estimation model 1221 accumulates each ISCM at all observation times using a training set of noise-affected observed signals under various moving sound source conditions and target speaker clean signals, and calculates a time-varying weight c v t The signal processing device 10 is trained to output the time-varying weight c v t Based on this, the range (area, time) for calculating the SCM of a certain time frame t among all ISCMs at all observation times is controlled, and the statistical values ​​are accumulated to calculate the SCM of the t-th frame.

[0096] Therefore, unlike conventional approaches, the signal processing device 10 assigns time-varying weights c v t As shown in the evaluation experiment, the signal processing device 10 uses the weight estimation model 1221 to calculate the optimal time-varying weight c v t By calculating the time-varying SCM by calculating the time-varying SCM, stable and high performance was obtained both when the sound source was moving and when it was not moving.

[0097] As described above, according to this embodiment, audio signals can be processed with high accuracy even when there is a change in the environment of the sound source.

[0098] [System configuration of the embodiment] The components of the signal processing device 10 and the learning device 20 are conceptual and functionally independent, and do not necessarily have to be physically configured as shown in the drawings. In other words, the specific forms of distribution and integration of the functions of the signal processing device 10 and the learning device 20 are not limited to those shown in the drawings, and all or part of them can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0099] Furthermore, all or any part of the processes performed in the signal processing device 10 and the learning device 20 may be realized by a CPU, a GPU (Graphics Processing Unit), and a program analyzed and executed by the CPU and the GPU. Furthermore, each process performed in the signal processing device 10 and the learning device 20 may be realized as hardware using wired logic.

[0100] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated can be changed as appropriate unless otherwise specified.

[0101] [program] 10 is a diagram showing an example of a computer in which a signal processing device 10 and a learning device 20 are realized by executing a program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0102] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0103] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes of the signal processing device 10 and the learning device 20 are implemented as program modules 1093 in which code executable by the computer 1000 is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configurations of the signal processing device 10 and the learning device 20 are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0104] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes them.

[0105] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0106] Although the present invention has been described above as an embodiment, the present invention is not limited to the descriptions and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention. [Explanation of symbols]

[0107] 10. Signal Processing Device 11,21 ISCM calculation section 12,22 Time-varying SCM calculation section 13,23 Speech enhancement unit 20 Learning Device 24 Parameter update section 121,221 Feature conversion unit 122,222 Time-varying weight estimation part 123,223 SCM calculation section 1221,2221 Weight estimation model

Claims

1. a first calculation unit that estimates a time-frequency mask for an observed signal and calculates instantaneous spatial covariance matrices (ISCMs) calculated only for an observation frame at a certain time over all observation times of the observed signal; a second calculation unit that estimates the time-varying weights of each ISCM calculated by the first calculation unit using a model that estimates the time-varying weights of each ISCM using each ISCM at all observation times as an input, and calculates time-varying Spatial Covariance Matrices (SCMs), which are weighted sums of each ISCM at all observation times; a speech enhancement unit that constructs a time-varying beamformer based on the time-varying SCM and generates an enhancement signal that enhances the speaker's speech from the input observation signal; A signal processing device comprising:

2. The second calculation unit a feature transformation unit that transforms each ISCM into a real-valued vector; an estimation unit that estimates time-varying weights of each ISCM using the real-valued vectors of the ISCM as inputs to the model; a third calculation unit that calculates the sum of each ISCM weighted by the corresponding time-varying weight as the time-varying SCM; 2. The signal processing device according to claim 1, further comprising:

3. 2. The signal processing device according to claim 1, wherein the model smoothes each estimated time-varying weight and outputs the smoothed weight.

4. a first calculation unit that estimates a time-frequency mask for a noisy observed signal that involves an environmental change of a sound source, and calculates Instantaneous Spatial Covariance Matrices (ISCMs) calculated only for an observation frame at a certain time over all observation times of the noisy observed signal; a second calculation unit that estimates the time-varying weight of each ISCM calculated by the first calculation unit using a model that estimates the time-varying weight of each ISCM using the ISCM at all observation times as input, and calculates time-varying Spatial Covariance Matrices (SCM), which are the weighted sum of the ISCM at all observation times; a speech enhancement unit that constructs a time-varying beamformer based on the time-varying SCM and generates an enhancement signal that enhances the speaker's speech from the input noise-added observation signal; an updating unit that updates parameters of the model so that the enhanced signal approaches a target speaker clean signal; A learning device comprising:

5. A signal processing method executed by a signal processing device, comprising: a step of estimating a time-frequency mask for an observed signal and calculating instantaneous spatial covariance matrices (ISCMs) calculated only for an observation frame at a certain time over all observation times of the observed signal; a step of estimating time-varying weights for each ISCM using a model that uses the ISCMs at all observation times as input and estimates time-varying weights for each ISCM, and calculating time-varying SCMs (Spatial Covariance Matrices), which are weighted sums of the ISCMs at all observation times; constructing a time-varying beamformer based on the time-varying SCM and generating an enhancement signal that enhances the speaker's voice from the input observation signal; A signal processing method comprising:

6. A learning method executed by a learning device, A step of estimating a time-frequency mask for a noisy observed signal that involves an environmental transformation of a sound source, and calculating Instantaneous Spatial Covariance Matrices (ISCMs) calculated only for an observation frame at a certain time over all observation times of the noisy observed signal; a step of estimating time-varying weights for each ISCM using a model that uses the ISCMs at all observation times as input and estimates time-varying weights for each ISCM, and calculating time-varying SCMs (Spatial Covariance Matrices), which are weighted sums of the ISCMs at all observation times; constructing a time-varying beamformer based on the time-varying SCM and generating an enhancement signal that enhances the speaker's voice from the input noisy observation signal; updating the parameters of the model so that the enhanced signal approaches the target speaker clean signal; A learning method comprising:

7. A step of estimating a time-frequency mask for an observed signal and calculating Instantaneous Spatial Covariance Matrices (ISCMs) calculated only for an observation frame at a certain time over all observation times of the observed signal; a step of estimating time-varying weights for each ISCM using a model that uses the ISCMs at all observation times as input and estimates time-varying weights for each ISCM, and calculating time-varying SCMs (Spatial Covariance Matrices), which are weighted sums of the ISCMs at all observation times; constructing a time-varying beamformer based on the time-varying SCM and generating an enhancement signal that enhances the speaker's voice from the input observation signal; A signal processing program for executing the above on a computer.

8. A step of estimating a time-frequency mask for a noisy observed signal that involves an environmental transformation of a sound source, and calculating Instantaneous Spatial Covariance Matrices (ISCMs) calculated only for an observation frame at a certain time over all observation times of the noisy observed signal; a step of estimating time-varying weights for each ISCM using a model that uses the ISCMs at all observation times as input and estimates time-varying weights for each ISCM, and calculating time-varying SCMs (Spatial Covariance Matrices), which are weighted sums of the ISCMs at all observation times; constructing a time-varying beamformer based on the time-varying SCM and generating an enhancement signal that enhances the speaker's voice from the input noisy observation signal; updating the parameters of the model so that the enhanced signal approaches the target speaker clean signal; A learning program that allows a computer to execute the above.

Images