Program, information processing method, recording medium, and information processing device.

Abstract

In order to reduce the number of calculations without decreasing sound source separation performance, a program according to the present invention causes a computer to implement an information processing method in which a neural network unit generates sound source separation information for separating a prescribed sound source signal from a mixed sound signal including a plurality of sound source signals, an encoder of the neural network unit transforms a feature value extracted from the mixed sound signal, the processing result from the encoder is inputted to each of a plurality of sub-neural-network units of the neural network unit, and the processing result from the encoder and the processing results from each of the plurality of sub-neural-network units are inputted to a decoder of the neural network unit.

Description

Technical Field

[0001] The present disclosure relates to a program, an information processing method, a recording medium, and an information processing apparatus.

Background Art

[0002] There is known a sound source separation technique for extracting a target sound source signal from a mixed sound signal including a plurality of sound source signals. For example, Patent Document 1 discloses a sound source separation technique using a DNN (Deep Neural Network).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Although a method using a DNN can obtain high sound source separation performance, the amount of calculations such as multiplication and addition increases. In addition, in a DNN that can obtain high sound source separation performance, the number of coefficients used increases, and thus there is a problem that the capacity of the memory for storing the coefficients also needs to be increased.

[0005] One of the objectives of the present disclosure is to provide a program, an information processing method, a recording medium, and an information processing apparatus that suppress the amount of calculations as much as possible while obtaining sound source separation performance above a certain level.

Means for Solving the Problems

[0006] The present disclosure, for example, a neural network unit generates sound source separation information for separating a predetermined sound source signal from a mixed sound signal including a plurality of sound source signals, an encoder included in the neural network unit converts a feature amount extracted from the mixed sound signal, The encoder's processing results are input to each of the multiple sub-neural network sections of the neural network section. The decoder in the neural network section receives the processing results of the encoder and the processing results of each of the multiple sub-neural networks as input. It is a program that instructs a computer to execute information processing methods.

[0007] This disclosure includes, for example, The neural network unit generates sound source separation information to separate a predetermined sound source signal from a mixed sound signal containing multiple sound source signals. The encoder in the neural network section converts the feature quantities extracted from the mixed sound signal. The encoder's processing results are input to each of the multiple sub-neural network sections of the neural network section. The decoder in the neural network section receives the processing results of the encoder and the processing results of each of the multiple sub-neural networks as input. It is an information processing method.

[0008] This disclosure includes, for example, The neural network unit generates sound source separation information to separate a predetermined sound source signal from a mixed sound signal containing multiple sound source signals. The encoder in the neural network section converts the feature quantities extracted from the mixed sound signal. The encoder's processing results are input to each of the multiple sub-neural network sections of the neural network section. The decoder in the neural network section receives the processing results of the encoder and the processing results of each of the multiple sub-neural networks as input. It is a recording medium that stores a program that causes a computer to execute an information processing method.

[0009] This disclosure includes, for example, It includes a neural network unit that generates sound source separation information for separating a predetermined sound source signal from a mixed sound signal containing multiple sound source signals, The neural network section is, An encoder that transforms the feature quantities extracted from a mixed sound signal, Multiple sub-neural network sections that receive the processing results of the encoder, The decoder receives the processing results of the encoder and the processing results of each of the multiple subneural networks as input. Equipped with It is an information processing device.

[0010] This disclosure includes, for example, Each of the multiple neural network sections generates sound source separation information to separate different sound source signals from a mixed sound signal containing multiple sound source signals. An encoder in one of the multiple neural network sections converts the feature quantities extracted from the mixed sound signal. The encoder's processing results are input to each of the sub-neural network sections of the multiple neural network sections. It is a program that instructs a computer to execute information processing methods.

[0011] This disclosure includes, for example, Each of the multiple neural network sections generates sound source separation information to separate different sound source signals from a mixed sound signal containing multiple sound source signals. An encoder in one of the multiple neural network sections converts the feature quantities extracted from the mixed sound signal. The encoder's processing results are input to each of the sub-neural network sections of the multiple neural network sections. It is an information processing method.

[0012] This disclosure includes, for example, Each of a plurality of neural network units generates sound source separation information for separating different sound source signals from a mixed sound signal including a plurality of sound source signals. An encoder included in one of the plurality of neural network units converts feature amounts extracted from the mixed sound signal. The processing result of the encoder is input to each of the sub-neural network units included in the plurality of neural network units. A recording medium recording a program for causing a computer to execute an information processing method.

[0013] The present disclosure is, for example, provided with a plurality of neural network units that generate sound source separation information for separating a predetermined sound source signal from a mixed sound signal including a plurality of sound source signals. Each neural network unit includes a sub-neural network unit and a decoder to which the processing result of the sub-neural network is input Department and is provided with One of the plurality of neural network units includes an encoder that converts feature amounts extracted from the mixed sound signal. The processing result of the encoder is input to each of the sub-neural network units included in the plurality of neural network units. An information processing apparatus.

Brief Description of Drawings

[0014] [Figure 1] FIG. 1 is a block diagram referred to when explaining the technology related to the present disclosure. [Figure 2] FIG. 2 is a block diagram referred to when explaining the technology related to the present disclosure. [Figure 3] FIG. 3 is a block diagram referred to when explaining the technology related to the present disclosure. [Figure 4]Figure 4 is a block diagram showing an example configuration of an information processing device according to the first embodiment. [Figure 5] Figure 5 is a flowchart showing the processing flow performed by the information processing device according to the first embodiment. [Figure 6] Figure 6 is a diagram that will be referenced when explaining the effects obtained by the first embodiment. [Figure 7] Figure 7 is a block diagram showing an example configuration of an information processing device according to the second embodiment. [Figure 8] Figure 8 is a flowchart showing the processing flow performed in the information processing device according to the second embodiment. [Figure 9] Figure 9 is a block diagram showing an example configuration of an information processing device according to the third embodiment. [Figure 10] Figure 10 is a diagram illustrating an example of the effects obtained in the embodiment. [Figure 11] Figure 11 is a diagram illustrating a modified example. [Modes for carrying out the invention]

[0015] The embodiments of this disclosure will be described below with reference to the drawings. The description will be given in the following order. <Technologies related to this disclosure> <First Embodiment> <Second Embodiment> <Third Embodiment> <Variation> The embodiments described below are preferred examples of the present disclosure, and the content of the present disclosure is not limited to these embodiments.

[0016] <Technologies related to this disclosure> First, to facilitate understanding of this disclosure, the technologies related to this disclosure will be described. Figure 1 is a block diagram showing an example configuration of an information processing device (information processing device 1A) related to the technologies related to this disclosure. The information processing device 1A is a sound source separation device that separates a desired sound source signal from a mixed sound signal containing multiple sound source signals (for example, the sounds of each instrument that make up the vocals and accompaniment sounds). Specifically, the information processing device 1A is incorporated into smartphones, personal computers, and in-vehicle equipment. For example, using the information processing device 1A, an accompaniment sound signal is separated from a mixed sound signal stored on media such as a CD (Compact Disc) or semiconductor memory, or from a mixed sound signal distributed via a network such as the Internet. The separated accompaniment sound signal is played back. The user sings along with the playback of the accompaniment sound signal. This makes it possible for the user to easily enjoy karaoke without having to prepare the accompaniment sound signal itself. Of course, the use of the information processing device 1A is not limited to karaoke. The sound source separation results of the information processing device 1A may also be used for text transcription processing, etc. The sound source separation processing performed by the information processing device 1A may be performed online (real-time) or offline (batch) processing.

[0017] As shown in Figure 1, the information processing device 1A generally comprises a feature extraction unit 2, a DNN unit 3, a multiplication unit 4 (an example of an arithmetic unit), and a separated sound source signal generation unit 5. The mixed sound signal is input to the feature extraction unit 2. The separated sound source signal generation unit 5 outputs a sound source signal separated from the mixed sound signal (hereinafter also referred to as the separated sound source signal SA as appropriate). As described above, the mixed sound signal is a signal obtained by mixing multiple sound source signals and is a signal digitized by PCM (Pulse Code Modulation), etc. The source of the mixed sound signal can be anything, such as a recording medium or a server device on a network.

[0018] The feature extraction unit 2 performs feature extraction processing to extract features from the mixed sound signal. For example, the feature extraction unit 2 cuts out the data of the mixed sound signal into fixed intervals (frames) of a predetermined length, and performs a frequency transformation (e.g., short-time Fourier transform) on each extracted frame. This frequency transformation process yields a time-series signal of the frequency spectrum. For example, if the frame length is 2048, the frequency transformation length is also set to 2048, and the signal is transformed into a frequency spectrum of 1025, which is less than or equal to the alias frequency. In other words, the processing of the feature extraction unit 2 yields a frequency spectrum, specifically a multidimensional vector (in this example, a vector with 1025 dimensions), as an example of a feature. The processing results of the feature extraction unit 2 are supplied to the subsequent DNN unit 3.

[0019] DNN section 3 generates source separation information to separate a predetermined source signal from a mixed sound signal. Specifically, DNN3 is a multi-layered algorithm modeled after the human brain's neural network, which is designed using machine learning to generate source separation information.

[0020] The DNN unit 3 includes an encoder 31 that converts the features extracted from the mixed sound signal by the feature extraction unit 2, a subneural network unit 32 to which the processing results of the encoder 31 are input, and a decoder 33 to which the processing results of the encoder 31 and the respective processing results of the subneural network unit 32 are input.

[0021] The encoder 31 comprises one or more affine transformation units. The affine transformation unit performs the process shown in equation (1) below. y = f(Wx + b) ... (1) However, in equation (1), x is the input vector, y is the output vector, W is the resulting weight coefficient, b is the bias coefficient, and f is a nonlinear function. The values ​​of W and b are numerical values ​​obtained by pre-training using a large dataset. As the nonlinear function f, for example, the ReLU (Rectified Linear Unit) function, the Sigmoid function, etc., can be applied.

[0022] In this example, the encoder 31 comprises a first affine transformation unit 31A and a second affine transformation unit 31B. The number of affine transformation units in the encoder 31 is appropriately set to ensure a certain level of performance in sound source separation. The encoder 31 performs feature transformation, for example, by compressing the size of the feature quantities. More specifically, the encoder 31 compresses the number of dimensions of a multidimensional vector.

[0023] The sub-neural network section 32 is a neural network located within the DNN section 3. The sub-neural network section 32 can utilize a recurrent neural network (RNN) that uses processing results obtained at least in the past and future for the current input. Future processing results can be used in batch processing. As the recurrent neural network, a neural network using GRU (Gated Recurrent Unit) or LSTM (Long Short-Term Memory) as the algorithm can be applied.

[0024] The subneural network unit 32 comprises a first RNN unit 32A, a second RNN unit 32B, and a third RNN unit 32C. The number of RNN units in the subneural network unit 32 is appropriately set to ensure a certain level of performance in sound source separation. Each RNN unit uses different parameters, and these parameters are stored in the ROM (Read Only Memory) or RAM (Random Access Memory) of each RNN unit (these are not shown in the diagram). In the following description, ROM and RAM will be referred to as memory cells unless there is a need to distinguish between them. The first RNN unit 32A, the second RNN unit 32B, and the third RNN unit 32C sequentially process the processing result of the encoder 31.

[0025] The decoder 33 generates sound source separation information based on the processing results of the encoder 31 and the processing results of the subneural network unit 32. The decoder 33 includes, for example, a third affine transform unit 33A and a fourth affine transform unit 33B. The third affine transform unit 33A connects the processing result of the encoder 31, i.e., the processing result after skipping the subneural network unit 32, with the output of the subneural network unit 32 (also called a skip connection). The fourth affine transform unit 33B performs the affine transform shown in the number (1) above on the processing result of the third affine transform unit 33A. Through the processing of the third and fourth affine transform units 33A and 33B, the feature quantities compressed by the encoder 31 are restored, thereby obtaining a mask, which is an example of sound source separation information. The mask information is output from the DNN unit 3 and supplied to the multiplication unit 4.

[0026] The multiplication unit 4 multiplies the features extracted by the feature extraction unit 2 by the mask supplied by the DNN unit 3. By multiplying the frequency spectrum by the mask, it is possible to either pass the signal in that frequency band directly (a predetermined value in the mask = 1) or block it (a predetermined value in the mask = 0). In other words, the DNN unit 3 estimates a mask that allows only the frequency spectrum of the sound source to be separated to pass through and blocks the frequency spectrum of the unseparated sound.

[0027] The separated sound source signal generation unit 5 performs a process to convert the calculation result of the multiplication unit 4 back into a signal on the time axis (for example, an inverse short-time Fourier transform). This generates the desired sound source signal (the sound source signal to be separated, which is a signal on the time axis). The separated sound source signal SA generated by the separated sound source signal generation unit 5 is used for purposes according to the application.

[0028] Figure 2 shows an example of the input / output sizes of each module constituting the DNN unit 3. The first affine transformation unit 31A receives a 1025-dimensional frequency spectrum as input, and the first affine transformation unit 31A outputs a 256-dimensional vector by performing an affine transformation on it. The second affine transformation unit 31B receives a 256-dimensional frequency spectrum (output of the first affine transformation unit 31A) as input, and the second affine transformation unit 31B outputs a 256-dimensional vector by performing an affine transformation on it. In this way, the first affine transformation unit 31A and the second affine transformation unit 31B reduce the size (number of dimensions) of the multidimensional vector input to the subneural network unit 32 in this embodiment. This improves the generalization performance of the DNN unit 3.

[0029] The first RNN section 32A, the second RNN section 32B, and the third RNN section 32C take a 256-dimensional multidimensional vector as input and output it with the same number of dimensions.

[0030] The third affine transformation unit 33A takes a 512-dimensional vector as input, which is obtained by concatenating the outputs of the second affine transformation unit 31B and the third RNN unit 32C. The performance of the DNN unit 3 can be improved by concatenating the vector before processing by the subneural network unit 32. The third affine transformation unit 33A takes a 512-dimensional vector as input and outputs a 256-dimensional vector by performing an affine transformation on this input. The fourth affine transformation unit 33B takes a 256-dimensional vector as input and outputs a 1025-dimensional vector by performing an affine transformation on this input. The 1025-dimensional vector corresponds to the mask that is multiplied by the multiplication unit 4 on the frequency spectrum supplied from the feature extraction unit 2. Note that the number of modules constituting the DNN unit 3 and the vector sizes of each input and output are examples, and the effective configuration will differ depending on the dataset.

[0031] Figure 3 is a block diagram showing an example configuration of another information processing device (information processing device 1B). While information processing device 1A was configured to separate one sound source signal from a mixed sound signal, information processing device 1B separates two sound source signals from a mixed sound signal. For example, information processing device 1B separates a separated sound source signal SA and a separated sound source signal SB from a mixed sound signal.

[0032] As shown in Figure 3, the information processing device 1B includes, in addition to the configuration of the information processing device 1A, a DNN unit 6, a multiplication unit 7, and a separated sound source signal generation unit 8. The DNN unit 6 includes an encoder 61, a subneural network unit 62, and a decoder 63. The encoder 61 has a first affine transformation unit 61A and a second affine transformation unit 61B. The subneural network unit 62 includes a first RNN unit 62A, a second RNN unit 62B, and a third RNN unit 62C. The decoder 63 includes a third affine transformation unit 63A and a fourth affine transformation unit 63B.

[0033] The operation flow of the DNN unit 6 is, in general terms, almost the same as that of the DNN unit 3. That is, the DNN unit 6 performs the same processing on the feature quantities of the mixed sound signal extracted by the feature quantity extraction unit 2 as the DNN unit 3. This generates a mask for obtaining the separated sound source signal SB. This mask is multiplied by the feature quantities of the mixed sound signal in the multiplication unit 7. The result of the multiplication is converted into a signal on the time axis by the separated sound source signal generation unit 8, thereby generating the separated sound source signal SB.

[0034] Furthermore, the learning of DNN3 and DNN section 6 is performed individually. That is, even if the arrangement of modules in each DNN section is the same, the values ​​of the weight coefficients and bias coefficients in the affine transform section and the coefficients used in the RNN section are different and are optimized according to the sound source signal to be separated. In this way, when the number of sound source signals to be separated increases by N times, the number of sum-of-products operations required for the DNN section and the amount of memory cells used also increase by N times. Details of this disclosure made in view of these points will be described in detail by the embodiments.

[0035] <First Embodiment> [Example of an information processing device configuration] Figure 4 is a block diagram showing an example configuration of an information processing device (information processing device 100) according to the first embodiment. Note that for configurations of the information processing device 100 that are the same as those of information processing device 1A or information processing device 1B, the same reference numerals are used, and redundant explanations are omitted as appropriate. Furthermore, unless otherwise specified, the matters described for information processing devices 1A and 1B are applicable to each embodiment.

[0036] The information processing device 100 includes a DNN unit 11 in place of the DNN unit 3. The DNN unit 11 generates a mask for separating and outputting a predetermined sound source signal (for example, a separated sound source signal SA) from a mixed sound signal.

[0037] The DNN unit 11 includes the encoder 31 and decoder 33 described above. The DNN unit 11 also includes multiple sub-neural network units, specifically two sub-neural network units (sub-neural network units 12 and 13) arranged in parallel. Sub-neural network unit 12 includes a first RNN unit 12A, a second RNN unit 12B, and a third RNN unit 12C. Sub-neural network unit 13 includes a first RNN unit 13A, a second RNN unit 13B, and a third RNN unit 13C. Each sub-neural network unit performs RNN-based processing on its input.

[0038] The output of encoder 31 is split. If encoder 31 outputs a 256-dimensional vector (see Figure 2), the number of dimensions of the vector is divided into two to generate a 128-dimensional first vector and a 128-dimensional second vector. This process is performed, for example, by encoder 31. The first vector is input to, for example, subneural network unit 12, and the second vector is input to, for example, subneural network unit 13. Subneural network unit 12 outputs a 128-dimensional vector by performing RNN-based processing on the first vector. Similarly, subneural network unit 13 outputs a 128-dimensional vector by performing RNN-based processing on the second vector.

[0039] Next, the third affine transformation unit 33A of the decoder 33 concatenates the 128-dimensional vector output from the subneural network unit 12, the 128-dimensional vector output from the subneural network unit 13, and the 256-dimensional vector output from the encoder 31, and performs an affine transformation on the concatenated vector. The other processing is the same as that of the information processing device 1A, so redundant explanations are omitted.

[0040] [Process Flow] The processing flow performed by the information processing device 100 will be explained with reference to the flowchart shown in Figure 5.

[0041] When processing begins, in step ST1, each module constituting the DNN unit 3 reads coefficients stored in a ROM (not shown) or the like. Then, the process proceeds to step ST2.

[0042] In step ST2, the mixed sound signal is input to the information processing device 100. Then, the processing proceeds to step ST3.

[0043] In step ST3, the feature extraction unit 2 extracts feature vectors from the mixed sound signal. For example, a 1025-dimensional feature vector is input to the encoder 31 of the DNN unit 11. Then, the process proceeds to step ST4.

[0044] In step ST4, encoding processing is performed by the encoder 31, specifically the first affine transformation unit 31A and the second affine transformation unit 31B. As a result of the processing, the second affine transformation unit 31B outputs, for example, a 256-dimensional vector. Then the process proceeds to step ST5.

[0045] In step ST5, the 256-dimensional vector is equally divided into two, generating two 128-dimensional vectors (the first and second vectors). The first vector is input to the subneural network unit 12, and the second vector is input to the subneural network unit 13. Note that the processing in step ST5 may be included in the encoding process of step ST4. Then the process proceeds to steps ST6 and ST7.

[0046] In step ST6, processing is performed by the subneural network unit 12 using the first vector. Then, in step ST7, processing is performed by the subneural network unit 13 using the second vector. Note that the processing in steps ST6 and ST7 may be performed in parallel or sequentially. Then the processing proceeds to step ST8.

[0047] In step ST8, the vectors are concatenated. This process is performed, for example, by the decoder 33. The third affine transformation unit 33A generates a 512-dimensional vector by concatenating the 256-dimensional vector output from the second affine transformation unit 31B, the 128-dimensional vector output from the subneural network unit 12, and the 128-dimensional vector output from the subneural network unit 13. Then the process proceeds to step ST9.

[0048] In step ST9, decoding is performed by the third affine transformation unit 33A and the fourth affine transformation unit 33B of the decoder 33. As a result of the decoding process, the fourth affine transformation unit 33B outputs a mask represented by a 1025-dimensional vector. Note that the processing in step ST8 described above may be included in the decoding process of step ST9. Then the process proceeds to step ST10.

[0049] In step ST10, a multiplication process is performed. Specifically, the multiplication unit 4 multiplies the vector output from the feature extraction unit 2 by the mask obtained in the DNN unit 11. Then, the process proceeds to step ST11.

[0050] In step ST11, the separated sound source signal generation process is performed. Specifically, the separated sound source signal generation unit 5 converts the frequency spectrum obtained by the multiplication unit 4 into a signal on the time axis. Then, the process proceeds to step ST12.

[0051] In step ST12, it is determined whether or not the input of the mixed sound signal is continuing. This determination is made, for example, by a CPU (Central Processing Unit) (not shown) that comprehensively controls the operation of the information processing device 100. If there is no input of the mixed sound signal (No), the process ends. If the input of the mixed sound signal is continuing (Yes), the process returns to step ST2, and the process described above is repeated.

[0052] [Effects obtained by this embodiment] An example of the effects that can be obtained by the embodiment described above will now be explained. The sum of the dimensions of the partitioned vectors is 128 + 128 = 256 dimensions, so it appears to be the same as before partitioning. However, the amount of coefficients stored in DNN11 and the number of multiply-accumulate operations can be reduced. A specific example will be given below to explain this.

[0053] For example, consider the vector multiplication (matrix operation) performed in the subneural network section 12 (and similarly in the subneural network section 13). A matrix operation with 256-dimensional vector inputs and 256-dimensional vector outputs results in 256 × 256 = 65536 multiplications. On the other hand, if the matrix is ​​divided into two 128-dimensional sections, the multiplication of the 128-dimensional matrices only needs to be performed twice, so the number of multiplications becomes (128 × 128) × 2 = 32768, which is a smaller value compared to the case without division. Thus, it can be seen that using multiple small matrices is computationally advantageous compared to using one large matrix. Since there are multiple matrix operations within the RNN section modules such as GRU and LSTM that depend on the input and output vector sizes, the configuration according to this embodiment can effectively reduce the amount of computation.

[0054] On the other hand, even if the number of calculations can be reduced, it is undesirable if this results in a decrease in the accuracy of sound source separation. However, in this embodiment, the decrease in the accuracy of sound source separation can be suppressed as much as possible. This point will be explained in detail with reference to Figure 6.

[0055] Figure 6 is a graph showing the relationship between the number of coefficients in the DNN section and the sound source separation performance. The horizontal axis of the graph (Number of Weights) represents the number of coefficients present in the DNN section (affine transformation section and subneural network section), and is a value that is approximately proportional to the number of calculations and the memory cell capacity required for processing in the DNN section. The vertical axis of the graph represents the SDR (Signal to Distortion Ratio) [dB]. SDR is an index that represents the accuracy of separating the target sound source, and a larger value indicates higher separation performance. Therefore, in the graph shown in Figure 6, the further to the upper left the data is plotted, the less computational resources are used and the higher the sound source separation performance.

[0056] We investigated how the number of coefficients and SDR change when the configuration of the DNN is altered. As a result, we obtained four plots (hereinafter referred to as patterns PA, PB, PC, and PD as appropriate) as shown in Figure 6. In this example, we describe an example using GRU as the algorithm for the RNN, but similar results can be obtained using other algorithms.

[0057] Pattern PA in Figure 6 corresponds to the case where the input / output vector size of the subneural network section is 256 dimensions using the general configuration (configuration shown in Figure 1) (1 Grouped-GRU

[0256] ). Pattern PB in Figure 6 corresponds to the case where the input / output vector size of the subneural network section is 84 dimensions using the general configuration (configuration shown in Figure 1) (1 Grouped-GRU

[84] ). Pattern PC in Figure 6 corresponds to the case where two subneural network sections are used as in the configuration of this embodiment, and the input / output vector sizes of each are divided equally (128 dimensions) (2 Grouped-GRU [128,128]). Pattern PD in Figure 6 corresponds to the case where four subneural network sections are used, and the input / output vector sizes of each are divided unequally (128 dimensions, 64 dimensions, 32 dimensions, 32 dimensions) (4 Grouped-GRU [128,64,32,32]).

[0058] In the case of a configuration and vector size corresponding to pattern PA, the number of coefficients was approximately 2,000,000, and the SDR was approximately 12.4. Although the sound source separation performance was high, the large number of coefficients resulted in a large number of calculations. On the other hand, when the configuration and vector size corresponding to pattern PB was used, i.e., the configuration of the DNN section was the same as in the pattern PA case, but the vector size was reduced, the number of coefficients became slightly less than 500,000, and the number of calculations could be reduced. However, the SDR in the case of pattern PB was approximately 11.9, and the sound source separation performance was lower compared to the pattern PA case. Therefore, simply reducing the number of coefficients will result in a decrease in sound source separation performance.

[0059] For the configuration and vector size corresponding to pattern PC, the number of coefficients was approximately 1,500,000. This allowed for fewer coefficients than pattern PA, thus reducing the number of calculations. Furthermore, the SDR for the configuration and vector size corresponding to pattern PC was approximately 12.5, achieving higher sound source separation performance than pattern PA in a typical configuration. In addition, for the configuration and vector size corresponding to pattern PD, the number of coefficients could be reduced compared to pattern PA (approximately 1,500,000), and the SDR also surpassed that of pattern PA. Furthermore, for the configuration and vector size corresponding to pattern PD, the number of coefficients could be reduced even further than that of pattern PC, and the SDR was approximately the same. Thus, both pattern PC and PD are located to the upper left of the line connecting pattern PA and PB, confirming that high sound source separation performance is achieved while reducing the number of calculations compared to conventional methods.

[0060] From the above, it has been confirmed that, according to the information processing device of this embodiment, the number of calculations can be reduced compared to an information processing device with a general configuration, and the sound source separation performance can be improved without degrading it.

[0061] Furthermore, the results shown in Figure 6 confirm that the number of subneural network sections is not limited to two, and that the vector sizes input to each subneural network section can be different (i.e., they can be unevenly divided).

[0062] <Second Embodiment> Next, a second embodiment will be described. Unless otherwise specified, the matters described in the first embodiment, etc., are also applicable to the second embodiment.

[0063] Figure 7 is a block diagram showing an example configuration of an information processing device (information processing device 200) according to the second embodiment. Note that in Figure 7, the configuration related to the DNN unit 3 has been simplified as appropriate due to space limitations. The information processing device 200 is a configuration that accommodates cases where there are multiple sound sources to be separated (for example, the configuration of the information processing device 1B shown in Figure 3), with the encoder configuration standardized.

[0064] In the information processing device 1B shown in Figure 3, the encoders were separated into encoders 31 and 61, but the processing content was the same: compressing the vector size (number of dimensions in this example) of the feature vector extracted from the mixed sound signal. Therefore, as shown in Figure 6, the information processing device 200 shared the encoders in multiple DNN units (for example, DNN units 3 and 6). This makes it possible to reduce the computational load in the information processing device 200. The output of encoder 31 is input to the subneural network unit 32 and decoder 33 of DNN unit 3, and to the subneural network unit 62 and decoder 63 of DNN unit 6. Other processing is basically the same as in the information processing device 1B, so redundant explanations are omitted.

[0065] [Process Flow] The processing flow performed by the information processing device 200 will be explained with reference to the flowchart shown in Figure 8.

[0066] Once processing begins, in step ST21, each module constituting the DNN unit 3 reads coefficients stored in a ROM (not shown) or the like. Then, the process proceeds to step ST22.

[0067] In step ST22, the mixed sound signal is input to the information processing device 200. Then, the processing proceeds to step ST23.

[0068] In step ST23, the feature extraction unit 2 extracts feature vectors from the mixed sound signal. For example, a 1025-dimensional feature vector is input to the encoder 31 of the DNN unit 11. Then, the process proceeds to step ST24.

[0069] In step ST24, encoding processing is performed by the encoder 31, specifically the first affine transformation unit 31A and the second affine transformation unit 31B. As a result of the processing, the second affine transformation unit 31B outputs a vector that has been compressed to, for example, 256 dimensions. This vector is input to the subneural network unit 32 and decoder 33 of the DNN unit 3, and to the subneural network unit 62 and decoder 63 of the DNN unit 6. Then the processing proceeds to steps ST25 and ST29.

[0070] The processing related to steps ST25 to ST28 consists of processing performed by the subneural network unit 32, decoding processing performed by the decoder 33, multiplication processing performed by the multiplication unit 4, and separated sound source signal generation processing performed by the separated sound source signal generation unit 5. The separated sound source signal SA is generated in the separated sound source signal generation processing. Furthermore, the processing related to steps ST29 to ST32 consists of processing performed by the subneural network unit 62, decoding processing performed by the decoder 63, multiplication processing performed by the multiplication unit 7, and separated sound source signal generation processing performed by the separated sound source signal generation unit 8. The separated sound source signal SB is generated in the separated sound source signal generation processing. The contents of each process have already been explained, so redundant explanations will be omitted as appropriate. Regarding the processing related to steps ST28 and ST32, the processing related to step ST33 is performed.

[0071] In step ST33, it is determined whether or not the input of the mixed sound signal is continuing. This determination is made, for example, by a CPU (not shown) that comprehensively controls the operation of the information processing device 200. If there is no input of the mixed sound signal (No), the process ends. If the input of the mixed sound signal is continuing (Yes), the process returns to step ST22, and the process described above is repeated.

[0072] In addition, the decoder and decoder 63 may be shared in the information processing device 200. However, the inputs to decoder 33 and decoder 63 each pass through a sub-neural network section whose coefficients are optimized for the respective sound source signals to be separated. Therefore, it is preferable from the viewpoint of not degrading the sound source separation performance if the coefficients of decoder 33 are also optimized for the sound source signals to be separated. Accordingly, it is preferable that decoder 33 and decoder 63 are provided in a configuration corresponding to each sound source signal to be separated.

[0073] <Third Embodiment> Next, a third embodiment will be described. Unless otherwise specified, the matters described in the first and second embodiments are also applicable to the third embodiment. The third embodiment is, in general terms, a combination of the first and second embodiments.

[0074] Figure 9 is a block diagram showing an example configuration of an information processing device (information processing device 300) according to the third embodiment. In the information processing device 300, the DNN unit 11 described in the first embodiment is used instead of the DNN unit 3 of the information processing device 200 described above. Also, in the information processing device 300, the DNN unit 6A is used instead of the DNN unit 6 of the information processing device 200 described above. The configuration of the DNN unit 6A differs from that of the DNN unit 6 and the subneural network unit. That is, the DNN unit 6A includes a plurality of subneural network units, similar to the first embodiment. The DNN unit 6A includes, for example, a subneural network unit 65 and a subneural network unit 66. The subneural network unit 65 includes a first RNN unit 65A, a second RNN unit 65B, and a third RNN unit 65C. The subneural network unit 66 includes a first RNN unit 66A, a second RNN unit 66B, and a third RNN unit 66C. The DNN unit 6A is the same as the DNN unit 6 in that it includes a decoder 63. The processing performed by the information processing device 300 has been explained in the first and second embodiments, so a redundant explanation will be omitted. The same effects as the first and second embodiments can be obtained in the third embodiment.

[0075] <Summary of effects obtained in the embodiment> Figure 10 shows specific numerical examples of the number of coefficients used in the DNN section for the first to third embodiments described above. There are four basic configurations: a general configuration (see Figure 1), a configuration with multiple subneural network sections (see Figure 4), a configuration with a common encoder (see Figure 7), and a configuration with multiple subneural network sections and a common encoder (see Figure 9). The number of sound sources to be separated was set to 2 or 10, and the configuration included subneural network sections corresponding to the number of sound sources to be separated.

[0076] As shown in Figure 10, in a typical configuration, when the number of sound sources to be separated was 2, the number of coefficients used in the DNN section was approximately 4,000,000. In a typical configuration, when the number of sound sources to be separated was 10, the number of coefficients used in the DNN section was approximately 20,000,000. The number of coefficients used in the DNN section in other configurations is shown as a relative value (with the number of coefficients used in the DNN section in the typical configuration set to 100%) and as an approximate number of coefficients. The GRU algorithm was used for each RNN section, and for configurations with multiple subneural network sections, the values ​​were calculated assuming the input / output vector sizes were equally divided.

[0077] In a configuration with multiple subneural network sections, the number of coefficients used in the DNN section was approximately 3,100,000 (about 76%) when there were two sound sources to be separated, and approximately 15,400,000 (about 76%) when there were ten sound sources to be separated. In other words, the number of coefficients could be reduced compared to a typical configuration. To put it another way, the number of calculations could be reduced.

[0078] In a configuration with a common encoder, the number of coefficients used in the DNN section could be reduced as the number of sound sources increased. (For 2 sound sources, the number of coefficients used in the DNN section was approximately 3,600,000 (about 76%), and for 10 sound sources to be separated, the number of coefficients used in the DNN section was approximately 16,200,000 (about 80%).)

[0079] In a configuration that includes multiple subneural network sections and uses a common encoder, the number of coefficients used in the DNN section could be further reduced. (For two sound sources, the number of coefficients used in the DNN section was approximately 2,630,000 (about 65%), and for 10 sound sources to be separated, the number of coefficients used in the DNN section was approximately 11,300,000 (about 56%).)

[0080] <Variation> Although several embodiments of this disclosure have been described above, this disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of this disclosure.

[0081] As shown in Figure 11, the information processing device 300 may be configured to include a filter unit 9 (post-filter) after the multiplier units 4 and 7. The filter unit 9 uses a plurality of separated sound source signals (two sound sources in the example shown in Figure 11) to separate the desired sound source signal with higher precision. For example, suppose the multiplier unit 4 outputs a separated vocal signal and the multiplier unit 7 outputs a separated piano accompaniment signal. The filter unit 9, while referring to the piano accompaniment signal, removes the residual components (noise components) of the piano accompaniment signal contained in the vocal signal, thereby separating the vocal signal (an example of a separated sound source signal SA) with higher precision. As the filter unit 9, a known filter such as a single-channel Wiener filter can be used.

[0082] This disclosure also allows for the adoption of a cloud computing configuration in which a single function is shared and processed collaboratively by multiple devices via a network. For example, a feature extraction unit may be provided on a server device, and the feature extraction process may be performed on the server device.

[0083] Furthermore, this disclosure can be implemented in any form, such as an apparatus, method, program, recording medium containing the program, or system. For example, a program that performs the functions described in the embodiments described above can be made downloadable, and an apparatus that does not have the functions described in the embodiments can download and install the program, thereby enabling the apparatus to perform the control described in the embodiments. This disclosure can also be implemented by a server that distributes such a program. In addition, the matters described in each embodiment and modification can be combined as appropriate. Furthermore, the effects exemplified herein should not be interpreted as limiting the content of this disclosure.

[0084] This disclosure may also be structured as follows: (1) The neural network unit generates sound source separation information to separate a predetermined sound source signal from a mixed sound signal containing multiple sound source signals. The encoder in the neural network unit converts the feature quantities extracted from the mixed sound signal, The processing result of the encoder is input to each of the multiple sub-neural network sections of the neural network section. The decoder of the neural network unit receives the processing result of the encoder and the processing results of each of the multiple sub-neural network units as input. A program that instructs a computer to execute information processing methods. (2) The subneural network section is a recurrent neural network that utilizes processing results obtained in at least one of the past and future time periods for the current input. The program described in (1). (3) The aforementioned recurrent neural network is a neural network that uses GRU (Gated Recurrent Unit) or LSTM (Long Short-Term Memory) as its algorithm. The program described in (2). (4) The encoder performs the conversion by compressing the size of the feature quantities. The program described in any of (1) through (3). (5) The aforementioned feature and its size are defined by a multidimensional vector and the number of dimensions of the vector. The encoder compresses the number of dimensions of the vector. The program described in (4). (6) The size of the feature quantities is divided equally to correspond to the number of subneural network sections. Each of the divided feature quantities is input to the corresponding subneural network section. The program described in (4) or (5). (7) The size of the aforementioned feature is divided unevenly, Each of the divided feature quantities is input to the corresponding subneural network section. The program described in (4) or (5). (8) The encoder is composed of one or more affine conversion units. The program described in any of (1) through (7). (9) The decoder generates the sound source separation information based on the processing results of the encoder and the processing results of each of the multiple sub-neural networks. The program described in any of (4) through (7). (10) The decoder is composed of one or more affine conversion units. A program described in any of (1) through (9). (11) The feature extraction unit extracts the feature quantities from the mixed sound signal. The program described in any of (1) through (10). (12) The calculation unit multiplies the feature quantities of the mixed sound signal by the sound source separation information output from the decoder. The program described in any of (1) through (11). (13) The separated sound source signal generation unit generates the predetermined sound source signal based on the calculation result of the calculation unit. The program described in (12). (14) The neural network unit generates sound source separation information to separate a predetermined sound source signal from a mixed sound signal containing multiple sound source signals. The encoder in the neural network unit converts the feature quantities extracted from the mixed sound signal, The processing result of the encoder is input to each of the multiple sub-neural network sections of the neural network section. The decoder of the neural network unit receives the processing result of the encoder and the processing results of each of the multiple sub-neural network units as input. Information processing methods. (15) The neural network unit generates sound source separation information to separate a predetermined sound source signal from a mixed sound signal containing multiple sound source signals. The encoder in the neural network unit converts the feature quantities extracted from the mixed sound signal, The processing result of the encoder is input to each of the multiple sub-neural network sections of the neural network section. The decoder of the neural network unit receives the processing result of the encoder and the processing results of each of the multiple sub-neural network units as input. A recording medium that stores a program that causes a computer to execute information processing methods. (16) It includes a neural network unit that generates sound source separation information for separating a predetermined sound source signal from a mixed sound signal containing multiple sound source signals, The aforementioned neural network unit is An encoder that converts the feature quantities extracted from the aforementioned mixed sound signal, Multiple sub-neural network units to which the processing results of the encoder are input, A decoder to which the processing results of the encoder and the processing results of each of the multiple subneural network sections are input. Equipped with Information processing device. (17) Each of the multiple neural network sections generates sound source separation information to separate different sound source signals from a mixed sound signal containing multiple sound source signals. An encoder provided in one of the aforementioned neural network units converts the feature quantities extracted from the mixed sound signal. The processing result of the encoder is input to each of the sub-neural network units provided by the multiple neural network units. A program that causes a computer to execute an information processing method. (18) Each of the aforementioned neural network units comprises a plurality of the aforementioned sub-neural network units. The processing result of the encoder is input to each of the multiple subneural network sections. The program described in (17). (19) Each of the neural network units has a processing unit that multiplies the feature quantities of the mixed sound signal by the sound source separation information output from the decoder, The filter unit separates the predetermined sound source signal based on the processing results of the multiple calculation units. The program described in (17) or (18). (20) Each of the multiple neural network sections generates sound source separation information to separate different sound source signals from a mixed sound signal containing multiple sound source signals. An encoder provided in one of the aforementioned neural network units converts the feature quantities extracted from the mixed sound signal. The processing result of the encoder is input to each of the sub-neural network units provided by the multiple neural network units. Information processing methods. (twenty one) Each of the multiple neural network sections generates sound source separation information to separate different sound source signals from a mixed sound signal containing multiple sound source signals. An encoder provided in one of the aforementioned neural network units converts the feature quantities extracted from the mixed sound signal. The processing result of the encoder is input to each of the sub-neural network units provided by the multiple neural network units. A recording medium that stores a program that causes a computer to execute information processing methods. (twenty two) It comprises multiple neural network units that generate sound source separation information for separating a predetermined sound source signal from a mixed sound signal containing multiple sound source signals. Each of the aforementioned neural network sections is: Subneural network section, A decoder to which the processing result of the subneural network is input. Equipped with, One of the aforementioned neural network units is: The system includes an encoder that converts the feature quantities extracted from the mixed sound signal, The processing result of the encoder is input to each of the sub-neural network units provided by the plurality of neural network units. Information processing device. [Explanation of Symbols]

[0085] 2. Feature extraction section 4, 7... Multiplication section 5, 8...Separated signal generation section 6, 11...DNN section 9. Filter section 12, 13... Subneural Network Section 31... Encoder 32...Decoder 100, 200, 300... Information processing device

Claims

1. The neural network unit generates sound source separation information to separate a predetermined sound source signal from a mixed sound signal containing multiple sound source signals. The encoder in the neural network unit converts the feature quantities extracted from the mixed sound signal, The processing result of the encoder is input to each of the multiple sub-neural network sections of the neural network section. The decoder of the neural network unit receives the processing result of the encoder and the processing results of each of the multiple sub-neural network units as input. A program that instructs a computer to execute information processing methods.

2. The subneural network section is a recurrent neural network that utilizes processing results obtained in at least one of the past and future time periods for the current input. The program according to claim 1.

3. The aforementioned recurrent neural network is a neural network that uses GRU (Gated Recurrent Unit) or LSTM (Long Short-Term Memory) as its algorithm. The program according to claim 2.

4. The encoder performs the conversion by compressing the size of the feature quantities. The program according to claim 1.

5. The aforementioned feature and its size are defined by a multidimensional vector and the number of dimensions of the vector. The encoder compresses the number of dimensions of the vector. The program according to claim 4.

6. The size of the feature quantities is divided equally to correspond to the number of subneural network sections. Each of the divided feature quantities is input to the corresponding subneural network section. The program according to claim 4.

7. The size of the aforementioned feature is divided unevenly, Each of the divided feature quantities is input to the corresponding subneural network section. The program according to claim 4.

8. The encoder is composed of one or more affine conversion units. The program according to claim 1.

9. The decoder generates the sound source separation information based on the processing results of the encoder and the processing results of each of the multiple sub-neural network sections. The program according to claim 4.

10. The decoder is composed of one or more affine conversion units. The program according to claim 1.

11. The feature extraction unit extracts the feature quantities from the mixed sound signal. The program according to claim 1.

12. The calculation unit multiplies the feature quantities of the mixed sound signal by the sound source separation information output from the decoder. The program according to claim 1.

13. The separated sound source signal generation unit generates the predetermined sound source signal based on the calculation result of the calculation unit. The program according to claim 12.

14. The neural network unit generates sound source separation information to separate a predetermined sound source signal from a mixed sound signal containing multiple sound source signals. The encoder in the neural network unit converts the feature quantities extracted from the mixed sound signal, The processing result of the encoder is input to each of the multiple sub-neural network sections of the neural network section. The decoder of the neural network unit receives the processing result of the encoder and the processing results of each of the multiple sub-neural network units as input. Information processing methods.

15. The neural network unit generates sound source separation information to separate a predetermined sound source signal from a mixed sound signal containing multiple sound source signals. The encoder in the neural network unit converts the feature quantities extracted from the mixed sound signal, The processing result of the encoder is input to each of the multiple sub-neural network sections of the neural network section. The decoder of the neural network unit receives the processing result of the encoder and the processing results of each of the multiple sub-neural network units as input. A recording medium that stores a program that causes a computer to execute information processing methods.

16. It includes a neural network unit that generates sound source separation information for separating a predetermined sound source signal from a mixed sound signal containing multiple sound source signals, The aforementioned neural network unit is An encoder that converts the feature quantities extracted from the aforementioned mixed sound signal, Multiple sub-neural network units to which the processing results of the encoder are input, A decoder to which the processing results of the encoder and the processing results of each of the multiple subneural network sections are input. Equipped with Information processing device.

17. Each of the multiple neural network sections generates sound source separation information to separate different sound source signals from a mixed sound signal containing multiple sound source signals. An encoder provided in one of the aforementioned neural network units converts the feature quantities extracted from the mixed sound signal. The processing result of the encoder is input to each of the sub-neural network units provided by the multiple neural network units. A program that instructs a computer to execute information processing methods.

18. Each of the aforementioned neural network units comprises a plurality of the aforementioned sub-neural network units. The processing result of the encoder is input to each of the multiple subneural network sections. The program according to claim 17.

19. Each of the neural network units has a processing unit that multiplies the feature quantities of the mixed sound signal by the sound source separation information. The filter unit separates a predetermined sound source signal based on the processing results of multiple calculation units. The program according to claim 18.

20. Each of the multiple neural network sections generates sound source separation information to separate different sound source signals from a mixed sound signal containing multiple sound source signals. An encoder provided in one of the aforementioned neural network units converts the feature quantities extracted from the mixed sound signal. The processing result of the encoder is input to each of the sub-neural network units provided by the multiple neural network units. Information processing methods.

21. Each of the multiple neural network sections generates sound source separation information to separate different sound source signals from a mixed sound signal containing multiple sound source signals. An encoder provided in one of the aforementioned neural network units converts the feature quantities extracted from the mixed sound signal. The processing result of the encoder is input to each of the sub-neural network units provided by the multiple neural network units. A recording medium that stores a program that causes a computer to execute information processing methods.

22. It comprises multiple neural network units that generate sound source separation information for separating a predetermined sound source signal from a mixed sound signal containing multiple sound source signals. Each of the aforementioned neural network sections is: Subneural network section, The decoder to which the processing result of the subneural network unit is input Equipped with, One of the aforementioned neural network units is: The system includes an encoder that converts the feature quantities extracted from the mixed sound signal, The processing result of the encoder is input to each of the sub-neural network units provided by the plurality of neural network units. Information processing device.

Images