Information processing device, sound source separation processing method, and program
The information processing device employs a single learning model to categorize and separate multiple sound sources based on common characteristics, addressing the high costs and memory issues of conventional methods, enabling efficient sound source separation on small devices.
Patent Information
- Application Number
- PCT/JP2024/015606
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-01
- Filing Date
- 2024-04-19
- Publication Date
- 2024-11-07
AI Technical Summary
Conventional sound source separation processing using neural networks requires generating sound source-specific models for each type of sound, leading to increased learning costs and memory requirements, making it difficult for small devices to process multiple sound sources effectively.
An information processing device and method that uses a single learning model to extract multiple types of musical instrument sounds, including vocal, guitar, and piano sounds, by categorizing these sounds into common sound characteristics, reducing the need for multiple specific models and lowering memory requirements.
This approach reduces learning costs and memory needs, enabling efficient sound source separation on devices with limited resources while maintaining effective separation performance for various sound sources.
Smart Images

Figure JP2024015606_07112024_PF_FP_ABST
Abstract
Description
Information processing device, sound source separation processing method, and program
[0001] The present disclosure relates to an information processing device, a sound source separation processing method, and a program, and more particularly to an information processing device, a sound source separation processing method, and a program that execute sound source separation processing.
[0002] Sound source separation processing is known as processing for separating sound data consisting of many different instrument sounds, such as an orchestra, into sound source-based sound data according to the type of instrument, such as only violin sounds or only trumpet sounds.
[0003] With the recent advances in AI (Artificial Intelligence) technology, it has become possible to perform sound source separation processing using a learning model such as a neural network, for example, a deep neural network (DNN). Patent Document 1 (International Publication No. WO2023 / 276235) is an example of a conventional technique that discloses sound source separation processing using a DNN.
[0004] However, in a configuration in which sound source separation processing is performed using a neural network such as a DNN, it is necessary to generate a neural network specific to the type of sound source to be separated, such as a human voice, guitar, piano, violin, etc.
[0005] For example, to separate only human voices from audio data containing a mixture of sounds from various sound sources, it is necessary to generate and use a neural network with parameters specific to human voices.Similarly, to separate only guitar sounds from audio data containing a mixture of sounds from various sound sources, it is necessary to generate and use a neural network with parameters specific to guitar sounds.
[0006] When performing sound source separation processing using a neural network such as a DNN, it is necessary to generate a neural network specific to the sound source according to the type of sound source to be separated.
[0007] However, in order to generate a sound source-specific learning model, such as a sound source-specific neural network corresponding to the type of sound source, a learning process using a large amount of learning data is required. That is, a variety of different speech data is input as learning data into a DNN or the like to perform a learning process, and a process of calculating separation parameters specialized for the type of sound source to be separated is required. Therefore, as the number of sound source types to be separated increases, the amount of learning process required also increases according to the number of sound source types, resulting in a problem of enormous learning process costs.
[0008] Sound source-specific parameters are associated with learning models such as neural networks that are generated by the learning process. The amount of data, such as parameters associated with these sound source-specific learning models, increases in proportion to the number of sound sources to be separated, and therefore the memory capacity for storing this data also increases.
[0009] Therefore, for example, small devices with limited memory capacity may not be able to record learning models and parameters corresponding to many sound sources, which may cause the small device to have difficulty separating various sound sources.
[0010] International Publication No. WO2023 / 276235
[0011] The present disclosure has been made in consideration of, for example, the above-mentioned problems, and aims to provide an information processing device, a sound source separation processing method, and a program that reduce learning costs and enable a reduction in storage memory capacity for learning models, parameters, etc. in a configuration that performs sound source separation processing using a learning model such as DNN.
[0012] A first aspect of the present disclosure is an information processing device that includes a sound source separation processing execution unit that executes sound source separation processing, the sound source separation processing execution unit having a configuration that executes sound source separation processing using a learning model, and that executes processing to extract a plurality of different types of instrument sounds using the same learning model.
[0013] Furthermore, a second aspect of the present disclosure is a sound source separation processing method executed in an information processing device, wherein the information processing device has a sound source separation processing execution unit that executes sound source separation processing, and the sound source separation processing execution unit executes processing to extract a plurality of different types of instrument sounds using the same learning model.
[0014] Furthermore, a third aspect of the present disclosure is a program for causing an information processing device to execute a sound source separation process, the information processing device having a sound source separation process execution unit that executes the sound source separation process, the program causing the sound source separation process execution unit to execute a process of extracting a plurality of different types of musical instrument sounds using the same learning model.
[0015] The program of the present disclosure is, for example, a program that can be provided in a computer-readable format via a storage medium or a communication medium to an information processing device or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or the computer system.
[0016] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.
[0017] According to one embodiment of the present disclosure, a configuration is realized that executes a sound source separation process that extracts a plurality of different instrument sounds using a single learning model. Specifically, for example, the configuration includes a sound source separation process execution unit that executes the sound source separation process, and the sound source separation process execution unit has a configuration that executes the sound source separation process using a learning model, and executes a process that extracts a plurality of different types of instrument sounds using the same learning model. The sound source separation process execution unit has three types of learning models: (a) a learning model that extracts vocal sounds (singing sounds) and voice sounds (conversation sounds), (b) a learning model that extracts guitar sounds and piano sounds, and (c) a learning model that extracts noise sounds. This configuration realizes a configuration that executes a sound source separation process that extracts a plurality of different instrument sounds using a single learning model. Note that the effects described in this specification are merely examples and are not limited to those described herein, and additional effects may also be present.
[0018] 1 is a diagram illustrating an example of sound source separation processing. FIG. 1 is a diagram illustrating a specific configuration example of a sound source separation processing execution unit, and specific processing executed by the sound source separation processing execution unit. FIG. 2 is a diagram illustrating a specific example of processing for generating a sound source-specific learning model (DNN). FIG. 3 is a diagram illustrating a configuration example of learning data consisting of pairs of input data and supervised data. FIG. 4 is a diagram illustrating a learning process for generating a guitar sound separation unit (DNN for guitar sound separation) in the sound source separation processing execution unit shown in FIG. 2. FIG. 5 is a diagram illustrating a configuration example of learning data consisting of pairs of input data and supervised data. FIG. 6 is a diagram illustrating a configuration example for separating vocal sounds and piano sounds. FIG. 7 is a diagram illustrating a specific configuration example of a sound source separation processing execution unit, and specific processing executed by the sound source separation processing execution unit. FIG. 8 is a diagram illustrating a learning process for generating a piano sound separation unit (DNN for piano sound separation). FIG. 9 is a diagram illustrating a configuration example of learning data consisting of pairs of input data and supervised data. FIG. 10 is a diagram illustrating a specific configuration example of a sound source separation processing execution unit that performs separation processing on input data in which sounds of three or more sound sources are recorded, and specific processing executed. 13 is a diagram illustrating a configuration and processing for executing processing for extracting and separating noise sounds in addition to processing for separating vocal sounds, guitar sounds, and piano sounds from input data. FIG. 14 is a diagram illustrating an example of learning data required for generating four neural networks that constitute the four sound source separation units shown in FIG. 12. FIG. 15 is a diagram illustrating an example of sound source separation processing using an information processing device of the present disclosure. FIG. 16 is a diagram illustrating an example of a UI for instructing the start of sound source separation processing. FIG. 17 is a diagram illustrating an example of a UI of an editing screen displayed on a display unit of an information processing device. FIG. 18 is a block diagram illustrating an example of a configuration of an information processing device of the present disclosure. FIG. 19 is a diagram illustrating types of audio data (recorded data) that are the target of sound source separation processing executed by an information processing device of the present disclosure. FIG. 20 is a diagram illustrating sound source separation processing of three types of sound source data executed by an information processing device of the present disclosure. FIG. 21 is a diagram illustrating an example of a configuration of a sound source separation processing execution unit of an information processing device of the present disclosure and the processing to be executed. FIG. 22 is a diagram illustrating an example of learning data used in learning processing (supervised learning) for generating three types of learning models (DNN) used by the sound source separation processing execution unit of an information processing device of the present disclosure.1 is a diagram illustrating a learning process for generating a vocal & voice sound separation unit (vocal & voice sound separation DNN) that extracts vocal sounds (singing sounds) or voice sounds (conversation sounds) from input data. FIG. 2 is a diagram illustrating a learning process for generating a guitar & piano sound separation unit (guitar & piano sound separation DNN) that extracts guitar sounds and piano sounds from input data. FIG. 3 is a diagram illustrating a learning process for generating a noise sound separation unit (noise sound separation DNN) that extracts noise sounds from input data. FIG. 4 is a diagram illustrating a flowchart illustrating a sequence of sound source separation processing executed by an information processing device of the present disclosure. FIG. 5 is a diagram illustrating an example of a UI that instructs the start of sound source separation processing. FIG. 6 is a diagram illustrating an example of a UI that instructs the start of sound source separation processing. FIG. 7 is a diagram illustrating an example of a UI that instructs the start of sound source separation processing. FIG. 8 is a diagram illustrating an example of a start instruction for sound source separation processing using a UI that instructs the start of sound source separation processing. FIG. 9 is a diagram illustrating a processing mode of sound source separation processing executed by a sound source separation processing execution unit of an information processing device. FIG. 10 is a diagram illustrating an example of a start instruction for sound source separation processing using a UI that instructs the start of sound source separation processing. Fig. 1 is a diagram illustrating a processing mode of sound source separation processing executed by a sound source separation processing execution unit of an information processing device. Fig. 2 is a diagram illustrating an example of display data for playback processing or editing processing of audio data in sound source units. Fig. 3 is a diagram illustrating an example of a system that executes sound source separation processing using an external device. Fig. 4 is a diagram illustrating an example of the configuration of each device in a system that executes sound source separation processing using an external device. Fig. 5 is a diagram illustrating an example of the hardware configuration of an information processing device of the present disclosure.
[0019] Hereinafter, the information processing device, sound source separation processing method, and program of the present disclosure will be described in detail with reference to the drawings. The description will be made according to the following items: 1. General configuration example of sound source separation processing using a learning model 2. Overview of sound source separation processing using an information processing device of the present disclosure 3. Configuration example of an information processing device of the present disclosure 4. Details of sound source separation processing performed by an information processing device of the present disclosure 5. Sequence of sound source separation processing performed by an information processing device of the present disclosure 6. Configuration example for performing sound source separation processing in an external device 7. Hardware configuration example of an information processing device of the present disclosure 8. Summary of the configuration of the present disclosure
[0020] [1. General Configuration Example of Sound Source Separation Processing Using Learning Model] First, a general configuration example of sound source separation processing using a learning model will be described.
[0021] An example of a general sound source separation process will be described with reference to Fig. 1. Fig. 1 shows a user 1 singing while playing the guitar. The vocal sounds produced by the user 1 and the guitar sounds produced by the guitar are input via a microphone and recorded as an audio data file in a storage unit 10.
[0022] The audio data file recorded in the storage unit 10 is an audio data file containing a mixture of vocal sounds from the user 1 and guitar sounds, which are sounds played on a guitar.
[0023] The sound source separation processing execution unit 20 reads out the audio data stored in the storage unit 10 and executes the sound source separation processing. That is, from the audio data in which vocal sounds uttered by the user 1 and guitar sounds are mixed, the sound source separation processing execution unit 20 executes processing to extract only the vocal sounds and processing to extract only the guitar sounds.
[0024] A specific configuration example of the sound source separation processing execution unit 20 and specific processing executed by the sound source separation processing execution unit 20 will be described with reference to FIG.
[0025] 2 from the storage unit 10. The input data is audio data that contains a mixture of vocal sounds uttered by the user 1 and guitar sounds, and changes along the time axis.
[0026] The sound source separation processing execution unit 20 receives the audio data containing a mixture of vocal sounds and guitar sounds, and executes a process of extracting only the vocal sounds and a process of extracting only the guitar sounds. The sound source separation processing execution unit 20 performs the sound source separation processing using a learning model made up of a neural network, such as a deep neural network (DNN).
[0027] As shown in FIG. 2 , the sound source separation processing execution unit 20 has a series of data processing units consisting of a pre-processing unit (STFT) 21, a vocal sound separation unit (DNN for vocal sound separation) 22, and a post-processing unit (ISTFT) 23, and a further series of data processing units consisting of a pre-processing unit (STFT) 31, a guitar sound separation unit (DNN for guitar sound separation) 32, and a post-processing unit (ISTFT) 33.
[0028] The vocal sounds contained in the input data are extracted by a pre-processing unit (STFT) 21, a vocal sound separation unit (DNN for vocal sound separation) 22, and a post-processing unit (ISTFT) 23 shown in the upper row. On the other hand, the guitar sounds contained in the input data are extracted by a pre-processing unit (STFT) 31, a guitar sound separation unit (DNN for guitar sound separation) 32, and a post-processing unit (ISTFT) 33 shown in the lower row.
[0029] The pre-processing unit (STFT) 21 shown in the upper part performs a short-time Fourier transform (STFT) on the input data. This STFT process converts the input data, which is time-domain data in which the audio signal changes along the time axis, into frequency-domain data (frequency spectrum). This converted data is input to the vocal sound separation unit (DNN for vocal sound separation) 22.
[0030] The vocal sound separation unit (DNN for vocal sound separation) 22 is a sound source separation unit configured using a neural network, specifically a learning model such as a DNN (Deep Neural Network).
[0031] The vocal sound separation unit (DNN for vocal sound separation) 22 is a neural network (learning model) that extracts sound data having vocal sound feature quantities (frequency, etc.) corresponding to a human voice from the audio data of various sound sources included in the input data. This neural network (learning model) is generated by a learning process executed in advance, and is a neural network (learning model) in which parameters for extracting sound data having vocal sound feature quantities corresponding to a human voice are set.
[0032] In this vocal sound separation unit (DNN for vocal sound separation) 22, audio data corresponding to vocal sounds that correspond to human voices is extracted from audio data of various sound sources included in the input data.
[0033] The audio data extracted by the vocal sound separation unit (DNN for vocal sound separation) 22 is input to a post-processing unit (ISTFT) 23. The post-processing unit (ISTFT) 23 executes the inverse process of the pre-processing unit (STFT) 21 described above. That is, it executes an inverse short-time Fourier transform (ISTFT).
[0034] This ISTFT processing converts the frequency domain data (frequency spectrum) into time domain data in which the audio signal changes along the original time axis. The output of this post-processing unit (ISTFT) 23 is the audio data extracted by the vocal sound separation unit (DNN for vocal sound separation) 22, i.e., the audio data of the vocal sound.
[0035] Meanwhile, the pre-processing unit (STFT) 31 shown in the lower part also performs a short-time Fourier transform (STFT) on the input data to convert the input data, which is time-domain data, into frequency-domain data (frequency spectrum). This converted data is input to the guitar sound separation unit (DNN for guitar sound separation) 32.
[0036] The guitar sound separation unit (DNN for guitar sound separation) 32 is also a sound source separation unit configured using a neural network, specifically a learning model such as a DNN (Deep Neural Network).
[0037] The guitar sound separation unit (DNN for guitar sound separation) 32 is a neural network (learning model) that extracts sound data having the characteristics (frequency, etc.) of guitar sounds from the audio data of various sound sources included in the input data. This neural network (learning model) is also generated by a learning process executed in advance, and is a neural network (learning model) in which parameters for extracting sound data having the characteristics of guitar sounds are set.
[0038] In this guitar sound separation unit (DNN for guitar sound separation) 32, audio data corresponding to the sound of the guitar is extracted from audio data of various sound sources included in the input data.
[0039] The audio data extracted by the guitar sound separation unit (DNN for guitar sound separation) 32 is input to a post-processing unit (ISTFT) 33. The post-processing unit (ISTFT) 33 performs the inverse process of the pre-processing unit (STFT) 31. In other words, it performs an inverse short-time Fourier transform (ISTFT).
[0040] This ISTFT processing converts the frequency domain data (frequency spectrum) into time domain data in which the audio signal changes along the original time axis. The output of this post-processing unit (ISTFT) 33 is the audio data extracted by the guitar sound separation unit (DNN for guitar sound separation) 32, i.e., the audio data of the guitar sound.
[0041] In this way, in a configuration in which sound source separation processing is performed using a neural network such as a DNN, a neural network specific to the sound source is used, corresponding to the type of sound source to be separated, such as a human voice, guitar, piano, violin, etc. In other words, a neural network is used in which parameters specific to the sound source to be separated are set.
[0042] In order to generate a learning model specific to a sound source, such as a neural network specific to a sound source type, a learning process using a large amount of learning data is required, i.e., various different speech data must be input as learning data to a DNN or the like to perform the learning process.
[0043] A specific example of a process for generating a sound source-specific learning model (DNN) will be described with reference to Fig. 3. The example shown in Fig. 3 is a diagram illustrating a learning process for generating a vocal sound separation unit (DNN for vocal sound separation) 22 for extracting vocal sounds corresponding to human voices from input data.
[0044] The learning process shown in Fig. 3 is a supervised learning process, which uses a large number of "input data" consisting of a mixture of various sound source data and a data set of "ground truth data" which is extracted data of vocal sound data included in each input data.
[0045] In this way, supervised learning processing involves preparing correct answer data for input data in advance, inputting the input data to a vocal sound separation unit (DNN for vocal sound separation) 22 via a pre-processing unit (STFT) 21, and outputting the separation processing results in the vocal sound separation unit (DNN for vocal sound separation) 22 via a post-processing unit (ISTFT) 23 to obtain "output data."
[0046] This "output data" is compared with the "correct data," and a process is executed to set and update the parameters of the neural network that constitutes the vocal sound separation unit (DNN for vocal sound separation) 22 so that the difference between these two data is small.
[0047] A large number of training data consisting of pairs of "input data" and "correct answer data" are prepared. For example, as shown in Figure 4, it is necessary to generate in advance a large number of different pairs of "input data" and "correct answer data" with training data identifiers 00001 to nnnnn, that is, training data consisting of combinations of "input data" consisting of mixed data of various sound sources and "correct answer data" consisting only of vocal sounds included in the input data.
[0048] These large amounts of input data are sequentially input to the vocal sound separation unit (DNN for vocal sound separation) 22, and the parameters of the neural network that constitutes the vocal sound separation unit (DNN for vocal sound separation) 22 are sequentially updated so that the difference between the "output data" and the "correct answer data" becomes smaller. By repeating this process, it is possible to gradually improve the separation performance of the vocal sound separation unit (DNN for vocal sound separation) 22. When a predetermined acceptable level of separation performance is obtained, the learning process can be terminated.
[0049] By performing such a learning process, one neural network in the sound source separation process execution unit 20 shown in FIG. 2, that is, the vocal sound separation unit (DNN for vocal sound separation) 22, can be generated.
[0050] FIG. 5 is a diagram illustrating the learning process for generating another neural network in the sound source separation process execution unit 20 shown in FIG. 2, that is, the guitar sound separation unit (DNN for guitar sound separation) 32.
[0051] The learning process shown in Figure 5 is also a supervised learning process. Supervised learning is performed using a large number of "input data" consisting of a mixture of various sound source data and a data set of "ground truth data" in which only guitar sounds are extracted from each input data.
[0052] In this case, too, it is necessary to prepare a large number of pairs of training data consisting of "input data" and "correct answer data." For example, as shown in Figure 6, it is necessary to generate in advance a large number of different pairs of "input data" and "correct answer data" with training data identifiers 00001 to nnnnn, that is, training data consisting of combinations of "input data" consisting of mixed data of various sound sources and "correct answer data" consisting only of guitar sounds included in the input data.
[0053] This large amount of input data is sequentially input to the guitar sound separation unit (DNN for guitar sound separation) 32 via the pre-processing unit (STFT) 31 shown in FIG. 5 , and the "output data" output from the post-processing unit (ISTFT) 33 is compared with the "correct data." The parameters of the neural network that constitutes the guitar sound separation unit (DNN for guitar sound separation) 32 are sequentially updated so that the difference between the "output data" and the "correct data" becomes smaller. By repeating this process, the separation performance of the guitar sound separation unit (DNN for guitar sound separation) 32 can be gradually improved. When a predetermined acceptable level of separation performance is obtained, the learning process can be terminated.
[0054] By performing such a learning process, it is possible to generate another neural network in the sound source separation process execution unit 20 shown in FIG. 2, that is, the guitar sound separation unit (DNN for guitar sound separation) 32.
[0055] In this way, the neural network configured in the sound source separation processing execution unit 20 is generated as a neural network specific to a sound source in which parameters specific to the sound source are set for each sound source.
[0056] In order to separate vocal sounds and guitar sounds, the sound source separation processing execution unit 20 shown in FIG. 2 needs to generate a vocal sound separation unit (DNN for vocal sound separation) 22, which is a neural network dedicated to vocal sound separation processing, and a guitar sound separation unit (DNN for guitar sound separation) 32, which is a neural network dedicated to guitar sound separation processing.
[0057] For these neural network generation processes, it is necessary to prepare a large number of training data sets consisting of "input data" and "correct answer data," and then input the prepared training data sets into the neural network sequentially to set and update the parameters.
[0058] For example, a configuration for separating vocal sounds and piano sounds, rather than a combination of vocal sounds and guitar sounds, is as shown in Fig. 7. Fig. 7 is a diagram illustrating an example of sound source separation processing for recorded data in which a user sings while playing the piano.
[0059] The user's vocal sound and the piano sound, which is the sound of the piano performance, are input via a microphone and recorded as an audio data file in the storage unit 10. The audio data file stored in the storage unit 10 is a recording file of audio data that contains a mixture of the user's vocal sound and the piano sound, which is the sound of the piano performance.
[0060] The sound source separation processing execution unit 20 reads out the audio data stored in the storage unit 10 and executes the sound source separation processing. That is, it executes processing to extract only the vocal sound and processing to extract only the piano sound from the audio data in which the user's vocal sound and the piano sound are mixed.
[0061] Fig. 8 explains a specific configuration example of the sound source separation processing execution unit 20 and specific processing executed by the sound source separation processing execution unit 20. The sound source separation processing execution unit 20 receives input data shown in the upper left of Fig. 8 from the storage unit 10. The input data is audio data containing a mixture of a user's vocal sound and piano sound, and is audio data that changes along the time axis.
[0062] The sound source separation processing execution unit 20 receives the audio data containing a mixture of vocal sounds and piano sounds, and executes processing to extract only the vocal sounds and processing to extract only the piano sounds.
[0063] The sound source separation processing execution unit 20 performs the sound source separation processing using a learning model made up of a neural network, such as a deep neural network (DNN), in the same manner as the configuration described above with reference to FIG.
[0064] 8, vocal sounds contained in input data are extracted by a configuration including a pre-processing unit (STFT) 21, a vocal sound separation unit (DNN for vocal sound separation) 22, and a post-processing unit (ISTFT) 23 shown in the upper row, while piano sounds contained in input data are extracted by a configuration including a pre-processing unit (STFT) 41, a piano sound separation unit (DNN for piano sound separation) 42, and a post-processing unit (ISTFT) 43 shown in the lower row.
[0065] The pre-processing unit (STFT) 21, vocal sound separation unit (DNN for vocal sound separation) 22, and post-processing unit (ISTFT) 23 shown in the upper part have the same configuration as that described above with reference to FIG.
[0066] On the other hand, among the pre-processing unit (STFT) 41, piano sound separation unit (DNN for piano sound separation) 42, and post-processing unit (ISTFT) 43 shown in the lower row, the piano sound separation unit (DNN for piano sound separation) 42 is a neural network (learning model) that extracts sound data having the characteristics of piano sounds from the audio data of various sound sources included in the input data.
[0067] This neural network (learning model) is also generated by a learning process executed in advance, and is a neural network (learning model) in which parameters are set to extract sound data having characteristics (frequency, etc.) of piano sounds.
[0068] In this piano sound separation unit (DNN for piano sound separation) 42, audio data corresponding to the sound of the piano is extracted from audio data of various sound sources included in the input data.
[0069] The learning process for generating the piano sound separation unit (DNN for piano sound separation) 42 will be described with reference to Fig. 9. The learning process shown in Fig. 9 is also the supervised learning process described above. The supervised learning process is performed using a large amount of "input data" consisting of a mixture of various sound source data and a data set of "ground truth data" in which only the piano sounds are extracted from each input data.
[0070] In this case, too, it is necessary to prepare a large number of pairs of training data consisting of "input data" and "correct answer data." For example, as shown in Figure 10, it is necessary to generate in advance a large number of different pairs of "input data" and "correct answer data" with training data identifiers 00001 to nnnnn, that is, training data consisting of combinations of "input data" consisting of mixed data of various sound sources and "correct answer data" consisting only of piano sounds included in the input data.
[0071] These numerous pieces of input data are sequentially input to the piano sound separation unit (DNN for piano sound separation) 42 via the pre-processing unit (STFT) 41 shown in FIG. 9 , and the "output data" output from the post-processing unit (ISTFT) 43 is compared with the "correct data." The parameters of the neural network constituting the piano sound separation unit (DNN for piano sound separation) 42 are sequentially updated so that the difference between the "output data" and the "correct data" becomes smaller. By repeating this process, the separation performance of the piano sound separation unit (DNN for piano sound separation) 32 can be gradually improved. When a predetermined acceptable level of separation performance is obtained, the learning process can be terminated.
[0072] By performing such learning processing, it is possible to generate the neural network in the sound source separation processing execution unit 20 shown in FIG. 8, that is, the piano sound separation unit (DNN for piano sound separation) 42.
[0073] In this way, the neural network configured in the sound source separation processing execution unit 20 needs to be generated as a sound source-specific neural network in which sound source-specific parameters are set for each sound source.
[0074] The sound source separation process described so far is an example of separation processing of input data in which sounds from two sound sources are recorded, i.e., separation processing of input data in which vocal sounds and guitar sounds are recorded, and separation processing of input data in which vocal sounds and piano sounds are recorded.
[0075] When three or more sound sources are included in the input data, it is necessary to generate a neural network specific to each of the three or more sound sources. With reference to Figure 11 and subsequent figures, a configuration example of the sound source separation processing execution unit 20 that performs separation processing on input data in which sounds from three or more sound sources are recorded will be described.
[0076] FIG. 11 is a diagram illustrating a specific configuration example of the sound source separation processing execution unit 20 that performs separation processing on input data in which sounds from three or more sound sources are recorded, and the specific processing that the sound source separation processing execution unit 20 performs.
[0077] The sound source separation processing execution unit 20 reads out the audio data stored in the storage unit 10 and executes the audio source separation processing. In the example shown in Fig. 11 , the sound source separation processing execution unit 20 inputs the input data shown in the upper left of Fig. 11 from the storage unit 10. The input data is audio data containing a mixture of the user's vocal sound, guitar sound, and piano sound, and is audio data that changes along the time axis.
[0078] The sound source separation processing execution unit 20 shown in FIG. 11 inputs audio data in which vocal sounds, guitar sounds, piano sounds, or a mixture of these sounds are recorded, and executes processing to extract vocal sounds, guitar sounds, and piano sounds from this input data.
[0079] The sound source separation processing execution unit 20 uses a neural network specific to each sound source, for example, a deep neural network (DNN), for the separation processing of each sound source.
[0080] In the example shown in Figure 11, the vocal sounds contained in the input data are extracted by a configuration consisting of a pre-processing unit (STFT) 21, a vocal sound separation unit (DNN for vocal sound separation) 22, and a post-processing unit (ISTFT) 23 shown in the upper part.
[0081] The guitar sounds contained in the input data are extracted by a pre-processing unit (STFT) 31, a guitar sound separation unit (DNN for guitar sound separation) 32, and a post-processing unit (ISTFT) 33 shown in the middle row. The piano sounds contained in the input data are extracted by a pre-processing unit (STFT) 41, a piano sound separation unit (DNN for piano sound separation) 42, and a post-processing unit (ISTFT) 43 shown in the bottom row.
[0082] The upper vocal separation unit (DNN for vocal separation) 22 is a neural network (learning model) that extracts sound data having vocal feature quantities from audio data of various sound sources included in the input data. In other words, it is a vocal-specific neural network in which vocal-specific parameters specialized for vocal extraction are set.
[0083] The guitar sound separation unit (DNN for guitar sound separation) 32 in the middle stage is a neural network (learning model) that extracts sound data having features of guitar sounds from audio data of various sound sources included in the input data. In other words, it is a neural network specific to guitar sounds, in which parameters specific to guitar sounds that are specialized for guitar sound extraction are set.
[0084] Furthermore, the piano sound separation unit (DNN for piano sound separation) 42 at the bottom is a neural network (learning model) that extracts sound data having piano sound features from the audio data of various sound sources included in the input data. In other words, it is a piano sound-specific neural network in which piano sound-specific parameters specialized for piano sound extraction are set.
[0085] As described above, in order to generate the vocal sound separation unit (DNN for vocal sound separation) 22, guitar sound separation unit (DNN for guitar sound separation) 32, and piano sound separation unit (DNN for piano sound separation) 42, a learning process using learning data for generating each neural network is required.
[0086] Furthermore, there is also a separation processing configuration that takes into account noise contained in the input data. A specific example is shown in Fig. 12. The sound source separation processing execution unit 20 shown in Fig. 12 inputs input data similar to the configuration shown in Fig. 11 described above, i.e., audio data containing a mixture of a user's vocal sound, guitar sound, and piano sound, and executes the sound source separation processing.
[0087] The difference from the configuration shown in FIG. 11 is that in addition to the process of separating vocal sounds, guitar sounds, and piano sounds from the input data, a process of extracting and separating noise sounds contained in the input data is executed.
[0088] 12, a vocal sound separation unit (DNN for vocal sound separation) 22 extracts vocal sounds contained in the input data, a guitar sound separation unit (DNN for guitar sound separation) 32 extracts guitar sounds contained in the input data, and a piano sound separation unit (DNN for piano sound separation) 42 extracts piano sounds contained in the input data.
[0089] Furthermore, noise sounds contained in the input data are extracted by a pre-processing unit (STFT) 51, a noise sound separation unit (DNN for noise sound separation) 52, and a post-processing unit (ISTFT) 53 shown in FIG.
[0090] The noise separation unit (DNN for noise separation) 52 is a neural network (learning model) that extracts sound data having features that do not correspond to vocal sounds, guitar sounds, or piano sounds from the audio data of various sound sources included in the input data. In other words, it is a noise-specific neural network in which noise-specific parameters specialized for noise extraction are set.
[0091] As described above, in order to generate the vocal sound separation unit (DNN for vocal sound separation) 22, guitar sound separation unit (DNN for guitar sound separation) 32, piano sound separation unit (DNN for piano sound separation) 42, and noise sound separation unit (DNN for noise sound separation) 52, a learning process using learning data for generating each of the neural networks is required.
[0092] An example of training data required to generate these four neural networks is shown in Fig. 13. The training data shown in Fig. 13 is an example of training data required to generate the four neural networks that constitute the four sound source separation units shown in Fig. 12, and is an example of training data required to perform the supervised training process described above.
[0093] Specifically, supervised learning processing is performed using a large number of "input data" consisting of mixed data of various sound source data and "correct answer data" obtained by extracting only each sound source from each input data, namely: (b1) correct answer data for vocal sounds, (b2) correct answer data for guitar sounds, (b3) correct answer data for piano sounds, and (b4) correct answer data for noise sounds.
[0094] A large number of training data consisting of pairs of "input data" and "correct answer data" are prepared, and as explained above with reference to Figure 3 etc., the "input data" is sequentially input to each sound source separation unit (DNN) via the pre-processing unit (STFT), the "output data" output from the post-processing unit (ISTFT) is compared with the "correct answer data", and the parameters of the neural network constituting each sound source separation unit (DNN) are sequentially updated so that the difference between the "output data" and the "correct answer data" becomes smaller.
[0095] By repeating this process, the separation performance of each sound source separation unit (DNN) can be gradually improved. When a predetermined acceptable level of separation performance is obtained, the learning process ends.
[0096] Such a learning process needs to be performed for each sound source. As the number of sound sources to be separated increases, the amount of required learning data and the number of learning process steps increase, resulting in a problem that the learning cost becomes enormous depending on the number of sound sources to be separated. Furthermore, as described above, the configuration and parameters to be set of a neural network (learning model) such as a DNN corresponding to each sound source are different for each sound source, resulting in a problem that the amount of memory required to record this data becomes enormous. The present disclosure is intended to solve these problems, and the configuration and processing of the present disclosure will be described below.
[0097] 2. Overview of Sound Source Separation Processing Using the Information Processing Device of the Present Disclosure An overview of sound source separation processing using the information processing device of the present disclosure will be described below.
[0098] First, an example of sound source separation processing using the information processing device 100 of the present disclosure will be described with reference to Fig. 14. The information processing device 100 shown in Fig. 14 is a smartphone. However, the smartphone is just one example, and the information processing device 100 of the present disclosure is not limited to a smartphone and may be various devices such as a PC or a tablet terminal.
[0099] User 1 shown in Fig. 14 is singing along with the guitar he is playing. The vocal sound of user 1 and the guitar sound are input to microphone 101 of information processing device (smartphone) 100 shown in Fig. 1 and recorded (audio recording) in memory (storage unit) 102 inside information processing device 100.
[0100] Once the recording is complete, the user 1 can then start a sound source separation process for the audio data recorded in the memory (storage unit) 102 inside the information processing device 100. A UI (user interface) such as that shown in Fig. 15 is displayed on the display unit of the information processing device 100, and the user uses this UI to instruct the start of the sound source separation process.
[0101] The UI displayed on the display unit of the information processing device 100 is a UI for selecting and specifying the type of sound source included in the recorded audio data: (a) Vocal / Voice (b) Guitar / Piano (c) Vocal & Guitar / Vocal & Piano One of these three options is selected.
[0102] "(a) Vocal / Voice" is selected when the sound source contained in the recorded audio data is singing sound (vocals) without any instruments, or conversation sound (voice). "(b) Guitar / Piano" is selected when the sound source contained in the recorded audio data is guitar sound or piano sound without vocals. "(c) Vocal & Guitar / Vocal & Piano" is selected when the sound source contained in the recorded audio data is vocals and guitar sound, or vocals and piano sound.
[0103] The information processing device 100 of the present disclosure executes sound source separation processing according to the type of recorded sound source designated by the user. Details of the sound source separation processing executed by the information processing device 100 of the present disclosure will be described later.
[0104] 15 to select a sound source type included in the recorded audio data and issue an instruction to start audio source separation, a data processing unit (sound source separation processing execution unit) in the information processing device 100 performs audio source separation processing. As a result of this processing, an audio data file for each audio source is generated, and the generated audio data file for each audio source is stored in a memory (storage unit).
[0105] When this sound source separation process is completed, the user can perform editing process on each piece of sound source audio data. An editing screen such as that shown in Fig. 16 is displayed on the display unit of the information processing device 100.
[0106] 16 shows an example in which vocal sound data generated by sound source separation processing and guitar sound data are displayed side by side. Using this editing screen, user 1 can edit the sound data for each sound source. For example, various editing processes can be performed on each sound source data, such as adjusting the volume, pitch, and timing of the vocal sound alone.
[0107] 3. Configuration Example of Information Processing Apparatus of the Present Disclosure Next, a configuration example of an information processing apparatus of the present disclosure will be described.
[0108] Fig. 17 is a block diagram showing an example configuration of an information processing device 100 according to the present disclosure. As shown in Fig. 17 , the information processing device 100 according to the present disclosure includes an audio input unit (microphone) 111, a storage unit 112, a sound source separation processing unit 113, a learning model (DNN) 114, an audio data editing processing unit 115, a playback control unit 116, a display unit 117, and an audio output unit (speaker) 118.
[0109] Note that the components (processing blocks) of the information processing device 100 shown in FIG. 17 are the main components required to perform the processing of the present disclosure. The information processing device 100 also includes various other components in addition to these components. For example, it includes various components such as a communication unit, an input unit, a control unit, and a camera. Furthermore, the information processing device 100 may be configured to include an "audio data file output unit" that outputs an audio source data file storing audio source data of each audio source separated by the audio source separation processing execution unit 113 to an external device. The "audio data file output unit" outputs audio source data files (e.g., WAV files) of each audio source after the audio source separation process to an external device (e.g., a PC or editing device), thereby enabling editing of the audio source data (e.g., volume adjustment, sound quality adjustment) in the external device. Furthermore, it is also possible to output an edited composite data file generated by recombining audio source data after editing processing in the audio data editing processing unit 115 in the information processing device 100 from the "audio data file output unit" to an external device.
[0110] The components of the information processing device 100 shown in Fig. 17 will be described in detail below. The audio input unit (microphone) 111 is a microphone that inputs the user's voice or the sound of an instrument such as a guitar. The storage unit 112 records the audio data input via the audio input unit (microphone) 111.
[0111] The sound source separation processing execution unit 113 inputs the audio data recorded in the storage unit 112 and executes a sound source separation process to separate the audio data into data corresponding to each sound source included in the audio data. Details of this sound source separation process will be described later. The audio data files for each sound source separated by the sound source separation processing execution unit 113 are stored in the storage unit 112.
[0112] The learning model (DNN) 114 is a learning model, for example, a neural network such as a DNN (Deep Neural Network), used in the sound source separation process executed by the sound source separation process execution unit 113. Details of the learning model (DNN) 114 used by the information processing device of the present disclosure will be described later.
[0113] The audio data editing processing unit 115 is a data processing unit that displays, for example, a UI as described above with reference to Fig. 16 and executes editing processing of audio data for each sound source in accordance with user instructions. The audio data editing processing unit 115 reads out the audio data files for each sound source separated by the sound source separation processing execution unit 113 from the storage unit 112, displays the audio data information for each sound source as described with reference to Fig. 16 on the display unit 117, changes and edits the audio data for each sound source in accordance with user instructions, and stores the results in the storage unit 112.
[0114] The playback control unit 116 executes a process of reading out and playing back audio data stored in the memory unit 112, such as audio data at the time of recording, and audio data files in sound source units separated by the processing execution unit 113, from the memory unit 112.
[0115] The display unit 117 displays, for example, a UI such as that described with reference to Fig. 15 and Fig. 16. The display unit 117 is a touch panel type display unit, and is configured to enable various input processes and instructions to be given by user operation.
[0116] The audio output unit (speaker) 118 is a speaker that outputs audio data and the like that is reproduced by the processing of the reproduction control unit 116, for example.
[0117] 4. Details of the Sound Source Separation Process Executed by the Information Processing Device of the Present Disclosure Next, details of the sound source separation process executed by the information processing device of the present disclosure will be described.
[0118] The sound source separation process performed by the information processing device 100 of the present disclosure is assumed to be used in the manner described above with reference to Figure 14, i.e., when a user 1 performs sound source separation process on audio data recorded using an information processing device 100 such as a smartphone, and is configured to limit the types of sound sources to be separated and separate multiple different musical instruments, etc. together using a single learning model (DNN, etc.).
[0119] That is, it is assumed that the main usage will be personal voice recording by user 1. Specifically, the system is configured to record any of the following: user 1's singing voice (vocals), conversation voice (voice), guitar or piano performance, or singing (vocals) accompanied by guitar or piano performance, and then perform sound source separation processing on any of these recording data.
[0120] FIG. 18 shows types of audio data (recorded data) that are the target of sound source separation processing executed by the information processing device 100 of the present disclosure.
[0121] The audio data (recorded data) that is the target of the sound source separation process executed by the information processing device 100 of the present disclosure is any of the following data: (R01) Vocal sound (singing sound) (R02) Voice sound (conversation sound) (R03) Guitar sound (R04) Piano sound (R05) Vocal sound (singing sound) + guitar sound (R06) Vocal sound (singing sound) + piano sound
[0122] "(R01) Vocal sound (singing sound)" is a singing sound without any instruments. "(R02) Voice sound (conversation sound)" is a conversation sound, not a singing sound. "(R03) Guitar sound" is a guitar performance sound without singing. "(R04) Piano sound" is a piano performance sound without singing. "(R05) Vocal sound (singing sound) + guitar sound" is a mixed sound of a singing sound and a guitar performance sound. "(R06) Vocal sound (singing sound) + piano sound" is a mixed sound of a singing sound and a piano performance sound.
[0123] The audio data (recorded data) that is the target of the sound source separation process executed by the information processing device 100 of the present disclosure is limited to any one of the data (R01) to (R06) shown in FIG.
[0124] Furthermore, the sound source separation process performed by the information processing device 100 of the present disclosure uses one learning model (DNN, etc.) for each category, for example, by grouping multiple different musical instruments into one category. Specifically, as shown in Fig. 19, sound sources are divided into three categories (three sound source types), and separation processing is performed for each of these categories (types). That is, as shown in Fig. 19, (C01) Vocal sound (singing sound) or voice sound (conversation sound) (C02) Guitar sound or piano sound (C03) Noise sound These three types of sound source separation are performed using a learning model (DNN, etc.) corresponding to each category (type).
[0125] That is, vocal sounds (singing sounds) and voice sounds (conversation sounds) are not distinguished from each other, but are treated as sounds (voices) of the same category (type) and are separated using a single learning model (DNN, etc.). Similarly, guitar sounds and piano sounds are not distinguished from each other, but are treated as sounds of the same category (musical instrument sounds) and are separated using a single learning model (DNN, etc.). In addition to these two types of sound source categories, noise sounds are also treated as sounds of the same category and are separated using a single learning model (DNN, etc.).
[0126] Due to such classification of sound source types, the information processing device 100 of the present disclosure uses only three types of learning models (neural networks such as DNN) for sound source separation processing. That is, as shown in item (c) of Fig. 19 , the following three types of learning models (neural networks such as DNN) are generated and used:
[0127] (C01) A voice-compatible learning model (DNN, etc.) with parameters set to extract vocal sounds (singing sounds) or voice sounds (conversational sounds) (C02) A guitar and piano sound-compatible learning model (DNN, etc.) with parameters set to extract guitar sounds or piano sounds (C03) A noise sound-compatible learning model (DNN, etc.) with parameters set to extract noise sounds The information processing device 100 of the present disclosure uses only these three types of learning models (neural networks such as DNN) for sound source separation processing for its sound source separation processing.
[0128] In this way, the information processing device 100 of the present disclosure generates and uses a learning model (neural network such as DNN) that does not distinguish between vocal sounds (singing sounds) and voice sounds (conversation sounds) but separates them as sounds (voices) of the same category (type).It also generates and uses a learning model (neural network such as DNN) that does not distinguish between guitar sounds and piano sounds but separates them as sounds of the same category (musical instrument sounds).
[0129] The reason why a learning model (neural network such as DNN) can be generated and used for each sound source category is that the sounds in each category have common characteristics. That is, there are common sound characteristics between vocal sounds (singing sounds) and voice sounds (conversational sounds), and there are also common sound characteristics between guitar sounds and piano sounds.
[0130] The information processing device 100 of the present disclosure extracts speech data having common characteristics on a category-by-category basis by using one learning model (DNN) on a category-by-category basis.
[0131] The sound characteristics common to vocal sounds (singing sounds) and voice sounds (conversational sounds) include, for example, the following features: (Common feature 1) They are sounds that accompany human speech. (Common feature 2) They have identifiable vowels. (Common feature 3) They have identifiable consonants. (Common feature 4) They have respiratory sounds from the oral cavity. (Common feature 5) They have respiratory sounds from the nose. (Common feature 6) They are accompanied by sounds that escape from the nose when spoken. (Common feature 7) They are accompanied by reverberation from the head and chest when spoken.
[0132] The vowels shown in (common feature point 2) are sounds that have a harmonic structure due to the vibration of the vocal cords and the reverberation and resonance of the oral cavity, while the consonants shown in (common feature point 3) are sounds made by the tongue, lips, teeth, etc. colliding with each other and the air passing over them.
[0133] As such, since vocal sounds (singing sounds) and voice sounds (conversational sounds) have the same sound characteristics as described above, it is possible to generate and use a single learning model (a neural network such as a DNN) that extracts vocal sounds (singing sounds) and voice sounds (conversational sounds).
[0134] In order to verify the above theory, a learning model (DNN) trained to separate vocal sounds (singing sounds), guitar sounds, and noise sounds was used.
[0135] The verification process involves inputting pure tones (audio data without other sounds) of voice sounds (conversation sounds) into a learning model (DNN) trained to separate vocal sounds (singing sounds), guitar sounds, and noise sounds, and analyzing the separation results. As a result of this analysis, it was confirmed that the voice sounds (conversation sounds) were almost completely separated as vocal sounds (singing sounds), and were hardly separated at all as guitar sounds or noise sounds.
[0136] In other words, it was confirmed that voice sounds (conversation sounds) and vocal sounds (singing sounds) can be separated as audio data of a single sound source by sound source separation processing using a single learning model (DNN).
[0137] Similarly, the sound characteristics common to guitar and piano sounds include, for example, the following: (Common Feature 1) A sound in which the same frequency tone is sustained for a certain period of time due to string vibration. (Common Feature 2) Has frequency characteristics with an overtone harmonic structure. (Common Feature 3) Has regular exponential decay characteristics (regular changes in volume and timbre) for each harmonic. (Common Feature 4) Has linear sound characteristics. (Common Feature 5) Has a constant pitch (fundamental frequency) with no time variation (such as vibrato). (Common Feature 6) Has a 12-tone scale (no intervals other than the 12-tone scale). (Common Feature 7) Has a chord (multiple intervals sounding simultaneously). (Common Feature 8) Has a constant resonance.
[0138] The linear sound characteristics shown in (common feature point 4) refer to sound characteristics that are distortion-free, that produce a loud sound when played hard, and that produce an added sound when multiple instruments are played simultaneously. Furthermore, the sound with a certain resonance shown in (common feature point 8) refers to a sound that is not simply the result of string vibration, but is produced when vibrations are reflected, resonated, or partially absorbed by the wooden body or soundboard.
[0139] As described above, because guitar sounds and piano sounds have common sound characteristics, it is possible to generate and use a single learning model (a neural network such as a DNN) that extracts guitar sounds and piano sounds.
[0140] In order to verify the above theory, a verification was carried out using a learning model (DNN) trained to separate vocal sounds (singing sounds), guitar sounds, and noise sounds. The verification process involved inputting pure piano tones (audio data not mixed with other sounds) into the learning model (DNN) trained to separate vocal sounds (singing sounds), guitar sounds, and noise sounds, and analyzing the separation results. As a result of this analysis, it was confirmed that the piano sounds were almost completely separated as guitar sounds, and were hardly separated at all as vocal sounds (singing sounds) or noise sounds.
[0141] That is, it was confirmed that the piano sound and the guitar sound can be separated as audio data of a single sound source by the sound source separation process using one learning model (DNN).
[0142] Based on these verifications, the information processing device 100 of the present disclosure does not distinguish between vocal sounds (singing sounds) and voice sounds (conversation sounds), but performs a process of separating them as sounds (voices) of the same category (type) using a single learning model (a neural network such as a DNN).
[0143] Similarly, guitar sounds and piano sounds are not distinguished from each other, but are treated as sounds in the same category (musical instrument sounds), and a process of separating them is performed using a single learning model (a neural network such as a DNN).
[0144] By categorizing the sound source types in this way, the information processing device 100 of the present disclosure can limit the number of learning models (neural networks such as DNN) used in the sound source separation process to only three types.
[0145] That is, as shown in item (c) of Fig. 19 , the configuration is such that the following three types of learning models (neural networks such as DNN) are generated and used: (C01) A voice-compatible learning model (DNN, etc.) in which parameters for extracting vocal sounds (singing sounds) or voice sounds (conversational sounds) are set (C02) A guitar and piano sound-compatible learning model (DNN, etc.) in which parameters for extracting guitar sounds or piano sounds are set (C03) A noise sound-compatible learning model (DNN, etc.) in which parameters for extracting noise sounds are set The information processing device 100 of the present disclosure uses only these three types of learning models (neural networks such as DNN) for sound source separation processing in the information processing device 100 of the present disclosure.
[0146] Next, with reference to FIG. 20 , a configuration example of the sound source separation processing execution unit 113 of the information processing device 100 of the present disclosure and the processing to be executed will be described.
[0147] The sound source separation processing execution unit 113 shown in Fig. 20 receives input data shown in the upper left of Fig. 20 from the storage unit 112. The input data is any of the following data described with reference to Fig. 18: (R01) Vocal sound (singing sound) (R02) Voice sound (conversation sound) (R03) Guitar sound (R04) Piano sound (R05) Vocal sound (singing sound) + guitar sound (R06) Vocal sound (singing sound) + piano sound
[0148] The sound source separation processing execution unit 113 inputs any one of these audio data (R01) to (R06) and executes processing to separate the data into sound source unit data. However, as described with reference to Fig. 19, vocal sounds (singing sounds) and voice sounds (conversation sounds) are not distinguished and are separated as sounds (voices) of the same category (type). Similarly, guitar sounds and piano sounds are not distinguished and are separated as sounds (instrument sounds) of the same category.
[0149] That is, the sound source separation processing unit 113 of the information processing device 100 of the present disclosure performs processing to separate the sound into the following three types of sound sources (C01) to (C03): (C01) Vocal sound (singing sound) or voice sound (conversation sound) (C02) Guitar sound or piano sound (C03) Noise sound The sound source separation processing execution unit 113 of the information processing device 100 of the present disclosure performs these three types of sound source separation processing.
[0150] That is, vocal sounds (singing sounds) and voice sounds (conversation sounds) are not distinguished and are separated as the same category (type) of sound (voice). Similarly, guitar sounds and piano sounds are not distinguished and are separated as the same category of sound (instrument sounds). In addition to these two sound source categories, noise sounds are also separated as a single category of sound.
[0151] The sound source separation processing execution unit 113 shown in FIG. 20 performs sound source separation processing using a learning model made up of a neural network, such as a deep neural network (DNN).
[0152] As shown in FIG. 20 , the sound source separation processing execution unit 113 has three types of data processing units: a series of data processing units consisting of a pre-processing unit (STFT) 121, a vocal & voice sound separation unit (DNN for vocal & voice sound separation) 122, and a post-processing unit (ISTFT) 123; a series of data processing units consisting of a pre-processing unit (STFT) 131, a guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132, and a post-processing unit (ISTFT) 133; and a series of data processing units consisting of a pre-processing unit (STFT) 141, a noise sound separation unit (DNN for noise sound separation) 142, and a post-processing unit (ISTFT) 143.
[0153] The pre-processing unit (STFT) 121, vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122, and post-processing unit (ISTFT) 123 shown in the upper part extract vocal sounds (singing sounds) or voice sounds (conversation sounds) contained in the input data.
[0154] Also, the pre-processing unit (STFT) 131, guitar and piano sound separation unit (DNN for guitar and piano sound separation) 132, and post-processing unit (ISTFT) 133 shown in the middle section extract guitar or piano sounds contained in the input data.
[0155] Furthermore, noise contained in the input data is extracted by a pre-processing unit (STFT) 141, a noise separation unit (DNN for noise separation) 142, and a post-processing unit (ISTFT) 143.
[0156] The preprocessing units (STFT) 121, 131, and 141 shown in each row perform a short-time Fourier transform (STFT) on the input data. This STFT process converts the input data, which is time-domain data in which the audio signal changes along the time axis, into frequency-domain data (frequency spectrum). This converted data is input to the sound separation units (sound separation DNNs) 122, 132, and 142. Note that the preprocessing units (STFT) 121, 131, and 141 shown in each row may be combined into a single preprocessing unit (STFT) if they are configured to perform the same processing on the same input.
[0157] Each of the sound separation units (sound separation DNN) 122, 132, and 142 is a sound source separation unit configured by a neural network, specifically a learning model such as a DNN (Deep Neural Network).
[0158] The vocal & voice sound separation unit (DNN for vocal & voice sound separation) 122 uses a learning model such as a DNN to extract vocal sounds (singing sounds) or voice sounds (conversation sounds) included in the input data. The guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132 uses a learning model such as a DNN to extract guitar sounds or piano sounds included in the input data. The noise sound separation unit (DNN for noise sound separation) 142 uses a learning model such as a DNN to extract noise sounds included in the input data.
[0159] The sound source data, which is the processed data of each sound separation unit (sound separation DNN) 122 , 132 , 142 , is input to post-processing units (ISTFT) 123 , 133 , 143 .
[0160] The post-processing units (ISTFT) 123, 133, and 143 perform the inverse processing of the above-described pre-processing units (STFT) 121, 131, and 141. That is, they perform an inverse short-time Fourier transform (ISTFT).
[0161] This ISTFT process converts the frequency domain data (frequency spectrum) into time domain data in which the audio signal changes along the original time axis.
[0162] The output of the upper post-processing unit (ISTFT) 123 is the audio data extracted by the vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122, i.e., audio data of vocal sound (singing sound) or voice sound (conversation sound).
[0163] The output of the middle-stage post-processing section (ISTFT) 133 is the audio data extracted by the guitar and piano sound separation section (DNN for guitar and piano sound separation) 132, that is, the audio data of the guitar sound or piano sound.
[0164] Furthermore, the output of the post-processing unit (ISTFT) 143 at the lower stage is the audio data extracted in the noise sound separation unit (DNN for noise sound separation) 142, that is, the audio data of the noise sound.
[0165] In this way, the sound source separation processing execution unit 113 of the information processing device 100 of the present disclosure executes sound source separation processing using three types of sound source separation units: a vocal & voice sound separation unit (DNN for vocal & voice sound separation) 122, a guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132, and a noise sound separation unit (DNN for noise sound separation) 142.
[0166] That is, by using these three types of sound source separation units, a process of separating into the following three types of sound sources (C01) to (C03) is executed: (C01) Vocal sound (singing sound) or voice sound (conversation sound) (C02) Guitar sound or piano sound (C03) Noise sound
[0167] The sound source separation processing execution unit 113 of the information processing device 100 of the present disclosure does not distinguish between vocal sounds (singing sounds) and voice sounds (conversation sounds), but separates them as sounds (voices) of the same category (type).Similarly, it does not distinguish between guitar sounds and piano sounds, but separates them as sounds (instrument sounds) of the same category.
[0168] The vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122 is composed of a neural network (learning model) that extracts sound data having features (frequency, etc.) of vocal sounds (singing sounds) or voice sounds (conversation sounds) corresponding to human voices from the audio data of various sound sources included in the input data.
[0169] The learning model (neural network such as DNN) that constitutes this vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122 is a neural network that extracts sound data that has sound characteristics common to the vocal sound (singing sound) and voice sound (conversation sound) described above, and that has a frequency equivalent to that of a human voice.
[0170] That is, it is a neural network that extracts audio data that has the following sound features common to vocal sounds (singing sounds) and voice sounds (conversational sounds): (Common feature 1) Sounds that accompany human speech. (Common feature 2) Has identifiable vowels. (Common feature 3) Has identifiable consonants. (Common feature 4) Has respiratory sounds from the oral cavity. (Common feature 5) Has respiratory sounds from the nose. (Common feature 6) Accompanied by sounds that escape from the nose when spoken. (Common feature 7) Accompanied by reverberation from the head and chest when spoken.
[0171] In addition, the guitar and piano sound separation unit (DNN for guitar and piano sound separation) 132 is composed of a neural network (learning model) that extracts sound data having the characteristics of guitar sounds or piano sounds from the audio data of various sound sources included in the input data.
[0172] The learning model (neural network such as DNN) that constitutes this guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132 is a neural network that extracts sound data with frequencies corresponding to guitar sounds or piano sounds, and further has sound characteristics (frequency, etc.) that are common to guitar sounds and piano sounds, as described above.
[0173] That is, it is a neural network that extracts audio data that has the following sound features common to guitar and piano sounds: (Common feature 1) A sound in which the same frequency tone is sustained for a certain period of time due to string vibration. (Common feature 2) It has frequency characteristics with an overtone harmonic structure. (Common feature 3) It has regular exponential decay characteristics (regular changes in volume and timbre) for each harmonic. (Common feature 4) It has linear sound characteristics. (Common feature 5) It has a constant pitch (fundamental frequency) and no time changes (such as vibrato). (Common feature 6) It has a 12-tone scale (no intervals other than the 12-tone scale). (Common feature 7) It has a chord (multiple intervals sounded simultaneously). (Common feature 8) It is a sound with a consistent resonance.
[0174] Furthermore, the noise separation unit (DNN for noise separation) 142 is configured by a neural network (learning model) that extracts sound data having noise feature quantities from the audio data of various sound sources included in the input data.
[0175] To generate these learning models (DNN), a learning process is required. The sound source separation process execution unit 113 of the information processing device 100 according to the present disclosure uses only three types of learning models (DNN), thereby reducing the learning process cost.
[0176] The learning process for generating the learning model (DNN) is the "supervised learning process" described above. That is, the learning process is performed using a large number of "input data" consisting of mixed data of various sound source data and a data set of "ground truth data" which is extracted data of the sound source data to be separated contained in each input data.
[0177] As described above, in supervised learning processing, correct answer data for input data is prepared in advance, the input data is input to the sound separation unit (sound separation DNN) via the pre-processing unit (STFT), and the separation processing results in the sound separation unit (sound separation DNN) are output via the post-processing unit (ISTFT) to obtain "output data." This "output data" is compared with the "correct answer data," and processing is executed to set and update the parameters of the neural network that makes up the sound separation unit (sound separation DNN) so that the difference between these two pieces of data is small.
[0178] Referring to Figure 21, an example of learning data used in the learning process (supervised learning) for generating three types of learning models (DNN) used by the sound source separation processing execution unit 113 of the information processing device 100 of the present disclosure will be described.
[0179] The training data shown in Figure 21 is an example of training data required to generate the three neural networks that make up the three sound source separation units shown in Figure 20, and is an example of training data required to perform supervised learning processing.
[0180] Specifically, supervised learning processing is performed using a large number of "input data" consisting of mixed data of various sound source data and "correct answer data" obtained by extracting only each sound source from each input data, namely: (b1) Correct answer data for vocal sounds (singing sounds) or voice sounds (conversation sounds) (b2) Correct answer data for guitar sounds or piano sounds (b3) Correct answer data for noise sounds.
[0181] A large number of training data consisting of pairs of "input data" and "correct answer data" are prepared, and as explained above with reference to Figure 3 etc., the "input data" is sequentially input to each sound source separation unit (DNN) via the pre-processing unit (STFT), the "output data" output from the post-processing unit (ISTFT) is compared with the "correct answer data", and the parameters of the neural network constituting each sound source separation unit (DNN) are sequentially updated so that the difference between the "output data" and the "correct answer data" becomes smaller.
[0182] By repeating this process, the separation performance of each sound source separation unit (DNN) can be gradually improved. When a predetermined acceptable level of separation performance is obtained, the learning process ends.
[0183] In the configuration of the present disclosure, such a learning process is executed to generate the following three types of learning models (DNN): (1) a learning model (DNN) for extracting vocal sounds (singing sounds) or voice sounds (conversation sounds), (2) a learning model (DNN) for extracting guitar sounds or piano sounds, and (3) a learning model (DNN) for extracting noise sounds.
[0184] Instead of performing learning processing for each sound source as in the past, it is possible to generate a learning model that does not distinguish between vocal sounds (singing sounds) and voice sounds (conversation sounds), and furthermore, a learning model that does not distinguish between guitar sounds and piano sounds, thereby reducing learning processing costs.
[0185] Referring to Figure 22 and subsequent figures, the details of the generation process of the following three types of learning models (DNN), namely: (1) learning model (DNN) for extracting vocal sounds (singing sounds) or voice sounds (conversation sounds) (2) learning model (DNN) for extracting guitar sounds or piano sounds (DNN) (3) learning model (DNN) for extracting noise sounds will be described.
[0186] FIG. 22 is a diagram illustrating the learning process for generating a vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122 for extracting vocal sounds (singing sounds) or voice sounds (conversation sounds) corresponding to human voices from input data.
[0187] The learning process shown in Fig. 22 is a supervised learning process. That is, the learning process is performed using a data set of a large number of "input data" consisting of mixed data of various sound source data and "correct answer data" which is extracted data of vocal sounds (singing sounds) or voice sounds (conversation sounds) included in each input data. That is, the learning process is performed using teacher data in which both vocal sounds (singing sounds) and voice sounds (conversation sounds) are set as correct answer data.
[0188] In this way, supervised learning processing involves preparing correct answer data for input data in advance, inputting the input data to a vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122 via a pre-processing unit (STFT) 121, and outputting the results of the separation processing in the vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122 via a post-processing unit (ISTFT) 123 to obtain "output data."
[0189] This "output data" is compared with the "correct data," and the parameters of the neural network that constitutes the vocal and voice separation unit (DNN for vocal and voice separation) 122 are set and updated so that the difference between these two pieces of data becomes small.
[0190] For example, a large number of different "(a) input data" with learning data identifiers 00001 to nnnnn and "(b1) correct output data (vocal sound or voice sound)" shown in FIG. 21 are used.
[0191] A large number of different "(a) input data" with learning data identifiers 00001 to nnnnn are sequentially input to vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122, and the parameters of the neural network constituting vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122 are sequentially updated so that the difference between the "output data" and "(b1) correct output data (vocal sound or voice sound)" becomes small.
[0192] By repeating this process, it is possible to gradually improve the separation performance of the vocal and voice sound separator (DNN for vocal and voice sound separation) 122. When a predetermined acceptable level of separation performance is obtained, the learning process can be terminated.
[0193] By performing such a learning process, it is possible to generate one neural network in the sound source separation process execution unit 113 shown in FIG. 20, i.e., the vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122.
[0194] The vocal and voice sound separator (vocal and voice sound separator DNN) 122 generated by this learning process is a neural network that extracts, from mixed audio data that is a mixture of sounds from various sound sources, audio data that is a sound with a frequency corresponding to a human voice and that has the following sound characteristics common to the vocal sounds (singing sounds) and voice sounds (conversational sounds) described above: (Common feature 1) It is a sound associated with human vocalization. (Common feature 2) It has identifiable vowels. (Common feature 3) It has identifiable consonants. (Common feature 4) It has respiratory sounds from the oral cavity. (Common feature 5) It has respiratory sounds from the nose. (Common feature 6) It is accompanied by sounds that escape from the nose when vocalized. (Common feature 7) It is accompanied by reverberation from the head and chest when vocalized.
[0195] FIG. 23 is a diagram illustrating the learning process for generating the guitar and piano sound separation unit (guitar and piano sound separation DNN) 132 in the sound source separation process execution unit 113 shown in FIG.
[0196] The learning process shown in Fig. 23 is also a supervised learning process. The supervised learning process is performed using a large amount of "input data" consisting of a mixture of various sound source data and a data set of "ground truth data" in which only guitar sounds and piano sounds are extracted from each input data.
[0197] In this case, as shown in FIG. 21, a large number of different data sets of "(a) input data" with learning data identifiers 00001 to nnnnn and "(b2) correct output data (guitar sound or piano sound)" are used as the learning data.
[0198] A large number of different "(a) input data" with learning data identifiers 00001 to nnnnn are input sequentially to a guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132 via a pre-processing unit (STFT) 131 shown in FIG. 23, and the "output data" output from a post-processing unit (ISTFT) 133 is compared with "(b2) correct output data (guitar sound or piano sound)," and the parameters of the neural network constituting the guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132 are updated sequentially so that the difference between the "output data" and the "correct data" becomes smaller.
[0199] By repeating this process, it is possible to gradually improve the separation performance of the guitar & piano sound separator (guitar & piano sound separation DNN) 132. When a preset acceptable level of separation performance is obtained, the learning process can be terminated.
[0200] By performing such learning processing, it is possible to generate the guitar and piano sound separation section (DNN for guitar and piano sound separation) 132 in the sound source separation processing execution section 113 shown in FIG.
[0201] The guitar and piano sound separation unit (guitar and piano sound separation DNN) 132 generated by this learning process is a neural network that extracts, from mixed audio data that is a mixture of sounds from various sound sources, audio data that is of frequencies corresponding to guitar sounds and piano sounds and that has the following sound characteristics common to guitar sounds and piano sounds described above: (Common feature 1) A sound in which the same frequency is sustained for a certain period of time due to string vibration. (Common feature 2) Has frequency characteristics with an overtone harmonic structure. (Common feature 3) Has regular exponential decay characteristics (regular changes in volume and timbre) for each harmonic. (Common feature 4) Has linear sound characteristics. (Common feature 5) Has a constant pitch (fundamental frequency) with no time variation (such as vibrato). (Common feature 6) Has a 12-tone scale (no intervals other than the 12-tone scale). (Common feature 7) Has a chord (multiple intervals sounded simultaneously). (Common feature 8) It is a sound with a certain resonance.
[0202] FIG. 24 is a diagram illustrating the learning process for generating the noise separation unit (DNN for noise separation) 142 in the sound source separation process execution unit 113 shown in FIG.
[0203] The learning process shown in Fig. 23 is also a supervised learning process. The supervised learning process is performed using a large number of "input data" consisting of a mixture of various sound source data and a data set of "ground truth data" in which only noise sounds are extracted from each input data.
[0204] In this case, too, as the learning data, a large number of different "(a) input data" with learning data identifiers 00001 to nnnnn and "(b3) correct output data (noise sound)" data sets are used, as shown in Fig. 21. In other words, the learning process is performed using teacher data in which both guitar sounds and piano sounds are set as correct data.
[0205] A large number of different "(a) input data" with learning data identifiers 00001 to nnnnn are input sequentially to a noise sound separation unit (DNN for noise sound separation) 142 via a pre-processing unit (STFT) 141 shown in FIG. 24, and the "output data" output from a post-processing unit (ISTFT) 143 is compared with the "correct data." The parameters of the neural network constituting the noise sound separation unit (DNN for noise sound separation) 142 are sequentially updated so that the difference between the "output data" and the "correct data" becomes smaller. By repeating this process, the separation performance of the noise sound separation unit (DNN for noise sound separation) 142 can be gradually improved. When a predetermined acceptable level of separation performance is obtained, the learning process can be terminated.
[0206] By performing such learning processing, it is possible to generate the noise separation unit (DNN for noise separation) 142 in the sound source separation processing execution unit 113 shown in FIG.
[0207] In the information processing device 100 of the present disclosure, it is only necessary to generate these three types of learning models (DNN), namely: (1) learning model (DNN) for extracting vocal sounds (singing sounds) or voice sounds (conversation sounds); (2) learning model (DNN) for extracting guitar sounds or piano sounds; and (3) learning model (DNN) for extracting noise sounds, thereby reducing the cost of the learning process.
[0208] Furthermore, compared to conventional configurations in which neural networks are used on a sound source basis, for example, a guitar sound separation learning model and a piano sound separation learning model are used as separate models, the configuration of the present disclosure reduces the amount of data for the neural network and parameters, and also makes it possible to reduce the amount of memory required to store this data.
[0209] 5. Sequence of sound source separation processing executed by the information processing device of the present disclosure Next, a sequence of sound source separation processing executed by the information processing device of the present disclosure will be described.
[0210] The sequence of sound source separation processing executed by the information processing device of the present disclosure will be described with reference to the flowchart shown in Fig. 25. The flowchart shown in Fig. 25 is executed under the control of a control unit in accordance with a program stored in a storage unit of the information processing device 100 of the present disclosure. The control unit has a processor such as a CPU having a program execution function, and controls processing according to the flow by program execution processing using the processor.
[0211] The processing of each step of the flow shown in Fig. 25 will be described in order below. It is assumed that before the flow shown in Fig. 25 starts, the audio data to be subjected to sound source separation has already been stored in the memory (storage unit) of the information processing device 100. The user starts the sound source separation processing of the audio data recorded (stored) in the memory (storage unit) inside the information processing device 100.
[0212] (Step S101) First, in step S101, the information processing device 100 inputs selection information of the type of sound source included in the audio data (recorded data) that is the target for executing the sound source separation process.
[0213] A specific example of this sound source type selection process will be described with reference to Fig. 26. A UI (user interface) such as that shown in Fig. 26 is displayed on the display unit of the information processing device 100, and the user uses this UI to instruct the start of sound source separation process.
[0214] The input process of the sound source type selection information may be performed at the start of the sound source separation process, or may be performed, for example, when the user starts recording. In this case, the sound source type selection information is recorded as metadata for the recorded audio data stored in the storage unit.
[0215] Here, an example of processing will be described in which selection information for the type of sound source contained in recorded audio data is input at the start of the sound source separation processing. As shown in Fig. 26, the UI displayed on the display unit of the information processing device 100 is a UI for selecting and specifying the type of sound source contained in the audio data that is the target of the sound source separation processing. (a) Vocal / Voice (b) Guitar / Piano (c) Vocal & Guitar / Vocal & Piano One of these three options is selected.
[0216] "(a) Vocal / Voice" is selected when the sound source contained in the recorded audio data is singing sound (vocals) without any instruments, or conversation sound (voice). "(b) Guitar / Piano" is selected when the sound source contained in the recorded audio data is guitar sound or piano sound without vocals. "(c) Vocal & Guitar / Vocal & Piano" is selected when the sound source contained in the recorded audio data is vocals and guitar sound, or vocals and piano sound.
[0217] As described above, the information processing device 100 of the present disclosure does not distinguish between vocal sounds (singing sounds) and voice sounds (conversational sounds), but extracts them as sounds of the same category using a single learning model (e.g., DNN).Furthermore, it does not distinguish between guitar sounds and piano sounds, but extracts them as sounds of the same category using a single learning model (e.g., DNN).
[0218] That is, in the configuration of the present disclosure, each sound source separation process is performed using the following three types of learning models (DNN): (1) Learning model (DNN) for extracting vocal sounds (singing sounds) or voice sounds (conversation sounds) (2) Learning model (DNN) for extracting guitar sounds or piano sounds (3) Learning model (DNN) for extracting noise sounds
[0219] 26 is configured as an option that does not distinguish between vocal sounds (singing sounds) and voice sounds (conversation sounds) when the sound source to be separated includes vocal sounds (singing sounds) and voice sounds (conversation sounds).Furthermore, when the sound source to be separated includes guitar sounds or piano sounds, the option is also configured as an option that does not distinguish between these.
[0220] When the user selects one of the options (a) to (c) from the UI shown in FIG. 26 and touches "Start sound source separation processing," the information processing device 100 starts the sound source separation processing.
[0221] Note that the UI shown in Fig. 26 is an example, and UIs with other configurations may be displayed and used. For example, the UI shown in Fig. 27 is a UI that allows selection from the following five sound source types as the type of sound source included in the audio data that is the target of sound source separation processing: (a) Vocal / Voice (b) Guitar (c) Piano (d) Vocal & Guitar (e) Vocal & Piano It is configured as a UI that allows selection of one of these three options.
[0222] However, as described above, the information processing device 100 of the present disclosure does not distinguish between guitar sounds and piano sounds, but extracts them as sound sources of the same category (same type) using a single learning model (DNN, etc.).
[0223] Therefore, the sound source separation process executed in the information processing device is the same process using the same single learning model (DNN, etc.) whether "(b) guitar" is selected or "(c) piano" is selected. That is, the guitar and piano sound separation unit (DNN for guitar and piano sound separation) 132 of the sound source separation process execution unit 113 shown in Fig. 20 is used to execute the process of extracting the guitar sound or piano sound.
[0224] The guitar and piano sound separation unit (DNN for guitar and piano sound separation) 132 executes a process of extracting guitar or piano sounds using a learning model (DNN) for extracting guitar or piano sounds.
[0225] Similarly, the information processing device 100 of the present disclosure executes the same sound source separation process when "(d) Vocals & Guitar" is selected and when "(e) Vocals & Piano" is selected. That is, the vocal & voice sound separation unit (DNN for vocal & voice sound separation) 122 and the guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132 of the sound source separation process execution unit 113 shown in Fig. 20 are used to execute the process of separating vocal sounds (singing sounds) and guitar sounds, or the process of separating vocal sounds (singing sounds) and piano sounds.
[0226] 27 is a UI in which guitar and piano are set as separate options to make it easier for the user to intuitively select the instrument that they actually used. The processing actually executed by the information processing device 100 is the same whether guitar or piano is selected.
[0227] Furthermore, a UI including a warning message as shown in Fig. 28 may be used. The UI shown in Fig. 28 includes the following warning message: "Warning: Instrument sounds other than guitar and piano may be erased."
[0228] As described above with reference to Fig. 20 , the information processing device 100 of the present disclosure performs sound source separation processing using the following three sound source separation units: a vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122, a guitar and piano sound separation unit (DNN for guitar and piano sound separation) 132, and a noise sound separation unit (DNN for noise sound separation) 142.
[0229] Therefore, there is a possibility that instrument sounds other than guitar and piano will be included in the noise sounds output by the above three separation units, will be discarded by the system, and will be unavailable to the user. To notify the user of this, a warning message such as that shown in Figure 28 may be displayed on the UI.
[0230] Returning to the flowchart shown in Fig. 25, the description of the flowchart will continue. (Step S102) In step S101, the information processing device 100 inputs selection information for the type of sound source included in the audio data (recorded data) that is the target of the sound source separation process. Next, in step S102, the information processing device 100 starts the sound source separation process.
[0231] Specifically, the information processing device 100 executes the sound source separation process in the sound source separation process execution unit 113 having the configuration described with reference to FIG.
[0232] The processing mode of the sound source separation processing executed by the sound source separation processing execution unit 113 of the information processing device 100 differs depending on the type of sound source included in the audio data to be separated selected by the user.
[0233] For example, as shown in FIG. 29, the processing when the user selects option (a) vocal / voice as the type of sound source contained in the audio data to be subjected to sound source separation processing will be described.
[0234] In this case, the processing mode of the sound source separation processing executed by the sound source separation processing execution unit 113 of the information processing device 100 is the processing mode shown in Fig. 30. That is, as shown in Fig. 30, processing is executed using two sound source separation units, namely, a vocal & voice sound separation unit (DNN for vocal & voice sound separation) 122 and a noise sound separation unit (DNN for noise sound separation) 142 within the sound source separation processing execution unit 113. The guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132 is not used.
[0235] The pre-processing unit (STFT) 121, vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122, and post-processing unit (ISTFT) 123 shown in the upper part perform extraction processing of vocal sounds (singing sounds) or voice sounds (conversation sounds) contained in the input data.
[0236] Furthermore, the pre-processing unit (STFT) 141, noise sound separation unit (DNN for noise sound separation) 142, and post-processing unit (ISTFT) 143 shown in the lower part perform extraction processing of noise sounds contained in the input data.
[0237] Note that the audio data files for each sound source generated by these sound source separation processes, i.e., an audio data file consisting of vocal sounds (singing sounds) or voice sounds (conversation sounds), and an audio data file consisting of noise sounds, are stored in the storage unit 112 within the information processing device 100. Note that in the processing example shown in Fig. 30 , processing is executed using only the vocal & voice sound separation unit (vocal & voice sound separation DNN) 122 and the noise sound separation unit (noise sound separation DNN) 142 within the sound source separation processing execution unit 113, and is described as an example of processing that does not use the guitar & piano sound separation unit (guitar & piano sound separation DNN) 132, but even when "(a) vocal / voice" is selected in the UI shown in Fig. 29 , the separation operation of the guitar & piano sound separation unit (guitar & piano sound separation DNN) 132 may be configured to operate without stopping.
[0238] Next, with reference to Figs. 31 and 32, a processing example will be described in which the user selects different options as the types of sound sources included in the audio data that is the target of the sound source separation processing.
[0239] For example, as shown in FIG. 31, the processing when the user selects option (c) vocals & guitar / vocals & piano as the type of sound source contained in the audio data to be subjected to sound source separation processing will be described.
[0240] In this case, the processing mode of the sound source separation processing executed by the sound source separation processing execution unit 113 of the information processing device 100 is the processing mode shown in Fig. 32. That is, as shown in Fig. 32, processing is executed using three sound source separation units in the sound source separation processing execution unit 113: a vocal & voice sound separation unit (DNN for vocal & voice sound separation) 122, a guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132, and a noise sound separation unit (DNN for noise sound separation) 142.
[0241] The pre-processing unit (STFT) 121, vocal and voice sound separation unit (DNN for vocal and voice sound separation) 122, and post-processing unit (ISTFT) 123 shown in the upper part perform extraction processing of vocal sounds (singing sounds) or voice sounds (conversation sounds) contained in the input data.
[0242] Furthermore, the middle stage pre-processing unit (STFT) 131, guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132, and post-processing unit (ISTFT) 133 are configured to extract the guitar or piano sounds contained in the input data.
[0243] Furthermore, the pre-processing unit (STFT) 141, noise sound separation unit (DNN for noise sound separation) 142, and post-processing unit (ISTFT) 143 shown in the lower part perform extraction processing of noise sounds contained in the input data.
[0244] The audio data files generated by these sound source separation processes, i.e., an audio data file consisting of vocal sounds (singing sounds) or voice sounds (conversation sounds), an audio data file consisting of guitar sounds or piano sounds, and an audio data file consisting of noise sounds, are stored in the memory unit 112 within the information processing device 100.
[0245] Continuing the explanation of the flowchart shown in Figure 25 (Step S103) In step S103, the information processing device 100 determines whether the sound source separation processing started in step S102 has ended, and if it determines that the sound source separation processing has ended, proceeds to step S104.
[0246] (Step S104) Next, in step S104, the information processing device 100 displays, on the display unit, display data for playback processing or editing processing of the sound data for each sound source generated by the sound source separation processing executed in step S102.
[0247] An example of display data for playback processing or editing processing of specific sound source-based sound data is shown in Fig. 33. In the example shown in Fig. 33, the sound source-based sound data generated by the sound source separation processing executed in step S102 is sound data for three types of sound sources: vocal sound (singing sound), guitar sound, and noise sound.
[0248] The sound data of these three types of sound source units are arranged and displayed along a synchronized time axis on the display unit 117 of the information processing device 100. Note that, although the display example shown in Fig. 33 shows a display example of sound data of three types of sound source units, namely vocal sound (singing sound), guitar sound, and noise sound, the noise sound is unlikely to be an editing target, and therefore the display of the noise sound may be omitted, and only the guitar sound and noise sound, namely these two types of sound data, may be displayed side by side.
[0249] (Step S105) In step S105, the user performs playback and editing of the audio data for each sound source unit using the editing display data displayed in step S104. To perform playback of the audio data for each sound source unit, the user simply touches the playback button set to the right of the audio data for each sound source unit.
[0250] To edit the audio data for each sound source, the user simply touches the edit button located to the right of the audio data for that sound source. This displays a UI that allows the user to perform various data edits, such as volume adjustment, pitch adjustment, or timing adjustment, for each sound source.
[0251] (Step S106) When the data editing process for each sound source in step S105 is completed, the information processing device 100 stores the edited data file for each sound source in the storage unit 106 in step S106, and ends the process.
[0252] 6. Example of a configuration in which sound source separation processing is executed in an external device Next, an example of a configuration in which sound source separation processing is executed in an external device will be described.
[0253] In the above-described embodiment, a configuration example has been described in which the information processing device 100, such as a smartphone, performs the sound source separation process using the sound source separation process execution unit 113 configured inside the information processing device 100.
[0254] That is, as explained above with reference to Figure 14, the voice data input by user 1 via the microphone of an information processing device 100 such as a smartphone is recorded in a storage unit 112 (corresponding to the memory (storage unit) 102 in Figure 14) of the information processing device 100 in Figure 17, and the sound source separation processing is executed on this recorded voice data in a sound source separation processing execution unit 113 inside the information processing device 100.
[0255] This sound source separation process is not limited to being performed within the information processing device 100, but may be performed in an external device that can communicate with the information processing device 100, such as a data processing server.
[0256] An example of a system that performs sound source separation processing using an external device will be described with reference to Fig. 34. Fig. 34 shows an information processing device 100, such as a smartphone used by a user, and a data processing server 200, which is an external device that can communicate with the information processing device 100.
[0257] 34, the sound source separation process is executed by the data processing server 200, not the information processing device 100. The data processing server 200 has a sound source separation process execution unit similar to the sound source separation process execution unit 113 described above with reference to FIG.
[0258] The information processing device 100 transmits voice data stored in a storage unit inside the information processing device 100 to the data processing server 200. The voice data to be transmitted is, for example, data recorded in advance by the user of the information processing device 100.
[0259] When transmitting audio data to the data processing server 200, the information processing device 100 adds information indicating the type of sound source included in the audio data to be transmitted as metadata and transmits the data. The information indicating the type of sound source included in the audio data can be information specified by the user using the UI described above with reference to Figures 26 to 28, for example.
[0260] The data processing server 200 receives audio data and metadata, which is information indicating the type of sound source included in the audio data, from the information processing device 100. The data processing server 200 performs sound source separation processing on the audio data based on the metadata received from the information processing device 100.
[0261] This sound source separation process is the same as the process described above with reference to Figures 30 and 32. The sound data for each sound source generated as a result of the sound source separation process executed in the data processing server 200 is transmitted from the data processing server 200 to the information processing device 100 and stored in a storage unit of the information processing device 100. Note that the noise data generated as a result of the sound source separation process may be discarded on the data processing server 200 side without being transmitted to the information processing device 100.
[0262] After this, the information processing device 100 can perform playback and editing of the audio data for each sound source. That is, the UI described above with reference to Fig. 33 can be displayed, and playback and editing of the audio data for each sound source can be performed.
[0263] In this way, the sound source separation process may be configured to be executed not inside the information processing device 100 but in an external device that can communicate with the information processing device 100, such as a data processing server.
[0264] Next, with reference to FIG. 35, a configuration of the information processing device 100 when the sound source separation process is executed in the data processing server 200 and an example of the configuration of the data processing server 200 will be described.
[0265] The information processing device 100, which is configured using a smartphone or the like used by a user, has an audio input unit (microphone) 111, a memory unit 112, an audio data editing processing unit 115, a playback control unit 116, a display unit 117, an audio output unit (speaker) 118, and a communication unit 119, as shown in FIG. 35.
[0266] The voice input unit (microphone) 111 is a microphone that inputs the user's voice or the sound of an instrument such as a guitar. The storage unit 112 records the voice data input via the voice input unit (microphone) 111.
[0267] The audio data editing processing unit 115 is a data processing unit that displays, for example, a UI as described above with reference to Fig. 33 and executes editing processing of audio data for each sound source unit in accordance with user instructions. The audio data editing processing unit 115 reads out audio data files for each sound source separated in the data processing server 200 from the storage unit 112, displays audio data information for each sound source as described with reference to Fig. 33 on the display unit 117, changes and edits the audio data for each sound source in accordance with user instructions, and stores the results in the storage unit 112.
[0268] The playback control unit 116 executes a process of reading out and playing back audio data stored in the memory unit 112, such as audio data at the time of recording, and audio data files in sound source units separated by the processing execution unit 113, from the memory unit 112.
[0269] The display unit 117 displays, for example, a UI such as those described with reference to Figures 26 to 28 and 33. The display unit 117 is a touch panel type display unit, and is configured to enable various input processes and instructions to be performed by user operations.
[0270] The audio output unit (speaker) 118 is a speaker that outputs audio data and the like that is reproduced by the processing of the reproduction control unit 116. The communication unit 119 executes communication with an external device, for example, the data processing server 200.
[0271] On the other hand, the data processing server 200 includes a communication unit 201 , a sound source separation processing execution unit 202 , and a learning model 203 .
[0272] The communication unit 201 communicates with the information processing device 100, such as a smartphone. The sound source separation processing execution unit 202 inputs audio data received from the information processing device 100 and executes sound source separation processing to separate the audio data into data corresponding to each sound source included in the audio data. The audio data files for each sound source separated by the sound source separation processing execution unit 202 are transmitted to the information processing device 100 via the communication unit 201.
[0273] The learning model (DNN) 203 is a learning model used in the sound source separation process executed by the sound source separation process execution unit 202, for example, a neural network such as a DNN.
[0274] The sound source separation processing execution unit 202 and the learning model (DNN) 203 in the data processing server 200 have the same configuration as when the sound source separation processing is executed in the previously described information processing device 100. That is, the data processing server 200 has the same sound source separation processing execution unit as previously described with reference to FIG. 20 etc.
[0275] Specifically, it has the same configuration as that described above with reference to Fig. 20 and is configured by the following three sound source separation units: (a) a vocal & voice sound separation unit (DNN for vocal & voice sound separation) 122 that executes processing to extract vocal sounds (singing sounds) or voice sounds (conversation sounds) using a learning model (DNN) for extracting vocal sounds (singing sounds) or voice sounds (conversation sounds), (b) a guitar & piano sound separation unit (DNN for guitar & piano sound separation) 132 that executes processing to extract guitar sounds or piano sounds using a learning model (DNN) for extracting guitar sounds or piano sounds, and (3) a noise sound separation unit (DNN for noise sound separation) 142 that executes processing to extract noise sounds using a learning model (DNN) for extracting noise sounds.
[0276] In this way, vocal sounds (singing sounds) and voice sounds (conversation sounds) are not distinguished from each other and are separated as sounds (voices) of the same category (type). Similarly, guitar sounds and piano sounds are not distinguished from each other and are separated as sounds (instrument sounds) of the same category. The sound source separation processing execution unit 202 in the data processing server 200 is configured to execute such sound source separation processing.
[0277] 7. Hardware Configuration Example of Information Processing Device of the Present Disclosure Next, a hardware configuration example of an information processing device of the present disclosure will be described with reference to FIG.
[0278] Each element of the hardware configuration shown in Fig. 36 will be described below. A CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes according to programs stored in a ROM (Read Only Memory) 302 or a storage unit 308. For example, it executes processes according to the sequences described in the above-mentioned embodiments. A RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are interconnected by a bus 304.
[0279] The CPU 301 is connected to an input / output interface 305 via a bus 304, and the input / output interface 305 is connected to an input unit 306 consisting of various sensors, a camera, a switch, a keyboard, a mouse, a microphone, etc., and an output unit 307 consisting of a display, a speaker, etc.
[0280] The storage unit 308 connected to the input / output interface 305 is composed of, for example, a USB memory, an SD card, a hard disk, etc., and stores various data and programs executed by the CPU 301. The communication unit 309 functions as a transmission / reception unit for data communication via a network such as the Internet or a local area network, and communicates with external devices.
[0281] A drive 310 connected to the input / output interface 305 drives removable media 311 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory such as a memory card, and executes recording or reading of data.
[0282] [8. Summary of the Configuration of the Present Disclosure] The embodiments of the present disclosure have been described above in detail with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.
[0283] The technology disclosed in this specification can be configured as follows: (1) An information processing device having a sound source separation processing execution unit that executes sound source separation processing, the sound source separation processing execution unit having a configuration that executes sound source separation processing using a learning model, and that executes processing to extract a plurality of different types of musical instrument sounds using the same learning model.
[0284] (2) The information processing device according to (1), wherein the sound source separation processing execution unit executes processing to extract guitar sounds and piano sounds using the same learning model.
[0285] (3) The information processing device according to (1) or (2), wherein the sound source separation processing execution unit further executes processing to extract vocal sounds (singing sounds) and voice sounds (conversation sounds) using the same learning model.
[0286] (4) The information processing device according to any one of (1) to (3), wherein the sound source separation processing execution unit has three types of learning models (a) to (c): (a) a learning model that extracts vocal sounds (singing sounds) and voice sounds (conversation sounds); (b) a learning model that extracts guitar sounds and piano sounds; and (c) a learning model that extracts noise sounds.
[0287] (5) The information processing device according to any one of (1) to (4), wherein the learning model is a learning model having a DNN (Deep Neural Network).
[0288] (6) The information processing device is configured to display, on a display unit, a UI (user interface) including sound source type options for selecting the type of sound source included in audio data to be subjected to sound source separation before starting sound source separation processing using the sound source separation processing unit, and the UI is a UI in which multiple different types of instrument sounds are set as one sound source type option. The information processing device is described in any one of (1) to (5).
[0289] (7) The information processing device according to (6), wherein, when starting recording of audio data to be subjected to sound source separation, the information processing device displays a UI (user interface) on a display unit, the UI including sound source type options for allowing the user to select the type of sound source included in the audio data to be recorded.
[0290] (8) The information processing device according to (6) or (7), wherein the UI is a UI in which a guitar sound and a piano sound are set as one sound source type option.
[0291] (9) The information processing device according to (8), wherein the UI further includes a vocal sound (singing sound) and a voice sound (conversation sound) set as one sound source type option.
[0292] (10) The information processing device described in any one of (1) to (9), wherein the sound source separation processing execution unit is configured to extract guitar sounds and piano sounds using the same learning model, and the learning model for extracting guitar sounds and piano sounds is a learning model in which parameters are set to extract sounds having sound characteristics common to guitar sounds and piano sounds.
[0293] (11) The information processing device described in any one of (1) to (10), wherein the sound source separation processing execution unit is configured to extract vocal sounds (singing sounds) and voice sounds (conversation sounds) using the same learning model, and the learning model that extracts vocal sounds (singing sounds) and voice sounds (conversation sounds) is a learning model in which parameters are set to extract sounds that have sound characteristics common to vocal sounds (singing sounds) and voice sounds (conversation sounds).
[0294] (12) The information processing device according to any one of (1) to (11), wherein the sound source separation processing execution unit has a preprocessing unit, a sound source separation unit using a learning model, and a postprocessing unit.
[0295] (13) The information processing device according to (12), wherein the preprocessing unit is a data processing unit that performs a short-time Fourier transform (STFT) on input audio data, and the postprocessing unit is a data processing unit that performs an inverse short-time Fourier transform (ISTFT) on an output from the sound source separation unit.
[0296] (14) The information processing device according to any one of (1) to (13), wherein the learning model is a learning model generated by a supervised learning process.
[0297] (15) The information processing device according to (14), wherein the learning model generated by the supervised learning process is a learning model that extracts guitar sounds and piano sounds, and is generated by a learning process using teacher data in which both guitar sounds and piano sounds are set as correct answer data.
[0298] (16) The information processing device according to (14) or (15), wherein the learning model generated by the supervised learning process is a learning model that extracts vocal sounds (singing sounds) and voice sounds (conversational sounds), and is generated by a learning process using teacher data in which both vocal sounds (singing sounds) and voice sounds (conversational sounds) are set as correct answer data.
[0299] (17) A sound source separation processing method executed in an information processing device, wherein the information processing device has a sound source separation processing execution unit that executes sound source separation processing, and the sound source separation processing execution unit executes processing to extract a plurality of different types of musical instrument sounds using the same learning model.
[0300] (18) A program for causing an information processing device to execute a sound source separation process, the information processing device having a sound source separation process execution unit that executes the sound source separation process, the program causing the sound source separation process execution unit to execute a process of extracting a plurality of different types of musical instrument sounds using the same learning model.
[0301] Furthermore, the series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program from the recording medium to the computer, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.
[0302] The various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are all located in the same housing.
[0303] As described above, according to one embodiment of the present disclosure, a configuration is realized that executes a sound source separation process that extracts a plurality of different instrument sounds using a single learning model. Specifically, for example, the configuration includes a sound source separation process execution unit that executes the sound source separation process, and the sound source separation process execution unit has a configuration that executes the sound source separation process using a learning model, and executes a process that extracts a plurality of different types of instrument sounds using the same learning model. The sound source separation process execution unit has three types of learning models: (a) a learning model that extracts vocal sounds (singing sounds) and voice sounds (conversation sounds), (b) a learning model that extracts guitar sounds and piano sounds, and (c) a learning model that extracts noise sounds. With this configuration, a configuration is realized that executes a sound source separation process that extracts a plurality of different instrument sounds using a single learning model.
[0304] 1 User 20 Sound source separation processing execution unit 21, 31, 41, 51 Pre-processing unit 23, 33, 43, 53 Post-processing unit 22 Vocal sound separation unit (DNN for vocal sound separation) 32 Guitar sound separation unit (DNN for guitar sound separation) 42 Piano sound separation unit (DNN for piano sound separation) 53 Noise sound separation unit (DNN for noise sound separation) 100 Information processing device 101 Microphone 102 Memory (storage unit) 111 Audio input unit (microphone) 112 Storage unit 113 Sound source separation processing execution unit 114 Learning model (DNN) 115 Audio data editing processing unit 116 Playback control unit 117 Display unit 118 Audio output unit (speaker) 119 Communication unit 121, 131, 141 Pre-processing unit 123, 133, 143 Post-processing unit 122 Vocal & voice sound separation unit (DNN for vocal & voice sound separation) 132 Guitar & piano sound separation unit (DNN for guitar & piano sound separation) 142 Noise sound separation unit (DNN for noise sound separation) 200 Data processing server 201 Communication unit 202 Sound source separation processing execution unit 203 Learning model (DNN) 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / output interface 306 Input unit 307 Output unit 308 Storage unit 309 Communication unit 310 Drive 311 Removable media
Claims
1. An information processing device having a sound source separation processing execution unit that executes sound source separation processing, the sound source separation processing execution unit having a configuration for executing sound source separation processing using a learning model, and executing a process of extracting multiple different types of musical instrument sounds using the same learning model.
2. The information processing device according to claim 1, wherein the sound source separation processing execution unit executes a process for extracting guitar sounds and piano sounds using a same learning model.
3. The information processing device according to claim 1, wherein the sound source separation processing execution unit further executes a process of extracting vocal sounds (singing sounds) and voice sounds (conversation sounds) using the same learning model.
4. The information processing device of claim 1, wherein the sound source separation processing execution unit is configured to have three types of learning models: (a) a learning model that extracts vocal sounds (singing sounds) and voice sounds (conversation sounds), (b) a learning model that extracts guitar sounds and piano sounds, and (c) a learning model that extracts noise sounds.
5. The information processing device according to claim 1, wherein the learning model is a learning model having a DNN (Deep Neural Network).
6. The information processing device according to claim 1, wherein the information processing device is configured to display, on a display unit, a UI (user interface) including sound source type options for selecting the type of sound source contained in the audio data to be subjected to sound source separation before starting sound source separation processing using the sound source separation processing unit, the UI being a UI in which a plurality of different types of musical instrument sounds are set as one sound source type option.
7. The information processing device according to claim 6, wherein when starting recording of audio data to be subjected to sound source separation, the information processing device displays on a display unit a UI (user interface) including sound source type options for allowing the user to select the type of sound source contained in the audio data to be recorded.
8. The information processing device according to claim 6, wherein the UI is a UI in which a guitar sound and a piano sound are set as one sound source type option.
9. The information processing device according to claim 8, wherein the UI further includes a vocal sound (singing sound) and a voice sound (conversation sound) set as one sound source type option.
10. The information processing device according to claim 1, wherein the sound source separation processing execution unit is configured to extract guitar sounds and piano sounds using the same learning model, and the learning model for extracting guitar sounds and piano sounds is a learning model in which parameters are set to extract sounds having sound characteristics common to guitar sounds and piano sounds.
11. The information processing device according to claim 1, wherein the sound source separation processing execution unit is configured to extract vocal sounds (singing sounds) and voice sounds (conversation sounds) using the same learning model, and the learning model for extracting vocal sounds (singing sounds) and voice sounds (conversation sounds) is a learning model having parameters set for extracting sounds having sound characteristics common to vocal sounds (singing sounds) and voice sounds (conversation sounds).
12. The information processing device according to claim 1, wherein the sound source separation processing execution unit has a pre-processing unit, a sound source separation unit using a learning model, and a post-processing unit.
13. The information processing device according to claim 12, wherein the pre-processing unit is a data processing unit that performs a short-time Fourier transform (STFT) on input audio data, and the post-processing unit is a data processing unit that performs an inverse short-time Fourier transform (ISTFT) on output from the sound source separation unit.
14. The information processing device according to claim 1, wherein the learning model is a learning model generated by a supervised learning process.
15. The information processing device according to claim 14, wherein the learning model generated by the supervised learning process is a learning model that extracts guitar sounds and piano sounds, and is generated by a learning process using teacher data in which both guitar sounds and piano sounds are set as correct answer data.
16. The information processing device according to claim 14, wherein the learning model generated by the supervised learning process is a learning model that extracts vocal sounds (singing sounds) and voice sounds (conversational sounds), and is a learning model generated by a learning process using teacher data in which both vocal sounds (singing sounds) and voice sounds (conversational sounds) are set as correct answer data.
17. A sound source separation processing method executed in an information processing device, wherein the information processing device has a sound source separation processing execution unit that executes sound source separation processing, and the sound source separation processing execution unit executes a process of extracting a plurality of different types of musical instrument sounds using the same learning model.
18. A program for causing an information processing device to execute sound source separation processing, the information processing device having a sound source separation processing execution unit that executes the sound source separation processing, the program causing the sound source separation processing execution unit to execute a process of extracting a plurality of different types of musical instrument sounds using the same learning model.
Citation Information
Patent Citations
Program, information processing method, recording medium, and information processing device
WO2023276235A1
Sound source separation program, sound source separation device, sound source separation method, and generation program
JP2021128252A
System and Method for Hierarchical Audio Source Separation
US20220101869A1
Signal processing device, method, and program
WO2020195924A1
Signal processing device, method, and program
WO2022014326A1
