Audio extraction device and program

Abstract

To provide a sound extracting apparatus and a program therefor which, when treatable sound is changed, can reduce the number of learning while keeping extraction accuracy, and can make processing faster.SOLUTION: A sound extracting apparatus 1 comprises: a model generating unit 32B for carrying out deep learning based on a created learning data set to generate a basic model Mo for extracting environmental sounds per kind, a user original basic model Mo' for extracting predetermined sounds which the basic model Mo does not treat, and a first state extraction model M for extracting a new sound; and an extracting unit 13 for extracting a new sound at a first stage from an input sound by use of the first stage extraction model M and by calculation of a neural network, and extracting individual environmental sounds and a predetermined sound at a second stage from the sound extracted at the first stage by use of the basic model Mo and the user original basic model Mo' and by calculation of the neural network.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a voice extraction device and a program for extracting a given sound or voice from an input voice in which various voices are superimposed. [Background technology]

[0002] There is known a technique for extracting any sound or voice from an input voice using a trained machine learning model.

[0003] Patent Document 1 describes an audio extraction device having an audio extraction unit that extracts either accompaniment sounds or vocal sounds from first channel audio data, second channel audio data, and center cut audio data using a trained machine learning model. The audio extraction device of Patent Document 1 provides an acoustic processing technique for extracting specific audio components from audio data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-3537 A Summary of the Invention [Problem to be solved by the invention]

[0005] The acoustic processing technology described in Patent Document 1 needs to redo all learning when the number of compatible voices is increased. For example, if three types of voices are currently supported, and the number of compatible voices is increased to four, learning using deep learning must be performed four times. This is even more serious when there are many compatible voices.

[0006] The present invention has been made in consideration of such problems, and has an object to provide a voice extraction device and a program therefor that can reduce the number of learning rounds while maintaining extraction accuracy when the compatible voices are changed, thereby speeding up processing. [Means for solving the problem]

[0007] In order to solve the above problems, the voice extraction device according to the present invention includes a data set creation means for creating a learning data set including a plurality of types of environmental sounds and a predetermined voice, data created by synthesizing these sounds as a target, and each sound that is a single voice before synthesis and used to create the target data; and a deep learning method based on the created learning data set. The system includes a model generating means for generating a basic model for extracting the environmental sounds by type, a user original basic model for extracting a predetermined sound not supported by the basic model, and a first-stage extraction model for extracting a new sound by learning through learning, a model storage means for storing the generated basic model, the user original basic model, and the first-stage extraction model, a model management means for managing the basic model, the user original basic model, and the first-stage extraction model stored in the model storage means, and an extraction means for extracting individual environmental sounds or the predetermined sound from an input sound by a neural network operation using the basic model, the user original basic model, and the first-stage extraction model passed from the model management means, wherein the extraction means extracts new sounds from the input sound by a neural network operation in a first stage using the first-stage extraction model, and extracts individual environmental sounds and the predetermined sound in a second stage from the sound extracted in the first stage by the neural network operation using the basic model and the user original basic model. The data set creation means creates a new training data set by combining the new voice with the target data in the training data set, and the model generation means generates the first-stage extraction model by training the new training data set created by the data set creation means. The voice extraction device is characterized by the above.

[0008] In such a configuration, the voice extraction device inputs a plurality of types of environmental sounds and a predetermined voice (e.g., a human voice) by the data set creation means, synthesizes these sounds to create a learning data set consisting of the data to be the subject and each sound that is a single voice before synthesis used to create the subject data. Note that the predetermined voice includes a human voice, and the environmental sound may be a sound other than a human voice.

[0009] In addition, the data set creation means creates a new training data set by combining the new voice with the target data from the training data set.

[0010] Then, the voice extraction device uses the model generation means to learn through deep learning based on the created learning dataset, and generates a basic model for extracting environmental sounds by type, a user's original basic model for extracting specific sounds that are not supported by the basic model, and a first-stage extraction model for extracting new sounds.

[0011] In addition, the model generation means generates a first-stage extraction model by training the model using the new training data set created by the data set creation means.

[0012] Then, the voice extraction device stores the generated basic model, the user's original basic model, and the first-stage extraction model in the model storage means.

[0013] The voice extraction device uses a model management means to manage the basic model, the user original basic model, and the first-stage extraction model stored in the model storage means. When new voice is input, the model management means outputs the first-stage extraction model to the extraction means. When the extracted voice can be handled by the basic model and the user original basic model by extracting new voice, the model management means outputs the basic model and the user original basic model to the extraction means in response to an instruction from the user, for example.

[0014] The voice extraction device then uses the basic model, the user original basic model, and the first-stage extraction model passed from the model management means to extract individual environmental sounds or predetermined sounds from the input voice through neural network calculations by the extraction means. The extraction means uses the first-stage extraction model to extract new sounds from the input voice in a first stage through neural network calculations, and uses the basic model and the user original basic model to extract individual environmental sounds and the predetermined sounds from the voice extracted in the first stage through neural network calculations in a second stage.

[0015] As a result, when the voice extraction device changes the available voices, it is possible to reduce the number of learning times while maintaining the extraction accuracy, thereby enabling the processing speed to be increased.

[0016] The voice extraction device can be operated by a program that causes a computer to function as each of the means of the voice extraction device. Effect of the Invention

[0017] The present invention provides the following excellent effects. According to the present invention, even when new sounds are extracted through two-stage extraction, it is not necessary to learn all sounds, and learning for separating new sounds from environmental sounds is performed only once. Therefore, in the present invention, when the available voices are changed, the number of learning times can be reduced while maintaining the extraction accuracy, and the processing speed can be increased. [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram showing a configuration of a voice extraction device according to an embodiment of the present invention. [Diagram 2] 1A to 1C are diagrams illustrating an example of the structures of a basic model Mo, a user's original basic model Mo', and a first-stage extraction model M, which are configured using a deep neural network of a voice extraction device according to an embodiment of the present invention. [Diagram 3] 1 is a diagram illustrating an overview of an extraction system of a voice extraction device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a hardware configuration diagram showing an example of a computer that realizes the processing of the voice extraction method according to the embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram illustrating the creation of a learning dataset for the voice extraction device according to the embodiment of the present invention. [Figure 6] FIG. 1 is a diagram illustrating a method for learning a learning dataset created by a voice extraction device according to an embodiment of the present invention through Deep Learning. [Figure 7] 1A to 1C are diagrams illustrating an extraction method for extracting individual environmental sounds or human voices from input voice data using each model of the voice extraction device according to an embodiment of the present invention. [Figure 8] FIG. 11 is a diagram providing additional explanation of the training dataset of the voice extraction device according to an embodiment of the present invention, where (a) shows the training dataset combined with a task and a model answer for a human voice, (b) shows the training dataset combined with a task and a model answer for the sound of rain, and (c) shows the training dataset combined with a task and a model answer for the sound of a car. [Figure 9] 11A and 11B are diagrams for providing a supplementary explanation of the sound extraction method of the voice extraction device according to the embodiment of the present invention. [Figure 10] FIG. 2 is a diagram illustrating an extraction method (part 1) of the voice extraction device according to the embodiment of the present invention. [Figure 11] FIG. 4 is a diagram illustrating an extraction method (part 2) of the voice extraction device according to the embodiment of the present invention. [Figure 12] 4 is a flowchart showing an operation of a voice extraction method of the voice extraction device according to the embodiment of the present invention. [Figure 13] 10 is a flowchart showing an extraction operation of a specific example of a voice extraction method of the voice extraction device according to the embodiment of the present invention. [Figure 14] FIG. 2 is a diagram for simply explaining functions in an extraction process of the voice extraction device according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Parameters for voice extraction device] The parameters of the voice extraction device according to the embodiment of the present invention will be described.

[0020] <Dataset> First, the data set used in the voice extraction device will be described. A dataset is a large number of pairs of "challenge data" and "model answer data." A model is the output of learning from this dataset using deep learning. In other words, a model is the parameters of a neural network used in the final extraction process.

[0021] In this embodiment, the voice extraction device has a basic model Mo which is an environmental sound model, a user original basic model Mo' which is a human voice model, and a first-stage extraction model M which is a model for separating new voice and environmental sound for separating new voice and environmental sound. The basic model Mo, the user original basic model Mo', and the first-stage extraction model M are essentially the same, with only differences in the contents of the data sets used for learning.

[0022] <Basic model Mo> Next, we will explain how to create and use the basic model Mo. The base model Mo is a model generated by learning with deep learning based on a learning dataset of environmental sounds, which are sounds other than human voices. Here, environmental sounds are sounds that change less (almost no) compared to human voices. The base model Mo is generated based on the model answers of the learning dataset for such environmental sounds, and once it is generated, it is unlikely to be altered. The basic model Mo may be pre-installed in the audio extraction device. The basic model Mo is an environmental sound model, and is a parameter for extracting and separating environmental sounds other than human voices by type. When performing extraction processing using this basic model Mo, it is assumed that the sound to be extracted does not include human voices.

[0023] The training dataset for the basic model Mo consists of "challenge data" created by synthesizing multiple environmental sounds, and "model answer data" consisting of each environmental sound (single audio before synthesis) used to create the "challenge data."

[0024] For example, let us take the following example: sounds that should be handled when recorded inside a car: "driving sounds / air conditioner / turn signals." A composite of the three sounds of driving sounds / air conditioner / turn signals will be the "challenge data." Additionally, a single audio sound of driving sounds will be the "model answer data." By learning using a learning dataset that pairs these, it will be possible to create a "model that extracts driving sounds from audio inside a car."

[0025] Next, the "model answer data" is changed to a single sound of an air conditioner, and the "challenge data" is left unchanged and remains the same. In other words, it is a combination of three sounds: driving sounds, air conditioner sounds, and turn signals. By training with a training dataset that pairs these, it is possible to create a "model that extracts the sound of an air conditioner from sounds inside a car." Similarly, by changing the "model answer data" to the turn signal, and training with a training dataset that pairs the same unaltered "challenge data," it is possible to create a model that extracts the sound of a turn signal from sounds inside a car. By performing extraction processing using the three models created in this way (basic model Mo), it is possible to separate the audio recorded "inside the car" into different types.

[0026] <User original basic model Mo'> Next, we will explain how to create and use the user's original basic model Mo'. The user original basic model Mo' is a predetermined voice (especially a human voice) model that is used by the user, and is a parameter for extracting and breaking down human voices by type (e.g., by individual). The learning dataset for the user original basic model Mo' is "challenge data" created by synthesizing human voices. Synthesized human voices are created by synthesizing multiple voices, and do not include voices created from scratch by a single human (such as Vocaloid). In addition, the "model answer data" is the human voice (single voice before synthesis) used to create the "challenge data".

[0027] <Model M for first stage extraction> Next, we will explain how to create and use the first-stage extraction model M. The first-stage extraction model M is a "model for separating new sounds and environmental sounds" for separating new sounds and environmental sounds. The first stage extraction model M is a means of making it possible to accommodate "new audio" (such as the voice of a new announcer or unexpected environmental sounds) that cannot be handled by the pre-installed basic model Mo.

[0028] The relationship between the basic model Mo and the first-stage extraction model M will be explained. As in the above example, in the basic model Mo, the "problem data" is a synthesis of multiple environmental sounds (three sounds: driving sounds, air conditioner sounds, and turn signals). For example, we try to extract the voice of a new announcer as the "new voice." In this case, in order to synthesize the "new voice," we create new "challenge data" for the first-stage extraction model M. Here, the "challenge data" is a synthesis of four sounds: driving sounds, air conditioner sounds, turn signals, and the voice of the new announcer. The "model answer data" that pairs with this new "challenge data" is the "new voice" and the "challenge data when the basic model Mo was created." Two first-stage extraction models M are created by training using these two datasets.

[0029] By using these two first-stage extraction models M for extraction, it is possible to separate "new voice" and "voice that can be handled by the basic model Mo." This completes the extraction of "new voice," and since "voice that can be handled by the basic model Mo" can be handled by the basic model Mo, the extraction process described below is carried out. This allows all voices to be extracted by type and separated. The above describes the relationship between the pre-installed basic model Mo and the first-stage extraction model M. The relationship between the user's original basic model Mo' and the first-stage extraction model M is similar.

[0030] <Environmental sounds> Environmental sounds are sounds other than human voices. Environmental sounds do not change (or change very little), whereas human voices change for each person (e.g., announcer). For this reason, environmental sounds once recorded can be reused as a learning dataset. Typical environmental sounds include, for example, the sounds of water (including rain), wind, cars, insects, and forests, but furthermore, as described below, a set of sounds expected to occur at each location where recording (such as the recording of a television program) is likely to take place may be introduced.

[0031] Examples of model sets (sets of voices) include the following: - Sounds that should be recorded in the forest: Set of tree sounds, bird sounds that may live in the forest, river sounds, and insect sounds (cicadas, crickets, bell crickets) - Sounds that should be recorded at sea: Set of wave sounds, sea bird sounds, and ship sounds - Sounds that should be handled when recording in the city: sounds of cars, sounds of ambulances and police cars, sounds of traffic lights, sounds of trains, footsteps, noise of people, and a set of sounds of cicadas - Sounds that should be recorded during construction: Set of fan sounds, machine sounds (whining sounds), and impact sounds (hammer sounds, etc.) - Sounds to be supported when recording animals: Set of dog / cat / bird / monkey / cow / pig / sheep / goat sounds - Sounds that need to be handled when recording at an amusement park: attraction sounds / screams / people noise / BGM set - Sounds that should be handled when recording in a car: Driving sounds / air conditioner / turn signal settings The above set of sounds includes white noise, helicopter sounds, wind sounds, etc. By preparing model sets (audio sets) of the locations listed here, it is believed that most filming locations and stopover points can be covered. Since there is a trade-off between the type of audio and accuracy, accuracy can be improved by narrowing down the compatible audio depending on the recording location.

[0032] [Configuration of the audio extraction device] Next, a configuration of a voice extraction device 1 according to an embodiment of the present invention will be described with reference to Fig. 1. The voice extraction device of this embodiment is an example applied to a multi-track system that extracts any sound or voice from an original sound (input voice) in which each sound or voice is superimposed. As shown in FIG. 1, the voice extraction device 1 includes an extraction system 10, a model management unit 20 (model management means), a model storage unit 25 (model storage means), and a learning system 30.

[0033] <Extraction system 10> The extraction system 10 includes an input unit 11, a Fourier transform unit 12, an extraction unit 13 (extraction means), a MIX (mixing) unit 14 (output means), an inverse Fourier transform unit 15, and an output unit 16 (output means). The input unit 11 receives the voice data for extraction. The Fourier transform unit 12 performs a Fourier transform to convert the audio data into frequency components for signal processing. The inverse Fourier transform unit 15 performs an inverse Fourier transform to convert the data that has been converted into frequency components back into audio data.

[0034] The extraction unit 13 uses the basic model Mo, the user's original basic model Mo', and the first-stage extraction model M passed from the model management unit 20 to extract individual environmental sounds or human voices from the input voice by neural network calculations.

[0035] When new voice is input, the extraction unit 13 uses the first-stage extraction model M to extract the new voice from the input voice through neural network calculations (first-stage extraction). By first extracting new voice using the first-stage extraction model M, the extracted voice can be handled by the basic model Mo. Then, the extraction unit 13 uses the basic model Mo and the user original basic model Mo' to extract individual environmental sounds or predetermined voices in a second stage from the voice extracted in the first stage.

[0036] The extraction unit 13 performs the actual extraction process (multiplication of the matrix of the neural network based on the model). The extraction unit 13 outputs the extracted voice based on the presence or absence of MIX (user designation). The MIX unit 14 mixes the multiple pieces of audio data sent from the extraction unit 13 in accordance with a ratio designated by the user. The output unit 16 adjusts the levels of each environmental sound or human voice extracted by the extraction unit 13 at a predetermined ratio, adds them together, and outputs them. Note that whether or not the output unit 16 performs the above-mentioned addition is based on an instruction from the model management unit 20 that has received an instruction from the user. The output unit 16 may also be in a mode in which it outputs the sounds without adjusting their levels or adding each sound together.

[0037] <Model Management Department 20> The model management unit 20 manages the basic models Mo stored in the basic model database 26 of the model storage unit 25, the user original basic models Mo' stored in the user original basic model database 27, and the first-stage extraction models M stored in the first-stage extraction model database 28. That is, the model management unit 20 manages the pre-installed basic models Mo, the user original basic models Mo' created by the learning system 30, and the first-stage extraction models M.

[0038] When the extraction unit 13 extracts a voice, the model management unit 20 selects an appropriate model from the databases (basic model database 26, user original basic model database 27, first stage extraction model database 28) and passes it to the extraction unit 13. Specifically, when new voice is input, the model management unit 20 outputs the first stage extraction model M, which is a model for separating the new voice and environmental sounds, to the extraction unit 13 of the extraction system 10. Furthermore, when the model management unit 20 extracts new voice first and the voice after extraction can be handled by the basic model Mo, it outputs the basic model Mo and the user original basic model Mo' from the model storage unit 25 in response to an instruction from the user.

[0039] <Model storage unit 25> The model storage unit 25 includes a basic model database 26, a user original basic model database 27, and a first-stage extraction model database . The models stored in the basic model database 26, the user original basic model database 27, and the first stage extraction model database 28 are all extraction models.

[0040] The basic model database 26 is a pre-installed database that stores basic models Mo of existing environmental sounds. The model management unit 20 passes the basic model Mo stored in the basic model database 26 of the model storage unit 25 to the extraction unit 13, and the extraction unit 13 uses the basic model Mo to extract a specific sound in a second stage from the environmental sounds extracted in the first stage. For example, the extraction unit 13 extracts "wind" from the environmental sounds of a forest (wind@forest environmental sounds) (hereinafter, △@□ represents the extraction of △ from □) based on the basic model Mo stored in the basic model database 26. Also, it extracts "insects" from the environmental sounds of a forest (insects@forest environmental sounds).

[0041] The user original basic model database 27 is a database that stores user original basic models Mo' of voices including human voices (for example, the voice of an announcer). When there are multiple new voices, the user original basic models Mo' are models for further extracting new voices using the user original basic models Mo' after extraction using the first-stage extraction model M, and separating the new voices by type.

[0042] The model management unit 20 extracts a specific person's voice (here, an announcer) from the people's voices in the user original basic model database 27. For example, the model management unit 20 extracts the voice of "Mr. B" from the announcers (Mr. B@announcer) and the voice of "Mr. C" from the announcers (Mr. C@announcer) based on the user original basic model database 27. Note that while Mr. A (described below) is an existing announcer (already trained), Mr. B and Mr. C are new announcers (new and untrained).

[0043] The user original basic model database 27 can also store models of voices (predetermined voices) other than human voices. For example, if D and a cat are in a car, the basic model Mo assuming the inside of a car does not correspond to the cat's meowing. In this case, the model (user original basic model) Mo' becomes "cat@D+cat" or "D@D+cat", and is stored in the user original basic model database 27.

[0044] The first-stage extraction model database 28 is a database that stores the first-stage extraction model M. The first-stage extraction model database 28 is a database that stores the first-stage extraction model M that performs extraction processing so that it can be handled by the first stage in this system that performs extraction in two stages, that is, the basic model Mo and the user's original basic model Mo'.

[0045] The extraction model database 28 stores, for example, “Mr. A@Mr. A+forest environmental sounds”, “Announcer@Announcer+forest environmental sounds”, etc. as examples of the first-stage extraction model M. When a new voice is input, the model management unit 20 reads out the first-stage extraction model M from the first-stage extraction model database 28 and outputs it to the extraction unit 13 of the extraction system 10.

[0046] The actual processing steps will now be described with reference to FIG. Let us take as an example the case where the voices of B and C (B and C are new announcers) were recorded in a forest. As a premise, the extraction unit 13 shown in Fig. 1 extracts "wind" from forest environmental sounds (wind@forest environmental sounds) and extracts "insects" from forest environmental sounds (insects@forest environmental sounds) based on the basic model database 26. The extraction unit 13 also extracts "Mr. B" from announcers (Mr. B@announcer) and "Mr. C" from announcers (Mr. C@announcer) based on the user original basic model database 27. The first stage extraction model database 28 also stores, for example, "Mr. A@Mr. A+forest environmental sounds", "announcer@announcer+forest environmental sounds", etc.

[0047] 1. Extract new audio (1) The extraction unit 13 extracts new speech (Mr. B + Mr. C = announcer) that has been newly trained using a model corresponding to the recording environment (the model of “announcer @ announcer + forest environmental sounds”) from the input speech. (2) The extraction unit 13 extracts audio consisting of only environmental sounds (input audio-new audio=forest environmental sounds).

[0048] 2. Extract the audio from 1.(1)(2) above by type. (3) The audio is compatible (only forest ambient sounds are included). (4) Extract environmental sounds in the same recording environment in order. For example, extract "wind@forest environmental sounds" and "birds@forest environmental sounds" using all models of "~@forest environmental sounds". (5) Similarly, extract the voice of a new announcer. For example, extract "only Mr. B" and "only Mr. C" from "Announcer".

[0049] 3. The extraction unit 13 outputs the extracted audio to the next block (the output unit 16). (6) Based on the user's specification, the signal is selected to pass through the MIX unit 14 or not, and is output to the output unit 16 (the signal is directly connected to the inverse Fourier transform unit 15, but is not shown in the figure because it reaches the output unit 16).

[0050] <Learning 30> The learning system 30 learns the speech that the user wants to be able to newly extract, and sends the generated model to the model management unit 20.

[0051] The learning system 30 includes an input unit 31, a data set creation unit 32A (data set creation means), a model generation unit 32B (model generation means), and a voice database 33. The input unit 31 is a learning input unit, and receives, for example, a voice sample such as recorded voice data.

[0052] <Data Set Creation Unit 32A> The data set creation unit 32A inputs a plurality of types of environmental sounds and a predetermined sound, synthesizes these sounds to create a training data set consisting of task data and each sound that is a single sound before synthesis used to create the task data. Here, the predetermined sound includes a human voice, and the environmental sound may be a sound other than a human voice. Specifically, the data set creation unit 32A creates each training data set according to the following (1-1) to (1-3).

[0053] (1-1) The data set creation unit 32A creates a learning data set that uses as model answers the individual sounds of the single voices before synthesis used to create the task data A, and task data created by synthesizing various non-human voices (this task data is referred to as data A). The data set creation unit 32A stores the data A and the single voices before synthesis in the voice database 33. A basic model Mo is generated by learning with this training data set (a training data set that uses as model answers the individual sounds of the single voices before synthesis used to create the task data A).

[0054] (1-2) The data set creation unit 32A creates a learning data set that uses as model answers the individual sounds of the single voices before synthesis used to create the task data B, and data set created by synthesizing a plurality of new voices (this task data is referred to as data B). The data set creation unit 32A stores data B and the single voices before synthesis in the voice database 33. A user original basic model Mo' is generated by learning with this learning data set (a learning data set that uses as model answers the individual sounds of the single voices before synthesis used to create the task data B).

[0055] When there is one new voice, the data B is a single new voice, and the user original basic model Mo' is not created.

[0056] (1-3) The data set creation unit 32A treats data A and data B as a single sound, and synthesizes them to create new data to be used as a task (this task data is referred to as data C). The data set creation unit 32A creates a training data set that uses data C and each sound (data A and data B) that is a single sound before synthesis as a model answer. A first-stage extraction model M is generated by learning with this training data set.

[0057] <Model Generation Unit 32B> The model generation unit 32B performs learning by deep learning based on the learning dataset created by the dataset creation unit 32A, and generates a basic model Mo for extracting environmental sounds by type, a user original basic model Mo' for extracting a predetermined sound not supported by the basic model Mo, and a first-stage extraction model M for extracting a new sound. Specifically, the model generation unit 32B creates each model according to the following (2-1) to (2-3).

[0058] (2-1) The model generation unit 32B learns from the training dataset created by the dataset creation unit 32A in (1-1) above (a training dataset in which each sound, which is a single pre-synthesized voice used to create the task data A, is used as a model answer) and generates a basic model Mo.

[0059] (2-2) The model generation unit 32B learns from the training dataset created by the dataset creation unit 32A in (1-2) above (a training dataset in which each sound, which is a single pre-synthesized voice used to create the task data B, is used as a model answer) and generates a user original basic model Mo'.

[0060] (2-3) The model generation unit 32B learns from the training dataset created by the dataset creation unit 32A in (1-3) above (training dataset in which data C and each sound (data A and data B) that is a single voice before synthesis are used as model answers) and generates a first-stage extraction model M.

[0061] After learning, the model generation unit 32B sends the basic model Mo, the user's original basic model Mo′, and the first-stage extraction model M to the model management unit 20.

[0062] <Audio Database 33> The audio database 33 accumulates data that is a subject of the learning data set. The audio database 33 accumulates, for example, environmental sounds of a forest, environmental sounds of the sea, environmental sounds of a town (urban area), etc. Here, the environmental sounds of a forest are the sounds of trees, the voices of birds that are likely to be found in a forest, the sounds of a river, and the voices of insects (such as cicadas, crickets, and bell crickets). Moreover, the environmental sounds of the sea are the sounds of waves, the voices of birds that are likely to be found in the sea, and the sounds of a ship.

[0063] [Structures of the base model Mo, the user's original base model Mo', and the first-stage extraction model M] The basic model Mo, the user's original basic model Mo', and the first-stage extraction model M are sound separation models that model parameters consisting of the connection matrix that connects the layers of the neural network and bias terms. The model generation unit 32B of the learning system 30 (see FIG. 1) learns a deep neural network (DNN) as the basic model Mo, the user's original basic model Mo′, and the model M for first-stage extraction.

[0064] FIG. 2 is a diagram showing an example of the structure of a basic model Mo, a user original basic model Mo′, and a first-stage extraction model M, which are configured using a deep neural network (DNN). As shown in FIG. 2, a deep neural network (DNN) model (model parameters) is composed of an input layer 41, an arbitrary number of hidden layers 42 which are intermediate layers, and an output layer 43. The input layer 41 has a plurality of input nodes (neurons) (here, eight). The hidden layer 42 has a plurality of hidden layers (here, three layers: hidden layer1, hidden layer2, hidden layer3). In reality, the number of layers n of the hidden layer 42 reaches, for example, 20 to 100. The output layer 43 has output nodes (neurons) equal to the number of objects to be classified (here, four). Note that the number of layers and the number of nodes (neurons) shown in FIG. 2 are merely examples. In the first-stage extraction model M, all nodes in the input layer 41 and hidden layer 42 are connected, and all nodes in the hidden layer 42 and output layer 43 are connected.

[0065] The model generation unit 32B (model learning means) of the learning system 30 learns the first-stage extraction model M by a Feed Forward Neural Network (FFNN) composed of an input layer 41, a hidden layer 42, and an output layer 43.

[0066] The input layer 41, hidden layer 42, and output layer 43 each have an arbitrary number of nodes (see circles in Figure 2). These nodes are functions that receive input and output values. The input layer 41 has a bias node that inputs an independent value separate from the input node. The configuration of a deep neural network is constructed by stacking layers with multiple nodes. In a deep neural network, propagation is performed by applying a weight to the received input, converting the received input with an activation function, and outputting it to the next layer. The activation function can be a nonlinear function such as a sigmoid function or a tanh function, or a ReLU (Rectified Linear Unit function). By increasing the number of nodes, the number of variables to be handled can be increased, and values / boundaries can be determined by taking into account a large number of factors. By increasing the number of layers, combinations of straight line boundaries and complex boundaries can be expressed.

[0067] Learning in a deep neural network involves calculating an error and adjusting the weights of each layer based on that error. Learning is solving an optimization problem that minimizes the error, and for example, backpropagation is used for learning the first-stage extraction model M. Note that a mean square error function is used as the loss error function that calculates the error. The backpropagation propagates the error from the output layer 43 and adjusts the weights of each layer.

[0068] In the pre-learning, the model generating unit 32B may perform the learning a predetermined number of times, or may end the learning when the parameter error converges to within a predetermined error. The model generating unit 32B writes and stores the learned first-stage extraction model M and basic model Mo (model parameters) in the model storage means of the model managing unit 20. It is assumed that the trained model will be evaluated against the input data, so the weights are already obtained as the trained results.

[0069] [Configuration of the extraction system of the voice extraction device] Fig. 3 is a diagram for explaining an outline of an extraction system 10 of a voice extraction device 1 according to an embodiment of the present invention. The same components as those in Fig. 1 are given the same reference numerals. Note that Fig. 3 omits the fact that extraction is performed in two stages. As shown in FIG. 3, the extraction system 10 of the audio extraction device 1 includes an extraction section 51, a level adjustment section 52, and an adder . The input unit 11 inputs an input audio wav file to the extraction unit 51 as audio data. The extraction unit 51 corresponds to the extraction unit 13 in Fig. 1, and the level adjustment unit 52 and the adder 54 correspond to the MIX unit 14 in Fig. 1. Since Fig. 3 is a configuration diagram showing an overview of the voice extraction device 1, the Fourier transform unit 12 and the inverse Fourier transform unit 15 in Fig. 1 are omitted. In addition, the input unit 11 and the output unit 16 are illustrated outside the extraction system 10.

[0070] The extraction unit 51 extracts human voices, water sounds, car sounds, and wind sounds (see the icons of people, water drops, cars, and wind in FIG. 3; the same applies below) from the input voice data. Here, for the framework for the neural network in the extraction unit 51, for example, float32 precision CNN of Chainer (registered trademark), which is framework software for deep neural networks, is used.

[0071] The level adjustment unit 52 adjusts the level of each of the extracted sounds (human voice, water sound, car sound, wind sound) based on a ratio 53 designated by the user. The adder 54 adds the audio data whose levels have been adjusted for each sound (human voice, water sound, car sound, wind sound) and outputs the added audio data to the output unit 16. As will be described later, the adder 54 may output the audio data separately without adding them. The output unit 16 outputs the added output through, for example, a speaker.

[0072] [Hardware configuration] The audio extraction method according to this embodiment is realized by a computer 900, which is a physical device having a configuration as shown in FIG. 4 is a hardware configuration diagram showing an example of a computer that realizes the processing of the voice extraction method according to the embodiment of the present invention. The computer 900 has a CPU (Central Processing Unit) 901, a GPU (Graphics Processing Unit) 902, a ROM (Read Only Memory) 903, a RAM (Random Access Memory) 904, a HDD (Hard Disk Drive) 905, an input / output I / F (Interface) 906, a communication I / F 907, and a media I / F 908.

[0073] The CPU 901 and the GPU 902 execute various processes of the voice extraction device 1, which will be described later. The CPU 901 operates based on a program stored in the ROM 903 or the HDD 905, and controls each processing unit of the voice extraction device 1 shown in Fig. 1. The ROM 903 stores a boot program executed by the CPU 901 when the computer 900 is started up, programs related to the hardware of the computer 900, and the like.

[0074] The CPU 901 controls an input device 910 such as a mouse or a keyboard, and an output device 911 such as a display, via an input / output I / F 906. The CPU 901 acquires data from the input device 910 via the input / output I / F 906, and outputs generated data to the output device 911.

[0075] The GPU 902 executes learning processes such as matrix operations in machine learning. The HDD 905 stores programs executed by the CPU 901 and data used by the programs, etc. The communication I / F 907 receives data from other devices via a communication network (e.g., NW (Network) 920) and outputs the data to the CPU 901, and also transmits data generated by the CPU 901 to other devices via the communication network.

[0076] The media I / F 908 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 904. The CPU 901 loads a program related to a target process from the recording medium 912 onto the RAM 904 via the media I / F 908, and executes the loaded program. The recording medium 912 is an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto Optical disk), a magnetic recording medium, a conductive memory tape medium, a semiconductor memory, or the like.

[0077] For example, when the computer 900 functions as the voice extraction device 1 configured as one device according to this embodiment, the CPU 901 of the computer 900 executes a program loaded onto the RAM 904 to realize the functions of the voice extraction device 1. Furthermore, the HDD 905 stores data in the RAM 904. The CPU 901 reads and executes a program relating to a target process from the recording medium 912. Additionally, the CPU 901 may read a program relating to a target process from another device via a communication network (NW 920).

[0078] The configuration of the voice extraction device 1 according to the embodiment of the present invention has been described above. The voice extraction device 1 can be operated by a program (multi-track program) that causes a computer to function as each of the above-mentioned means.

[0079] [Audio extraction device in action] The operation of the voice extraction device 1 configured as above will now be described. (Background Information) This assumes that you are reporting on something related to cars. During recording, especially when recording outdoors, unwanted sounds can get mixed in. For example, a strong wind may blow during an interview. What is needed for this interview is the sound of people's voices and cars, not the sound of the wind. If it were possible to extract only the necessary sounds, it would be possible to produce better broadcast audio.

[0080] Since the voice extraction device 1 of this embodiment uses Deep Learning, first, a learning dataset (see FIG. 5) is created.

[0081] <Creating a learning dataset> First, the creation of the learning dataset will be described. FIG. 5 is a diagram for explaining the creation of a learning dataset. The learning data set 90 is a large number of pairs of "tasks" and "model answers" for neural network learning. As shown in Fig. 5, the single sounds 60 are, for example, a human voice 60a, a water sound 60b, a car sound 60c, a wind sound 60d, etc., and are recorded in advance. The task 70 is created by synthesizing the single sounds 60 such as the human voice 60a, the water sound 60b, the car sound 60c, and the wind sound 60d.

[0082] A learning dataset 90 is created by pairing a task 70 with a single voice of the voice to be extracted as a model answer 80. For example, a learning dataset 90a for extracting a human voice becomes a learning dataset for extracting a human voice by pairing the task 70 shown in Fig. 5 with the model answer 80a for the human voice. Similarly, a learning dataset 90b for extracting the sound of water becomes a learning dataset for extracting the sound of water by pairing the task 70 shown in Fig. 5 with the model answer 80b for the sound of water. This training dataset 90 is used to reproduce audio recorded in a noisy location.

[0083] <Learning from training datasets using deep learning> Next, a method for training the created training data set 90 using deep learning will be described. FIG. 6 is a diagram for explaining a method for learning the created learning data set 90 by using deep learning. First, task 70-1, which is a Fourier transform of task 70, is processed by a neural network (see FIG. 2).

[0084] The neural network processing result is compared with a model answer 80a-1 obtained by Fourier transforming a model answer 80 (here, a model answer 80a of a human voice). As shown by symbol a in FIG. 6, the comparison results are fed back to the neural network to modify the neural network parameters (the connection matrix connecting the layers of the network and the bias terms). The neural network is optimized by repeating this process as many times as the number of pairs in the training data set 90 (see symbol b in Fig. 6). In Fig. 6, in order to reproduce human voices recorded in a noisy place, human voices, sounds of water, cars, and wind are trained using deep learning based on the training data set 90a to generate each model (basic model Mo, user original basic model Mo', and first-stage extraction model M). Each model uses the same learning method, with the only difference being the contents of the training data set.

[0085] <Method for extracting individual environmental sounds or human voices> Next, a method for extracting individual environmental sounds or human voices from input audio data will be described. FIG. 7 is a diagram for explaining a method of extracting a human voice from input voice data using each model (basic model Mo, user original basic model Mo', and first-stage extraction model M). As in the case of learning in Fig. 6, each model uses the main extraction flow. The model management unit 20 (see Fig. 1) can repeat the main extraction flow several times while switching between models, for example, to perform first-stage extraction or to extract individual sounds separately. First, an input voice 71-1 obtained by Fourier transforming an input voice 71 is processed by a neural network (see FIG. 2). Here, as shown by symbol c in Fig. 7, the neural network has been trained and optimized by the learning shown in Fig. 6 (for example, an optimized first-stage extraction model M has been generated. The same is true for all models, not just for the first-stage extraction). Then, the output of the neural network (here, the extracted output 81a-1 of the human voice) is converted back into voice data 81a by inverse Fourier transform. By carrying out the above processing, the target voice can be extracted.

[0086] <Supplementary explanation about the training dataset> Fig. 8 is a diagram for providing a supplementary explanation of the training data set 90. The same components as those in Fig. 5 are given the same reference numerals. Fig. 8(a) shows the training data set 90a combined with the task 70 and the model answer 80a for human voices, Fig. 8(b) shows the training data set 90b combined with the task 70 and the model answer 80b for the sound of rain, and Fig. 8(c) shows the training data set 90c combined with the task 70 and the model answer 80c for the sound of cars. The task 70 shown in FIG. 8 is created by synthesizing single sounds 60, including a human voice 60a, a water sound 60b, and a car sound 60c.

[0087] The training data set 90a shown in Fig. 8(a) is a training data set for extracting a human voice by pairing the task 70 with a model answer 80a of a human voice. The training data set 90b shown in Fig. 8(b) is a training data set for extracting a water sound by pairing the task 70 with a model answer 80b of a water sound. The training data set 90c shown in Fig. 8(c) is a training data set for extracting a car sound by pairing the task 70 with a model answer 80c of a car sound.

[0088] In this way, the number of training data sets 90 is equal to the number of sounds to be matched. In this example, three training data sets 90a to 90c are created to match three sounds: a human voice, the sound of water, and the sound of a car. The model answer speech data is speech data in which the sounds to be matched are recorded individually, and these are referred to as single sounds. In Fig. 8, three single sounds, a human voice, a water sound, and a car sound, are prepared as model answers 80a to 80c for the learning data sets 90a to 90c, respectively.

[0089] The speech data used as the tasks are all synthesized single sounds (human voice, water sound, car sound) prepared as model answers 80a to 80c. This synthetic speech is used as a task for all learning data sets 90a to 90c. In Figure 8, task 70 is used three times. As described above, the three training data sets 90a to 90c are used for training to generate three first-stage extraction models M. At this time, the voice data of the training data set 90 is subjected to a short-time Fourier transform before being used for training (see FIG. 6).

[0090] <Additional information on extraction method> Fig. 9 is a diagram for explaining the sound extraction method in addition. The same components as those in Fig. 5 and Fig. 8 are given the same reference numerals. The actual extraction method is to apply a short-time Fourier transform to the input speech, then use each model (first-stage extraction model M, basic model Mo, user original basic model Mo') to extract each speech, and then apply an inverse short-time Fourier transform to return it to speech data. By performing the above extraction process on one input voice using all the models (first stage extraction model M, base model Mo, user original base model Mo'), the voice can be classified into types. In Figure 9, the model for the model answer was extracted three times using three models: a human voice model, a water sound model, and a car sound model.

[0091] (Traditional training datasets and training methods) Generally, when the number of supported voices is increased, all learning must be redone. Specifically, as shown in Figures 8 and 9, if three types of voices are supported, learning with Deep Learning must be performed four times to support four types of voices. When increasing the number of voices from 100 to 101, learning with Deep Learning must be performed 101 times.

[0092] (Learning Data Set, Model Generation, and Extraction Method of Voice Extraction Device 1) <Learning Dataset and Model Generation (Part 1)> The data set creation unit 32A (see FIG. 1) of the speech extraction device 1 creates a new training data set by combining the new speech with the target data in the training data set. The first-stage extraction model M is created by training with this training data set. The target data in the training data set is stored in a speech database 33 (see FIG. 1).

[0093] Fig. 10 is a diagram for explaining an extraction method (part 1) of the voice extraction device 1 (see Fig. 1). The same components as those in Figs. 8 and 9 are given the same reference numerals. 10 is data to be used as a task in a training data set 90 of only environmental sounds, which is trained by synthesizing single sounds other than human voices, here the sounds of water, cars, and wind, and a model generated by further synthesizing a user's voice 80a and training with the training data set 90 consisting of this newly synthesized voice and the newly added human voice is set as a first-stage extraction model M. Using the first-stage extraction model M, a human voice 81a is first extracted from the input voice by neural network calculations (see symbol d in FIG. 10).

[0094] Then, based on the basic model Mo of the learning data set 90 containing only environmental sounds, each environmental sound, here the sound of water 81b, the sound of a car 81c, and the sound of wind 81d, are extracted (see symbol e in FIG. 10). In this way, the system learns all kinds of sounds other than human voices. By then learning the user's voice, it becomes possible to perform highly accurate extraction specific to the user.

[0095] <Learning Dataset and Model Generation (Part 2)> The data set creation unit 32A (see FIG. 1) of the voice extraction device 1 inputs a plurality of types of environmental sounds and a predetermined voice, synthesizes these sounds to create a learning data set consisting of the data to be challenged and each sound that is a single voice before synthesis and used to create the target data. The model generation unit 32B (see FIG. 1) learns by deep learning based on the learning data set created by the data set creation unit 32A, and generates a basic model Mo for extracting environmental sounds by type, a user original basic model Mo' for extracting a predetermined voice not supported by the basic model Mo, and a first-stage extraction model M for extracting a new voice.

[0096] 11 is a diagram for explaining an extraction method (part 2) of the voice extraction device 1. The same components as those in FIG. 10 are given the same reference numerals. As shown in FIG. 11, the user voices 180 are multiple people (for example, three people), and each of the voices is a user voice 180a, a user voice 180b, and a user voice 180c. The newly added multiple user voices 180 (user voice 180a, user voice 180b, and user voice 180c) are combined with the subject data of the learning dataset 90 of only environmental sounds to create a new first-stage extraction model M. Using the first-stage extraction model M, first, the voices 181 (181a to 181c) of three users are collectively extracted from the input voice by neural network calculations (see symbol f in FIG. 11).

[0097] After that, a part is prepared in which the voice of each of the three users is extracted from the voices of the three users as a new learning data set (see the overlapping part of symbols f and h in FIG. 11). Here, the data obtained by synthesizing the voices of users a, b, and c is the challenge data, and the model answer is the synthesis of a single voice of only user a. Then, a user original basic model Mo' is generated by training with this learning data set. Similarly, learning data sets are created for users b and c, and trained to generate a total of three user original basic models Mo'. Using these user original basic models Mo', individual user voices 181a to 181c are extracted from the input voice by neural network calculations (see symbol h in FIG. 11).

[0098] Further, based on the basic model Mo of the learning data set 90 containing only environmental sounds, each environmental sound, here the sound of water 81b, the sound of a car 81c, and the sound of wind 81d, are extracted (see symbol g in FIG. 10). In this way, the voices of the three users are first extracted together. Then, the voices of each user are extracted from the voices of the three users. This makes it possible to separate individual voices from environmental sounds such as the sound of water.

[0099] (Voice Extraction Operation in Extraction Unit 51 of Voice Extraction Device 1) With reference to FIG. 3, the actual processing steps in the extraction method of FIG. 7 will be described. As shown in FIG. 3, the input unit 11 inputs an input audio wav file to the extraction unit 51 as audio data. The extraction unit 51 extracts each sound from the input sound data using the first stage extraction model M, which is an AI trained by deep learning, the basic model Mo, and the user original basic model Mo'. For example, human voice is extracted using the deep learning shown in Fig. 7. In addition, the extraction unit 51 extracts individual environmental sounds such as the sound of water, the sound of cars, and the sound of wind.

[0100] A level adjustment unit 52 adjusts the level of each of the extracted sounds (human voice, water sound, car sound, wind sound) based on a ratio 53 specified by the user. The adder 54 adds the level-adjusted sounds (human voice, water sound, car sound, wind sound) and outputs the result as a wav file.

[0101] Here, the reason for mixing by the level adjustment unit 52 and the adder 54 (MIX unit 14 in FIG. 1) will be described. It has been confirmed that the voice extraction device 1 can process one minute of voice in 10 seconds or less. The fact that processing can be completed in under 10 seconds and that mixing is possible are independent from each other when viewed as a system. Because both of these processes can be completed in a short time, better broadcast audio can be produced even in a limited amount of time.

[0102] In this way, it is possible to mix the extracted sounds by extracting only the necessary sounds and suppressing other sounds. This makes it possible to produce better broadcast sound, especially in limited time such as when responding to interviews. Furthermore, in cases where a long production time is available, such as when shooting a drama, the extracted audio can be output as is without being mixed.

[0103] (flowchart) 12 is a flowchart showing the operation of the voice extraction device 1 (see FIG. 1). This flow is executed during or immediately before step S14, step S16, and step S18 in FIG. In step S1, the dataset creation unit 32A (see Figure 1) inputs multiple types of environmental sounds from the audio database 33 (see Figure 1) and a specified audio from the input unit 31 (see Figure 1), and creates a learning dataset consisting of task data created by synthesizing these sounds and each sound, which is a single audio before synthesis and used to create the task data.

[0104] In step S2, the model generation unit 32B (see Figure 1) uses deep learning to learn from the created learning dataset, and generates a basic model Mo for extracting environmental sounds by type, a user original basic model Mo' for extracting specific sounds that are not supported by the basic model Mo, and a first-stage extraction model M for extracting new sounds.

[0105] In step S3, the model storage unit 25 (see FIG. 1) stores the generated basic model Mo, user original basic model Mo', and first-stage extraction model M in the basic model database 26, user original basic model database 27, and extraction model database 28, respectively.

[0106] In step S4, the model management unit 20 (see FIG. 1) selects an appropriate model when the extraction unit 13 (see FIG. 1) extracts a voice, and passes it to the extraction unit 13. Specifically, when a new voice is added, the model management unit 20 outputs the first-stage extraction model M to the extraction unit 13, and when the extracted voice can be handled by the basic model Mo and the user original basic model Mo' by extracting the new voice, the model management unit 20 outputs the basic model Mo and the user original basic model Mo' to the extraction unit 13.

[0107] In step S5, the extraction unit 13 extracts individual environmental sounds or predetermined sounds from the input sound by neural network calculations using the basic model Mo, the user original basic model Mo', and the first-stage extraction model M passed from the model management unit 20. Specifically, the extraction unit 13 extracts new sounds from the input sound in a first stage by neural network calculations using the first-stage extraction model M, and extracts individual environmental sounds and predetermined sounds from the sound extracted in the first stage by neural network calculations in a second stage using the basic model Mo and the user original basic model Mo'.

[0108] (Extraction operation flowchart) FIG. 13 is a flowchart showing the extraction operation of a specific example of the voice extraction method of the voice extraction device 1 (see FIG. 1). FIG. 13 is a detailed flow of the extraction operation of step S5 in FIG. 12. In the notation of sounds in FIG. 13, sounds surrounded by rectangles indicate that they are extracted. For example, "wind, insects, trees" indicates that wind, insects, and trees are environmental sounds that have not been separated, and "wind", "insects", and "trees" indicate that these have been extracted individually. Similarly, "Mr. A and Mr. B" indicates that human voices have not been separated, and "Mr. A" and "Mr. B" indicate that human voices have been extracted individually. Furthermore, the size of the rectangle surrounding the sound indicates the volume of the sound after extraction (the larger the rectangle, the louder the volume).

[0109] Let's imagine that Mr. A and Mr. B are filming in a forest. In FIG. 13, A and B are assumed to be existing announcers (already trained), and models of "A@A+B" and "B@A+B" are stored in a user original basic model database 27. In addition, sounds of wind, insects, and trees have been trained as environmental sounds of the forest, and are stored in a basic model database 26. In addition, "A+B" @ "A+B+forest environmental sounds" is saved in a first stage extraction model database 28.

[0110] In step S11, the voices of A and B and the environmental sounds of the forest (sounds of wind, insects, and trees) are inputted from the input unit 11 (see FIG. 1) (see symbol i in FIG. 13). In step S12, the model management unit 20 (see FIG. 1) performs various settings. The various settings are as follows. (1) Select whether to extract by recording environment and type For example, select the location from forest / sea / city, etc. If you select forest environmental sounds, set whether to extract wind, insects, and trees separately.

[0111] (2) Select whether to extract individual announcers For example, if you have Mr. A and Mr. B, you can choose to separate them as individuals (Mr. A / Mr. B) or extract them together (Mr. A & Mr. B).

[0112] (3) Output format Select MIX (including ratio) or multi-track. In step S13, the Fourier transform unit 12 (see FIG. 1) performs a Fourier transform to convert the input voice into frequency components.

[0113] In step S14, the model management unit 20 (see FIG. 1) separates the input sound into the environmental sounds of the forest and the announcer's voice. Step S14 does not proceed to either one, but to both. That is, the sound input in step S11 is separated into two, the environmental sounds and the announcer's voice, in step S14, and then the announcer's voice and the environmental sounds proceed to their respective routes (steps S15 and S17). The extraction unit 13 uses the model passed from the model management unit 20 to separate the input sound into the environmental sounds of the forest and the announcer's voice through neural network calculations. Here, in the "various settings" of step S12, by selecting "Recording environment: Forest" and "Announcers: Mr. A & Mr. B", processing can be performed using the corresponding model.

[0114] If the environmental sounds of the forest are extracted (see symbol k in FIG. 13, “wind, insects, trees”), proceed to step S15, and if the voice of an announcer is extracted (see symbol j in FIG. 13, “Mr. A and Mr. B”), proceed to step S17. In step S15, it is determined whether or not to extract forest environmental sounds by type based on the settings made in step S12. If forest environmental sounds are to be extracted by type (S15: Yes), in step S16 the extraction unit 13 extracts each environmental sound (see "wind", "insects", and "trees" indicated by symbol l in FIG. 13) and proceeds to step S19. If forest environmental sounds are not to be extracted by type (S15: No), proceeds to step S19.

[0115] On the other hand, if the announcer's voice is extracted in step S14, then in step S17, it is determined whether or not to extract the announcer's voice for each individual person, based on the settings made in step S12.

[0116] If extraction is to be performed for each individual, the model newly trained by the user in step S18 is used to extract each announcer individually from the input voice by neural network calculation (see "Mr. A" and "Mr. B" in symbol m of FIG. 13), and the process proceeds to step S19. If extraction is not to be performed for each individual, the process proceeds to step S19.

[0117] In step S19, the extraction unit 13 determines the output format based on the settings made in step S12. Now, in the "various settings" in step S12, it is assumed that the following settings are set: environmental sound: extraction by type, and announcer voice: not extraction by individual. In the case of multi-track, environmental sounds are output in multiple tracks, classified as "wind," "insects," and "trees," and announcers are output as "Mr. A and Mr. B" (see symbol n in FIG. 13). In the case of MIX, in step S20, the MIX unit 14 (see FIG. 1) mixes the types "wind" and "trees" (without "insects") with the announcers "Mr. A and Mr. B" and outputs the mixed results.

[0118] In the case of MIX, the volume of each sound can be specified by the user. Here, the voices of the announcers "Mr. A and Mr. B" are made the loudest. Also, among the environmental sounds "wind," "insects," and "trees," the "sound of trees" is made louder, the "sound of wind" is made quieter, and the "sound of insects" is deleted (see symbol o in Figure 13).

[0119] In step S21, the inverse Fourier transform unit 15 (see FIG. 1) performs an inverse Fourier transform to convert the data that has been converted into frequency components back into audio data, and the process of this flow ends.

[0120] (Specific examples) The recorded voice will be described with reference to FIG. Fig. 14 is a diagram for simply explaining the functions of the extraction process of the voice extraction device 1 according to the embodiment of the present invention. Fig. 14 does not depend on the type of voice, such as recorded voice or pseudo recorded voice. The same components as those in Fig. 3 are given the same reference numerals. An input voice (pseudo recorded voice) containing a mixture of various voices is input to the input unit 11 (see symbol p in FIG. 14). As a result of learning using the learning data set, each model (basic model Mo, user original basic model Mo', and first-stage extraction model M) is generated. The extraction unit 51 uses each model to learn through deep learning and extracts input voice by type (here, human voice, water sound, car sound, and wind sound).

[0121] A level adjustment unit 52 adjusts the level of each of the extracted sounds (human voice, water sound, car sound, wind sound) based on a ratio 53 designated by the user.

[0122] The adder 54 adds the level-adjusted sound data (human voice, water sound, car sound, wind sound) and outputs the result to the output unit 16. The output unit 16 outputs the added output as an output sound. This output voice is a voice obtained by mixing the extracted voices at an arbitrary ratio.

[0123] The level adjustment section 52, the ratio 53, the adder 54 and the output section 16 will be illustrated. For example, as indicated by the symbol s in Fig. 14, the level adjustment unit 52 adjusts the level of the sound to make the human voice 0 (an example of a ratio 53) and emphasize the wind sound. The adder 54 adds the respective sounds and outputs the result to the output unit 16. The output sound output from the output unit 16 is a sound obtained by mixing the extracted sounds at an arbitrary ratio, for example, a human voice 81a:0, the sound of rain 81b:2, the sound of a car 81c:1, and the sound of wind 81d:6.

[0124] Alternatively, as shown by the symbol t in Fig. 14, the level adjustment unit 52 can adjust the level so that the human voice is set to 10 (an example of a ratio 53) and the wind sound is muted. The adder 54 adds the respective sounds and outputs the result to the output unit 16. The output sound output from the output unit 16 is a sound in which the extracted sounds are mixed at an arbitrary ratio, for example, human voice 81a:10, rain sound 81b:0, car sound 81c:1, and wind sound 81d:0. This sound is adjusted so that the human voice is the main focus, with some car sound being heard in the background.

[0125] [effect] As described above, the voice extraction device 1 (see FIG. 1) of this embodiment includes a data set creation unit 32A (see FIG. 1) that creates a training data set including task data created by inputting a plurality of types of environmental sounds and a predetermined voice (e.g., a human voice) and synthesizing these sounds, and each sound that is a single voice before synthesis and used to create the task data; and a Deep Learning unit 32B (see FIG. 1) that performs Deep Learning based on the created training data set. The system includes a model generation unit 32B (see FIG. 1) that learns through learning and generates a basic model Mo for extracting environmental sounds by type, a user original basic model Mo' for extracting specified sounds not supported by the basic model Mo, and a first-stage extraction model M for extracting new sounds; a model management unit 20 that outputs the first-stage extraction model M to the extraction unit 13 when a new sound is added, and outputs the basic model Mo and the user original basic model Mo' to the extraction unit when the extracted sound can be supported by the basic model Mo and the user original basic model Mo' by extracting the new sound; and an extraction unit 13 (see FIG. 1) that uses the first-stage extraction model M to extract new sounds from the input sound by neural network calculation in a first stage, and uses the basic model Mo and the user original basic model Mo' to extract individual environmental sounds and specified sounds in a second stage from the sound extracted in the first stage by neural network calculation.

[0126] In conventional systems, when the number of supported voices is increased, learning has to be restarted from the beginning. For example, if the number of supported voices is currently n, the number of times learning is n+1. For example, if 100 types of voices are supported, and you want to increase the number of supported voices to 101, you have to restart all learning, which means you have to perform Deep Learning 101 times.

[0127] In contrast, in this embodiment, as described in steps S14 and S17 of the flow charts in Fig. 12 and 13, when new voice is input, the first-stage extraction model M is output to the extraction unit 13, and the new voice is extracted, so that the extracted voice can be handled by the basic model Mo and the user original basic model Mo'. That is, environmental sounds such as the sound of wind and the sound of trees that have already been learned can be extracted by using existing learning data (basic model Mo), while new voices (for example, a new announcer) can be extracted from the original voice by learning and setting the user original basic model Mo' as a learned data set for the voice of the newly added person.

[0128] With this two-stage extraction, even when extracting new audio, it is only necessary to do it once to separate the new audio from environmental sounds, without having to learn all the audio. Compared to the above conventional example, if the number of compatible audio is increased under the same conditions as the conventional example, the number of learning times will be n+1. If there is existing learning data (basic model Mo) that supports 100 types of audio, only learning to extract the newly added audio is required. In other words, only one learning time is required, and only one additional learning of the newly added person's audio is required.

[0129] To explain in more detail, the newly added voice is separated into two parts: the new voice and the 100 voices that were already supported. This separation requires one learning step.

[0130] The first stage of extraction is performed to separate the input audio (original audio / extraction target) into two: new audio and audio that can be handled by the basic model. The first stage of extraction is the process of extracting environmental sounds and new audio. These processes are performed simultaneously, so the extraction process is performed once.

[0131] In particular, even if the number of supported voice types is increased from 3 to 4, or from 100 to 101, only one learning session is required.

[0132] In the voice extraction device 1 of this embodiment, the learning method has the following features. A training dataset is created using the task data A, which is created by synthesizing various non-human voices, and each sound, which is a single pre-synthesized voice used to create the task data A, as a model answer. A basic model Mo is generated by training with this training dataset. For example, multiple basic models Mo are set in advance based on the expected recording, and trained in advance (for example, "in front of the station" or "sand dune").

[0133] · Create a training dataset with the model answers being the task data B created by synthesizing multiple new voices and the single unsynthesized voices used to create the task data B. Generate the user's original base model Mo' by learning with this training dataset.

[0134] · The above data A and data B are treated as single sounds, and these are synthesized to create new data C as the task. A training dataset is created with the above data C and each sound (data A and data B) that is the single sound before synthesis as model answers. A first-stage extraction model M is generated by training with this training dataset. As an example, a new training dataset is created by combining the voice of a newly added person (for example, the voice of a new announcer) with the synthetic voice that is the task data of the dataset used when generating the basic model Mo, and a new model, the first-stage extraction model M, is generated by training using this dataset.

[0135] In the voice extraction device 1 of this embodiment, the extraction unit 13 (see FIG. 1) determines whether or not to pass through the MIX unit 14 (see FIG. 1) based on user specification. The output unit 16 (see FIG. 1) outputs each environmental sound or human voice extracted by the extraction unit 13 via the MIX unit 14 and / or without passing through the MIX unit 14. If the sounds pass through the MIX unit 14, the sounds are added after level adjustment at a ratio specified by the user, and output in a state where the volume of each sound is mixed at an arbitrary level. If the sounds do not pass through the MIX unit 14, the sounds are output in multi-track.

[0136] [Variations] Although the configuration, operation, and evaluation of the voice extraction device 1 according to the embodiment of the present invention have been described above, the present invention is not limited to this embodiment. The voice extraction device 1 performs, in one device, the operations of a data set creation unit 32A and a model generation unit 32B that generate a basic model Mo and create a learning data set for it, generate a user original basic model Mo' and create a learning data set for it, and generate a one-stage extraction model M and create a learning data set for it, and the operation of a model management unit 20 that reads each generated model from a model storage unit 25 and passes it to the extraction unit 13. However, these operations may be performed by separate devices. The model generation unit 32B can be called a model learning means.

[0137] The above-mentioned embodiment examples have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all the configurations described. In addition, a part of the configuration of one embodiment example can be replaced with the configuration of another embodiment example, and the configuration of one embodiment example can be added to the configuration of another embodiment example. In addition, the embodiment examples can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0138] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0139] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized by software for a processor to interpret and execute a program that realizes each function. Information on the programs, tables, files, etc. that realize each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC (Integrated Circuit) card, SD (Secure Digital) card, or optical disk. In addition, in the above embodiment, the name "audio extraction device" is used, but this is for convenience of explanation, and the name may be "multi-tracking device," "multi-tracking system," etc. Furthermore, the method and program may be "multi-tracking method," "multi-tracking program," etc. [Explanation of symbols]

[0140] 1. Audio Extraction Device 10 Extraction system 11 Input section 12 Fourier transform section 13,51 Extraction part (extraction means) 14 MIX section (output means) 15 Inverse Fourier transform section 16 Output section (output means) 20 Model Management Unit (Model Management Means) 25 Model storage unit (model storage means) 26 Basic model database (model storage means) 27 User original basic model database (model storage means) 28 Model database for first stage extraction (model storage means) 30 Learning 31 Input section 32A Data set creation unit (data set creation means) 32B Model generation unit (model generation means) (model learning means) 33 Voice Database 41 Input layer 42 Any number of intermediate hidden layers 43 Output layer 52 Level adjustment section 54 Adder 70 Challenges 80 Model answer 90,90a~90c Training dataset Mo Basic Model Mo' User Original Basic Model M Model for first stage extraction

Claims

1. A data set creation means for creating a learning data set including task data created by inputting a plurality of types of environmental sounds and a predetermined sound, synthesizing these sounds, and each sound being a single sound before synthesis used to create the task data; A model generation means for generating a basic model for extracting the environmental sound by type, a user original basic model for extracting a predetermined sound not supported by the basic model, and a first stage extraction model for extracting a new sound by learning using deep learning based on the created learning dataset; a model storage means for storing the generated basic model, the user original basic model, and the first-stage extraction model; a model management means for managing the basic model, the user original basic model, and the first-stage extraction model stored in the model storage means; an extraction means for extracting an individual environmental sound or the predetermined sound from an input sound by a neural network operation using the basic model, the user original basic model, and the first stage extraction model transferred from the model management means, The extraction means includes: Using the first-stage extraction model, a new speech is extracted from the input speech by a neural network operation in a first stage; extracting individual environmental sounds and the predetermined sound in a second stage from the sound extracted in the first stage by a neural network operation using the basic model and the user original basic model; The data set creation means creates a new training data set by combining the new speech with the subject data in the training data set; The model generation means generates the first-stage extraction model by training the model on the new training data set created by the data set creation means.

1. A voice extraction device comprising:

2. The model management means outputs the first stage extraction model to the extraction means when an input voice is input, and outputs the basic model and the user original basic model to the extraction means when the extracted voice can be handled by the basic model and the user original basic model by extracting new voice.

2. The audio extraction device according to claim 1 .

3. The model management means outputs the basic model and the user original basic model to the extraction means in response to an instruction from a user.

3. The audio extraction device according to claim 2.

4. The predetermined sound includes a human voice, and the environmental sound is a sound other than a human voice.

2. The audio extraction device according to claim 1 .

5. an output unit that performs level adjustment at a predetermined ratio for each of the environmental sounds or the predetermined sounds extracted by the extraction unit, adds them together, and outputs them; 2. The audio extraction device according to claim 1 .

6. A program for causing a computer to function as the voice extraction device according to any one of claims 1 to 5.

Images