Learning model creation method, learning model creation device, background noise removing method, and background noise removing device

Abstract

To provide a background noise removing method that can achieve good background noise removal performance.SOLUTION: In a learning model creation method of the present invention, a first two-dimensional image is created using a target voice, a second two-dimensional image is created using a superimposed voice obtained by superimposing a removal target voice on the target voice, the first two-dimensional image is used as teacher data, the second two-dimensional image is input to a deep learning network, and learning is performed so that output data of the deep learning network matches the teacher data to create a learning model for removing the removal target voice. The second two-dimensional image is created using each of multiple superimposed voices created by changing SN ratio with voice pitch of the removal target voice superimposed.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] The present invention relates to a background sound removal method for removing background sound using a learning model. [Background technology]

[0002] In recent years, telecommuting, working from home, and distance learning have rapidly spread, allowing people to participate in meetings and classes without worrying about location or movement. However, when streaming video from home, the voices of people living in the same house, sounds of daily life, pets barking, and babies crying can get mixed in, making it difficult for participants in meetings and classes to hear the audio.

[0003] Noise removal methods have been proposed to deal with noise that is mixed into videos distributed online during meetings and classes. For example, Non-Patent Document 1 proposes a method of converting audio data into a spectrogram image by short-time Fourier transform and removing noise such as train running sounds using U-Net. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Hayashi et al.: "Study on noise removal and speech recognition performance improvement using U-Net-Focusing on background noise of train running", IEICE Techniques SeMI2022-26, Vol. 122, No. 108, pp. 34-39 (2022) Summary of the Invention [Problem to be solved by the invention]

[0005] When a user distributes a video, sounds that are mixed in vary depending on the environment in which the user distributes the video, such as the barking of a pet or the crying of a baby, and it is expected that various sounds will be mixed in with the voice of the speaker distributing the video. In a background sound removal method that targets general noise, such as a conventional background sound removal method, good removal performance may not be obtained when various sounds that differ from user to user are targeted for removal.

[0006] The present invention has been made to solve the above problems, and has an object to provide a background sound removal method that is capable of obtaining good background sound removal performance when removing various background sounds that a user wishes to remove. [Means for solving the problem]

[0007] In order to solve the above problems, the learning model creation method of the present invention is a learning model creation method executed in a learning model creation device having a learning data creation unit and a learning model creation unit, wherein the learning data creation unit creates a first two-dimensional image using first voice data, creates third voice data by superimposing at least one second voice data on the first voice data, and creates a second two-dimensional image using the third voice data, the learning model creation unit uses the first two-dimensional image as teacher data and the second two-dimensional image as input data, and trains the deep learning network so that output data when the input data is input to the deep learning network matches the teacher data, thereby creating a learning model for removing the second voice data, and the learning data creation unit creates multiple pieces of the third voice data by changing the voice pitch of the second voice data and the SN ratio at the time of the superposition, and creates the second two-dimensional image using each of the multiple pieces of the third voice data.

[0008] In order to solve the above problem, the learning model creation device of the present invention is a learning model creation device including a learning data creation unit and a learning model creation unit, wherein the learning data creation unit creates a first two-dimensional image using first voice data, creates third voice data by superimposing at least one second voice data on the first voice data, and creates a second two-dimensional image using the third voice data, the learning model creation unit uses the first two-dimensional image as teacher data and the second two-dimensional image as input data, and trains the deep learning network so that output data when the input data is input to the deep learning network matches the teacher data, thereby creating a learning model for removing the second voice data, and the learning data creation unit creates multiple pieces of the third voice data by changing the voice pitch of the second voice data and the SN ratio at the time of the superposition, and creates the second two-dimensional image using each of the multiple pieces of the third voice data.

[0009] In order to solve the above problems, the learning model creation method of the present invention is a learning model creation method executed in a learning model creation device having a learning data creation unit and a learning model creation unit, wherein the learning data creation unit creates a first two-dimensional image using first voice data, creates third voice data by superimposing at least one second voice data on the first voice data, and creates a second two-dimensional image using the third voice data, the learning model creation unit uses the first two-dimensional image as teacher data and the second two-dimensional image as input data, and trains the deep learning network so that output data when the input data is input to the deep learning network matches the teacher data, thereby creating a learning model for removing the second voice data, and the learning data creation unit creates the first two-dimensional image by cutting out an image from the two-dimensional image created using the first voice data so that some of the images overlap, and creates the second two-dimensional image by cutting out an image from the two-dimensional image created using the third voice data so that some of the images overlap.

[0010] In order to solve the above problem, the learning model creation device of the present invention is a learning model creation device including a learning data creation unit and a learning model creation unit, wherein the learning data creation unit creates a first two-dimensional image using first voice data, creates third voice data by superimposing at least one second voice data on the first voice data, and creates a second two-dimensional image using the third voice data, the learning model creation unit uses the first two-dimensional image as teacher data and the second two-dimensional image as input data, and trains the deep learning network so that output data when the input data is input to the deep learning network matches the teacher data, thereby creating a learning model for removing the second voice data, and the learning data creation unit creates the first two-dimensional image by cutting out an image from the two-dimensional image created using the first voice data so that some of the images overlap, and creates the second two-dimensional image by cutting out an image from the two-dimensional image created using the third voice data so that some of the images overlap.

[0011] In order to solve the above problem, the learning model creation method of the present invention is a learning model creation method executed in a learning model creation device having a learning data creation unit and a learning model creation unit, in which the learning data creation unit creates a first two-dimensional image using first voice data, creates third voice data by superimposing at least one second voice data on the first voice data, and creates a second two-dimensional image using the third voice data, the learning model creation unit uses the first two-dimensional image as teacher data and the second two-dimensional image as input data, and creates a learning model for removing the second voice data by training the deep learning network so that output data when the input data is input to the deep learning network matches the teacher data, and the learning data creation unit creates the first two-dimensional image by adding a two-dimensional image having a resolution different from the predetermined resolution to the two-dimensional image having a predetermined resolution created using the first voice data, and creates the second two-dimensional image by adding a two-dimensional image having a resolution different from the predetermined resolution to the two-dimensional image having a predetermined resolution created using the third voice data.

[0012] In order to solve the above problem, the learning model creation device of the present invention is a learning model creation device including a learning data creation unit and a learning model creation unit, wherein the learning data creation unit creates a first two-dimensional image using first voice data, creates third voice data by superimposing at least one second voice data on the first voice data, and creates a second two-dimensional image using the third voice data, the learning model creation unit uses the first two-dimensional image as teacher data and the second two-dimensional image as input data, and trains the deep learning network so that output data when the input data is input to the deep learning network matches the teacher data, thereby creating a learning model for removing the second voice data, and the learning data creation unit creates the first two-dimensional image by adding a two-dimensional image having a resolution different from the predetermined resolution to the two-dimensional image having a predetermined resolution created using the first voice data, and creates the second two-dimensional image by adding a two-dimensional image having a resolution different from the predetermined resolution to the two-dimensional image having a predetermined resolution created using the third voice data.

[0013] In order to solve the above problem, the voice removal method of the present invention is a voice removal method executed in a voice removal device that removes a voice to be removed from a superimposed voice in which a voice to be removed is superimposed on a voice of interest, using a learning model trained using input data and teacher data, wherein the teacher data is a first two-dimensional image created using first voice data corresponding to the voice of interest, the input data is a second two-dimensional image created using the third voice data by superimposing at least one second voice data corresponding to the voice to be removed on the first voice data, and the learning model is created by training the deep learning network so that output data when the input data is input to the deep learning network matches the teacher data, and the third voice data is created by superimposing each of the multiple second voice data created by changing the signal-to-noise ratio when superimposing it with the voice pitch of the second voice data on the first voice data.

[0014] In order to solve the above problem, the voice removal device of the present invention is a voice removal device that removes a voice to be removed from a superimposed voice in which a voice to be removed is superimposed on a voice of interest, using a learning model trained using input data and teacher data, wherein the teacher data is a first two-dimensional image created using first voice data corresponding to the voice of interest, the input data is a second two-dimensional image created using the third voice data by superimposing at least one second voice data corresponding to the voice to be removed on the first voice data, and the learning model is created by training the deep learning network so that output data when the input data is input to the deep learning network matches the teacher data, and the third voice data is created by superimposing each of the multiple second voice data created by changing the signal-to-noise ratio when superimposing it with the voice pitch of the second voice data on the first voice data. Effect of the Invention

[0015] According to the present invention, it is possible to provide a background sound removal method capable of obtaining good background sound removal performance when removing various background sounds that a user wishes to remove. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 shows an example of the configuration of a system for removing background sound from audio with background sound superimposed thereon. [Diagram 2] FIG. 2 is a diagram for explaining a superimposed sound on which a background sound is superimposed. [Diagram 3] FIG. 3 is a diagram for explaining a spectrogram image. [Figure 4] FIG. 4 is a diagram for explaining data expansion of the learning data. [Diagram 5] FIG. 5 is a diagram for explaining extraction of learning data. [Figure 6] FIG. 6 is a diagram for explaining data expansion of the learning data. [Figure 7] FIG. 7 is a diagram for explaining data expansion of the learning data. [Figure 8] FIG. 8 is a diagram for explaining the structure of a deep learning network (U-Net) for background sound removal. [Figure 9] FIG. 9 shows an example of the configuration of a system for evaluating the removal effect of a target voice. [Figure 10] FIG. 10 is a diagram for explaining a sound superimposed with a background sound for evaluating the removal effect of a sound to be removed. [Figure 11] FIG. 11 is a diagram showing the evaluation results of background sound removal. [Figure 12] FIG. 12 is a diagram illustrating an example of the configuration of a voice removal system. [Figure 13] FIG. 13 is a diagram illustrating an example of the configuration of a learning model creation device. [Figure 14] FIG. 14 is a diagram showing an operational flow of the learning model creation method. [Figure 15] FIG. 15 is a diagram illustrating an example of the configuration of a speech removal device. [Figure 16] FIG. 16 is a diagram showing an operation flow of the speech removal method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention can be embodied in many different forms, and should not be construed as being limited to the embodiment of the present invention described below.

[0018] <Example of system configuration for removing background noise> An example of the configuration of a system for removing background sound from background sound superimposed sound in which background sound is superimposed, which is the subject of this embodiment, is shown in Fig. 1. In this embodiment, a short-time Fourier transform (STFT) is applied to superimposed sound in which a target sound such as a speaker's voice is mixed with a target sound such as background sound to generate a two-dimensional image such as a spectrogram image, and the image is input to a deep learning model trained to remove the features of the target sound such as background sound that appears on the generated spectrogram image. By applying an inverse STFT to the output image of the deep learning model, the target sound such as a speaker's voice is restored from which the target sound such as background sound has been removed.

[0019] In this embodiment, a two-dimensional time-frequency image created by converting time-series sound intensity information is used as an image to be input to a deep learning model. As the two-dimensional image, in addition to a spectrogram image, a Mels spectrogram image or a scalogram image can be used. In the following description, a case where a spectrogram image is used as the two-dimensional image will be described.

[0020] In this embodiment, a learning model was created using U-Net as a deep learning network for removing the target speech. U-Net is one of the FCNs (Fully Convolutional Networks) and was announced for image segmentation in biomedical sciences.

[0021] Here, the target voice in this embodiment refers to a voice that is desired to be left after removing noise or to be emphasized by removing noise in a mixed voice in which various voices are mixed in. The target voice includes, but is not limited to, a voice of a specific speaker that is desired to be emphasized in an environment where background voices such as everyday sounds and noises occur, and a specific voice that is an observation target in a natural environment.

[0022] <Creating audio with background sound for creating a learning model> FIG. 2 is a diagram for explaining a superimposed voice on which a background sound for creating a learning model is superimposed. Various sounds such as a pet's cry, a baby's cry, and an intercom sound are assumed as background sounds to be removed in this embodiment. In this embodiment, as shown in FIG. 2, the same background sounds are arranged at a predetermined interval to create a voice to be removed, so that the data length is the same as that of the speaker's voice, and the created voice to be removed is superimposed on the speaker's voice to create a background sound superimposed voice for creating a learning model. In the example of FIG. 2, the background sounds are arranged at equal intervals, but they may be arranged at random intervals.

[0023] In this embodiment, audio data (third audio data) of superimposed audio with background sound superimposed is generated by superimposing audio data (first audio data) of a target audio such as a speaker's voice with audio to be removed (second audio data), and a two-dimensional image created using this is input to U-Net, which is trained to remove the characteristics of the background sound, thereby creating a learning model for removing background sound.

[0024] <Spectrogram image> In this embodiment, in order to input learning data to U-Net in the form of image data, STFT is applied to the time-series background sound superimposed voice in Fig. 2 to convert it into a spectrogram image. Fig. 3 is a diagram for explaining the spectrogram image to be input to U-Net. The parameters of STFT at the time of conversion may be, for example, a window function hann, a frame size 512, and a number of points 512.

[0025] <How to expand learning data> The cries of pets and babies may vary in volume and quality depending on emotions and situations, even if they are from the same individual or person. In this embodiment, in order to provide variation in the learning data and improve the background sound removal performance, data expansion is performed by changing the voice pitch of the background sound and the SN ratio when the background sound is superimposed.

[0026] Figure 4 shows an example of the audio waveform when the pitch of a cat's meow is changed, and a spectrogram image converted by applying STFT to the audio waveform. (a), (b), and (c) in Figure 4 show the audio waveform and spectrogram image when there is no change in the audio pitch, when the audio pitch is changed by -2 semitones, and when the audio pitch is changed by +2 semitones, respectively.

[0027] In this embodiment, the voice pitch is changed in five stages: no change, +1 semitone, -1 semitone, +2 semitones, -2 semitones, and the S / N ratio is changed in five stages in 10 dB increments from 40 dB to 0 dB. By such data expansion, a total of 25 types of background sound superimposed voices are generated for one background sound, including five types of pitch changes and five types of S / N ratio changes, and training data for input to U-Net is generated. It is expected that the background sound removal performance will be improved by performing training using the data-expanded training data.

[0028] <Creating learning data> FIG. 5 is a diagram for explaining the cutting out of learning data. In this embodiment, the background sound superimposed voice is converted into a spectrogram image for U-Net learning, but if the image size is long in the horizontal direction (time direction), it is not suitable for U-Net learning. Therefore, in this embodiment, the spectrogram image is cut out into a 256×256 image in accordance with the input of U-Net and input to U-Net. As shown in FIG. 5, the cut-out positions are aligned in the speaker voice, which is the target voice, and the superimposed voice in which the background sound is superimposed, and a data set for learning is created by using the former as teacher data (first spectrogram image) and the latter as input data to U-Net (second spectrogram image).

[0029] <How to expand other learning data> In the above-mentioned training data extension method, the data extension was performed by changing the S / N ratio when superimposing the pitch of the background sound, but other data extension methods may be used. Figure 6 is a diagram for explaining another data extension method for training data. In Figure 6, when creating a dataset for extracting images from a spectrogram image obtained by applying STFT to audio data and inputting them to U-Net, data extension is performed by extracting images so that parts of the spectrogram images overlap.

[0030] In the configuration example in Figure 6, the slide width is changed to 1 / 2 and 1 / 3 to cut out images, taking into consideration the occurrence of images that are extremely similar to each other. As shown in Figure 6(b), when the slide width is 1 / 2, an additional image can be created between the first and second images. By setting the slide width so that part of the data overlaps when cutting out images to be used in the dataset, it is possible to create multiple different datasets from the same audio data, and an increase in the amount of training data is expected to improve background sound removal performance.

[0031] When applying STFT to audio data to generate a spectrogram image, images with different resolutions can be generated by adjusting the STFT parameters. By using this, the amount of data can be increased by adding a spectrogram image created with a different resolution to a spectrogram image created with standard parameters.

[0032] 7 is a diagram for explaining another data extension of the learning data. In this embodiment, the parameters of the STFT are the window function hann, the frame size 512, and the number of points 512, and images with different resolutions to be added are created by changing the frame size of the window function, which has a particularly large effect on the change in resolution, based on these numerical values. Data extension may be performed by adding images with multiple resolutions.

[0033] In the example of Figure 7, data expansion is performed by adding images with resolutions of 128, 256, and 1024 to a reference image with resolution of 512. By adding spectrogram images created with different resolutions to a spectrogram image generated with reference parameters, it is possible to create multiple different data sets from the same audio data, so it is expected that the background sound removal performance will improve as the amount of training data increases.

[0034] <Deep learning network structure> 8 is a diagram for explaining the structure of a deep learning network for background sound removal. In this embodiment, a learning model for background sound removal is created by applying U-Net as a deep learning network.

[0035] In this embodiment, as shown in Fig. 8, a spectrogram image (size 256 x 256) of audio data with background sound superimposed is used as input data, and 9 convolutions and 4 Max Poolings are performed to extract features while compressing the image, then 4 Upsamplings and 9 deconvolutions are performed to return the image to its original size, and output data is output.

[0036] By training U-Net so that the output data matches the training data, which is the audio data before the background audio is superimposed (the error is reduced), a training model that removes the characteristics of the background audio (background audio removal filter) can be created. In the configuration example in Figure 8, U-Net is trained so that the mean squared error (MSE) between the output data and the training data is reduced.

[0037] <Background sound removal effect evaluation system> Fig. 9 is a configuration example of a system for evaluating the removal effect of a removal target voice. Fig. 10 is a diagram for explaining a background sound superimposed voice for evaluating the removal effect of a removal target voice. In this embodiment, a lecture video of the same person as used in learning that was not used for learning was separated into a lecture image and a speaker voice to extract the speaker voice, and background sound was superimposed at S / N ratios of 0 dB, 10 dB, and 20 dB at two points in the extracted speaker voice, the speaking part and the non-speaking part, to create a background sound superimposed voice for evaluation.

[0038] Lecture video X was created by combining the background sound superimposed audio created by superimposing background sound with the original lecture image. A spectrogram image obtained by performing STFT on the background sound superimposed audio with background sound superimposed was input into the created learning model, and an inverse STFT was performed on the spectrogram image output from the learning model to generate background sound removed audio, which was then combined with the original lecture image to create lecture video Y. A specified number of subjects (27 people) were asked to watch lecture video X and lecture video Y, and the effectiveness of removing background sound was evaluated by investigating how bothersome the background sound was.

[0039] Figure 11 shows the evaluation results of the background sound removal effect. Figures 11(a) and 11(b) show the evaluation results using a standalone model that targets the removal of a cat's meow and a baby's cry, respectively. For the cat's meow in Figure 11(a) and the baby's cry in Figure 11(b), the percentage of people who answered "it bothers me" increased as the S / N ratio of lecture video X, in which the background sound had not been removed, decreased. On the other hand, for lecture video Y, in which the background sound had been removed, a high percentage of people answered "I can't hear it" or "it doesn't bother me," and the viewing results by the subjects confirmed that a high removal effect was achieved regardless of the S / N ratio.

[0040] Figure 11(c) compares the evaluation results using a mixed model created to remove multiple background sounds, specifically, a cat's meow, a dog's barking, a baby's crying, and the sound of an intercom, with the evaluation results using the standalone models in Figures 11(a) and 11(b). In removing cat's meows, the mixed model gave similar evaluation results to those using the standalone models. On the other hand, in removing a baby's crying, the standalone model gave a higher evaluation result than the mixed model, but the evaluation results showed that "can't hear," "doesn't bother me," and "doesn't bother me much" accounted for a high percentage of over 80%.

[0041] <Background Sound Removal System> 12 is a diagram showing an example of the configuration of a background sound removal system. In the background sound removal system of this embodiment, a learning model (background sound removal filter) is created that is trained to remove the background sound to be removed by inputting various background sounds that a user wants to remove, and the created learning model (background sound removal filter) is used to remove the background sound from a background sound-mixed video in which the background sound is mixed into a target voice such as a speaker's voice, thereby providing a background sound-removed video.

[0042] In addition, when the voice of a specific speaker is mixed with the voice of other speakers, a learning model (voice enhancement filter) trained to emphasize the voice of the specific speaker can be created, and the created learning model (voice enhancement filter) can be used to output a video in which the voice of the specific speaker is emphasized in a mixed video in which the voice of the specific speaker is mixed with the voice of other speakers.

[0043] The background sounds that users want to remove can be various sounds depending on the environment in which the user distributes the video, such as the barking of a pet or the crying of an infant. By creating a learning model according to the various background sounds that users want to remove, it is possible to obtain good background sound removal performance when removing the various background sounds that users want to remove.

[0044] In the configuration example of Fig. 12, a learning model (background sound removal filter / voice emphasis filter) is created in the learning model creation device 10, and the background sound removal device 20 removes background sound from a video containing background sound provided from a speaker terminal 30. The creation of the learning model and the removal of background sound may be performed in one device. It is also possible to provide the learning model created in the learning model creation device 10 to a user.

[0045] <Selection of learning model according to sensor information> The background sound removal system of this embodiment is configured to be able to select from the created learning models according to sensor information such as the source of the background sound and the position information of the speaker 50, the operating status of the source of the background sound, and the status of the surrounding environment of the speaker 50. Learning models trained using various background sounds are created in advance, and it becomes possible to select a learning model with a higher background sound removal effect according to the sensor information detected by the sensor 40, etc.

[0046] In order to select a learning model according to the sensor information, it is sufficient to add information corresponding to the sensor information to the created learning model. For example, if the background sound to be removed is the sound of a train running, by adding to the learning model information indicating that the background sound to be removed is the sound of a train running and information for identifying the running section and running position of the train, which is the source of the sound to be removed, it becomes possible to select an appropriate learning model according to the position information of the speaker 50, which is the source of the target sound.

[0047] Here, the location information of the speaker 50 can be detected from the location information of the sensor 40 detected by a GPS sensor provided in the sensor 40. In addition, by detecting an SSID (Service Set Identifier) ​​transmitted from an access point of a wireless LAN received by the sensor 40, the location information of the speaker 50 can be estimated using the location information of the access point.

[0048] When the background sound to be removed is noise generated from a machine tool, etc., by adding to the learning model information that the background sound to be removed is noise generated from a machine tool, etc., and information for identifying the noise state (noise level, frequency components of the noise, etc.) generated from the machine tool, etc., which is the source of the sound to be removed, it becomes possible to select an appropriate learning model according to the noise state estimated from the vibration information from the vibration sensor.

[0049] When the background sound to be removed is noise generated from air conditioning equipment or the like, by adding to the learning model information that the background sound to be removed is noise generated from air conditioning equipment or the like and information for identifying the noise conditions generated by air conditioning equipment or the like that is the source of the sound to be removed, it becomes possible to select an appropriate learning model according to the noise conditions estimated from information from the temperature / humidity sensors.

[0050] When the background sound to be removed is noise in the speaker's surrounding environment, by providing the learning model with information indicating that the background sound to be removed is noise in the speaker's surrounding environment, which is the source of the target voice, and specifying the noise conditions in which it is mixed, it becomes possible to select an appropriate learning model depending on the noise conditions estimated by information from the people flow sensor.

[0051] Furthermore, when selecting a learning model (voice enhancement filter) that emphasizes the voice of a specific speaker, by adding information for identifying the speaker to the learning model, it becomes possible to select a learning model (voice enhancement filter) that emphasizes the voice of the specific speaker by identifying the speaker using a face recognition sensor, a speaker identification filter, etc. Identification of the speaker using a face recognition sensor, a speaker identification filter, etc. can be performed in the background sound removal device 20.

[0052] <Configuration of learning model creation device> 13 is a diagram showing a configuration example of a learning model creation device 10. The learning model creation device 10 includes an I / F unit 11 for transmitting and receiving information to and from an external device, a learning data creation unit for creating learning data, a central processing unit 12 functioning as a learning model creation unit for creating a learning model, and a storage unit 13 for storing teacher data, input data, the created learning model, a program for executing the processing of the central processing unit 12, etc.

[0053] Audio data of the background sound to be removed is input via the I / F unit 11, and a learning model (background sound removal filter) is created in the central processing unit 12 using the input audio data of the background sound. The created learning model is output via the I / F unit 11 and provided to the background sound removal device 20. As described in Figs. 2 to 5, the learning model can be created by having U-Net learn the data-extended input data and teacher data. The U-Net may be implemented in the learning model creation device 10, or may be trained in a U-Net implemented on the cloud.

[0054] <How the learning model is created> 14 is a diagram showing an operation flow of the learning model creation method. The learning model creation device 10 acquires, from a user, audio data of the background sound to be removed (second audio data) (S1-1), superimposes the acquired audio data of the background sound on audio data of the speaker's voice that is the target audio (first audio data) to create background sound superimposed audio (third audio data), and extracts a part of the background sound superimposed audio to create input data for U-Net (S1-2).

[0055] Here, the voice data of the speaker's voice, which is the voice of interest, may be provided by the user, or a speaker's voice prepared in advance in the learning model creation device 10 may be used. In addition, when the voice data of the background sound to be removed is voice data without individual differences due to the sound source, the learning model may be created using the voice data of the background sound prepared in advance.

[0056] For example, when creating a learning model for removing background sounds such as the sirens of ambulances, police cars, and fire engines, data on background sounds prepared in advance in the learning model creation device may be used.

[0057] The teacher data and input data can be created using the methods described in Figures 2 to 5. As for the input data, as described in Figure 4, a total of 25 types of background sound-overlapping voices are generated for one background sound, with five types of pitch changes and five types of S / N ratio changes, to generate input data to be input to U-Net.

[0058] The learning model creation device 10 applies STFT to a speaker voice provided by a user or a speaker voice prepared in advance to create a spectrogram image (first spectrogram image), cuts out a part of it to create teacher data, and applies STFT to the background sound superimposed voice to create a spectrogram image (second spectrogram image), and cuts out a part of it to create data.

[0059] The learning model creation device 10 creates a learning model that removes the background sound provided by the user by having U-Net learn the created input data and teacher data (S1-3), and outputs the created learning model to the background sound removal device 20 (S1-5).

[0060] <Configuration of background sound removal device> 15 is a diagram showing a configuration example of a voice removal device that removes background sound. The background sound removal device 20 includes an I / F unit 21 for transmitting and receiving information to and from an external device, a learning model selection unit, a central processing unit 22 that functions as a background sound removal processing unit, and a storage unit 23 that stores speaker data, sensor information, a learning model created by the learning model creation device 10, a program for executing the processing of the central processing unit 22, and the like.

[0061] The learning model created by the learning model creation device 10 is input via the I / F unit 21 and stored in the storage unit 23. In the central processing unit 12, a learning model is selected according to the sensor information acquired from the sensor 40, and the background sound-mixed video provided by the speaker terminal 30 is input to the selected learning model, thereby executing background sound removal processing.

[0062] The background sound elimination device 20 is configured to be able to select a learning model according to sensor information detected by the sensor 40. A learning model trained in advance using various background sounds is created, and the learning model is selected according to sensor information such as the source of the background sound detected by the sensor 40, the position information of the speaker, the operating status of the source of the background sound, and the status of the speaker's surrounding environment, making it possible to select a learning model with a higher background sound elimination effect and perform background sound elimination.

[0063] Possible sensors 40 that provide sensor information include a position sensor that detects the position information of the source of the background sound that is the sound to be removed or the speaker that is the source of the target sound, a vibration sensor that detects the operating status of the source of the background sound, a temperature / humidity sensor or a people flow / noise sensor that detects the status of the speaker's surrounding environment, a face recognition sensor or a speaker identification filter for identifying the speaker, etc. Depending on the type of background sound, the sensor information is detected by a sensor 40 that includes at least one of these.

[0064] When the background sound to be removed is the sound of a train running, the characteristics of the audio data of the background sound generated differ depending on the type of train and the section the train is running in. By utilizing this, the sound of a train running that is mixed into the voice of a speaker according to the position information of the speaker who emits the speaker's voice is estimated, and a learning model corresponding to the estimated sound of a train running is selected, making it possible to select a learning model with a higher background sound removal effect.

[0065] When the background noise to be removed is generated from a machine tool or the like, the characteristics of the audio data of the generated background noise differ depending on the operating status of the machine tool, which is the source of the noise. By utilizing this, the noise state generated from the machine tool or the like that is mixed into the speaker's voice can be estimated according to the vibration information of the machine tool, and a learning model corresponding to the estimated noise state of the machine tool can be selected.

[0066] When the background noise to be removed is noise generated from an air conditioner or the like, the characteristics of the audio data of the background noise generated differ depending on the temperature and humidity of the environment in which the air conditioner, which is the source of the noise, is installed. By utilizing this, the state of noise generated from the air conditioner or the like that is mixed into the speaker's voice can be estimated according to the temperature information and humidity information of the environment in which the air conditioner is installed, and a learning model corresponding to the estimated noise state of the air conditioner can be selected.

[0067] When the background sound to be removed is noise in the speaker's surrounding environment, the characteristics of the voice data of the background sound generated by the speaker differ depending on the congestion level around the speaker. By utilizing this, the noise condition mixed into the speaker's voice can be estimated based on the detection results of a people flow sensor that detects the congestion level around the speaker, and a learning model corresponding to the estimated noise condition of the surrounding environment can be selected.

[0068] Also, a speaker may be identified by a face recognition sensor or a speaker identification filter for identifying the speaker, and a learning model (voice enhancement filter) that enhances the voice of the specific speaker may be selected. By creating a learning model (voice enhancement filter) that enhances the voice of a specific speaker in advance and selecting a learning model (voice enhancement filter) according to the speaker, the level of voices other than the specific speaker is relatively reduced, and the same effect as removing voices other than the specific speaker can be obtained.

[0069] <Background sound removal method operation> 16 is a diagram showing an operation flow of the audio removal method for removing background sound. The background sound removal device 20 acquires data of a video containing background sound to be removed from a user (S2-1), and acquires sensor information for identifying the source of the background sound to be removed, the situation in which the background sound to be removed is generated, and the speaker (S2-2).

[0070] A learning model corresponding to the acquired sensor information is selected, and the background sound is removed by inputting audio data created using the audio data of the video containing background sound into the selected learning model (S2-4), and the background sound removed video from which the background sound has been removed is output (S2-5).

[0071] As described above, in this embodiment, when creating a learning model for removing background sound from background sound superimposed audio on which background sound is superimposed, the learning model is created by performing data expansion by changing the pitch of the audio data of the background sound and the S / N ratio when the background sound is superimposed.

[0072] By using a learning model trained by performing such data expansion, it is possible to obtain good background sound removal performance when removing various background sounds that users wish to remove.

[0073] In addition, in this embodiment, the learning model can be selected according to sensor information such as the source of the background sound to be removed detected by the sensor, the situation in which the background sound to be removed occurs, and information for identifying the speaker.

[0074] Depending on the source of the background sound to be removed detected by the sensor, the situation in which the background sound to be removed occurs, information for identifying the speaker, etc., a learning model that is more suitable for removing the background sound to be removed and has a higher background sound removal effect can be selected, making it possible to obtain good background sound removal performance when removing various background sounds that the user wishes to remove. [Industrial Applicability]

[0075] The present invention can be used in a background sound removal system for removing background sound mixed into a moving image. [Explanation of symbols]

[0076] 10... learning model creation device, 20... background sound removal device, 30... speaker terminal, 40... sensor.

Claims

1. A learning model creation method executed in a learning model creation device including a learning data creation unit and a learning model creation unit, The learning data creation unit: creating a first two-dimensional image using first audio data, creating third audio data by superimposing at least one second audio data on the first audio data, and creating a second two-dimensional image using the third audio data; The learning model creation unit is A learning model for removing the second voice data is created by training the deep learning network using the first two-dimensional image as training data and the second two-dimensional image as input data such that output data obtained when the input data is input to the deep learning network matches the training data; The learning data creation unit: A plurality of third voice data are generated by changing the voice pitch of the second voice data and the S / N ratio at the time of the superposition, and the second two-dimensional image is generated using each of the plurality of third voice data. How to create a learning model.

2. A learning model creation device including a learning data creation unit and a learning model creation unit, The learning data creation unit: creating a first two-dimensional image using first audio data, creating third audio data by superimposing at least one second audio data on the first audio data, and creating a second two-dimensional image using the third audio data; The learning model creation unit is A learning model for removing the second voice data is created by training the deep learning network using the first two-dimensional image as training data and the second two-dimensional image as input data such that output data obtained when the input data is input to the deep learning network matches the training data; The learning data creation unit: A plurality of third voice data are generated by changing the voice pitch of the second voice data and the S / N ratio at the time of the superposition, and the second two-dimensional image is generated using each of the plurality of third voice data. Learning model creation device.

3. A learning model creation method executed in a learning model creation device including a learning data creation unit and a learning model creation unit, The learning data creation unit: creating a first two-dimensional image using the first voice data, creating third voice data by superimposing at least one second voice data on the first voice data, and creating a second two-dimensional image using the third voice data; The learning model creation unit is A learning model for removing the second voice data is created by training the deep learning network using the first two-dimensional image as training data and the second two-dimensional image as input data such that output data obtained when the input data is input to the deep learning network matches the training data; The learning data creation unit: A first two-dimensional image is created by cutting out an image from a two-dimensional image created using the first voice data so that the image overlaps with another image, and a second two-dimensional image is created by cutting out an image from a two-dimensional image created using the third voice data so that the image overlaps with another image. How to create a learning model.

4. A learning model creation device including a learning data creation unit and a learning model creation unit, The learning data creation unit: creating a first two-dimensional image using the first voice data, creating third voice data by superimposing at least one second voice data on the first voice data, and creating a second two-dimensional image using the third voice data; The learning model creation unit is A learning model for removing the second voice data is created by training the deep learning network using the first two-dimensional image as training data and the second two-dimensional image as input data such that output data obtained when the input data is input to the deep learning network matches the training data; The learning data creation unit: A first two-dimensional image is created by cutting out an image from a two-dimensional image created using the first voice data so that the image overlaps with another image, and a second two-dimensional image is created by cutting out an image from a two-dimensional image created using the third voice data so that the image overlaps with another image. Learning model creation device.

5. A learning model creation method executed in a learning model creation device including a learning data creation unit and a learning model creation unit, The learning data creation unit: creating a first two-dimensional image using first audio data, creating third audio data by superimposing at least one second audio data on the first audio data, and creating a second two-dimensional image using the third audio data; The learning model creation unit is A learning model for removing the second voice data is created by training the deep learning network using the first two-dimensional image as training data and the second two-dimensional image as input data such that output data obtained when the input data is input to the deep learning network matches the training data; The learning data creation unit: A two-dimensional image having a resolution different from the predetermined resolution is added to a two-dimensional image having a predetermined resolution created using the first voice data to create the first two-dimensional image, and a two-dimensional image having a resolution different from the predetermined resolution is added to a two-dimensional image having a predetermined resolution created using the third voice data to create the second two-dimensional image. How to create a learning model.

6. A learning model creation device including a learning data creation unit and a learning model creation unit, The learning data creation unit: creating a first two-dimensional image using first audio data, creating third audio data by superimposing at least one second audio data on the first audio data, and creating a second two-dimensional image using the third audio data; The learning model creation unit is A learning model for removing the second voice data is created by training the deep learning network using the first two-dimensional image as training data and the second two-dimensional image as input data such that output data obtained when the input data is input to the deep learning network matches the training data; The learning data creation unit: A two-dimensional image having a resolution different from the predetermined resolution is added to a two-dimensional image having a predetermined resolution created using the first voice data to create the first two-dimensional image, and a two-dimensional image having a resolution different from the predetermined resolution is added to a two-dimensional image having a predetermined resolution created using the third voice data to create the second two-dimensional image. Learning model creation device.

7. A voice removal method executed in a voice removal device that removes a voice to be removed from a superimposed voice in which the voice to be removed is superimposed on a voice of interest, using a learning model that is trained using input data and teacher data, comprising: The teacher data is a first two-dimensional image created using first sound data corresponding to the target sound; The input data is a second two-dimensional image is created by superimposing at least one second sound data corresponding to the sound to be removed on the first sound data to create third sound data, and using the third sound data; The learning model is The deep learning network is trained so that output data when the input data is input to the deep learning network matches the teacher data, The third audio data is The second voice data is generated by superimposing each of the second voice data, which is generated by changing the S / N ratio when the second voice data is superimposed on the first voice data. Audio removal methods.

8. A speech removal device that removes a removal target speech from a superimposed speech in which the removal target speech is superimposed on a speech of interest, using a learning model that is trained using input data and teacher data, comprising: The teacher data is a first two-dimensional image created using first sound data corresponding to the target sound; The input data is a second two-dimensional image is created by superimposing at least one second sound data corresponding to the sound to be removed on the first sound data to create third sound data, and using the third sound data; The learning model is The deep learning network is trained so that output data when the input data is input to the deep learning network matches the teacher data, The third audio data is The second voice data is generated by superimposing each of the second voice data, which is generated by changing the S / N ratio when the second voice data is superimposed on the first voice data. Audio removal device.

9. Using sensor information detected by a sensor, the removal target sound mixed in the superimposed sound is estimated; The learning model for removing the second voice data corresponding to the estimated removal target voice is selected, and the removal target voice is removed from the superimposed voice using the selected learning model.

8. The method of claim 7.

10. The sensor includes: a position sensor for detecting position information of a source of the sound to be removed; a position sensor for detecting position information of a source of the target sound; a vibration sensor for detecting the operating status of the source of the sound to be removed; - a temperature / humidity sensor for detecting the conditions of the surrounding environment of the source of the target sound; Contains at least one of 10. The method of claim 9.

11. Using a face recognition sensor or a speaker identification filter, a speaker in the video containing the superimposed voice and background sound is estimated; Selecting the training model that emphasizes the first speech data corresponding to the estimated speaker.

8. The method of claim 7.

12. Using sensor information detected by a sensor, the removal target sound mixed in the superimposed sound is estimated; The learning model for removing the second voice data corresponding to the estimated removal target voice is selected, and the removal target voice is removed from the superimposed voice using the selected learning model.

9. The speech removal device of claim 8.

13. The sensor includes: a position sensor for detecting position information of a source of the sound to be removed; a position sensor for detecting position information of a source of the target sound; a vibration sensor for detecting the operating status of the source of the sound to be removed; - a temperature / humidity sensor for detecting the conditions of the surrounding environment of the source of the target sound; Contains at least one of 13. The speech removal device of claim 12.

14. Using a face recognition sensor or a speaker identification filter, a speaker in the video containing the superimposed voice and background sound is estimated; Selecting the training model that emphasizes the first speech data corresponding to the estimated speaker.

9. The speech removal device of claim 8.

Images