Learning device, noise removal device, and training data generation device

Abstract

The present invention provides a noise removal device and the like that can perform noise removal on sound field video images with higher precision than the prior art and perform favorable noise removal even on data of an unknown nature or data containing nonlinear noise. The noise removal device is formed from a DNN, accepts a sound field image as input, and outputs a sound field image from which noise included in the sound field image has been removed.

Description

Learning device, noise reduction device, and training data generation device

[0001] The present invention relates to sound field imaging technology.

[0002] Sound field imaging is a technique for measuring sound in a space and visualizing it as a moving image, using microphone arrays, optical measurement devices, etc. Sound field moving images are discrete data in which the value of each pixel in the moving image corresponds to the amplitude of sound in the measurement space, and are used to observe and measure the spatiotemporal behavior of sound and acoustic phenomena.

[0003] Noise removal is one of the signal processing tasks related to sound field video images. In Non-Patent Document 1, noise removal is performed on sound field video images using spatiotemporal frequency filtering. By using a filter designed to match the physical properties of the sound being measured, it is possible to remove noise and enhance the sound signal.

[0004] N. Chitanont, K. Yatabe, K. Ishikawa, and Y. Oikawa, "Spatio-temporal filter bank for visualizing audible sound field by schlieren method", Appl. Acoust. 115, 109-120 (2017).

[0005] However, conventional techniques have certain limitations in noise removal performance, require prior knowledge of the properties of the sound signal, and are not suitable for removing nonlinear noise.

[0006] The present invention aims to provide a noise removal device that can remove noise from sound field moving images with higher accuracy than conventional technology and can perform good noise removal even on data with unknown properties or data containing nonlinear noise, a learning device for models used in the noise removal device, and a training data generation device that generates training data to be used in the learning device.

[0007] In order to solve the above problem, according to one aspect of the present invention, a noise reduction device is composed of a DNN, takes a sound field image as input, and outputs a sound field image from which noise contained in the sound field image has been removed.

[0008] In order to solve the above problem, according to another aspect of the present invention, a learning device includes: a noise removal unit that is made of a DNN and uses a noise removal model that receives as input information based on a complex amplitude sound field image for a sound field image and outputs information based on the complex amplitude sound field image for a sound field image from which noise contained in the sound field image has been removed, to remove noise from the sound field image for learning input data included in training data and obtain the learning input data after noise removal; and an update unit that updates parameters of the noise removal model so as to reduce an error between the correct data included in the training data and the learning input data after noise removal.

[0009] In order to solve the above problems, according to another aspect of the present invention, a training data generation device includes a parameter generation unit that generates parameters to be used when generating correct data, and a training data generation unit that generates correct data by acoustic simulation using the parameters, uses the correct data to which arbitrary noise has been added as learning input data in accordance with a measurement system that acquires a sound field moving image, and obtains training data consisting of pairs of the correct data and the learning input data, the training data being made up of a DNN and being used when training a noise removal model that receives as input information based on a complex amplitude sound field image for a sound field image and outputs information based on the complex amplitude sound field image for a sound field image from which noise contained in the sound field image has been removed.

[0010] According to the present invention, noise removal from sound field moving images can be performed with higher accuracy than conventional techniques, and good noise removal can be performed even on data with unknown properties or data containing nonlinear noise.

[0011] 1 is a functional block diagram of a noise removal system according to a first embodiment. FIG. 2 is a diagram showing an example of a processing flow of the noise removal system according to the first embodiment. FIG. 3 is a functional block diagram of a training data generation device. FIG. 4 is a diagram showing an example of a processing flow of the training data generation device. FIG. 5 is a schematic diagram of a simulation when generating training data. FIG. 6 is a functional block diagram of a learning device. FIG. 7 is a diagram showing an example of a processing flow of the learning device. FIG. 8 is a functional block diagram of a noise removal device. FIG. 9 is a diagram showing an example of a processing flow of the noise removal device. FIG. 10 is a diagram showing an example of a noise removal result. FIG. 11 is a diagram showing an example of the configuration of a computer to which the present method is applied.

[0012] An embodiment of the present invention will be described below. In the drawings used in the following description, components having the same function and steps performing the same processing are denoted by the same reference numerals, and duplicate explanations will be omitted. In the following description, symbols such as "^" used in the text should normally be written directly above the character immediately preceding it, but due to limitations in text notation, they are written immediately after the character in question. In formulas, these symbols are written in their original positions. Furthermore, unless otherwise specified, processing performed on each element of a vector or matrix is ​​assumed to apply to all elements of that vector or matrix.

[0013] <Key Points of the First Embodiment> In this embodiment, a deep neural network (DNN) is used to achieve highly accurate noise removal. By using a DNN, it is possible to perform good noise removal even on data with unknown properties or data containing nonlinear noise.

[0014] In this embodiment, the sound field moving image is expressed as a complex amplitude by Fourier transform, and the sound field image expressed as a complex amplitude (hereinafter also referred to as a complex amplitude sound field image) is regarded as a two-channel image consisting of a real part and an imaginary part to make it easier to handle with DNN. This allows any DNN for image noise removal to be used for the purpose of sound field noise removal.

[0015] To train a DNN, a large amount of training data is required, but it is extremely difficult to collect a large amount of pairs of noise-free and noise-containing data in experiments for sound field video images. Therefore, in this embodiment, training data is generated using acoustic simulation.

[0016] First Embodiment Fig. 1 shows an example of the configuration of a noise reduction system according to a first embodiment, and Fig. 2 shows a processing flow of the noise reduction system.

[0017] The noise reduction system includes a training data generation device 100 , a learning device 200 , and a noise reduction device 300 .

[0018] The training data generation device 100 generates and outputs training data consisting of pairs of correct answer data and learning input data (S100). The correct answer data is noise-free data generated using acoustic simulation, while the learning input data is noise-containing data.

[0019] The learning device 200 receives training data as input, learns a noise removal model (S200), and outputs the learned noise removal model. The noise removal model is a model that is made up of a DNN, receives information based on a complex amplitude sound field image for a sound field image as input, and outputs information based on the complex amplitude sound field image for a sound field image from which noise contained in the sound field image has been removed.

[0020] The noise reduction device 300 receives a trained noise reduction model before noise reduction processing. The noise reduction device 300 receives a sound field moving image to be processed as input, removes noise contained in the sound field moving image (S300), and outputs the sound field moving image after noise reduction. A sound field moving image is made up of time-series sound field images, and the noise reduction device 300 can also be said to receive sound field images constituting the sound field moving image to be processed as input, remove noise contained in the sound field images constituting the sound field moving image, and output sound field images constituting the sound field moving image after noise reduction.

[0021] The training data generation device 100, the learning device 200, and the noise reduction device 300 are special devices configured by loading special programs into a publicly known or dedicated computer having, for example, a central processing unit (CPU), a main memory (RAM), etc. The training data generation device 100, the learning device 200, and the noise reduction device 300 execute their respective processes under the control of, for example, the central processing unit. Data input to the training data generation device 100, the learning device 200, and the noise reduction device 300 and data obtained by each process are stored in, for example, the main memory, and the data stored in the main memory is read out by the central processing unit as needed and used for other processes. At least a portion of each processing unit of the training data generation device 100, the learning device 200, and the noise reduction device 300 may be configured using hardware such as an integrated circuit. Each storage unit included in the training data generation device 100, the learning device 200, and the noise reduction device 300 can be configured, for example, as a main storage device such as a random access memory (RAM), or middleware such as a relational database or a key-value store. However, each storage unit does not necessarily need to be included inside the training data generation device 100, the learning device 200, and the noise reduction device 300. Instead, each storage unit can be configured as an auxiliary storage device implemented by a semiconductor memory element such as a hard disk, optical disk, or flash memory, and can be configured to be external to the training data generation device 100, the learning device 200, and the noise reduction device 300.

[0022] Each device will be described below.

[0023] <Training Data Generating Device 100> FIG. 3 is a functional block diagram of the training data generating device 100, and FIG. 4 shows an example of the processing flow thereof.

[0024] The training data generation device 100 includes a parameter generation unit 110 , a supervised data generation unit 120 , a noise generation unit 130 , and a noise addition unit 140 .

[0025] <Parameter Generation Unit 110 and Correct Data Generation Unit 120> To generate correct data, any theoretical formula or numerical simulation method that can numerically calculate sound field data can be used. When creating data, diverse training data is generated by determining any parameters of the generation method from a desired probability distribution. The following method is one example.

[0026] A two-dimensional sound field simulation with randomized parameters is used. Figure 5 shows a schematic diagram of the simulation. The inner rectangle is the measurement area, and the outer rectangle is the sound source area, with point sound sources randomly placed. To generate sound fields with a variety of spatial characteristics, from simple to complex, the number of point sound sources is set to 1 to 5, and the position and relative amplitude of each sound source are randomly assigned. The ground truth data is the superposition of sound waves generated from these point sound sources, and is calculated using the following formula. Here, r = (x, y) is the position, k is the magnitude of the acoustic wave number, A is a constant that determines the size of the entire sound field, N is the number of sound sources, a i and r i =(x i , y i ) are the relative amplitude and position of the ith sound source, respectively, and H (2) 0 is the zero-order Hankel function of the second kind. The part inside the summation symbol represents the product of the relative amplitude of the i-th sound source and the Green's function of the two-dimensional Helmholtz equation. The parameters are generated from the desired probability distribution, and the sound pressure at each point is calculated based on the above equation to generate the correct data.

[0027] The parameter generating unit 110 generates and outputs parameters used when generating the correct answer data (S110). The parameters include the relative amplitude a of each sound source, i and position r i For example, the position of each sound source is randomly assigned according to a desired probability distribution within the sound source region, and the relative amplitude is randomly assigned to each sound source according to a desired probability distribution within the range of expected relative amplitudes. Note that the numbers of measurement regions, sound source regions, and point sound sources are assumed to be given in advance.

[0028] The correct data generating unit 120 receives the parameters generated by the parameter generating unit 110 as input, and generates correct data by acoustic simulation using the parameters (S120), and outputs the correct data. For example, while changing the number N of sound sources from 1 to 5, the relative amplitude a i and position r i Using the above as input, the sound pressure at each point in the measurement area is calculated using equation (1), and correct data is generated.

[0029] <Noise Generator 130 and Noise Adder 140> Any noise can be added to the sound field moving image according to the measurement system used to acquire the moving image. For example, additive Gaussian white noise can be added. White noise with a desired amplitude is generated and added to the ground truth data to generate noisy data.

[0030] The noise generator 130 generates and outputs any noise (S130) in accordance with the measurement system that acquires the sound field moving image. For example, additive Gaussian white noise is generated.

[0031] The noise addition unit 140 receives the correct answer data and the generated noise as input, adds the generated noise to the correct answer data (S140), uses the correct answer data with the added noise as learning input data, and outputs training data consisting of a pair of the correct answer data and the learning input data. For example, the noise is added by adding white noise. The correct answer data generation unit 120, the noise generation unit 130, and the noise addition unit 140 are collectively referred to as the training data generation unit 150, and the processing in the training data generation unit 150 is also referred to as training data generation processing S150.

[0032] Furthermore, when considering noise removal from sound field imaging results using an optical system called a speckle interferometer, speckle noise can also be added. Speckle noise data can be generated using, for example, the method described in Reference 1. However, any method for generating data containing noise can be used, and the present invention is not limited to these. (Reference 1) Q. Fang, H. Xia, Q. Song, M. Zhang, R. Guo, S. Montresor, and P. Picart, "Speckle denoising based on deep learning via a conditional generative adversarial network in digital holographic interferometry," Opt. Express 30, 20666-20683 (2022). For example, when using the speckle noise data generation method, the training data generation unit 150 receives parameters generated by the parameter generation unit 110 as input, performs acoustic simulation using the parameters to generate ground truth data, and generates ground truth data with speckle noise added during the algorithm for generating the ground truth data (S150).

[0033] With this configuration, a large amount of training data can be generated by acoustic simulation without conducting experiments that require a huge amount of effort and time.

[0034] <Learning Device 200> FIG. 6 is a functional block diagram of the learning device 200, and FIG. 7 shows an example of the processing flow thereof.

[0035] The learning device 200 includes an image conversion unit 210 , a noise removal unit 220 , and an update unit 230 .

[0036] <Image conversion unit 210> The image conversion unit 210 receives training data as input. The learning input data and correct answer data included in the training data are each a complex amplitude sound field image. The elements of the complex amplitude sound field image represent complex amplitudes at corresponding spatial positions and Fourier frequencies. The image conversion unit 210 converts each complex amplitude sound field image into a two-channel image consisting of a real part and an imaginary part (S210). For example, if a complex amplitude sound field image consists of K elements, [a1, ..., a K ], and the kth element is a k =b k +c k When j, the image conversion unit 210 converts the complex amplitude sound field image [a1, ..., a K ] is an image consisting of real parts (hereinafter also referred to as real part image) [b1,...,b K ] and an image consisting of the imaginary parts (hereinafter also referred to as the imaginary part image) [c1,...,c K ]. Furthermore, the image conversion unit 210 converts the two-channel image (real part image [b1, ..., b K ] and the imaginary image [c1,…,c K ]) is normalized and output. Note that the normalization performed here is a process of adjusting the data so that it can be easily handled by the noise removal model, and is a process of multiplying the data by a normalization coefficient.

[0037] <Noise removal unit 220> The noise removal unit 220 receives the learning input data included in the training data as input, removes noise from the learning input data using a noise removal model (S220), and obtains and outputs the learning input data after noise removal.

[0038] In this embodiment, the noise elimination unit 220 converts the learning input data (complex amplitude sound field image) included in the training data into a two-channel image (real part image [b T,1 ,…,b T,K ] and the imaginary image [c T,1 ,…,c T,K ]) is used as input, and a two-channel image (real part image [b T,1 ,…,b T,K ] and the imaginary image [c T,1 ,…,c T,K ]) and remove noise from the two-channel image after noise removal (real part image [b' T,1,…,b' T,K ] and the imaginary image [c' T,1 ,…,c' T,K ]) is obtained. As described above, the noise removal model is a model that is made up of a DNN, receives information based on a complex amplitude sound field image for a sound field image as input, and outputs information based on the complex amplitude sound field image for a sound field image from which noise contained in the sound field image has been removed. In this embodiment, the noise removal model receives normalized real part images and imaginary part images as input, removes noise contained in the real part images and imaginary part images, and outputs the real part images and imaginary part images after noise removal.

[0039] In this embodiment, a DNN is used to remove noise from two-channel input images, so any network that removes image noise can be used. For example, a network with a Unet structure can be used. It is also possible to use a nonlinear activation free network (NAFNet) (see Reference 2), which has excellent performance in image noise removal and can operate with relatively small memory and training time. (Reference 2) L. Chen, X. Chu, X. Zhang, and J. Sun, "Simple baselines for image restoration," in Proceedings of the European 350 Conference on Computer Vision (ECCV), (Springer Nature Switzerland, Cham, 2022), pp. 17-33

[0040] <Update unit 230> The update unit 230 receives as input the correct answer data included in the training data and the learning input data after noise removal, and updates the parameters of the noise removal model so as to reduce the error between the correct answer data included in the training data and the learning input data after noise removal (S230).

[0041] In this embodiment, the correct data (complex amplitude sound field image) included in the training data is converted into a two-channel image, normalized, and compared with the two-channel image after noise removal (real part image [b' T,1 ,…,b' T,K ] and the imaginary image [c'T,1 ,…,c' T,K The parameters of the noise removal model are updated so that the error with the input signal is minimized. For example, RMSE (Root Mean Square Error) can be used for the loss, and steepest descent or Adam can be used for the optimization algorithm.

[0042] If the predetermined condition is not satisfied (NO in S230-1), the update unit 230 outputs the updated parameters to the noise reduction unit 220 and repeats S210 to S230.

[0043] If the predetermined condition is met (YES in S230-1), the update unit 230 outputs the noise removal model at that time as the trained noise removal model.

[0044] The specified condition can be set to determine whether the parameter updates have converged. For example, conditions such as whether learning has been repeated a certain number of times (e.g., several times), whether the difference in parameters before and after the update is below a specified threshold, and whether the loss is below a specified threshold can be used.

[0045] By adopting this learning method, any DNN for image noise removal can be used for the purpose of sound field noise removal.

[0046] <Noise Cancellation Device 300> FIG. 8 is a functional block diagram of the noise cancellation device 300, and FIG. 9 shows an example of the processing flow thereof.

[0047] The noise reduction device 300 includes a Fourier transform unit 310 , an image conversion unit 320 , a noise reduction unit 330 , an image inverse conversion unit 340 , and an inverse Fourier transform unit 350 .

[0048] The noise reduction device 300 uses the trained noise reduction model in the noise reduction unit 330 during noise reduction processing.

[0049] <Fourier transform unit 310> The Fourier transform unit 310 receives the sound field moving image to be processed as input, performs a time domain Fourier transform on all pixels of the sound field moving image (S310), and obtains and outputs a complex amplitude sound field image for each frequency. As the Fourier transform, a fast Fourier transform or the like can be used.

[0050] The image conversion unit 320 receives a complex amplitude sound field image for each frequency, converts the complex amplitude sound field image into a two-channel image consisting of a real part and an imaginary part (S320), normalizes the two-channel image, and outputs it. For example, the image conversion unit 320 performs the same processing as the image conversion unit 210.

[0051] <Noise removal unit 330> The noise removal unit 330 receives as input a sound field image for the sound field moving image to be processed, removes noise from the sound field moving image to be processed using a noise removal model (S330), and obtains and outputs a sound field image for the sound field moving image to be processed after noise removal.

[0052] In this embodiment, the noise removal unit 330 converts the complex amplitude sound field image for the sound field moving image to be processed into a two-channel image (real part image [b E,1 ,…,b E,K ] and the imaginary image [c E,1 ,…,c E,K ]) is used as input, and noise is removed from the two-channel image using a noise removal model. The two-channel image after noise removal (real part image [b' E,1 ,…,b' E,K ] and the imaginary image [c' E,1 ,…,c' E,K ])

[0053] <Image inverse transformation unit 340> The image inverse transformation unit 340 converts the noise-removed two-channel image (real part image [b' E,1 ,…,b' E,K ] and the imaginary image [c' E,1 ,…,c' E,K ]) is used as input, and the noise-removed two-channel image (real part image [b' E,1 ,…,b' E,K ] and the imaginary image [c' E,1 ,…,c' E,K ]) is multiplied by the inverse of the normalization coefficient multiplied by the image conversion unit 210. Next, the image inverse conversion unit 340 multiplies the two-channel image (real part image [b' E,1 ,…,b' E,K ] and the imaginary image [c' E,1 ,…,c' E,K ]) into the complex amplitude sound field image [a' E,1,…,a' E,K ] (S340) and output. For example, if a complex amplitude sound field image consists of K elements, K ], and the kth element is a k =b k +c k When j, the image inverse conversion unit 340 converts an image consisting of real parts (hereinafter also referred to as a real part image) [b1, ..., b K ] and an image consisting of the imaginary parts (hereinafter also referred to as the imaginary part image) [c1,...,c K ] and the complex amplitude sound field image [a1,…,a K ].

[0054] <Inverse Fourier Transform Unit 350> The inverse Fourier transform unit 350 converts the complex amplitude sound field image [a' E,1 ,…,a' E,K ] is input, and is subjected to an inverse Fourier transform in the time direction (S350), to obtain and output a noise-removed sound field moving image. For the inverse Fourier transform, a transform method corresponding to the Fourier transform performed by the Fourier transform unit 310 described above may be used.

[0055] <Effects> With the above configuration, noise removal from sound field moving images can be performed with higher accuracy than with conventional techniques, and good noise removal can be performed even for data with unknown properties or data containing nonlinear noise.

[0056] <Noise Removal Results> Next, an example of implementation and an example of noise removal results based on this embodiment will be described.

[0057] We used NAFNet, the same network as in Reference 1, except for the number of image channels. The network consisted of 32 blocks, with two image channels (real image, imaginary image), and an image size of 128 x 128. The loss was root mean square error (RMSE), the optimization algorithm was Adam, and the learning rate was set to 0.001. A total of 2,000 training data sets were created, with 400 sets for each number of sound sources. The training batch size was 32, and the number of epochs was 50. Additive white Gaussian noise was used as noise.

[0058] 10 shows an example of the noise removal result, which shows that the noise removal according to this embodiment significantly removes noise contained in the noisy data.

[0059] <Variation 1> This section will focus on differences from the first embodiment. In the first embodiment, the noise reduction device 300 receives a sound field moving image to be processed, removes noise from the sound field moving image (S300), and outputs the noise-removed sound field moving image. However, the processing target may be a single sound field image instead of a sound field moving image. In this case, the noise reduction device 300 receives the sound field image to be processed, removes noise from the sound field image (S300), and outputs the noise-removed sound field image. The sound field image to be processed is a single real-valued sound field image (e.g., a grayscale image). The noise reduction device 300 does not need to include the Fourier transform unit 310, the image transform unit 320, the image inverse transform unit 340, and the inverse Fourier transform unit 350; it is sufficient to include at least the noise reduction unit 330. Note that, if necessary, normalization processing (processing to adjust data to be more easily handled by a noise reduction model) is performed. The noise elimination unit 220 of the learning device 200 receives the learning input data included in the training data as input, and removes noise from the learning input data using a noise elimination model (S220), thereby obtaining and outputting the learning input data after noise removal. In this modification, the noise elimination unit 220 converts the learning input data (complex amplitude sound field image) included in the training data into a two-channel image (real part image [b T,1 ,…,b T,K ] and the imaginary image [c T,1 ,…,c T,K ]) contained in the real part image [b T,1 ,…,b T,K ] is taken as input, and the real part image [b T,1 ,…,b T,K ], and the noise is removed from the real part image [b' T,1 ,…,b' T,K] is obtained. As described above, the noise removal model is made up of a DNN, takes a sound field image (real part image) as input, and outputs a sound field image (real part image) from which noise contained in the sound field image (real part image) has been removed. However, in this modification, the noise removal model takes a normalized real part image as input, removes noise contained in the real part image, and outputs the real part image after noise removal. In this modification, a DNN is used to remove noise from a one-channel input image (real part image), so any network that removes noise from an image, as in the first embodiment, can be used.

[0060] <Update unit 230> The update unit 230 receives as input the correct answer data included in the training data and the learning input data after noise removal, and updates the parameters of the noise removal model so as to reduce the error between the real part of the correct answer data included in the training data and the learning input data (real part image) after noise removal (S230).

[0061] <Modification 2> The following mainly describes the differences from the first embodiment. In this modification, the noise removal model is a model that receives amplitude and phase corresponding to a complex amplitude sound field image as input and outputs amplitude and phase corresponding to the complex amplitude sound field image after noise removal.

[0062] <Image conversion unit 210> The image conversion unit 210 of the learning device 200 receives training data as input. The learning input data and correct answer data included in the training data are each a complex amplitude sound field image. Elements of the complex amplitude sound field image represent complex amplitudes at corresponding spatial positions and Fourier frequencies. The image conversion unit 210 converts each complex amplitude sound field image into amplitude and phase (S210). The image conversion unit 210 performs normalization processing (processing to adjust data to make it easier to handle with a noise reduction model) as necessary.

[0063] <Noise removal unit 220> The noise removal unit 220 of the learning device 200 receives as input the learning input data included in the training data, removes noise from the learning input data using a noise removal model (S220), and obtains and outputs the learning input data after noise removal.

[0064] In this modified example, the noise removal unit 220 receives as input the amplitude and phase obtained by converting the learning input data (complex amplitude sound field image) included in the training data, removes noise from the complex amplitude sound field image using a noise removal model, and obtains the amplitude and phase corresponding to the complex amplitude sound field image after noise removal.

[0065] In this modification, it is possible to use a network similar to that of the first embodiment. For example, a network with a Unet structure can be used, as in the first embodiment. It is also possible to use NAFNet.

[0066] <Update unit 230> The update unit 230 of the learning device 200 receives as input the correct answer data included in the training data and the learning input data after noise removal, and updates the parameters of the noise removal model so as to reduce the error between the correct answer data included in the training data and the learning input data after noise removal (S230).

[0067] In this modification, the parameters of the noise removal model are updated so that the error between the amplitude and phase corresponding to the ground truth data (complex amplitude sound field image) included in the training data and the amplitude and phase corresponding to the complex amplitude sound field image after noise removal is reduced. For example, the parameters can be updated by the same method as in the first embodiment.

[0068] The image conversion unit 320 of the noise reduction device 300 receives a complex amplitude sound field image for each frequency, converts the complex amplitude sound field image into amplitude and phase (S320), and outputs the converted image. For example, the image conversion unit 320 performs the same processing as the image conversion unit 210.

[0069] <Noise removal unit 330> The noise removal unit 330 of the noise removal device 300 receives as input a sound field image for the sound field moving image to be processed, removes noise from the sound field moving image to be processed using a noise removal model (S330), and obtains and outputs a sound field image for the sound field moving image to be processed after noise removal.

[0070] In this modified example, the noise removal unit 330 receives as input the amplitude and phase obtained by converting a complex amplitude sound field image for the sound field moving image to be processed, and removes noise from the complex amplitude sound field image using a noise removal model to obtain the amplitude and phase corresponding to the complex amplitude sound field image after noise removal.

[0071] <Image inverse conversion unit 340> The image inverse conversion unit 340 receives as input the amplitude and phase corresponding to the complex amplitude sound field image after noise removal, converts the amplitude and phase corresponding to the complex amplitude sound field image after noise removal into a complex amplitude sound field image (S340), and outputs it. If normalization processing has been performed in the image conversion unit 210, processing to restore the original state will be performed. <Variation 3> The following will focus on differences from the first embodiment. In this variation, the noise removal model is a DNN (complex neural network) that receives complex numbers as input, and is a model that receives a complex amplitude sound field image as input and outputs a complex amplitude sound field image after noise removal.

[0072] The learning device 200 includes a noise removal unit 220 and an update unit 230 .

[0073] <Noise removal unit 220> The noise removal unit 220 receives the learning input data included in the training data as input, removes noise from the learning input data using a noise removal model (S220), obtains and outputs the learning input data after the noise removal. Note that, if necessary, the noise removal unit 220 performs normalization processing on the learning input data (processing to make the data easier to handle with the noise removal model).

[0074] In this modified example, the noise removal unit 220 receives learning input data (complex amplitude sound field image) included in the training data as input, removes noise from the complex amplitude sound field image using a noise removal model, and obtains a complex amplitude sound field image after noise removal.

[0075] In this modification, a DNN is used to remove noise from a complex amplitude sound field image, so any complex neural network that removes noise from an image can be used.

[0076] <Update unit 230> The update unit 230 receives as input the correct answer data included in the training data and the learning input data after noise removal, and updates the parameters of the noise removal model so as to reduce the error between the correct answer data included in the training data and the learning input data after noise removal (S230).

[0077] In this modification, the parameters of the noise removal model are updated so as to reduce the error between the correct data (complex amplitude sound field image) included in the training data and the complex amplitude sound field image after noise removal.

[0078] <Noise Cancellation Device 300 > The noise cancellation device 300 includes a Fourier transform unit 310 , a noise cancellation unit 330 , and an inverse Fourier transform unit 350 .

[0079] <Noise removal unit 330> The noise removal unit 330 receives a sound field image for a sound field moving image to be processed as an input, removes noise from the sound field moving image to be processed using a noise removal model (S330), obtains a sound field image for the sound field moving image to be processed after the noise removal, and outputs it. Note that, similar to the noise removal unit 220, it performs normalization processing (processing to adjust the data to make it easier to handle with the noise removal model) on the learning input data as needed, and performs processing to restore the output of the noise removal model.

[0080] In this modified example, the noise removal unit 330 receives as input a complex amplitude sound field image for the sound field moving image to be processed, removes noise from the complex amplitude sound field image using a noise removal model, and obtains a complex amplitude sound field image after noise removal.

[0081] In any of the embodiments and modifications, the noise removal model can be said to be a model that receives information based on a complex amplitude sound field image for a sound field image as input, and outputs information based on the complex amplitude sound field image for a sound field moving image from which noise contained in the sound field image has been removed. In the first embodiment, the real part image and the imaginary part image correspond to information based on the complex amplitude sound field image, in modification 1, the real part image corresponds to information based on the complex amplitude sound field image, in modification 2, the phase and amplitude correspond to information based on the complex amplitude sound field image, and in modification 3, the complex amplitude sound field image itself corresponds to information based on the complex amplitude sound field image.

[0082] Furthermore, the image conversion unit 210 of the first embodiment may be considered as pre-processing for the noise removal unit 220, and the image conversion unit 210 may be considered as part of the noise removal unit 220. Furthermore, the image conversion unit 320 and the image inverse conversion unit 340 of the first embodiment may be considered as pre-processing and post-processing for the noise removal unit 330, respectively, and the image conversion unit 320 and the image inverse conversion unit 340 may be considered as part of the noise removal unit 330. The same applies to the second modification.

[0083] <Other Modifications> The present invention is not limited to the above-described embodiments and modifications. For example, the various processes described above may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as needed. Other modifications are possible as long as they do not deviate from the spirit of the present invention.

[0084] <Program and Recording Medium> The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in FIG. 11 and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0085] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0086] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0087] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with each program transferred from the server computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (e.g., data that is not a direct instruction to the computer but has properties that define computer processing).

[0088] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. A noise reduction device consisting of a DNN that takes a sound field image as input and outputs a sound field image with the noise contained in the sound field image removed.

2. A noise removal device according to claim 1, comprising: a Fourier transform unit that performs a time domain Fourier transform on all pixels of a sound field moving image to be processed to obtain a complex amplitude sound field image for each frequency; a noise removal unit that obtains a complex amplitude sound field image for the sound field moving image to be processed after noise removal, using a noise removal model consisting of a DNN, which receives as input information based on the complex amplitude sound field image for the sound field image and outputs information based on the complex amplitude sound field image for the sound field image from which noise contained in the sound field image has been removed; and an inverse Fourier transform unit that performs an inverse Fourier transform in the time direction on the complex amplitude sound field image for the sound field moving image to be processed after noise removal, to obtain a sound field moving image from which noise has been removed.

3. A learning device including: a noise removal unit that uses a noise removal model consisting of a DNN, which receives as input information based on a complex amplitude sound field image for a sound field image and outputs information based on the complex amplitude sound field image for a sound field image from which noise contained in the sound field image has been removed, to remove noise from a sound field image for learning input data included in training data, thereby obtaining learning input data after noise removal; and an update unit that updates parameters of the noise removal model so as to reduce an error between correct answer data included in the training data and the learning input data after noise removal.

4. A training data generating device including: a parameter generating unit that generates parameters used when generating correct answer data; and a training data generating unit that generates correct answer data by acoustic simulation using the parameters, and obtains training data consisting of a pair of correct answer data and learning input data by using the correct answer data to which any noise has been added as learning input data in accordance with a measurement system that acquires a sound field moving image, wherein the training data is composed of a DNN and is used when learning a noise removal model that receives as input information based on a complex amplitude sound field image for a sound field image and outputs information based on the complex amplitude sound field image for a sound field image from which noise contained in the sound field image has been removed.

Images