Signal generation device, signal generation method, and program

Abstract

One embodiment of the present invention is a signal generation device comprising a generation unit that, in a target signal acquisition process to obtain a target signal by performing conversion using a predetermined base for a processing object, executes a mathematical model to obtain the target signal. To the mathematical model is inputted time-dependent information that is the information indicating the time dependency of a time-dependent quantity. The mathematical model executes a plurality of feature amount extraction processes that are the plurality of processes executed in a neural network, and in which the dimensions of the data to be executed are different from each other. The feature amount extraction process is a process that, when d-dimensional data (d = integer 1 or greater) is given as the data to be executed, extracts a feature amount in each element of the d-dimensional data, and said processing object is the result of execution of the mathematical model.

Description

Signal generation device, signal generation method, and program

[0001] The present invention relates to a signal generating device, a signal generating method, and a program.

[0002] As one implementation of a signal generator that generates a target signal, a hybrid signal generator that combines a neural network and a time-frequency transform based on basis functions (see Non-Patent Document 1) has been proposed.

[0003] Unlike the signal generators configured solely with a neural network described in Non-Patent Document 2, etc., the signal generator described in Non-Patent Document 1 achieves high-speed and lightweight signal generation by configuring a part of the signal generator with time-frequency transform based on a high-speed and lightweight basis function. Examples of time-frequency transform based on a high-speed and lightweight basis function include the inverse short-time Fourier transform, the inverse wavelet transform, the inverse discrete cosine transform, and the inverse modified discrete cosine transform.

[0004] Takuhiro Kaneko, Kou Tanaka, Hirokazu Kameoka, Shogo Seki, “iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform,” in Proc. ICASSP, 2022. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,” in Adv. NeurIPS, 2020.

[0005] An example of the configuration of the signal generator described in Non-Patent Document 1 is shown in Fig. 11. This example shows a case where a two-stage processing approach that explicitly represents the intermediate representation is used, and further shows a case where a mel spectrogram is used as input data and a speech signal is used as the target signal.

[0006] In the technology described in Non-Patent Document 1, input data for time-frequency transformation based on basis functions is generated using only a neural network that processes one-dimensional data, as shown in Fig. 11. The neural network that processes one-dimensional data is specifically a one-dimensional convolutional neural network, and the time-frequency transformation based on basis functions is specifically an inverse short-time Fourier transform.

[0007] One-dimensional convolutional neural networks can operate relatively quickly because they only need to perform convolution in the time domain. However, they cannot perform transformations that focus only on local structures in the frequency domain, making it difficult to generate high-dimensional data in the frequency domain.

[0008] Therefore, in Non-Patent Document 1, significant upsampling is performed in the time direction in the neural network part, and the number of dimensions in the frequency direction is significantly reduced. In the example of Fig. 11, upsampling is performed by 64 times in the time direction, and the number of dimensions in the frequency direction is reduced from 513 dimensions to 9 dimensions.

[0009] This dimension reduction in the frequency direction is based on the following equation (1).

[0010]

[0011] iSTFT(·) represents the inverse short-time Fourier transform. ｓ represents the FFT size used in the inverse short-time Fourier transform after upsampling by s times in the time direction. ｓ represents the shift width used in the inverse short-time Fourier transform after upsampling by a factor of s in the time direction. ws represents the window width used in the inverse short-time Fourier transform after upsampling by a factor of s in the time direction.

[0012] Equation (1) is based on the tradeoff between time and frequency, and indicates that by multiplying by s in the time direction, the FFT size in the frequency direction can be multiplied by 1 / s. Note that the tradeoff between time and frequency is １ ×1 = f ｓ The relationship is ×s = constant.

[0013] However, the amount of calculation increases in proportion to the length in the time direction, so the large upsampling described above can sometimes increase the amount of calculation required to generate the target signal.

[0014] In view of the above circumstances, an object of the present invention is to provide a technique for reducing an increase in the amount of calculation required to generate a target signal.

[0015] One aspect of the present invention is a signal generation device including: a generation unit that executes a mathematical model to obtain a processing object of a target signal acquisition process that obtains a target signal, which is a target signal, by performing a transformation on a processing object using a predetermined basis; time-dependent information that is information indicating the time dependency of a time-dependent quantity is input to the mathematical model; the mathematical model executes a plurality of processes executed by a neural network, including a plurality of feature extraction processes, the feature extraction processes being for extracting features for each element of the d-dimensional data when d-dimensional (d is an integer greater than or equal to 1) data is given as the data to be executed; and the processing object is a result of execution of the mathematical model.

[0016] One aspect of the present invention is a signal generation method including a target signal acquisition process for obtaining a target signal, which is a target signal, by performing a transformation on a processing object using a predetermined basis, and a generation step of executing a mathematical model to obtain the processing object, wherein time-dependent information, which is information indicating the time dependency of a time-dependent quantity, is input to the mathematical model, and the mathematical model executes a plurality of processes executed by a neural network, including a plurality of feature extraction processes, the dimensions of which are different from each other, and the feature extraction processes are processes for extracting features for each element of the d-dimensional data when d-dimensional (d is an integer greater than or equal to 1) data is given as the data to be executed, and the processing object is a result of executing the mathematical model.

[0017] One aspect of the present invention is a program for causing a computer to function as the signal generating device described above.

[0018] According to the present invention, it is possible to reduce the increase in the amount of calculation required to generate a target signal, which is a target signal.

[0019] FIG. 1 is an explanatory diagram for explaining a signal generation system according to an embodiment. FIG. 2 is a diagram showing an example of a trained processing object acquisition model according to an embodiment. FIG. 3 is a diagram showing a first example of the configuration of 2D Blocks according to an embodiment. FIG. 4 is a diagram showing a second example of the configuration of 2D Blocks according to an embodiment. FIG. 5 is a diagram showing an example of a more specific hardware configuration of a learning device according to an embodiment. FIG. 6 is a diagram showing an example of a flow of processing executed by the learning device according to an embodiment. FIG. 7 is a diagram showing an example of a more specific hardware configuration of a signal generation device according to an embodiment. FIG. 8 is a diagram showing an example of a flow of processing executed by the signal generation device according to an embodiment. FIG. 9 is a diagram showing an example of the results of a first experiment according to an embodiment. FIG. 10 is a diagram showing an example of the results of a second experiment in a modified example. FIG. 11 is an explanatory diagram of the prior art.

[0020] 1 is an explanatory diagram illustrating a signal generation system 100 according to an embodiment. The signal generation system 100 includes a learning device 1 and a signal generation device 2. The signal generation device 2 generates a signal using a trained mathematical model obtained by the learning device 1.

[0021] The learning device 1 includes a control unit 11 including a processor 91, such as a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU), and a memory 92, connected via a bus. The control unit 11 executes a learning process, which is a process for learning a learning object. The learning object is a mathematical model (hereinafter referred to as a "processing object acquisition model") for a target signal acquisition process, which is a process for obtaining a target signal by performing a transformation on the processing object using a predetermined basis.

[0022] The transformation using a predetermined basis (hereinafter referred to as "analysis transformation") may be any transformation using a predetermined basis, such as an inverse short-time Fourier transform, an inverse wavelet transform, an inverse discrete cosine transform, or an inverse modified discrete cosine transform.

[0023] Time-dependent information is input to the processing object acquisition model. The time-dependent information is information that indicates the time dependency of a quantity that depends on time. Therefore, the time-dependent information is, for example, a time series. As will be described in detail later, the time series may be, for example, an audio signal, an acoustic signal, or text.

[0024] The processing object acquisition model executes a first process, a dimension increase process, and a second process. The first process is a process executed by a neural network, and is a process for extracting features from one-dimensional data.

[0025] The feature extraction process is a process executed by a neural network, and when d-dimensional (d is an integer equal to or greater than 1) data is given as the data to be executed, feature extraction is performed for each element of the d-dimensional data. Feature extraction is a process for extracting features. Therefore, the first process is a feature extraction process with d=1.

[0026] When extracting features, upsampling or downsampling may be performed one or more times in a predetermined dimension. In this case, the feature extraction process uses the feature of each element of the d-dimensional data after upsampling or downsampling as the final output.

[0027] Furthermore, when extracting features, a convolutional neural network, a recurrent neural network, or the like may be used to extract features for each element based on information about neighboring elements. More specifically, the first process is the process when d is 1 in the feature extraction process. That is, it is a process executed by a neural network, and extracts features for each element (each time) in the time direction from the time-dependent information input to the processing target acquisition model.

[0028] When extracting features, upsampling or downsampling may be performed one or more times in the time direction. When upsampling is performed, the increase in the amount of calculation in the time direction caused by the upsampling can be reduced by reducing the upsampling rate compared to the technique described in Non-Patent Document 1. The dimension increase process is a process of increasing the dimension of the result of the first process. The increased dimension is, for example, the frequency dimension.

[0029] The second process is a process executed by a neural network, which extracts features from two-dimensional data. Specifically, the second process is a process when d is 2 in the feature extraction process. More specifically, the second process is a process executed by a neural network, which extracts features for each element in the time direction (each time) and each element of the dimension obtained by the dimension increase process (each frequency, if the added dimension is a frequency dimension). Note that, in the second process, when extracting features, one or more upsampling processes or one or more downsampling processes may be performed on the dimension obtained by the dimension increase process. Note that, when upsampling processes are performed, the number of elements of the dimension obtained by the dimension increase process can be reduced in the part before upsampling, thereby reducing the increase in the amount of calculations. The result of the second process is the target of the target signal acquisition process.

[0030] The first process is performed by, for example, a convolutional neural network (CNN). For example, the neural network that performs the first process may be a one-dimensional convolutional neural network. The first process does not necessarily have to be performed by a convolutional neural network, but may be performed by a recurrent neural network (RNN), a transformer network, an attention network, a fully connected neural network (FNN), or a network that combines these.

[0031] The second process is performed by, for example, a convolutional neural network (CNN). For example, the neural network that performs the second process may be a two-dimensional convolutional neural network. The second process does not necessarily have to be performed by a convolutional neural network, but may be performed by a recurrent neural network (RNN), a transformer network, an attention network, a fully connected neural network (FNN), or a network that combines these.

[0032] For simplicity of explanation, the following description will be given taking as an example a case where upsampling is performed in the time direction in the first process and upsampling is performed in the non-time direction in the second process. However, as described in a modified example below, it is not necessarily the case that upsampling is performed in the time direction in the first process and upsampling is performed in the non-time direction in the second process.

[0033] The time-dependent information may be, for example, an audio signal, an acoustic signal, text, a multidimensional signal such as an image or video, a signal acquired using a sensor, or a combination of these. Furthermore, the time-dependent information may be, for example, an intermediate representation, which is a predetermined signal obtained by transforming an audio signal, an acoustic signal, text, or a multidimensional signal using a predetermined function. Furthermore, the time-dependent information may be a combination of intermediate representations. In this specification, "time" is a term that represents a certain dimension of data. For example, in the case of a still image, the vertical dimension or the horizontal dimension of the still image is treated as the "time dimension." In the case of a video, the vertical dimension or the horizontal dimension of the image, or the so-called time dimension, is treated as the "time dimension."

[0034] The intermediate representation is, for example, the result of performing an analytical transformation on the time-dependent information. If the time-dependent information is an audio signal, the result of performing the analytical transformation on the time-dependent information is, for example, time-dependent information indicating features such as a spectrogram or a logarithmic spectrogram. If the time-dependent information is an audio signal, the result of performing the analytical transformation on the time-dependent information is, for example, the result of performing a further scale transformation on the time-dependent information indicating features such as a spectrogram. If the time-dependent information is, for example, the result of performing a further scale transformation on the time-dependent information indicating features such as a spectrogram, the result is, for example, a mel spectrogram or a logarithmic mel spectrogram.

[0035] If the time-dependent information is an audio signal, the result of performing the analytical transformation on the time-dependent information may be, for example, time-dependent information indicative of features obtained by performing the analytical transformation on the result of performing the analytical transformation on the time-dependent information. The time-dependent information indicative of features obtained by performing the analytical transformation on the result of performing the analytical transformation on the time-dependent information is, for example, a cepstrum.

[0036] When the time-dependent information is an audio signal, the result of performing an analytical transformation on the time-dependent information may be, for example, time-dependent information indicating features obtained by performing a further analytical transformation on time-dependent information indicating features obtained by performing a scale transformation on the result of performing the analytical transformation on the time-dependent information. The time-dependent information indicating features obtained by performing a further analytical transformation on time-dependent information indicating features obtained by performing a scale transformation on the result of performing the analytical transformation on the time-dependent information may be, for example, a mel-cepstrum. Thus, when the time-dependent information is an audio signal, the result of performing an analytical transformation on the time-dependent information may be, for example, time-dependent information indicating features obtained by performing a predetermined type of analytical transformation (including a scale transformation) on the time-dependent information a predetermined number of times.

[0037] The intermediate representation may be, for example, a feature obtained from time-dependent information such as a fundamental frequency or an aperiodicity index. The intermediate representation may be, for example, time-dependent information indicating a feature obtained by applying a predetermined function, such as a function expressed by a neural network, to the time-dependent information. The intermediate representation may be, for example, time-dependent information indicating a feature obtained by applying a predetermined function, such as a function expressed by a neural network or a signal processing function, to time-dependent information indicating the feature obtained based on the time-dependent information. Furthermore, the intermediate representation may be a combination of the various types of time-dependent information indicating a feature exemplified above.

[0038] The neural network that obtains the intermediate representation may obtain time-dependent information with a scale reduced in the time direction compared to the target signal as the intermediate representation. In this case, the scale reduced in the time direction reduces the amount of calculation required to obtain the intermediate representation compared to when the scale is not reduced.

[0039] The target signal may be, for example, an audio signal, an acoustic signal, text, a multidimensional signal such as an image or video, a signal acquired using a sensor, or a combination of these.

[0040] In the learning process, the learning object is learned until a predetermined condition for terminating learning (hereinafter referred to as the "learning termination condition") is satisfied. The learning termination condition may be any condition for terminating learning, and may be, for example, a condition that the learning object has been updated a predetermined number of times. The learning termination condition may be, for example, a condition that the change in the output of the learning object due to the update is smaller than a predetermined change.

[0041] Generally, a mathematical model at the time when the learning termination condition is satisfied is called a trained mathematical model. Therefore, hereinafter, a processing object acquisition model at the time when the learning termination condition is satisfied is called a trained processing object acquisition model.

[0042] In the learning process, the learning target may be learned using any learning algorithm as long as it can learn the learning target. Therefore, the learning algorithm executed in the learning process may be, for example, a generative model such as a generative adversarial network (GAN), an autoregressive model, a flow model, a diffusion model, or a variational autoencoder (VAE), or a combination thereof.

[0043] FIG. 2 is a diagram showing an example of a trained processing object acquisition model in an embodiment. As described above, the information input to the processing object acquisition model may be time-dependent information other than intermediate representations such as speech signals, acoustic signals, and text. However, for simplicity, FIG. 2 illustrates an example in which an intermediate representation is input as time-dependent information. More specifically, FIG. 2 illustrates an example in which a logarithmic mel spectrogram is used as the time-dependent information input to the processing object acquisition model. Furthermore, FIG. 2 illustrates an example in which a speech signal is obtained as the target signal.

[0044] 2 shows that the processing object acquisition model is realized by multiple neural networks that process data with different dimensions. Specifically, in the example of Fig. 2, the processing object acquisition model is expressed by a neural network that executes a first process (hereinafter referred to as the "first network") and a neural network that executes a second process (hereinafter referred to as the "second network").

[0045] FIG. 2 indicates that a dimension increase process is performed between the first process and the second process. The dimension increase process in FIG. 2 is the same as the process performed immediately after "Output 1D Conv" in FIG. 11. Specifically, this process transforms the shape of the data. If the number of channels before the dimension increase process is C' x F', the dimension increase process transforms the data so that the number of elements in the increasing dimension (e.g., frequency dimension) becomes F' instead of changing the number of channels to C'. By performing this process, a fixed number of elements (F') can be assigned to the increasing dimension. Here, C' and F' are both integers greater than or equal to 1. The first network is a neural network that processes one-dimensional data, and the second network is a neural network that processes two-dimensional data.

[0046] In FIG. 2, "Conv" indicates a convolutional neural network. "ConvT" indicates a transposed convolutional neural network. "ResBlock" indicates a residual neural network block. "iSTFT" indicates an inverse short-time Fourier transform. Note that "2D" means two dimensions, and "1D" means one dimension.

[0047] In FIG. 2, "Time" indicates the length in the time direction. For example, "8T" means that the length in the time direction is eight times "T". The longer the length in the time direction, the more upsampling there is. Therefore, FIG. 2 shows that upsampling in the time direction is performed by the first network.

[0048] "Frequency" represents the number of dimensions in the frequency direction of the spectrogram to which the inverse short-time Fourier transform is applied. For example, "F ２ " indicates that the number of dimensions in the frequency direction is "F２ 2. This means that the number of dimensions is eight times larger than the number of dimensions in the frequency domain. As the number of dimensions increases, the degree of upsampling increases. Therefore, FIG. 2 shows that upsampling in the frequency domain is performed by the second network.

[0049] "Channel" indicates the number of channels. For example, "C / 2" means that the number of channels is half of "C". "Kernel size" means the size of the convolution kernel. The number of channels indicates the number of features. FIG. 2 shows that the first network extracts features at each time, and the second network extracts features at each time and each frequency.

[0050] "Tx32 iSTFT" in FIG. 2 means an inverse short-time Fourier transform that performs 32-fold upsampling in the time direction, and is an example of target signal acquisition processing.

[0051] <Effects of the Processing Object Acquisition Model> In this way, the processing object acquisition model is realized by a first network and a second network. In the second network, it is possible to perform a transformation that focuses on a local structure in a direction different from the time direction (hereinafter referred to as the "non-time direction"), and therefore it is possible to generate high-dimensional data in the non-time direction. The non-time direction is, for example, the frequency direction. The non-time direction is the dimension obtained by the dimension increase process. Note that a transformation that focuses on a local structure means a transformation that uses information indicating the local structure.

[0052] Therefore, the processing object acquisition model can generate higher-dimensional data in the non-time direction than the technology described in Non-Patent Document 1, and therefore can reduce upsampling in the time direction by a neural network compared to the technology described in Non-Patent Document 1. Specifically, the technology described in Non-Patent Document 1 shown in the example of Figure 11 was limited to generating 9-dimensional data, but the processing object acquisition model shown in the example of Figure 2 can generate 65-dimensional data. As a result, with the processing object acquisition model shown in the example of Figure 2, upsampling up to 8 times in the time direction is sufficient, and there is no need to perform upsampling up to 64 times in the time direction as with the technology described in Non-Patent Document 1 shown in the example of Figure 11.

[0053] As will be described in a modified example later, the processing object acquisition model does not necessarily have to be trained. However, if the parameters of the processing object acquisition model are not optimal for the training data, training allows for optimization based on the training data compared to not training, thereby improving the quality of generating a signal that is closer to the training data.

[0054] The amount of calculation increases in proportion to the length in the time direction. Therefore, the processing object acquisition model described in Non-Patent Document 1, which can reduce the significant upsampling, can reduce the increase in the amount of calculation required to generate the target signal.

[0055] In the example of FIG. 2 , in the second network, data reduced to 1 / 8 in the non-time direction is subjected to processing by “Input 2D Conv” and “2D Blocks,” and then upsampling by 8 times in the non-time direction is performed using three “2D ConvTs.” The “Input 2D Conv” is a two-dimensional convolutional neural network including a two-dimensional convolutional layer, and is applied to the result obtained by the dimensionality increase processing. The “2D Blocks” is a two-dimensional convolutional neural network including one or more two-dimensional convolutional layers, and is applied to the result obtained by the “Input 2D Conv” processing. Each “2D ConvT” performs upsampling by 2 times in the non-time direction. The upsampling in the non-time direction of the second network does not necessarily have to be performed in this manner.

[0056] However, such processing can mitigate the increase in the amount of calculation in the non-time direction. In fact, the amount of calculation increases in proportion to the number of dimensions in the non-time direction. Therefore, performing processing using data with a small number of dimensions in the non-time direction and then upsampling in the non-time direction can reduce the amount of calculation compared to upsampling without processing using data with a small number of dimensions in the non-time direction.

[0057] The second network in the example of Fig. 2 performs this. That is, the second network in the example of Fig. 2 performs a predetermined process on data with a small number of dimensions in the non-time direction, and then performs upsampling in the non-time direction. Note that the predetermined process is, for example, a process of applying a two-dimensional convolutional neural network (e.g., "Input 2D Conv" or "2D Blocks") that does not change the size in the non-time direction.

[0058] FIG. 3 is a diagram showing a first example of the configuration of 2D Blocks in an embodiment. In the "2D Blocks" in FIG. 2, nonlinear transformation processing and convolution processing are performed. The nonlinear transformation processing is performed using, for example, a Leaky Rectified Unit (LReLU). The convolution processing is performed using, for example, Conv. Note that "x×y Conv z" represents two-dimensional convolution processing with a kernel size of x×y and z output channels. Therefore, for example, "3×3 Conv 2C" represents two-dimensional convolution processing with a kernel size of 3×3 and 2C output channels.

[0059] 3 shows that the data input to "2D Blocks" is added to the output of the neural network processing of "2D Blocks." In this case, the neural network processing of "2D Blocks" in FIG. 3 is a process for obtaining a residual.

[0060] In addition, "x3" in FIG. 3 means that three neural networks with the same structure are connected in series.

[0061] FIG. 4 is a diagram showing a second example of the configuration of 2D Blocks in an embodiment. FIG. 4 shows a process in which, for features of data input to "2D Blocks" in FIG. 2, the features are shuffled in the channel direction, the features are divided in the channel direction, neural network processing is performed on only one of the divided parts, and finally the features are combined in the channel direction. In FIG. 4, "Channel Shuffle" represents the process of shuffling features in the channel direction. "Channel Split" represents the process of splitting features in the channel direction. "Channel Concat" represents the process of combining features in the channel direction.

[0062] The neural network processing executed on one of the divided parts is the processing of "LReLU", "3x3 Conv 2C", "LReLU", and "3x3 Conv C / 2" in FIG. 4 executed in order.

[0063] In the case of Fig. 4, the number of feature channels is the same as that of the first example of the 2D Blocks configuration. However, since the neural network is applied to only half (i.e., only one of the above), it is possible to reduce the number of model parameters and the amount of calculation while maintaining a higher expressive power than that of the first example of the 2D Blocks configuration.

[0064] In addition, "x3" in FIG. 4 means that three neural networks with the same structure are connected in series.

[0065] <Learning Example Using GAN> Here, an example of the learning process will be described using a GAN (Generative Adversarial Network) as the learning algorithm. For simplicity, the following description will be given using an example in which the time-dependent information input to the processing target acquisition model is an audio signal.

[0066] In GAN, optimization is performed using an adversarial loss function, where G denotes a signal generator and D denotes a signal discriminator that discriminates whether an input signal is a generated signal. The signal generator G is a mathematical model that executes a processing target acquisition model and a target signal acquisition process. The adversarial loss function is expressed, for example, by the following equations (2) and (3).

[0067]

[0068]

[0069] s represents the input data of the signal generator. For example, in the case of speech signal generation, s represents either the text or speech signal, or the intermediate representation of the target signal (hereinafter referred to as the "target intermediate representation"). x represents the correct data of the target signal of the signal generator (hereinafter referred to as the "real signal"). For example, in the case of speech signal generation, x represents the correct data of the speech signal. In learning, one or more paired data (x, s) of x and s are used as training data.

[0070] Signal discriminator D distinguishes between real signal x and generated signal G(s), which is a signal generated by a signal generator, by minimizing equation (2). Meanwhile, signal generator G makes generated signal G(s) indistinguishable from real signal x by minimizing equation (3). In this way, by optimizing under conditions where D, which attempts to separate the real signal from the generated signal, and G, which attempts to bring the real signal and the generated signal closer together, compete with each other, G becomes able to generate a signal that D cannot distinguish between a real signal and a generated signal.

[0071] The adversarial loss functions shown in Equations (2) and (3) are based on Least Squares GAN (LSGAN). However, the adversarial loss function used for training is not limited to this. The adversarial loss function may be based on, for example, Wasserstein GAN, Non-saturating GAN, Hinge GAN, or any combination thereof.

[0072] In addition to the adversarial loss function, an intermediate representation matching loss function may also be used in the training. The intermediate representation matching loss function is a loss function that represents the difference between the real signal and the generated signal in the feature space of the intermediate representation. By using the intermediate representation matching loss function, training becomes more stable than when the intermediate representation matching loss function is not used. The intermediate representation matching loss function is expressed, for example, by the following equation (4).

[0073]

[0074] φ represents a function for extracting an intermediate representation from a signal. Equation (4) is an example of expressing the intermediate representation matching loss function using the L1 distance, but the intermediate representation matching loss function may be expressed by a distance other than the L1 distance. The distance may be, for example, the L2 distance, the Lp distance, the Wasserstein distance, the Hinge function, or any other measure such as a combination thereof.

[0075] By using the intermediate representation matching loss function, it is possible to bring the generated signal G(s) and the real signal x closer together in the feature space of the intermediate representation. As a result, as described above, by using the intermediate representation matching loss function, learning becomes more stable than when the intermediate representation matching loss function is not used.

[0076] Note that, when an intermediate representation is used as an input to the signal generator, the extraction function φ of the intermediate representation may be the same as or different from the function used to extract the intermediate representation. Taking an audio signal as an example, the extraction function φ of the intermediate representation may be a function that acquires, for example, a feature obtained by applying a time-frequency transformation based on a basis function, such as a short-time Fourier transform, a wavelet transform, a discrete cosine transform, or a modified discrete cosine transform, to the audio signal, a feature obtained by further scaling the feature, or a feature obtained by further applying a time-frequency transformation based on a basis function, such as a Fourier transform, to a spectrogram, a mel spectrogram, or the like.

[0077] Note that features obtained by applying a time-frequency transform based on a basis function, such as a short-time Fourier transform, a wavelet transform, a discrete cosine transform, or a modified discrete cosine transform, to an audio signal are, for example, a spectrogram or a logarithmic spectrogram. Features obtained by further scaling features obtained by applying a time-frequency transform based on a basis function, such as a short-time Fourier transform, a wavelet transform, a discrete cosine transform, or a modified discrete cosine transform, to an audio signal are, for example, a mel spectrogram or a logarithmic mel spectrogram. Features obtained by further applying a time-frequency transform based on a basis function, such as a Fourier transform, to a spectrogram or a mel spectrogram are, for example, a cepstrum or a mel cepstrum.

[0078] Furthermore, the extraction function φ of the intermediate representation may be a function that acquires a feature extracted from the audio signal, such as a fundamental frequency or an aperiodicity index. The extraction function φ of the intermediate representation may be a function that acquires a feature obtained by applying a predetermined function, such as a function expressed by a neural network or a signal processing function, to the audio signal or its feature. The extraction function φ of the intermediate representation may be a function that acquires a feature that is a combination of these feature quantities.

[0079] In addition to the adversarial loss function and the intermediate representation matching loss function, a feature-matching loss function may also be used in the training. The feature-matching loss function is a loss function that represents the difference between the real signal and the generated signal in the feature space of the signal classifier. By using the feature-matching loss function, training becomes more stable than when the feature-matching loss function is not used. The feature-matching loss function is expressed, for example, by the following formula (5).

[0080]

[0081] T represents the number of layers of the signal discriminator D. ｉ represents the feature quantity of the i-th layer of the signal classifier D. ｉ represents the number of features in the i-th layer of the signal classifier D. The distance used to express the feature adaptation loss function is not limited to the L1 distance. The distance may be, for example, the L2 distance, the Lp distance, the Wasserstein distance, the Hinge function, or any other measure such as a combination of these.

[0082] Equation (5) includes information on the features of all layers of the signal classifier D. However, it is not necessary to include information on the features of all layers, and it may include information on the features of only some layers.

[0083] By using the feature adaptation loss function, it is possible to bring the generated signal G(s) and the real signal x closer together in the feature space of the signal classifier. As a result, as described above, using the feature adaptation loss function makes learning more stable than when the feature adaptation loss function is not used.

[0084] In the learning process, a function that combines an adversarial loss function, an intermediate representation matching loss function, and a feature matching loss function with predetermined weights may be used as the loss function. Therefore, in the learning process, for example, the functions expressed by the following formulas (6) and (7) may be used as the loss functions. Note that the predetermined weights may be constant values ​​or may be dynamically changed during learning.

[0085]

[0086]

[0087] λ ａｄｖ is the adversarial loss function L ａｄｖ is the weighting coefficient of λ ｉｍ is the intermediate representation matching loss function L ｉｍ is the weighting coefficient of λ ｆｍ is the feature adaptation loss function L ｆｍ G is the weighting coefficient of the loss function L Ｇ It is optimized by minimizing the loss function L Ｄ It is to be noted that in the training, it is not necessary to use all three of the adversarial loss function, intermediate representation matching loss function, and feature matching loss function, and only some of them may be used.

[0088] Furthermore, although the above describes an example in which one signal discriminator is used, multiple signal discriminators may be provided, and equations (2), (3), and (5) may be used for each of them. When multiple signal discriminators are provided, each discriminator may be configured to discriminate signals using a different scale. Discriminating signals using a different scale means discriminating signals based on, for example, different scales, different periods, or different time-frequency transformations.

[0089] The learning may be performed up to the target signal acquisition process and the mathematical model to be learned may be updated based on the output of the target signal acquisition process, or the mathematical model to be learned may be updated based on the signal input to the target signal acquisition process.

[0090] Furthermore, the processing target acquisition model may be a mathematical model that executes a two-stage processing approach that explicitly represents an intermediate representation, or may be a mathematical model that executes a consistent approach that does not explicitly represent an intermediate representation. Explicitly representing an intermediate representation means outputting the intermediate representation to a predetermined output destination such as the interface unit 12 or the storage unit 13, which will be described later.

[0091] For example, in the case of a consistent approach to generating speech from text, the conversion from input text to a target signal is performed without explicitly representing an intermediate representation, and in the case of speech conversion, the conversion from input speech signal to a target signal is performed without explicitly representing an intermediate representation.

[0092] 5 is a diagram showing an example of a more specific hardware configuration of the learning device 1 according to an embodiment. The learning device 1 includes a control unit 11 including a processor 91 such as a CPU, GPU, or NPU, and a memory 92, all connected via a bus as described above, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0093] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the learning device 1 to function as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0094] The control unit 11 controls the operation of each functional unit included in the learning device 1. The control unit 11 executes, for example, a learning process. In the learning process, the control unit 11 executes, for example, a processing object acquisition model. In the learning process, the control unit 11 may execute, for example, a target signal acquisition process. The control unit 11 controls, for example, the operation of the interface unit 12. The control unit 11 acquires data to be used for learning, for example, via the interface unit 12. The control unit 11 acquires, for example, information stored in the memory unit 13. The process of acquiring information stored in the memory unit 13 is specifically reading.

[0095] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits data used for learning. In such a case, the interface unit 12 acquires the data used for learning by communicating with the device that transmits the data used for learning.

[0096] The external device is, for example, the signal generating device 2. In such a case, the interface unit 12 transmits the trained processing object acquisition model to the signal generating device 2 through communication with the signal generating device 2. The signal generating device 2 generates a signal using the acquired trained processing object acquisition model.

[0097] The interface unit 12 may be configured to include input devices such as a mouse, keyboard, touch panel, microphone, etc. The interface unit 12 may be configured as an interface that connects these input devices to the study device 1. In this way, the interface unit 12 accepts input of various information to the study device 1 via the input device, either wired or wireless.

[0098] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 12 may be configured as an interface that connects these display devices to the learning device 1. The interface unit 12 may include, for example, a sound output device such as a speaker. The interface unit 12 may be configured as an interface that connects these sound output devices to the learning device 1. The interface unit 12 outputs, for example, information input to the interface unit 12.

[0099] The storage unit 13 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 stores information acquired by the interface unit 12, for example.

[0100] 6 is a diagram showing an example of the flow of processing executed by the learning device 1 in the embodiment. As an example, a case will be described in which a target signal acquisition process is also executed in the learning process, and the processing object acquisition model is updated based on the result of the target signal acquisition process. However, the target signal acquisition process does not necessarily have to be executed in learning the processing object acquisition model. If the target signal acquisition process is not executed in the learning process, the processing object acquisition model is updated based on the result of the second process.

[0101] Also, FIG. 6 shows an example in which the time-dependent information acquired via the interface unit 12 is not input directly to the processing object acquisition model, but is converted into time-dependent information of a different format, such as converting audio waveform data into a mel spectrogram, and then input to the processing object acquisition model.

[0102] However, such conversion into time-dependent information of a different format does not necessarily have to be performed, and such conversion into time-dependent information of a different format may have been performed by a device other than learning device 1, and the time-dependent information input to interface unit 12 may be the time-dependent information after such conversion.

[0103] Time-dependent information is input to the interface unit 12, and the control unit 11 acquires the time-dependent information (step S101). Next, the control unit 11 converts the time-dependent information acquired in step S101 into time-dependent information of a different format (step S102). Next, the control unit 11 executes a first process (step S103). Next, the control unit 11 executes a dimension increase process (step S104). Next, the control unit 11 executes a second process (step S105). Note that the processes of steps S103 to S105 are an example of a process for executing a processing target acquisition model. After step S105, the control unit 11 executes a target signal acquisition process (step S106).

[0104] Next, the control unit 11 determines whether the learning termination condition is satisfied (step S107). If the learning termination condition is satisfied (step S107: YES), the processing ends. On the other hand, if the learning termination condition is not satisfied (step S107: NO), the processing object acquisition model is updated according to a predetermined learning algorithm (step S108). The predetermined learning algorithm is, for example, the algorithm exemplified above, such as GAN. After step S108, the processing returns to step S101.

[0105] 7 is a diagram showing an example of a more specific hardware configuration of the signal generation device 2 in the embodiment. The signal generation device 2 includes a control unit 21 including a processor 93 such as a CPU, GPU, or NPU, and a memory 94, which are connected via a bus, and executes a program. By executing the program, the signal generation device 2 functions as a device including the control unit 21, an interface unit 22, and a storage unit 23.

[0106] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. The processor 93 executes the program stored in the memory 94, causing the signal generating device 2 to function as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0107] The control unit 21 controls the operation of each functional unit included in the signal generating device 2. The control unit 21, for example, executes a trained processing target acquisition model. The control unit 21, for example, executes a target signal acquisition process.

[0108] The control unit 21 controls, for example, the operation of the interface unit 22. The control unit 21 acquires, for example, time-dependent information via the interface unit 22. The control unit 21 acquires, for example, information stored in the memory unit 23. Specifically, the process of acquiring the information stored in the memory unit 23 is reading.

[0109] The interface unit 22 includes a communication interface for connecting the signal generating device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless communication. The external device is, for example, the learning device 1. In this case, the interface unit 22 acquires the trained processing object acquisition model obtained by the learning device 1 through communication with the learning device 1. The external device is, for example, a device that transmits time-dependent information. In this case, the interface unit 22 acquires the time-dependent information through communication with the device that transmits the time-dependent information.

[0110] The interface unit 22 may be configured to include input devices such as a mouse, a keyboard, a touch panel, a microphone, etc. The interface unit 22 may be configured as an interface that connects these input devices to the signal generating device 2. In this way, the interface unit 22 accepts input of various information to the signal generating device 2 via the input devices, either wired or wireless.

[0111] The interface unit 22 outputs, for example, various types of information. The interface unit 22 is configured to include a display device such as a CRT display, a liquid crystal display, or an organic EL display. The interface unit 22 may be configured as an interface that connects these display devices to the signal generating device 2. The interface unit 22 may be configured to include, for example, a sound output device such as a speaker. The interface unit 22 may be configured as an interface that connects these sound output devices to the signal generating device 2. The interface unit 22 outputs, for example, information input to the interface unit 22.

[0112] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the signal generating device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 stores information acquired by the interface unit 22, for example.

[0113] 8 is a diagram showing an example of the flow of processing executed by the signal generating device 2 in the embodiment. As an example, Fig. 8 shows an example in which the time-dependent information acquired via the interface unit 22 is not input directly to the trained processing object acquisition model, but is converted into time-dependent information of a different format, such as conversion from audio waveform data to a mel spectrogram, and then input to the trained processing object acquisition model.

[0114] However, such conversion into time-dependent information of a different format does not necessarily have to be performed. Alternatively, such conversion into time-dependent information of a different format may have been performed by a device other than the signal generating device 2, and the time-dependent information input to the interface unit 22 may be the time-dependent information after such conversion.

[0115] For simplicity, in FIG. 8, the signal generating device 2 will be described taking as an example a case where a pre-trained processing object acquisition model has been acquired.

[0116] Time-dependent information is input to the interface unit 22, and the control unit 21 acquires the time-dependent information (step S201). Next, the control unit 21 converts the time-dependent information acquired in step S201 into time-dependent information of a different format (step S202). Note that the time-dependent information after conversion in step S202 is time-dependent information of the format input to the processing object acquisition model in the learning process.

[0117] Next, the control unit 21 executes the learned processing target acquisition model (step S203), executes target signal acquisition processing (step S204), and outputs the target signal obtained by the target signal acquisition processing to a predetermined output destination (step S205).

[0118] <First Experimental Results> Here, an example of the first experiment using the signal generation system 100 and its results will be shown. In the first experiment, approximately 24 hours of speech signals from a female speaker were used as training data. The sampling frequency of the speech signals was 22.05 kHz. As the intermediate representation, an 80-dimensional logarithmic mel spectrogram was used, obtained by applying a short-time Fourier transform with an FFT size of 1024, a shift width of 256, and a window width of 1024 to the speech signals. In the first experiment, a signal generator that generates speech signals from the intermediate representation was trained using a GAN.

[0119] In the first experiment, the technology described in Non-Patent Document 1 was used as a technology to be compared with signal generation system 100. Hereinafter, the technology to be compared used in the first experiment will be referred to as the first comparison technology. Hereinafter, the signal generation system 100 in which the "2D Blocks" in FIG. 2 are those in FIG. 3 will be referred to as invention method 1. Hereinafter, the signal generation system 100 in which the "2D Blocks" in FIG. 2 are those in FIG. 4 will be referred to as invention method 2.

[0120] In the first experiment, the voice quality was evaluated by subjective evaluation. Specifically, the subjective evaluation was a Mean Opinion Score (MOS) test. The MOS evaluation is a five-point scale, with 5 being the best and 1 being the worst. Therefore, the higher the value, the better the voice quality.

[0121] In the first experiment, the processing speed was evaluated by the Real Time Factor (RTF). RTF is the time required to generate a signal divided by the length of the generated signal. RTF was measured using a CPU. A smaller RTF value indicates a faster processing speed.

[0122] In the first experiment, the model size was evaluated by calculating the number of parameters of the neural network included in the signal generator. A smaller value indicates a smaller model size.

[0123] FIG. 9 is a diagram showing an example of the results of the first experiment in the embodiment. FIG. 9 shows that in terms of voice quality (MOS), inventive methods 1 and 2 are better than the first comparison technology. FIG. 9 shows that in terms of processing speed, inventive methods 1 and 2 are faster than the first comparison technology. FIG. 9 shows that in terms of the number of parameters, inventive methods 1 and 2 are smaller than the first comparison technology. Thus, FIG. 9 shows that inventive methods 1 and 2, compared to the first comparison technology, faster processing speed and a reduced number of parameters are achieved without sacrificing voice quality.

[0124] The learning device 1 configured in this manner learns a processing object acquisition model, which is a mathematical model that executes not only the first process but also the second process. Therefore, by using the learned processing object acquisition model obtained by the learning device 1, it is possible to reduce the increase in the amount of calculation required to generate a target signal, as described above in <Effects of the processing object acquisition model>.

[0125] Furthermore, the signal generating device 2 configured in this manner obtains a target signal by executing the learned processing object acquisition model obtained by the learning device 1. Therefore, the signal generating device 2 can reduce an increase in the amount of calculation required to generate a target signal.

[0126] Furthermore, the signal generation system 100 configured in this manner includes the learning device 1. Therefore, the signal generation system 100 can reduce an increase in the amount of calculation required to generate a target signal.

[0127] (Variant) In the example of Figure 2, the first process is described as performing upsampling in the time direction, and the second process is described as performing upsampling in the non-time direction, but it is also possible not to perform upsampling in either the first process or the second process, or in both.

[0128] If the first process does not involve upsampling in the time direction, there is no need for upsampling up to 64 times as in the technology described in non-patent document 1, which reduces the amount of calculation required, but it becomes necessary to generate high-dimensional data in the non-time direction.

[0129] As described in "Effects of the Processing Target Acquisition Model," the technology of Non-Patent Document 1 cannot perform conversion focusing on local structures in the non-time direction. Therefore, it is not possible to generate high-dimensional data in the non-time direction, making it difficult to handle such cases.

[0130] On the other hand, the second network of the processing target acquisition model can perform transformations that focus on local structures in the non-time direction, making it possible to generate high-dimensional data in the non-time direction and respond to such cases.

[0131] In this way, compared to the technology of Non-Patent Document 1, the processing object acquisition model can generate high-dimensional data in the non-time direction, and is therefore effective in reducing the amount of calculation by reducing or eliminating upsampling in the time direction.

[0132] If upsampling in the non-time direction is not performed in the second process, the amount of calculation increases compared to when processing data with a small number of dimensions in the non-time direction and then upsampling in the non-time direction, as shown in the example of Figure 2. However, even in this case, the processing target acquisition model is effective for generating high-dimensional data in the non-time direction. Therefore, compared to the technology described in Non-Patent Document 1, this is effective in reducing the amount of calculation by reducing or eliminating upsampling in the time direction.

[0133] Note that the signal generation system 100 does not necessarily have to include the learning device 1, and the signal generation device 2 does not necessarily have to execute the trained processing object acquisition model obtained by the learning device 1 as a mathematical model for acquiring the processing object of the target signal acquisition processing. The signal generation device 2 only needs to execute the processing object acquisition model and perform the target signal acquisition processing on the processing object obtained by executing the processing object acquisition model, and it does not matter whether the processing object acquisition model to be executed is trained or not. Even in such a case, the signal generation device 2 can reduce the increase in the amount of calculation required to generate the target signal because it has the effects described in the above-mentioned <Effects of the processing object acquisition model>.

[0134] The target signal acquisition process does not necessarily have to be an inverse short-time Fourier transform, but may be other signal processing. The target signal acquisition process may be, for example, multiband modeling. For example, the multiband modeling may be a process of generating subband signals and then executing pseudo-quadrature mirror filters (PQMF).

[0135] The target signal acquisition process may be, for example, a process that combines an inverse short-time Fourier transform and multiband modeling. When the target signal acquisition process is a process that combines an inverse short-time Fourier transform and multiband modeling, if the number of subbands is represented as b, the dimension reduction in the frequency direction of Equation (1) is expressed by the following Equation (8).

[0136]

[0137] Equation (8) shows that when the number of subbands is multiplied by b, the number of dimensions in the frequency direction can be multiplied by 1 / b.

[0138] An experiment (hereinafter referred to as "second experiment") was conducted in which the target signal acquisition process combined inverse short-time Fourier transform and multiband modeling, and an example of this experiment and its results will be described below.

[0139] In the second experiment, a comparative technology was used, which is described in Non-Patent Document 1, in which four-fold upsampling is performed twice in the time direction using a neural network, four-fold upsampling is performed in the time direction using an inverse short-time Fourier transform, and four-fold upsampling is performed in the time direction using multiband modeling (hereinafter referred to as the "second comparative technology"). Note that in the technology described in Non-Patent Document 1, eight-fold upsampling is performed twice in the time direction using a neural network, and four-fold upsampling is performed in the time direction using an inverse short-time Fourier transform.

[0140] In the example of FIG. 2, the processing object acquisition model performed one 8-fold upsampling in the time direction, but in the second experiment, the processing object acquisition model performed one 4-fold upsampling in the time direction.

[0141] In addition, in the example of Figure 2, the target signal acquisition process performs 32-fold upsampling in the time direction using inverse short-time Fourier transform, but in the second experiment, the target signal acquisition process performs 16-fold upsampling in the time direction using inverse short-time Fourier transform.

[0142] In addition, in the second experiment, the “2D Blocks” in FIG. 2 in the processing target acquisition model was replaced with the “2D Blocks” in FIG. 4, in which the number of input channels of the “2D Blocks” was doubled and the number of output channels of the first convolutional layer was changed from 2C to C.

[0143] Also, in the second experiment, in order to generate multibands, the number of channels at the output of the second network in Figure 2 was 8 instead of 2. The number of subbands in the multiband modeling was 4.

[0144] Hereinafter, the signal generating system 100 used in the second experiment will be referred to as invention method 3.

[0145] Fig. 10 shows an example of the results of the second experiment in the modified example. Fig. 10 shows that in terms of voice quality (MOS), the invention method 3 is better than the second comparison technology. Fig. 10 also shows that in terms of processing speed and the number of parameters, the invention method 3 and the comparison technology are equivalent.

[0146] One method for improving speech quality is to increase the number of parameters. If this method is used to obtain speech quality equivalent to that of Invention Method 3 using the second comparison technology, the number of parameters will increase and processing speed will deteriorate. Therefore, if Invention Method 3 and the second comparison technology achieve the same speech quality, Invention Method 3 is faster and lighter.

[0147] Up to this point, the processing object acquisition model has executed a first process for extracting features of one-dimensional data and a second process for extracting features of two-dimensional data. However, when the time-dependent information is three-dimensional data such as a video, the processing object acquisition model may execute at least two of the first process, the second process, and the third process. The third process is executed by a neural network and extracts features of three-dimensional data. The third process is, for example, the execution of a three-dimensional convolutional neural network.

[0148] In this way, the processing object acquisition model may be any mathematical model as long as it executes a plurality of processes executed by a neural network, each of which is a plurality of feature extraction processes for data of different dimensions as the processing object. The result of the execution of the processing object acquisition model is the processing object of the target signal acquisition process.

[0149] Although the above description has been given of an example in which multiple feature extraction processes for data with different dimensions are executed in series, multiple feature extraction processes for data with different dimensions may also be executed in parallel. Specifically, when the first process and the second process are executed in parallel, for example, the first process and a dimension increase process may be applied to all or part of the time-dependent information, and in parallel with this, the dimension increase process and the second process may be applied to all or part of the time-dependent information, and the results of the two parallel processes may finally be integrated. Furthermore, multiple feature extraction processes for data with different dimensions may be executed in series or in parallel any number of times, in any order, or in any combination.

[0150] The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0151] The signal generating device 2 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.

[0152] Note that all or part of the functions of the learning device 1 and the signal generating device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0153] The control unit 21 is an example of a generating unit.

[0154] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0155] REFERENCE SIGNS LIST 100... signal generation system, 1... learning device, 2... signal generation device, 11... control unit, 22... interface unit, 23... storage unit, 21... control unit, 22... interface unit, 23... storage unit, 91... processor, 92... memory, 93... processor, 94... memory

Claims

1. A signal generation device comprising: a generation unit that executes a mathematical model to obtain a processing object of a target signal acquisition process that obtains a target signal, which is a target signal, by performing a transformation on a processing object using a predetermined basis, wherein time-dependent information that is information indicating the time dependency of a time-dependent quantity is input to the mathematical model, wherein the mathematical model executes a plurality of processes executed by a neural network, and a plurality of feature extraction processes having different dimensions of data to be executed, wherein the feature extraction process is a process for extracting features for each element of the d-dimensional data when d-dimensional (d is an integer equal to or greater than 1) data is given as the data to be executed, and the processing object is the result of executing the mathematical model.

2. The signal generating device of claim 1, wherein the mathematical model executes a first process, which is executed by a neural network, for extracting features in the time direction from the time-dependent information, a dimension increase process for increasing the dimension of the result of the first process, and a second process for extracting features for each element in the time direction and each element of the dimension obtained by the dimension increase process.

3. The signal generating device according to claim 2, wherein the neural network that performs the first processing is a one-dimensional convolutional neural network.

4. The signal generating device according to claim 2 or 3, wherein the neural network that performs the second processing is a two-dimensional convolutional neural network.

5. The signal generating device according to claim 1, wherein the mathematical model is obtained by learning.

6. A signal generation method comprising: a generation step of executing a mathematical model to obtain a target signal, which is a target signal, by performing a transformation using a predetermined basis on a processing object; wherein time-dependent information, which is information indicating the time dependency of a time-dependent quantity, is input to the mathematical model; the mathematical model executes a plurality of processes executed by a neural network, each of which executes a plurality of feature extraction processes having different dimensions of data to be executed; wherein the feature extraction process is a process for extracting features for each element of d-dimensional data when d-dimensional (d is an integer equal to or greater than 1) data is given as the data to be executed; and the processing object is a result of executing the mathematical model.

7. A program for causing a computer to function as the signal generating device according to claim 1.

Images