Encoder / decoder, method, and program

Abstract

To improve the performance of an auto-encoder.SOLUTION: A coder-decoder according to an embodiment of the present invention has a multiple auto-encoder that has a plurality of pairs of encoders and decoders, a machine learning model, and a control unit. The control unit causes the plurality of encoders to encode input signals, causes the plurality of decoders to decode the encoded signals, and causes the machine learning model to combine the decoded signals.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a coder / decoder, a method, and a program. [Background technology]

[0002] Conventionally, autoencoders, which are algorithms in machine learning, and their derivatives (e.g., variational autoencoders) are known (Patent Document 1). In an autoencoder, the dimensions of an input signal are reduced to extract features, and a signal restored to its original dimensions is output. Summary of the Invention [Problem to be solved by the invention]

[0003] However, when applying autoencoders to time-domain signals such as audio signals, if the number of dimensions is reduced while the signal (waveform) remains in the time domain, there is a problem that the amplitude, frequency, and phase information cannot be retained, resulting in signal degradation.

[0004] Even if pre-processing is performed using a speech coder and decoder (vocoder) before and after the autoencoder to convert the speech signal into frequency domain features, and post-processing is performed to convert the features back into a time domain signal (waveform), degradation of the speech signal occurs during the pre-processing and post-processing steps.

[0005] Therefore, an object of one embodiment of the present invention is to improve the performance of autoencoders. [Means for solving the problem]

[0006] A coding / decoding device according to one embodiment of the present invention comprises a multiple autoencoder having multiple pairs of encoders and decoders, a machine learning model, and a control unit, wherein the control unit encodes an input signal using each of the multiple encoders, decodes the encoded signal using each of the multiple decoders, and synthesizes the decoded signals using the machine learning model. Effect of the Invention

[0007] In one embodiment of the present invention, the performance of an autoencoder can be improved. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram for explaining a coding / decoding device according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a diagram for explaining a multiple autoencoder according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram illustrating a multiple variational encoder according to an embodiment of the present invention. [Figure 4] 4 is a flowchart showing the flow of encoding and decoding processing according to one embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram illustrating a hardware configuration of a coder / decoder according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0010] <Terminology explanation> As used herein, a "multiple autoencoder" refers to an autoencoder that has multiple pairs of encoders and decoders, each of which is independent. In this specification, "autoencoder" includes not only autoencoders but also autoencoders derived from autoencoders (e.g., variational autoencoders). In this specification, a "signal" may be any signal, for example a time domain signal such as an audio signal. The time domain signal may be a physical observation signal such as an electrocardiogram, a seismometer, a water level gauge, or statistical data such as stock prices, traffic volume, or the number of internet accesses. In other words, the time domain signal may be anything that indicates a time transition. In one embodiment of the present invention, physical observation signals such as an audio signal, an electrocardiogram, a seismometer, a water level gauge, or statistical data such as stock prices, traffic volume, or the number of internet accesses may be used.

[0011] <Encoder / Decoder> FIG. 1 is a diagram illustrating a coder / decoder 1 according to an embodiment of the present invention.

[0012] The encoder / decoder 1 has a multiple autoencoder 10 (e.g., a multiple autoencoder 11 described in FIG. 2 or a multiple variational autoencoder 12 described in FIG. 3), a machine learning model (e.g., a neural network such as a convolutional neural network) 20, and a control unit 30.

[0013] The parameters of the machine learning model (e.g., convolutional neural network) 20 are pre-optimized to statistically minimize the residual error between the input signal and the output signal of the encoder / decoder 1. The code vectors or latent vectors extracted from the multiple autoencoder 11 or multiple variational autoencoder 12 can be applied to various speech signal processing such as speech coding, speech analysis, and speech conversion.

[0014] In one embodiment of the present invention, the coder / decoder 1 may be implemented in an audio converter (also called a voice changer), an audio noise reducer, an audio data compression system, or the like.

[0015] <Multiple Autoencoders> FIG. 2 is a diagram for explaining the multiple autoencoder 11 according to an embodiment of the present invention.

[0016] In one embodiment of the present invention, the encoder / decoder 1 has a multiple autoencoder 11 and a synthetic reconstruction convolutional neural network 21. The multiple autoencoder 11 has p independent pairs of encoders 1, 2,...,p (111-1, 111-2,...,111-p) and decoders 1, 2,...,p (112-1, 112-2,...,112-p).

[0017] An input signal x is simultaneously input to encoders 1, 2, . . . , p (111-1, 111-2, . . . , 111-p).

[0018] Encoders 1, 2, . . . , p (111-1, 111-2, . . . , 111-p) encode the input signal x into lower dimensional vectors z1, z2, . . . , zp, respectively.

[0019] Decoders 1, 2, . . . , p (112-1, 112-2, . . . , 112-p) decode the code vectors z1, z2, . . . , zp, respectively, to obtain vectors v1, v2, . . . , vp of the same dimensions as the input signal.

[0020] The convolutional neural network 21 combines v1, v2, ..., vp and outputs an output signal y. Specifically, the control unit 30 inputs v1, v2, ..., vp to the convolutional neural network 21 and causes it to output the output signal y.

[0021] The network parameters of the encoders 1, 2, ..., p (111-1, 111-2, ..., 111-p), the decoders 1, 2, ..., p (112-1, 112-2, ..., 112-p) and the convolutional neural network 21 are optimized so as to minimize the error (for example, the mean squared error or the mean absolute error) between the input signal x and the output signal y.

[0022] <Multiple Variational Autoencoders> FIG. 3 is a diagram for explaining the multiple variational autoencoder 12 according to an embodiment of the present invention.

[0023] In one embodiment of the present invention, the encoder / decoder 1 includes a multiple variational autoencoder 12 and a synthetic reconstruction convolutional neural network 22. The multiple variational autoencoder 12 includes p independent pairs of encoders 1, 2,...,p (121-1, 121-2,...,121-p) and decoders 1, 2,...,p (122-1, 122-2,...,122-p).

[0024] An input signal x is simultaneously input to encoders 1, 2, . . . , p (121-1, 121-2, . . . , 121-p).

[0025] Encoders 1, 2, . . . , p (121-1, 121-2, . . . , 121-p) convert the input signal x into statistics μ1, σ1, μ2, σ2, . . . , μp, σp, respectively, which represent the probability distribution in a low-dimensional latent space.

[0026] Reparameterized latent vectors z1, z2, . . . , zp are generated based on the probability distributions μ1, σ1, μ2, σ2, . . . , μp, σp.

[0027] The decoders (122-1, 122-2, . . . , 122-p) decode the latent vectors z1, z2, . . . , zp, respectively, to obtain vectors v1, v2, . . . , vp of the same dimensions as the input signal.

[0028] The convolutional neural network 22 combines v1, v2, ..., vp and outputs an output signal y. Specifically, the control unit 30 inputs v1, v2, ..., vp to the convolutional neural network 22 and causes it to output the output signal y.

[0029] The network parameters of the encoders 1, 2, ..., p (121-1, 121-2, ..., 121-p) and the decoders 1, 2, ..., p (122-1, 122-2, ..., 122-p) and the convolutional neural network 22 are optimized to minimize the error (for example, the mean squared error or the mean absolute error) between the input signal x and the output signal y.

[0030] <Processing method> FIG. 4 is a flowchart showing the flow of encoding and decoding processing according to one embodiment of the present invention.

[0031] In step 1 (S1), the control unit 30 inputs an input signal. Specifically, the control unit 30 inputs the input signal to each of the multiple encoders (encoder 1, encoder 2, . . . , encoder p) of the multiple autoencoder 10.

[0032] In step 2 (S2), each of the multiple encoders (encoder 1, encoder 2, . . . , encoder p) of the multiple autoencoder 10 encodes the input signal input in S1.

[0033] In step 3 (S3), each of the multiple decoders (decoder 1, decoder 2, . . . , decoder p) of the multiple autoencoder 10 decodes the signal encoded in S2.

[0034] In step 4 (S4), the control unit 30 synthesizes the signals decoded in S3 using the convolutional neural network 20. Specifically, the control unit 30 inputs all the signals decoded in S3 to the convolutional neural network 20, and causes it to output an output signal y.

[0035] In step 5 (S5), the control unit 30 outputs the signal (output signal) synthesized in S4.

[0036] <Hardware configuration> FIG. 5 is a diagram showing a hardware configuration of a coder / decoder 1 according to an embodiment of the present invention.

[0037] The codec 1 includes a central processing unit (CPU) 101, a read only memory (ROM) 102, and a random access memory (RAM) 103. The CPU 101, the ROM 102, and the RAM 103 form a so-called computer.

[0038] The codec 1 may also include an auxiliary storage device 104, a display device 105, an operation device 106, an I / F (Interface) device 107, and a drive device .

[0039] The hardware components of the codec 1 are connected to one another via a bus B.

[0040] The CPU 101 is a computing device that executes various programs installed in the auxiliary storage device 104 .

[0041] The ROM 102 is a non-volatile memory. The ROM 102 functions as a main storage device that stores various programs, data, etc. required for the CPU 101 to execute various programs installed in the auxiliary storage device 104. Specifically, the ROM 102 functions as a main storage device that stores boot programs such as a Basic Input / Output System (BIOS) and an Extensible Firmware Interface (EFI).

[0042] The RAM 103 is a volatile memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 103 functions as a main storage device that provides a working area in which various programs installed in the auxiliary storage device 104 are expanded when the CPU 101 executes them.

[0043] The auxiliary storage device 104 is an auxiliary storage device that stores various programs and information used when the various programs are executed.

[0044] The display device 105 is a display device that displays the internal state of the codec 1 and the like.

[0045] The operation device 106 is an input device through which an operator of the codec 1 inputs various instructions to the codec 1 .

[0046] The I / F device 107 is a communication device for connecting to a network and communicating with other devices.

[0047] The drive device 108 is a device for setting the storage medium 109. The storage medium 109 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. The storage medium 109 may also include semiconductor memories that record information electrically, such as an EPROM (Erasable Programmable Read Only Memory), a flash memory, etc.

[0048] The various programs to be installed in the auxiliary storage device 104 are installed, for example, by setting the distributed storage medium 109 in the drive device 108 and reading out the various programs recorded in the storage medium 109 by the drive device 108. Alternatively, the various programs to be installed in the auxiliary storage device 104 may be installed by being downloaded from a network via the I / F device 107.

[0049] <Effects> In this manner, in one embodiment of the present invention, each individual encoder reduces the dimensionality of the input signal, while, as a whole, the information necessary to reconstruct the input signal is preserved.

[0050] In one embodiment of the present invention, it is possible to process signals in the time domain using an autoencoder or a variational autoencoder as input and output without converting the signals into frequency domain features, which causes degradation of the signals. Therefore, degradation of the restored signals can be reduced more than in the past.

[0051] A conventional autoencoder or variational autoencoder is composed of a pair of an encoder and a decoder, in which the encoder reduces the dimension of the input signal, and the decoder restores it to the original dimension. Therefore, when a signal such as a voice, which has complex information consisting of amplitude, frequency, and phase in the time domain and whose changes are sensitive, loses information by dimensional reduction, the quality of the restored voice is unavoidably significantly degraded. In one embodiment of the present invention, an autoencoder or variational autoencoder is composed of multiple independent pairs of an encoder and a decoder, and the input signal is restored by synthesizing the outputs of each decoder using a convolutional neural network. With this structure, each encoder cuts down the information of the input signal by dimensional compression, but the system as a whole can retain the information of the input signal, so that it is possible to restore the time domain voice signal without degradation.

[0052] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0053] Although the examples of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0054] 1. Encoder / Decoder 10 Multiple Autoencoders 11 Multiple Autoencoders 12 Multiple Variational Autoencoder 20 Convolutional Neural Networks 21 Convolutional Neural Network for Synthesis and Restoration 22 Convolutional Neural Network for Synthesis and Restoration 30 Control section 111 Encoder 112 Decoder 121 Encoder 122 Decoder 101 CPU 102 ROM 103 RAM 104 Auxiliary storage 105 Display device 106 Operating device 107 I / F device 108 Drive device 109 Storage medium [Prior art documents] [Patent documents]

[0055] [Patent Document 1] JP 2020-071808 A

Claims

1. A multi-autoencoder having multiple pairs of encoders and decoders, a convolutional neural network, and a controller; The control unit is encoding an input signal with each of the plurality of encoders; decoding the encoded signal with each of the plurality of decoders; A coding / decoding device that inputs all of the decoded signals to the convolutional neural network and outputs a signal in which all of the decoded signals are combined.

2. The coder / decoder according to claim 1 , wherein the signal is an audio signal.

3. The encoder / decoder according to claim 1 or 2, wherein the multiple autoencoder is an autoencoder or a variational autoencoder.

4. A coding / decoding device including a multiple autoencoder having a plurality of pairs of encoders and decoders, a convolutional neural network, and a control unit, encoding an input signal with each of the plurality of encoders; decoding the encoded signal by each of the plurality of decoders; inputting all of the decoded signals to the convolutional neural network, and outputting a signal in which all of the decoded signals are combined.

5. A method implemented by an encoder / decoder having a multiple autoencoder having multiple pairs of encoders and decoders, a convolutional neural network, and a controller, comprising: encoding an input signal with each of the plurality of encoders; decoding the encoded signal by each of the plurality of decoders; inputting all the decoded signals to the convolutional neural network and outputting a signal in which all the decoded signals are combined; The method includes:

Images