Learning device, conversion device, method, and program

Abstract

This learning device comprises: a spectrogram generation unit 1 that generates a spectrogram from a first sound signal, and generates a target spectrogram from a second sound signal; a patch generation unit 2 that generates multiple patches by dividing the spectrogram, and generates multiple target patches by dividing the target spectrogram; a mask processing unit 3 that selects, as a masked patch, some patches among the multiple patches; a reconstruction unit 4 that, using a visible patch that is a patch other than the some patches among the multiple patches, and a mask token corresponding to the masked patch, obtains multiple reconstructed patches obtained by reconstructing the multiple patches by processing by an encoder and a decoder; and a parameter updating unit 5 that updates parameters of the encoder and the decoder such that a target patch corresponding to the masked patch among the multiple target patches and a reconstructed patch corresponding to the masked patch among the multiple reconstructed patches approach each other.

Description

Learning device, conversion device, method and program

[0001] The present invention relates to deep learning technology.

[0002] In particular, in fields other than audio, there are representation learning frameworks called Masked Language Modeling (MLM) and Masked Image Modeling (MIM). These frameworks mask part of the input and use information obtained from the unmasked part to restore and predict the masked part. These frameworks aim to obtain a good information extraction model (also called an encoder, or abstractly called a "representation") by training to extract effective information from the unmasked part.

[0003] For sound signals, Non-Patent Document 1 proposes a learning method that combines mask restoration and classification for spectrograms.

[0004] Yuan Gong, Cheng-I Jeff Lai, Yu-An Chung, James Glass, "SSAST: Self-Supervised Audio Spectrogram Transformer", MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA 02139

[0005] The technique of Non-Patent Document 1 cannot be used to solve general tasks such as noise removal, super-resolution, style transfer, and anomaly detection.

[0006] An object of the present invention is to provide a learning device that learns a model that can be used to solve general tasks, a conversion device, method, and program for enabling the solution of general tasks.

[0007] A learning device according to one aspect of the present invention includes a spectrogram generation unit that generates a spectrogram from an input first sound signal and generates a target spectrogram from an input second sound signal; a patch generation unit that divides the generated spectrogram to generate a plurality of patches and divides the generated target spectrogram to generate a plurality of target patches; a mask processing unit that selects some of the plurality of patches as masked patches; a reconstruction unit that obtains a plurality of reconstructed patches by reconstructing the plurality of patches through encoder and decoder processing in a Transformer, which is a deep learning model, using visible patches, which are patches other than some of the plurality of patches, and mask tokens corresponding to the masked patches; and a parameter update unit that updates encoder parameters and decoder parameters so that a target patch corresponding to a masked patch among the plurality of target patches and a reconstructed patch corresponding to the masked patch among the plurality of reconstructed patches become closer to each other.

[0008] A conversion device according to one aspect of the present invention includes a memory unit in which an encoder and a decoder trained by a learning device are stored; a spectrogram generation unit that generates a spectrogram from an input sound signal; a patch generation unit that divides the generated spectrogram to generate a plurality of patches; a mask processing unit that selects some of the plurality of patches as masked patches; and a reconstruction unit that obtains a plurality of reconstructed patches by reconstructing the plurality of patches by processing the encoder and decoder read from the memory unit using visible patches, which are patches other than some of the plurality of patches, and mask tokens corresponding to the masked patches.

[0009] It is possible to train a model that can be used to solve general tasks, or it is possible to solve general tasks.

[0010] FIG. 1 is a diagram showing an example of the functional configuration of a learning device. FIG. 2 is a diagram showing an example of the processing procedure of a learning method. FIG. 3 is a diagram showing an outline of an example of the processing of a learning device and method. FIG. 4 is a diagram showing an example of the functional configuration of a computer. FIG. 5 is a diagram showing an example of the functional configuration of a conversion device. FIG. 6 is a diagram showing an example of the processing procedure of a conversion method. FIG. 7 is a diagram showing an example of mask processing. FIG. 8 is a diagram showing an example of mask processing. FIG. 9 is a diagram showing an example of mask processing. FIG. 10 is a diagram showing an example of the functional configuration of a conversion device.

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings, in which like reference numerals are used to designate like components having the same functions, and redundant description will be omitted.

[0012] [Learning Device and Method] As shown in FIG. 1, the learning device includes, for example, a spectrogram generating unit 1, a patch generating unit 2, a mask processing unit 3, a reconstructing unit 4, a parameter updating unit 5, and a storage unit 6.

[0013] The learning method is realized, for example, by each component of the learning device performing the processes of steps S1 to S5 shown in FIG.

[0014] <Spectrogram Generator 1> The spectrogram generator 1 receives a first sound signal and a second sound signal such as an acoustic signal or a voice signal.

[0015] The spectrogram generating unit 1 generates a spectrogram from an input first sound signal, and generates a target spectrogram from an input second sound signal (step S1).

[0016] The generated spectrogram and target spectrogram are output to the patch generation unit 2 .

[0017] For example, if the number of spectrogram bins is B and the number of frames of the sound signal is F, a spectrogram and a target spectrogram with a size of B x F are generated. B and F are predetermined positive integers. For example, B = 80, F = 304. The time length of one frame is 10 ms.

[0018] <Patch Generator 2> The patch generator 2 receives the spectrogram generated by the spectrogram generator 1 and the target spectrogram.

[0019] The patch generating unit 2 divides the spectrogram to generate a plurality of patches, and divides the target spectrogram to generate a plurality of target patches (step S2).

[0020] The generated patches are output to the mask processing unit 3, the reconstruction unit 4, and the parameter update unit 5.

[0021] The generated target patches are output to the parameter update unit 5 .

[0022] For example, the patch generation unit 2 divides a spectrogram of size B×F into patches of size b×f, where b and f are predetermined positive integers. b may be a divisor of B, and f may be a divisor of F. For example, if B=80, F=304, and b=16, f=16, then (80 / 16)×(304 / 16)=5×19=95 patches of size 16×16 are generated from the spectrogram of size 80×304. The same applies to the target spectrogram.

[0023] In this way, the patch generating unit 2 divides, for example, the spectrogram and the target spectrogram into a grid pattern.

[0024] <Mask Processing Unit 3> The mask processing unit 3 receives the plurality of patches generated by the patch generating unit 2 as input.

[0025] The mask processing unit 3 selects some patches from the plurality of patches as masked patches (step S3).

[0026] Information about the patch selected as the masked patch is output to the reconstruction unit 4 and the parameter update unit 5 .

[0027] The mask processing unit 3 selects, for example, X% of the patches as masked patches, where X is a real number between 0 and 100. For example, X is 75. The mask processing unit 3 may select 50% or more of the patches as masked patches.

[0028] The mask processing unit 3 may select patches randomly or may select patches according to a predetermined pattern.

[0029] An example of the processing of the learning device and method is outlined in Figure 3. In the example of Figure 3, the spectrogram is divided into 5 x 6 patches as shown in INPUT. In the example of Figure 3, the patches indicated by dashed lines are selected as masked patches as shown in INPUT.

[0030] <Reconstruction Unit 4> Information about the masked patch selected by the mask processing unit 3 is input to the reconstruction unit 4. The reconstruction unit 4 also receives a plurality of patches generated by the patch generation unit 2.

[0031] The reconstruction unit 4 uses visible patches, which are patches other than the masked patches among the multiple patches, and mask tokens corresponding to the masked patches, to obtain multiple reconstructed patches by processing the encoder and decoder in the transformer, which is a deep learning model (step S4).

[0032] The mask token is a vector. The initial value of each element of the mask token is, for example, a random real number. As will be described later, the mask token may be updated by the parameter update unit 5. In this case, the reconstruction unit 4 uses the updated mask token.

[0033] The obtained reconstructed patches are output to the parameter update unit 5 .

[0034] Below, examples 1 and 2 of the processing by the reconstruction unit 4 will be described.

[0035] <<Example 1>> The encoder processing unit 41 of the reconstruction unit 4 inputs the visible patch to an encoder in a transformer, which is a deep learning model, to obtain an encoding result.

[0036] The decoder processing unit 42 of the reconstruction unit 4 inputs the encoding result and the mask token to the decoder to obtain a decoding result, which is a plurality of reconstructed patches obtained by reconstructing a plurality of patches.

[0037] In the example of Fig. 3, visible patches, which are patches other than the masked patches indicated by dashed lines in INPUT, are input to the encoder, and the encoded result indicated by the dotted line is obtained. In the example of Fig. 3, the encoded result plus the mask token corresponding to the masked patch is input to the decoder, and the decoded result indicated by the diagonal stripe pattern sloping downward to the right is obtained.

[0038] <<Example 2>> The encoder processing unit 41 of the reconstruction unit 4 inputs the visible patches and mask tokens to an encoder in a transformer, which is a deep learning model, to obtain an encoding result.

[0039] The decoder processing unit 42 of the reconstruction unit 4 inputs the encoded result to the decoder to obtain a decoded result, which is a plurality of reconstructed patches obtained by reconstructing a plurality of patches.

[0040] <Parameter Update Unit 5> The parameter update unit 5 receives as input the multiple reconstructed patches obtained by the reconstruction unit 4. The parameter update unit 5 also receives as input information about the masked patch selected by the mask processing unit 3, the multiple patches generated by the patch generation unit 2, and the multiple target patches.

[0041] The parameter update unit 5 updates the encoder parameters and decoder parameters so that the target patch corresponding to the masked patch among the multiple target patches and the reconstructed patch corresponding to the masked patch among the multiple reconstructed patches become closer to each other (step S17).

[0042] The parameter update unit 5 updates the encoder parameters and decoder parameters using a technique such as backpropagation, for example, so as to reduce the magnitude of the error between a target patch corresponding to a masked patch among a plurality of target patches and a reconstructed patch corresponding to a masked patch among a plurality of reconstructed patches.

[0043] In the example of FIG. 3, the parameters are updated so as to reduce the magnitude of the error between the target patch corresponding to the masked patch indicated by the dashed line in TARGET and the reconstructed patch corresponding to the masked patch indicated by the dashed line in RECONSTRUCTION.

[0044] <<Repeated Processing>> The processing from step S1 to step S5 is repeated. For example, the processing from step S1 to step S5 is performed for each of a plurality of different sound signals.

[0045] For example, the processes from step S1 to step S5 are repeated until the encoder parameters and decoder parameters converge.

[0046] <Storage Unit 6> When the repetition is completed, the model, which is the encoder and decoder determined by the latest parameters updated by the parameter update unit 5, is stored in the storage unit 6.

[0047] <<Modifications of the Learning Apparatus and Method>> The time length of the spectrogram may be longer than a predetermined time length. In other words, the number of frames of the sound signal input to the spectrogram generator 1 may be greater than a predetermined number of frames.

[0048] For example, the time length of the spectrogram may be 4 seconds or more. Increasing the time length of the spectrogram improves the performance of the model generated by the learning device and method.

[0049] The patch size may be smaller than a predetermined size. For example, the patch size may be smaller than 16 × 8, or the patch size may be smaller than 16 × 4. Reducing the patch size increases the resolution, thereby improving the performance of the model generated by the learning device and method.

[0050] [Conversion Device and Method] As shown in FIG. 5, the conversion device includes, for example, a spectrogram generation unit 11, a patch generation unit 12, a mask processing unit 13, a reconstruction unit 14, an integration unit 15, a time domain conversion unit 16, and a storage unit 6.

[0051] The conversion method is realized, for example, by each component of the conversion device performing the processes from step S11 to step S16 described below and shown in FIG.

[0052] <Storage Unit 6> The storage unit 6 stores a model (encoder and decoder in a transformer) trained by the training device and method.

[0053] The spectrogram generator 11, patch generator 12, mask processor 13, and reconstructor 14 described below perform the same processing as the spectrogram generator 1, patch generator 2, mask processor 3, and reconstructor 4 in the learning device. Duplicate descriptions of the same parts as the spectrogram generator 1, patch generator 2, mask processor 3, and reconstructor 4 will be omitted.

[0054] <Spectrogram Generator 11> The spectrogram generator 11 receives as input a sound signal, such as an acoustic signal or a voice signal, to be converted.

[0055] The spectrogram generating unit 11 generates a spectrogram from the input sound signal (step S11).

[0056] is entered.

[0057] The generated spectrogram is output to the patch generating unit 12 .

[0058] <Patch Generator 12> The patch generator 12 receives the spectrogram generated by the spectrogram generator 11 .

[0059] The patch generating unit 12 divides the spectrogram to generate a plurality of patches (step S12).

[0060] The generated patches are output to the mask processing unit 13 and the reconstruction unit 14 .

[0061] <Mask Processing Unit 13> The mask processing unit 13 receives the plurality of patches generated by the patch generating unit 2 as input.

[0062] The mask processing unit 13 selects some patches from the plurality of patches as masked patches (step S13).

[0063] Information about the patch selected as the masked patch is output to the reconstruction unit 14 and the integration unit 15 .

[0064] <Reconstruction Unit 14> Information about the masked patch selected by the mask processing unit 13 is input to the reconstruction unit 14. In addition, the reconstruction unit 14 also receives a plurality of patches generated by the patch generation unit 12.

[0065] The reconstruction unit 14 obtains multiple reconstructed patches by reconstructing the multiple patches by processing the encoder and decoder read from the memory unit 6 using visible patches, which are patches other than some of the multiple patches, and mask tokens corresponding to the masked patches (step S14).

[0066] The resulting reconstructed patches are output to the integration unit 15 .

[0067] <<Repeated Processing of Mask Processing Unit 13 and Reconstruction Unit 14>> In the conversion device and method, the processing of the mask processing unit 13 and the reconstruction unit 14 is repeatedly performed. During the repeated processing, the mask processing unit 13 selects some or all of the patches that have not been selected so far from among the multiple patches as masked patches.

[0068] The processing by the mask processing unit 13 and the reconstructing unit 14 is performed until all of the patches among the plurality of patches are selected as masked patches and reconstructed patches corresponding to all of the patches among the plurality of patches are obtained.

[0069] For example, the mask processing unit 13 may select masked patches in a checkerboard pattern, assuming that the multiple patches are arranged on a two-dimensional plane. In this case, for example, in the first processing, the mask processing unit 13 selects the patches indicated by solid lines in Fig. 7 as masked patches. In addition, in the second processing, the mask processing unit 13 selects the patches indicated by dashed lines in Fig. 7 as masked patches.

[0070] Alternatively, the plurality of patches may be arranged on a two-dimensional plane, and the mask processing unit 13 may select every other masked patch in one coordinate axis direction of the coordinate system of the two-dimensional plane.

[0071] In this case, for example, in the first processing, the mask processing unit 13 selects the patches indicated by solid lines in Fig. 8 as masked patches. In addition, in the second processing, the mask processing unit 13 selects the patches indicated by dashed lines in Fig. 8 as masked patches. In other words, the mask processing unit 13 may select every other masked patch in the vertical direction.

[0072] In this case, for example, in the first processing, the mask processing unit 13 may select the patches indicated by solid lines in Fig. 9 as masked patches, and in the second processing, may select the patches indicated by dashed lines in Fig. 9 as masked patches. In other words, the mask processing unit 13 may select every other masked patch in the horizontal direction.

[0073] In the example described using Figures 7 to 9, all of the patches among the multiple patches are selected as masked patches in two processes, and reconstructed patches corresponding to all of the patches among the multiple patches are obtained.

[0074] <Integration Unit 15> The integration unit 15 receives information about the masked patch selected by the mask processing unit 13. The integration unit 15 also receives a plurality of reconstructed patches obtained by the reconstruction unit 14.

[0075] After the repetitive processing is completed, the integrating unit 15 performs processing to integrate the reconstructed patch corresponding to the masked patch among the plurality of reconstructed patches to generate a reconstructed spectrogram (step S15).

[0076] The generated reconstructed spectrogram is output to the time domain transform unit 16 .

[0077] <Time Domain Transformation Unit 16> The time domain transformation unit 16 receives the reconstructed spectrogram generated by the integration unit 15 as input.

[0078] The time domain transforming unit 16 transforms the reconstructed spectrogram into a sound signal in the time domain (step S16).

[0079] <<Conversion Example 1>> Assume that a model trained by providing a sound signal in which a desired sound and noise are mixed as the first sound signal and by providing only the desired sound as the second sound signal is stored in the memory unit 6.

[0080] In this case, by inputting the conversion device and method into a sound signal from which noise is to be removed, a sound signal with reduced noise can be obtained by the time domain conversion unit 16. In other words, in this case, the conversion device and method can convert a sound signal with noise into a sound signal with reduced noise.

[0081] <<Conversion Example 2>> It is assumed that a model trained by providing a low-resolution version of a desired sound as the first sound signal and a high-resolution version of the desired sound as the second sound signal is stored in the storage unit 6.

[0082] In this case, by inputting a low-resolution sound signal into the conversion device and method, a high-resolution sound signal can be obtained by the time domain conversion unit 16. In other words, in this case, the conversion device and method can convert a low-resolution sound signal into a high-resolution sound signal.

[0083] <<Conversion Example 3>> Assume that a model trained by providing a sound signal of a desired source style as the first sound signal and a sound signal of a desired destination style as the second sound signal is stored in the memory unit 6.

[0084] In this case, by inputting a sound signal in a source style into the conversion device and method, a sound signal in a destination style is obtained by the time domain conversion unit 16. In other words, in this case, the conversion device and method can convert a sound signal in a source style into a sound signal in a destination style.

[0085] Examples of (source style sound signal, destination style sound signal) are (speaker A's voice, speaker B's voice), (classical music, rock music), and (sound of knocking on a wooden door, sound of knocking on an iron door).

[0086] 10 , the conversion device is assumed to include an abnormal sound detection unit 17 instead of the time domain conversion unit 16. The abnormal sound detection unit 17 is assumed to receive as input the reconstructed spectrogram generated by the integrating unit 15 and the spectrogram generated by the spectrogram generation unit 11.

[0087] It is assumed that a model trained by providing normal sounds in an environment in which abnormal sounds are to be detected as the first sound signal and the second sound signal is stored in the storage unit 6.

[0088] In this case, by using the environmental sound of the environment in which abnormal sound is desired to be detected as input to the conversion device and method, the integration unit 15 can obtain a reconstructed spectrogram of the environmental sound of the environment in which abnormal sound is desired to be detected.

[0089] The abnormal sound detection unit 17 calculates the magnitude of the error between the spectrogram and the reconstructed spectrogram, and determines that there is an abnormality if the magnitude of the calculated error is greater than a predetermined threshold, and determines that there is a normality if not.

[0090] For example, suppose a model trained by providing the operating sounds of a normal machine as the first sound signal and the second sound signal is stored in the storage unit 6. Then, suppose a sound generated by a faulty machine is input to the conversion device and method. In this case, the sound generated by the faulty machine does not exist at the time of training, so the magnitude of the error between the spectrogram and the reconstructed spectrogram becomes large.

[0091] Similarly, suppose that a model trained by providing normal street sounds as the first sound signal and the second sound signal is stored in the storage unit 6. Then, suppose that street sounds including shouting are input to the conversion device and method. In this case, shouting is not present at the time of training, so the magnitude of the error between the spectrogram and the reconstructed spectrogram becomes large.

[0092] In this way, by utilizing the fact that the error increases when an abnormal sound is included, it is possible to detect an abnormality.

[0093] As in this example, the conversion device and method may be used as an abnormal sound detection device and method.

[0094] [Modifications] Although the embodiments of the present invention have been described above, the specific configurations are not limited to these embodiments, and it goes without saying that appropriate design changes, etc., within the scope of the spirit of the present invention, are also included in the present invention.

[0095] The various processes described in the embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as necessary.

[0096] For example, data exchange between components of the learning device and between components of the conversion device may be performed directly or via a storage unit (not shown).

[0097] [Program, Recording Medium] The processing of each unit of each of the above-mentioned devices may be realized by a computer, in which case the processing content of the functions to be possessed by each device is described by a program. Then, by loading this program into storage unit 1020 of computer 1000 shown in Figure 4 and operating arithmetic processing unit 1010, input unit 1030, output unit 1040, display unit 1060, etc., various processing functions of each of the above-mentioned devices are realized on the computer.

[0098] The program describing the processing contents can be recorded on a computer-readable recording medium, such as a non-transitory recording medium, specifically a magnetic recording device, an optical disk, or the like.

[0099] The program may be distributed, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to another computer via a network, thereby distributing the program.

[0100] A computer executing such a program, for example, first stores the program recorded on a portable recording medium or transferred from a server computer in its own non-transitory storage device, auxiliary storage unit 1050. Then, when executing a process, the computer loads the program stored in auxiliary storage unit 1050, its own non-transitory storage device, into storage unit 1020 and executes processing in accordance with the loaded program. Alternatively, as another form of execution of this program, the computer may load the program directly from a portable recording medium into storage unit 1020 and execute processing in accordance with the program. Furthermore, the computer may execute processing in accordance with the received program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the processing function by issuing an execution instruction and obtaining the results, thereby executing the above-described processing through a so-called ASP (Application Service Provider) type service. Note that, in this embodiment, the program includes information used for processing by a computer that is equivalent to a program (e.g., data that is not a direct instruction to the computer but has properties that define computer processing).

[0101] In this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing may be realized by hardware. For example, the spectrogram generator 1, patch generator 2, mask processor 3, reconstructor 4, parameter updater 5, spectrogram generator 11, patch generator 12, mask processor 13, reconstructor 14, integrator 15, and time domain transformer 16 may be configured by processing circuits.

[0102] It goes without saying that other modifications are possible without departing from the spirit of the present invention.

Claims

1. A learning device comprising: a spectrogram generation unit that generates a spectrogram from an input first sound signal and generates a target spectrogram from an input second sound signal; a patch generation unit that divides the generated spectrogram to generate a plurality of patches and divides the generated target spectrogram to generate a plurality of target patches; a mask processing unit that selects some of the plurality of patches as masked patches; a reconstruction unit that obtains a plurality of reconstructed patches by reconstructing the plurality of patches through processing by an encoder and a decoder in a Transformer that is a deep learning model, using visible patches that are patches other than the some of the patches among the plurality of patches and mask tokens corresponding to the masked patches; and a parameter update unit that updates parameters of the encoder and the decoder so that a target patch corresponding to the masked patch among the plurality of target patches and a reconstructed patch corresponding to the masked patch among the plurality of reconstructed patches become closer to each other.

2. A conversion device comprising: a storage unit in which an encoder and a decoder trained by the learning device of claim 1 are stored; a spectrogram generation unit that generates a spectrogram from an input sound signal; a patch generation unit that divides the generated spectrogram to generate a plurality of patches; a mask processing unit that selects some of the plurality of patches as masked patches; and a reconstruction unit that obtains a plurality of reconstructed patches by reconstructing the plurality of patches through processing by the encoder and decoder read from the storage unit using visible patches that are patches other than the some of the plurality of patches and mask tokens corresponding to the masked patches.

3. A conversion device according to claim 2, wherein the processing of the mask processing unit and the reconstruction unit is performed repeatedly, and during the repeated processing, the mask processing unit selects some or all of the patches that have not been selected so far from the plurality of patches as masked patches, and the conversion device further includes an integration unit that performs, after the repeated processing is completed, a process of integrating reconstructed patches corresponding to the masked patches from the plurality of reconstructed patches to generate a reconstructed spectrogram, and a time domain conversion unit that converts the reconstructed spectrogram into a sound signal in the time domain.

4. A conversion device according to claim 3, wherein the plurality of patches are arranged on a two-dimensional plane, and the mask processing unit selects masked patches in a checkerboard pattern.

5. A conversion device according to claim 3, wherein the plurality of patches are arranged on a two-dimensional plane, and the mask processing unit selects every other masked patch in one coordinate axis direction of a coordinate system of the two-dimensional plane.

6. A learning method including: a spectrogram generation step in which a spectrogram generation unit generates a spectrogram from an input first sound signal and generates a target spectrogram from an input second sound signal; a patch generation step in which a patch generation unit divides the generated spectrogram to generate a plurality of patches and divides the generated target spectrogram to generate a plurality of target patches; a mask processing step in which a mask processing unit selects some patches from the plurality of patches as masked patches; a reconstruction step in which a reconstruction unit obtains a plurality of reconstructed patches by reconstructing the plurality of patches through processing of an encoder and a decoder in a Transformer that is a deep learning model, using visible patches that are patches other than the some patches from the plurality of patches and mask tokens corresponding to the masked patches; and a parameter update step in which a parameter update unit updates parameters of the encoder and the decoder so that a target patch corresponding to the masked patch from the plurality of target patches and a reconstructed patch corresponding to the masked patch from the plurality of reconstructed patches become closer to each other.

7. A conversion method comprising: a spectrogram generation step in which a spectrogram generation unit generates a spectrogram from an input sound signal; a patch generation step in which a patch generation unit divides the generated spectrogram to generate a plurality of patches; a mask processing step in which a mask processing unit selects some patches from the plurality of patches as masked patches; and a reconstruction step in which a reconstruction unit obtains a plurality of reconstructed patches by reconstructing the plurality of patches by processing an encoder and a decoder read from a storage unit in which an encoder and a decoder trained by the learning device of claim 6 are stored, using visible patches which are patches other than the some patches from the plurality of patches and mask tokens corresponding to the masked patches.

8. A program for causing a computer to function as each part of the learning device of claim 1 or each part of the conversion device of claim 2.