Learning device, conversion device, method, and program

Abstract

This learning device comprises: a spectrogram generation unit 1 that generates a spectrogram from a first sound signal, and generates a target spectrogram from a second sound signal; a patch generation unit 2 that generates multiple patches by dividing the spectrogram, and generates multiple target patches by dividing the target spectrogram; a mask processing unit 3 that selects, as a masked patch, some patches among the multiple patches; a reconstruction unit 4 that, using a visible patch that is a patch other than the some patches among the multiple patches, and a mask token corresponding to the masked patch, obtains multiple reconstructed patches obtained by reconstructing the multiple patches by processing by an encoder and a decoder; and a parameter updating unit 5 that updates parameters of the encoder and the decoder such that a target patch corresponding to the masked patch among the multiple target patches and a reconstructed patch corresponding to the masked patch among the multiple reconstructed patches approach each other.

Description

Technical Field

[0001] The present invention relates to deep learning technology.

Background Art

[0002] Especially in fields other than sound, there are frameworks for representation learning such as Masked Language Modeling (MLM) and Masked Image Modeling (MIM). These are frameworks that mask a part of the input and use the information obtained from the unmasked part to restore and predict the masked part. These frameworks aim to obtain a good information extraction model (also called an Encoder and abstractly called a "representation") by learning to extract effective information from the unmasked part.

[0003] In the case of sound signals, Non-Patent Document 1 proposes a learning method that combines mask restoration and classification for spectrograms.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technology of Non-Patent Document 1 could not be used to solve general tasks such as noise removal, super-resolution, style conversion, and anomaly detection.

[0006] The object of this invention is to provide a learning device for learning a model that can be used to solve general tasks, a conversion device, a method, and a program for enabling the solving of general tasks. [Means for solving the problem]

[0007] A learning device according to one aspect of this invention includes: a spectrogram generation unit that generates a spectrogram from an input first sound signal and generates a target spectrogram from an input second sound signal; a patch generation unit that divides the generated spectrogram to generate a plurality of patches and divides the generated target spectrogram to generate a plurality of target patches; a mask processing unit that selects some of the plurality of patches as masked patches; a reconstruction unit that uses visible patches, which are patches other than some of the plurality of patches, and mask tokens corresponding to the masked patches to obtain a plurality of reconstructed patches by processing the encoder and decoder in a transformer, which is a deep learning model; and a parameter update unit that updates the encoder parameters and decoder parameters so that the target patch corresponding to the masked patch in the plurality of target patches and the reconstructed patch corresponding to the masked patch in the plurality of reconstructed patches are close together.

[0008] A conversion device according to one aspect of this invention is: Using visible patches (all patches except some) and mask tokens (all patches except some) from the multiple patches generated by dividing the spectrogram produced from the first tone signal, and mask tokens corresponding to the masked patches (some patches), multiple reconstructed patches are obtained by reconstructing the multiple patches, and the reconstructed patch corresponding to the masked patch among the multiple reconstructed patches approaches the target patch corresponding to the masked patch among the multiple target patches generated by dividing the target spectrogram produced from the second tone signal. The system comprises a storage unit that stores learned encoders and decoders, a spectrogram generation unit that generates a spectrogram from an input sound signal, a patch generation unit that divides the generated spectrogram to generate multiple patches, a mask processing unit that selects some of the patches from the multiple patches as masked patches, and a reconstruction unit that uses visible patches (which are the patches other than some of the multiple patches) and mask tokens corresponding to the masked patches to obtain multiple reconstructed patches by processing the encoders and decoders read from the storage unit. [Effects of the Invention]

[0009] It is possible to train a model that can be used to solve common tasks, or to enable the solution of common tasks. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows an example of the functional configuration of a learning device. [Figure 2] Figure 2 shows an example of the processing procedure for a learning method. [Figure 3] Figure 3 is a schematic diagram showing an example of a learning device and method. [Figure 4] Figure 4 shows an example of a computer's functional configuration. [Figure 5] Figure 5 shows an example of the functional configuration of a conversion device. [Figure 6] Figure 6 shows an example of the processing procedure for the conversion method. [Figure 7] Figure 7 shows an example of mask processing. [Figure 8] Figure 8 shows an example of mask processing. [Figure 9] Figure 9 shows an example of mask processing. [Figure 10] Figure 10 shows an example of the functional configuration of a conversion device. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described in detail below. In the drawings, components having the same function are given the same number, and redundant explanations are omitted.

[0012] [Learning device and method] As shown in Figure 1, the learning device includes, for example, a spectrogram generation unit 1, a patch generation unit 2, a mask processing unit 3, a reconstruction unit 4, a parameter update unit 5, and a storage unit 6.

[0013] The learning method is realized, for example, by each component of the learning device performing the processes from step S1 to step S5 shown in FIG. 2.

[0014] <Spectrogram generation unit 1> The spectrogram generation unit 1 receives a first sound signal and a second sound signal such as an acoustic signal and a voice signal.

[0015] The spectrogram generation unit 1 generates a spectrogram from the input first sound signal and generates a target spectrogram from the input second sound signal (step S1).

[0016] The generated spectrogram and target spectrogram are output to the patch generation unit 2.

[0017] For example, if the number of spectrogram bins is B and the number of frames of the sound signal is F, a spectrogram and a target spectrogram of size B×F are generated. B and F are predetermined positive integers. For example, B = 80 and F = 304. The time length of one frame is 10 ms.

[0018] <Patch generation unit 2> The patch generation unit 2 receives the spectrogram and the target spectrogram generated by the spectrogram generation unit 1.

[0019] The patch generation unit 2 divides the spectrogram to generate a plurality of patches and divides the target spectrogram to generate a plurality of target patches (step S2).

[0020] The generated plurality of patches are output to the mask processing unit 3, the reconstruction unit 4, and the parameter update unit 5.

[0021] The generated plurality of target patches are output to the parameter update unit 5.

[0022] For example, the patch generation unit 2 divides a spectrogram of size B × F into patches of size b × f. b and f are predetermined positive integers. b may be a divisor of B, and f may be a divisor of F. For example, if B=80, F=304, and b=16, f=16, then 95 patches of size 16 × 16 are generated from an 80 × 304 spectrogram (80 / 16) × (304 / 16) = 5 × 19 = 95. The same applies to the target spectrogram.

[0023] In this way, the patch generation unit 2 divides the spectrogram and the target spectrogram into a grid, for example.

[0024] <Mask processing 3> The mask processing unit 3 receives multiple patches generated by the patch generation unit 2 as input.

[0025] The mask processing unit 3 selects some of the patches from among multiple patches as masked patches (step S3).

[0026] Information about the patch selected as a masked patch is output to the reconstruction unit 4 and the parameter update unit 5.

[0027] The masking unit 3 selects, for example, X% of the patches from among multiple patches as masked patches. X is a real number between 0 and 100. For example, X is 75. The masking unit 3 may also select 50% or more of the patches from among multiple patches as masked patches.

[0028] The mask processing unit 3 may select patches randomly, or it may select patches according to a predetermined pattern.

[0029] Figure 3 shows a schematic of an example of the processing of the learning device and method. In the example in Figure 3, the spectrogram is divided into 5x6 patches, as shown in INPUT. In the example in Figure 3, the patches shown by the dashed lines are selected as masked patches, as shown in INPUT.

[0030] <Reconstruction section 4> The reconstruction unit 4 receives information about the masked patch selected by the mask processing unit 3. The reconstruction unit 4 also receives multiple patches generated by the patch generation unit 2.

[0031] The reconstruction unit 4 uses visible patches, which are patches other than masked patches among the multiple patches, and mask tokens corresponding to the masked patches, to obtain multiple reconstructed patches by processing the multiple patches using encoders and decoders in the transformer, which is a deep learning model (step S4).

[0032] The mask token is a vector. The initial value of each element of the mask token is, for example, a random real number. As will be described later, the mask token may be updated by the parameter update unit 5. In this case, the reconstruction unit 4 uses the updated mask token.

[0033] The multiple reconstruction patches obtained are output to the parameter update unit 5.

[0034] Examples 1 and 2 of the processing in the reconstruction unit 4 are described below.

[0035] <<Example 1>> The encoder processing unit 41 of the reconstruction unit 4 inputs the visible patch to the encoder in the transformer, which is a deep learning model, and obtains the encoding result.

[0036] The decoder processing unit 42 of the reconstruction unit 4 inputs the encoded result and the mask token to the decoder to obtain a decoded result. As a result of the decoded result, multiple reconstructed patches are obtained by reconstructing multiple patches.

[0037] In the example in Figure 3, visible patches, which are patches other than the masked patches shown by the dashed line in INPUT, are input to the encoder, and the encoded result shown by the dotted line is obtained. In the example in Figure 3, the encoded result plus the mask token corresponding to the masked patch is input to the decoder, and the decoded result shown by the diagonal stripe pattern sloping downwards to the right is obtained.

[0038] <<Example 2>> The encoder processing unit 41 of the reconstruction unit 4 inputs the visible patch and the mask token to the encoder in the deep learning model, which is a transformer, and obtains the encoding result.

[0039] The decoder processing unit 42 of the reconstruction unit 4 inputs the encoding result to the decoder and obtains a decoding result. As a decoding result, multiple reconstructed patches are obtained by reconstructing multiple patches.

[0040] <Parameter update section 5> The parameter update unit 5 receives multiple reconstruction patches obtained by the reconstruction unit 4. The parameter update unit 5 also receives information about the masked patch selected by the mask processing unit 3, multiple patches generated by the patch generation unit 2, and multiple target patches.

[0041] The parameter update unit 5 updates the encoder parameters and decoder parameters so that the target patch corresponding to the masked patch among the multiple target patches and the reconstructed patch corresponding to the masked patch among the multiple reconstructed patches are brought closer together (step S17).

[0042] The parameter update unit 5 updates the encoder parameters and decoder parameters using a method such as backpropagation, for example, so that the magnitude of the error between the target patch corresponding to the masked patch among the multiple target patches and the reconstructed patch corresponding to the masked patch among the multiple reconstructed patches is reduced.

[0043] In the example in Figure 3, the parameters are updated to minimize the error between the target patch corresponding to the masked patch shown by the dashed line in TARGET and the reconstructed patch corresponding to the masked patch shown by the dashed line in RECONSTRUCTION.

[0044] <<Repeat process>> The processes from step S1 to step S5 are repeated. For example, the processes from step S1 to step S5 are performed for each of several different audio signals.

[0045] For example, steps S1 to S5 are repeated until the encoder parameters and decoder parameters converge.

[0046] <Storage section 6> Once the iteration is complete, the encoder and decoder model, determined by the latest parameters updated by the parameter update unit 5, is stored in the storage unit 6.

[0047] <<Modified Examples of Learning Apparatus and Method>> The duration of the spectrogram may be longer than a predetermined duration. In other words, the number of frames of the sound signal input to the spectrogram generation unit 1 may be greater than a predetermined number of frames.

[0048] For example, the duration of the spectrogram may be set to 4 seconds or longer. Increasing the duration of the spectrogram improves the performance of the model generated by the learning device and method.

[0049] The patch size may be smaller than the specified size. For example, the patch size may be smaller than 16x8, or smaller than 16x4. Reducing the patch size increases the resolution, thus improving the performance of the model generated by the learning device and method.

[0050] [Conversion device and method] As shown in Figure 5, the conversion device includes, for example, a spectrogram generation unit 11, a patch generation unit 12, a mask processing unit 13, a reconstruction unit 14, an integration unit 15, a time-domain conversion unit 16, and a storage unit 6.

[0051] The conversion method is achieved, for example, by having each component of the conversion device perform the processing described below and shown in Figure 6, from step S11 to step S16.

[0052] <Storage section 6> The memory unit 6 stores models (encoders and decoders in a Transformer) that have been learned by the learning device and method.

[0053] The spectrogram generation unit 11, patch generation unit 12, mask processing unit 13, and reconstruction unit 14, described below, perform the same processing as the spectrogram generation unit 1, patch generation unit 2, mask processing unit 3, and reconstruction unit 4 in the learning device. Parts that are the same as those in the spectrogram generation unit 1, patch generation unit 2, mask processing unit 3, and reconstruction unit 4 will not be explained again.

[0054] <Spectrogram generation unit 11> The spectrogram generation unit 11 receives sound signals such as acoustic signals and speech signals that are to be converted.

[0055] The spectrogram generation unit 11 generates a spectrogram from the input sound signal (step S11).

[0057] The generated spectrogram is output to the patch generation unit 12.

[0058] <Patch generation section 12> The patch generation unit 12 receives the spectrogram generated by the spectrogram generation unit 11 as input.

[0059] The patch generation unit 12 divides the spectrogram and generates multiple patches (step S12).

[0060] The generated multiple patches are output to the mask processing unit 13 and the reconstruction unit 14.

[0061] <Mask processing 13> The mask processing unit 13 receives multiple patches generated by the patch generation unit 2 as input.

[0062] The mask processing unit 13 selects some of the patches from among multiple patches as masked patches (step S13).

[0063] Information about the patch selected as a masked patch is output to the reconstruction unit 14 and the integration unit 15.

[0064] <Reconstruction part 14> The reconstruction unit 14 receives information about the masked patch selected by the mask processing unit 13. The reconstruction unit 14 also receives multiple patches generated by the patch generation unit 12.

[0065] The reconstruction unit 14 uses visible patches, which are patches other than some of the patches among the multiple patches, and mask tokens corresponding to masked patches, to obtain multiple reconstructed patches by processing the encoder and decoder read from the storage unit 6 (step S14).

[0066] The multiple reconfiguration patches obtained are output to the integration unit 15.

[0067] <<Repeated processing of mask processing unit 13 and reconstruction unit 14>> In the conversion apparatus and method, the processing of the mask processing unit 13 and the reconstruction unit 14 is performed repeatedly. During this repeated processing, the mask processing unit 13 selects some or all of the patches that have not been selected so far from among the multiple patches as masked patches.

[0068] The mask processing unit 13 and the reconstruction unit 14 continue processing until all patches among the multiple patches are selected as masked patches and a reconstruction patch corresponding to all patches among the multiple patches is obtained.

[0069] For example, assuming that multiple patches are arranged on a two-dimensional plane, the mask processing unit 13 may select masked patches in a checkerboard pattern. In this case, for example, in the first processing step, the mask processing unit 13 selects the patches shown by solid lines in Figure 7 as masked patches. Then, in the second processing step, the mask processing unit 13 selects the patches shown by dashed lines in Figure 7 as masked patches.

[0070] Furthermore, assuming that multiple patches are arranged on a two-dimensional plane, the mask processing unit 13 may select every other masked patch along one of the coordinate axis directions of the two-dimensional coordinate system.

[0071] In this case, for example, in the first processing step, the mask processing unit 13 selects the patch shown by the solid line in Figure 8 as the masked patch. Then, in the second processing step, the mask processing unit 13 selects the patch shown by the dashed line in Figure 8 as the masked patch. In other words, the mask processing unit 13 may select masked patches every other patch in the vertical direction.

[0072] In this case, for example, in the first processing step, the mask processing unit 13 may select the patches shown with solid lines in Figure 9 as masked patches, and in the second processing step, it may select the patches shown with dashed lines in Figure 9 as masked patches. In other words, the mask processing unit 13 may select masked patches every other patch in the horizontal direction.

[0073] In the example explained using Figures 7 to 9, all patches among multiple patches are selected as masked patches in two processes, and a reconstructed patch corresponding to all patches among the multiple patches is obtained.

[0074] <Integration Department 15> The integration unit 15 receives information about the masked patch selected by the mask processing unit 13. The integration unit 15 also receives multiple reconstructed patches obtained by the reconstruction unit 14.

[0075] The integration unit 15 performs the process of integrating the reconstruction patches corresponding to the masked patches among the multiple reconstruction patches to generate a reconstruction spectrogram after the iterative processing is completed (step S15).

[0076] The generated reconstructed spectrogram is output to the time-domain conversion unit 16.

[0077] <Time Domain Conversion Unit 16> The time-domain conversion unit 16 receives the reconstructed spectrogram generated by the integration unit 15 as input.

[0078] The time-domain conversion unit 16 converts the reconstructed spectrogram into a time-domain sound signal (step S16).

[0079] <<Conversion Example 1>> Assume that a model learned by providing a first tone signal which is a mix of the desired sound and noise, and a second tone signal which is only the desired sound, is stored in the memory unit 6.

[0080] In this case, by setting the input of the conversion device and method to an audio signal from which noise is to be removed, the time-domain conversion unit 16 can obtain an audio signal with reduced noise. In other words, in this case, the conversion device and method can convert an audio signal containing noise into an audio signal with reduced noise.

[0081] <<Conversion Example 2>> Assume that a model learned by providing a low-resolution version of the desired sound as the first sound signal and a high-resolution version of the desired sound as the second sound signal is stored in the memory unit 6.

[0082] In this case, by using a low-resolution audio signal as the input to the conversion device and method, a high-resolution audio signal can be obtained by the time-domain conversion unit 16. In other words, in this case, the conversion device and method can convert a low-resolution audio signal into a high-resolution audio signal.

[0083] <<Conversion Example 3>> Assume that a model learned by providing a sound signal of the desired source style as the first sound signal and a sound signal of the desired destination style as the second sound signal is stored in the memory unit 6.

[0084] In this case, by using an audio signal of the source style as the input to the conversion device and method, the time-domain conversion unit 16 obtains an audio signal of the target style. In other words, in this case, the conversion device and method can convert an audio signal of the source style to an audio signal of the target style.

[0085] Examples of (source style audio signal, destination style audio signal) include (speaker A's voice, speaker B's voice), (classical music, rock music), and (sound of knocking on a wooden door, sound of knocking on an iron door).

[0086] <<Modified Examples of Conversion Devices and Methods>> As shown in Figure 10, the conversion device is assumed to include an abnormal sound detection unit 17 instead of a time-domain conversion unit 16. The abnormal sound detection unit 17 receives the reconstructed spectrogram generated by the integration unit 15 and the spectrogram generated by the spectrogram generation unit 11 as input.

[0087] Assume that a model learned by providing normal sounds in an environment where abnormal sounds are to be detected as the first and second sound signals is stored in the memory unit 6.

[0088] In this case, by setting the input of the conversion device and method to the ambient sound of the environment in which abnormal sounds are to be detected, the integration unit 15 obtains a reconstructed spectrogram of the ambient sound of the environment in which abnormal sounds are to be detected.

[0089] The abnormal sound detection unit 17 calculates the magnitude of the error between the spectrogram and the reconstructed spectrogram, and determines that there is an abnormality if the calculated magnitude of the error is greater than a predetermined threshold, and determines that there is a normal condition otherwise.

[0090] For example, suppose a model learned by providing normal machine operation sounds as the first and second sound signals is stored in the memory unit 6. Then, suppose a sound produced by a malfunctioning machine is input to the conversion device and method. In this case, since the sound produced by the machine malfunction does not exist during learning, the magnitude of the error between the spectrogram and the reconstructed spectrogram will be large.

[0091] Similarly, suppose a model learned by providing normal street sounds as the first and second sound signals is stored in the memory unit 6. Then, suppose a street sound containing a scream is input to the conversion device and method. In this case, since the scream does not exist during learning, the magnitude of the error between the spectrogram and the reconstructed spectrogram will be large.

[0092] In this way, by utilizing the fact that the error increases when abnormal sounds are included, it becomes possible to detect anomalies.

[0093] As in this example, the conversion device and method may also be used as an abnormal sound detection device and method.

[0094] [Differentiation] Although embodiments of the present invention have been described above, the specific configuration is not limited to these embodiments, and it goes without saying that any design modifications or other changes made as appropriate without departing from the spirit of the present invention are still included in the present invention.

[0095] The various processes described in the embodiments may be executed not only in chronological order according to the sequence described, but also in parallel or individually as needed, depending on the processing capacity of the device performing the processes.

[0096] For example, data exchange between components of a learning device and data exchange between components of a conversion device may be performed directly, or it may be performed via a storage unit (not shown).

[0097] [Programs, recording media] The processing of each part of the above-mentioned devices may be implemented by a computer. In this case, the processing content of the functions that each device should have is described by a program. This program is then loaded into the memory unit 1020 of the computer 1000 shown in Figure 4, and the arithmetic processing unit 1010, input unit 1030, output unit 1040, display unit 1060, etc. are operated, thereby realizing the various processing functions of each of the above-mentioned devices on the computer.

[0098] The program describing this process can be recorded on a computer-readable recording medium. Computer-readable recording media are, for example, non-temporary recording media, specifically magnetic recording devices, optical discs, etc.

[0099] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0100] A computer executing such a program first stores the program recorded on a portable recording medium or transferred from a server computer in its own non-temporary storage device, the auxiliary recording unit 1050. Then, when processing is to be executed, the computer reads the program stored in the auxiliary recording unit 1050 into the storage unit 1020 and executes the processing according to the loaded program. Alternatively, the computer may directly read the program from the portable recording medium into the storage unit 1020 and execute the processing according to that program. Furthermore, each time a program is transferred to this computer from a server computer, it may sequentially execute the processing according to the received program. Alternatively, the above processing may be executed by a so-called ASP (Application Service Provider) type service, where the server computer does not transfer programs to this computer, but the processing function is realized only by execution instructions and result acquisition. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (data that is not a direct instruction to the computer but has the property of defining the processing of the computer).

[0101] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processing contents may be implemented in hardware. For example, the spectrogram generation unit 1, patch generation unit 2, mask processing unit 3, reconstruction unit 4, parameter update unit 5, spectrogram generation unit 11, patch generation unit 12, mask processing unit 13, reconstruction unit 14, integration unit 15, and time-domain conversion unit 16 may be configured by processing circuits.

[0102] It goes without saying that the invention may be modified as appropriate without departing from its spirit.

Claims

1. A spectrogram generation unit that generates a spectrogram from the input first sound signal and generates a target spectrogram from the input second sound signal, A patch generation unit that divides the generated spectrogram to generate multiple patches, and divides the generated target spectrogram to generate multiple target patches, A masking processing unit that selects some of the aforementioned multiple patches as masked patches, A reconstruction unit obtains multiple reconstructed patches by processing the multiple patches using visible patches, which are patches other than some of the aforementioned patches, and mask tokens corresponding to the masked patches, through the processing of encoders and decoders in a deep learning model called a transformer. A parameter update unit updates the encoder parameters and decoder parameters so that the target patch corresponding to the masked patch among the plurality of target patches and the reconstructed patch corresponding to the masked patch among the plurality of reconstructed patches are brought closer together. A learning device that includes [this].

2. A storage unit that stores encoders and decoders which have been trained to reconstruct a plurality of patches by using visible patches, which are patches other than some of the patches among a plurality of patches generated by dividing a spectrogram generated from a first sound signal, and mask tokens corresponding to the masked patches, and which have been trained to reconstruct a plurality of patches by using the plurality of visible patches, which are patches other than some of the patches, and mask tokens corresponding to the masked patches, and which have been trained to reconstruct a plurality of patches by using the plurality of visible patches, which are some of the patches, and mask tokens corresponding to the masked patches, and which have been trained to reconstruct a plurality of patches by dividing a target spectrogram generated from a second sound signal, A spectrogram generation unit that generates a spectrogram from an input sound signal, A patch generation unit that divides the generated spectrogram to generate multiple patches, A masking processing unit that selects some of the aforementioned multiple patches as masked patches, A reconstruction unit obtains a plurality of reconstructed patches by processing an encoder and decoder read from the storage unit, using visible patches which are patches other than some of the aforementioned patches among the plurality of patches, and mask tokens corresponding to the masked patches, A conversion device that includes a converter.

3. A conversion device according to claim 2, The mask processing unit and the reconstruction unit process are performed repeatedly. During the aforementioned iterative process, the mask processing unit selects some or all of the previously unselected patches from the plurality of patches as masked patches. The system further includes an integration unit that, after the iterative process is completed, integrates the reconstruction patches corresponding to the masked patch among the plurality of reconstruction patches to generate a reconstructed spectrogram, and a time-domain conversion unit that converts the reconstructed spectrogram into a time-domain sound signal. Conversion device.

4. A conversion device according to claim 3, Assuming that the aforementioned multiple patches are arranged on a two-dimensional plane, The mask processing unit selects masked patches in a checkerboard pattern. Conversion device.

5. A conversion device according to claim 3, Assuming that the aforementioned multiple patches are arranged on a two-dimensional plane, The mask processing unit selects every other masked patch in one of the coordinate axis directions of the two-dimensional plane coordinate system. Conversion device.

6. The spectrogram generation unit generates a spectrogram from the input first sound signal and generates a target spectrogram from the input second sound signal in a spectrogram generation step, The patch generation unit divides the generated spectrogram to generate multiple patches, and divides the generated target spectrogram to generate multiple target patches in a patch generation step, A masking step in which the masking processing unit selects some of the patches from the plurality of patches as masked patches, The reconstruction unit uses visible patches, which are patches other than some of the patches among the plurality of patches, and mask tokens corresponding to the masked patches, to obtain a plurality of reconstructed patches by processing the plurality of patches using encoders and decoders in a deep learning model called a transformer. A parameter update step in which the parameter update unit updates the encoder parameters and decoder parameters so that the target patch corresponding to the masked patch among the plurality of target patches and the reconstructed patch corresponding to the masked patch among the plurality of reconstructed patches are brought closer together. Learning methods that include this.

7. The spectrogram generation unit generates a spectrogram from the input sound signal in a spectrogram generation step, The patch generation unit performs a patch generation step of dividing the generated spectrogram to generate multiple patches, A masking step in which the masking processing unit selects some of the patches from the plurality of patches as masked patches, The reconstruction step involves a reconstruction unit obtaining multiple reconstructed patches by reconstructing the multiple patches using visible patches, which are patches other than some of the patches among the multiple patches, and mask tokens corresponding to the masked patches, and processing encoders and decoders read from a storage unit that stores encoders and decoders learned to make the reconstructed patch corresponding to the masked patch among the multiple reconstructed patches approach the target patch corresponding to the masked patch among the multiple target patches generated by dividing the target spectrogram generated from the second sound signal, thereby obtaining multiple reconstructed patches by reconstructing the multiple patches. A conversion method that includes this.

8. A program for causing a computer to function as each part of the learning device according to claim 1 or each part of the conversion device according to claim 2.

Images