Generation device, generation method, and program

Abstract

The present technology relates to a generation device, a generation method, and a program that make it possible to use a simple, lightweight generative model to generate not-inputted data of related data of a plurality of modalities. A generation device according to the present invention uses a generative model that is a transformer model for a single modality to generate not-inputted data that is data that is not inputted of related data of a plurality of modalities. The present technology can be applied, for example, to a generation device or the like that uses a generative model that is a transformer model for a single modality to generate one or another or both of related image data and audio data.

Description

Generation device, generation method, and program

[0001] The present technology relates to a generation device, a generation method, and a program, and in particular to a generation device, a generation method, and a program that are capable of generating uninput data among data of multiple related modalities using a simple and lightweight generation model.

[0002] In recent years, generative AI (Artificial Intelligence) has become one of the most creative and dynamic fields in the machine learning industry. For example, numerous still image generation models capable of synthesizing photographic images have been developed, and many still image generation models that enable individual control over the required generation have also been developed. Among still image generation models, models based on diffusion models, which have excellent performance and scalability, are the mainstream (see, for example, Patent Document 1).

[0003] Along with the advancement of still image generation technology, research is also progressing on generation technologies for other modalities such as video images, audio (speech and music), text, depth images, and segmentation maps. In the following, when there is no need to particularly distinguish between still images and video images, they will be collectively referred to as images. As for image and audio generation technology, conditional generation technology using only text as input (text2image, text2audio) is rapidly advancing.

[0004] On the other hand, multi-modality generation technologies other than image and audio and text have little progress, despite their high practical value in improving creativity and machine cognition. For example, the progress of technologies for generating images from audio (audio2image, audio2video) and images from audio (image2audio, video2audio) has been relatively slow.

[0005] Techniques for generating audio from images include Im2Wav, which generates audio using cascaded autoregressive transformer models conditioned on CLIP image features. Another technique for generating audio from images is DiffFoley, which uses a model pre-trained on contrasting audio and video to train a latent diffusion model to encode video information. Techniques for generating images from audio include Sound2Scene, which first learns to align image and audio features and then uses the audio features to generate images using a pre-trained BigGAN.

[0006] To achieve more flexible generation, attempts have been made to achieve arbitrary cross-modality generation (any-to-any generation) using a single generative model. Examples of single generative models for arbitrary cross-modality generation include well-pretrained large language models (LLMs), combinations of multiple single-modality diffusion models, and large multi-modality transformer models. These generative models are trained using data from multiple modalities, such as text, images, and speech, enabling multi-dimensional conditional generation, thereby enabling superior arbitrary cross-modality generation.

[0007] However, these generative models typically require large amounts of training data, huge models with over a billion parameters (especially when LLMs are employed), and extensive engineering techniques for large-scale training.

[0008] US Patent Application Publication No. 2024 / 0037812

[0009] Therefore, there is a need for a method to generate non-input data, which is data that is not input from multiple modalities such as related images and audio, using a simple and lightweight generative model, but such a method has not been considered.

[0010] The present technology was developed in light of this situation, and makes it possible to generate uninput data from data of multiple related modalities using a simple and lightweight generative model.

[0011] A generating device according to one aspect of the present technology is a generating device that includes a generating unit that generates non-input data, which is data that is not input, from data of multiple associated modalities using a generating model that is a transformer model for a single modality.

[0012] A generation method according to one aspect of the present technology is a generation method that includes a generation device using a generation model that is a transformer model for a single modality to generate non-input data, which is data that is not input, from data of multiple associated modalities.

[0013] A program according to one aspect of the present technology is a program for causing a computer to execute a process including generating non-input data, which is data that is not input, from data of multiple associated modalities using a generative model that is a transformer model for a single modality.

[0014] In one aspect of the present technology, a generative model that is a transformer model for a single modality is used to generate non-input data, which is data that is not input among data of multiple associated modalities.

[0015] The generating device may be a stand-alone device or a module that is incorporated into another device.

[0016] FIG. 10 is a block diagram showing an example configuration of a learning device. FIG. 11 is a diagram explaining the flow of data in the learning device. FIG. 12 is a flowchart explaining learning processing. FIG. 13 is a block diagram showing an example configuration of a first embodiment of a generation device to which the present technology is applied. FIG. 14 is a diagram explaining the flow of data in the generation device of FIG. 4. FIG. 15 is a flowchart explaining generation processing. FIG. 16 is a block diagram showing an example configuration of a second embodiment of a generation device to which the present technology is applied. FIG. 17 is a diagram explaining the flow of data in the generation device of FIG. 7. FIG. 18 is a first diagram explaining the effect of the generation device. FIG. 19 is a second diagram explaining the effect of the generation device. FIG. 19 is a block diagram showing an example configuration of computer hardware.

[0017] Hereinafter, modes for implementing the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order: 1. Learning device 2. First embodiment (generation device using a generative model that does not use CFG (Classifier-Free Guidance)) 3. Second embodiment (generation device using a generative model that uses CFG) 4. Description of effects 5. Description of computer to which the present technology is applied

[0018] In the following, we will use the example of generating uninputted data from related image and audio data as an explanation, but this technology can also be applied to generating uninputted data from data of any multiple related modalities.

[0019] 1. Learning Device Configuration Example of Learning Device FIG. 1 is a block diagram showing a configuration example of a learning device that learns a generative model that is a transformer model for a single modality used in a generation device to which the present technology is applied.

[0020] The learning device 10 in FIG. 1 includes an audio preprocessing unit 11, an audio VQGAN (Vector-Quantized Generative Adversarial Network) encoder 12, a mask adding unit 13, an image preprocessing unit 14, an image VQGAN encoder 15, and a mask adding unit 16. The learning device 10 also includes a concatenation unit 17, a transformer encoder 18, a substitution unit 19, a transformer decoder 20, and a loss calculation unit 21. Pairs of related image data and audio data are input to the learning device 10 as training data. Here, the training data are assumed to be pairs of image data and audio data for 10 seconds captured simultaneously by a video camera or the like.

[0021] The audio preprocessing unit 11 converts the audio data from the input learning data into a mel spectrogram A∈R Ｃａ×Ｔ where Ca is the mel channel (frequency), and is, for example, 80 here. T is the number of frames (time), and is, for example, 848 here. The audio preprocessing unit 11 supplies the mel spectrogram A to the audio VQGAN encoder 12.

[0022] The audio VQGAN encoder 12 is an encoder for the audio VQGAN, and converts the mel spectrogram A supplied from the audio preprocessing unit 11 into an audio token, which is a discrete index matrix. Here, for example, the audio token is a 5x53 index matrix. The audio VQGAN is, for example, a SpecVQGAN pre-trained with AudioSet and is not updated during training. Details of SpecVQGAN are described, for example, in Vladimir Iashin and Esa Rahtu, "Taming visually guided sound generation," BMVC 2021.

[0023] Here, the size Csa of the audio VQGAN codebook is, for example, 1024, and the embedding dimension is 256. The audio VQGAN encoder 12 converts the generated audio tokens into a one-dimensional audio token sequence a∈{1, ..., Csa} La Here, La is the length of the audio token, which is 265 (= 5 × 53) in this example. The audio VQGAN encoder 12 supplies the reconstructed audio token sequence a to the mask adding unit 13.

[0024] The mask adding unit 13 randomly adds masks at a predetermined masking rate to the audio tokens that make up the audio token sequence a supplied from the audio VQGAN encoder 12. This masking rate is determined from a truncated Gaussian distribution (truncated normal distribution) having a hyperparameter μ a in the range of 0.2 to 1 as the mean value, for example, with reference to MAGE (MAsked Generative Encoder).

[0025] The masking unit 13 randomly deletes a predetermined percentage of the audio token sequence after the masking. At this time, audio tokens are deleted randomly regardless of whether they are masked or not. This reduces the amount of calculation required during training. Here, the deletion percentage is 50%, but can be any value. The masking unit 13 supplies the audio token sequence after the deletion to the concatenation unit 17.

[0026] The image preprocessing unit 14 uniformly samples, for example, 10 frames from the image data of the input learning data, and supplies image data V of any one of the 10 frames to the image VQGAN encoder 15. The resolution of the image data is, for example, 256 × 256 pixels.

[0027] The image VQGAN encoder 15 is an encoder for the image VQGAN, and converts the image data V supplied from the image pre-processing unit 14 into image tokens, which are discrete index matrices. Here, for example, the image tokens are 16 × 16 index matrices. Note that, hereinafter, when there is no need to particularly distinguish between audio tokens and image tokens, they will be collectively referred to as tokens. The image VQGAN is a VQGAN that has been pre-trained on, for example, ImageNet, and is not updated during training.

[0028] Here, the codebook size Csv and embedding dimension of the image VQGAN are the same as the codebook size Csa and embedding dimension of the audio VQGAN, but may be different. The image VQGAN encoder 15 converts the generated image tokens into a one-dimensional image token sequence v∈{1,...,Csv} Lv where Lv is the length of the image token, which is 256 (=16×16) in this example. The image VQGAN encoder 15 supplies the reconstructed image token sequence v to the mask adding unit 16.

[0029] The mask adding unit 16 randomly adds masks at a predetermined mask rate to the image tokens that make up the image token sequence v supplied from the image VQGAN encoder 15. This mask rate is determined from a truncated Gaussian distribution with a hyperparameter μv ranging from 0.2 to 1 as the mean value, for example, with reference to MAGE.

[0030] The mask adding unit 16 randomly deletes a predetermined percentage of the image token sequence after the mask is added. At this time, image tokens are randomly deleted regardless of whether a mask is added or not. This reduces the amount of calculation during training. Here, the deletion percentage is 50%, but it can be set to any value. The mask adding unit 16 supplies the image token sequence after the deletion to the concatenation unit 17.

[0031] The concatenation unit 17 converts the discrete audio token sequence provided by the mask addition unit 13 into a token embedding representation that is easy to process with a transformer such as BERT (Bidirectional Encoder Representations from Transformers). The concatenation unit 17 adds a position embedding and a modality embedding as biases to the resulting token embedding representation to generate an audio embedding representation fa. The position embedding and modality embedding representation are randomly initialized, trainable, and optimized during the training process.

[0032] The concatenation unit 17 converts the discrete image token sequence provided by the mask addition unit 16 into a token embedding representation. The concatenation unit 17 adds the location embedding representation and modality embedding representation as biases to the resulting token embedding representation to generate an image embedding representation fv. The location embedding representation and modality embedding representation are randomly initialized, trainable, and optimized during the training process.

[0033] The concatenation unit 17 concatenates the audio embedded representation fa and the image embedded representation fv to generate a concatenated embedded representation (fa, fv). This concatenation is performed by adding one embedded layer for audio BEa and one embedded layer for image BEv. Therefore, the audio embedded representation fa can be expressed as BEa(a), and the image embedded representation fv can be expressed as BEv(v). The concatenation unit 17 supplies the concatenated embedded representation (fa, fv) to the transformer encoder 18.

[0034] The transformer encoder 18 is an encoder of the generative model and encodes the concatenated embedded representation (fa, fv) supplied from the concatenation unit 17. The architecture of this generative model can be, for example, the architecture of the ViT (Vision Transformer) model, which is a non-autoregressive transformer model that does not use the [cls] token. Therefore, this generative model is a simple model in which no modules have been added or the architecture has been changed compared to the ViT model. Note that attention in all attention layers of the generative model is full attention, which differs from the attention with causal mask used in autoregressive transformers. The transformer encoder 18 supplies the encoded results to the substitution unit 19.

[0035] The replacement unit 19 embeds a learnable token embedded representation m that is common to audio and image in the position of the token embedded representation of the deleted token in the encoding result supplied from the transform encoder 18. The replacement unit 19 adds a position embedded representation and a modality embedded representation specific to the modality for each modality to the encoding result in which the token embedded representation m has been embedded and replaced, and supplies the result to the transform decoder 20 as a decoding target.

[0036] The Transform Decoder 20 is a decoder of the generative model, and decodes the decoding target provided by the permutation unit 19 to obtain an embedded representation. The Transform Decoder 20 quantizes this embedded representation by calculating a dot product with a BERT-style embedding that differs for each modality, and then performs a softmax operation. As a result, the Transform Decoder 20 calculates the similarity between the embeddings, i.e., the predicted value (probability) of the audio token, p∈R. Ｌａ×Ｃｓａ and the predicted value of the image token pv∈R Ｌｖ×Ｃｓｖ The predicted values ​​p a and p v correspond to the outputs of the audio VQGAN encoder 12 and the image VQGAN encoder 15, respectively. The transformer decoder 20 supplies the predicted values ​​p a and p v to the loss calculation unit 21.

[0037] The loss calculation unit 21 trains a generative model based on the predicted values ​​p and p so as to minimize the cross-entropy loss L of the masked tokens. This makes it possible to train a generative model that reconstructs information at masked positions using information from unmasked positions in both audio and image modalities.

[0038] The cross-entropy loss L of the token to which the mask is added is calculated by equation (1).

[0039]

[0040] In formula (1), A ｉ ,V ｉ indicate the mel spectrogram and image data of the audio data included in the i-th training data, respectively. ｉ ａ ∈R Ｌａ×Ｃｓａ is the predicted value p a corresponding to the i-th training data, and p ｉ ｖ ∈R Ｌｖ×Ｃｓｖ is the predicted value pv corresponding to the i-th learning data. ｉ ∈{1,...,Csa} La denotes the audio token sequence a output from the audio VQGAN encoder 12 corresponding to the i-th training data, and v ｉ ∈{1,... ,Csv} Lv denotes the image token sequence v output from the image VQGAN encoder 15 corresponding to the i-th training data.

[0041] L ｃｅ (p ｉ ａ , a ｉ ) is the predicted value p ｉ ａ and the audio token sequence a as the ground truth. ｉ represents the cross entropy loss between ｃｅ (p ｉ ｖ , v ｉ ) is the predicted value p ｉ ｖand the image token sequence v as the ground truth ｉ represents the cross entropy loss between ｉ ａ , M ｉ ｖ and indicate whether or not the audio token and image token are masked. ｉ ａ and M ｉ ｖ In this case, 1 is set for tokens to which a mask is added, and 0 is set for tokens to which no mask is added. As described above, the cross-entropy loss L of tokens to which a mask is added is calculated according to equation (1).

[0042] Note that the lower bound of the range of the truncated Gaussian distributions corresponding to the masking rates for audio and image tokens is 0.2 instead of 0.5, which prevents continuous masking at very high rates for both audio and image tokens, preventing training instability due to little information being available to reconstruct one from the other.

[0043] Predicted value p ｉ ａ is the image token sequence v that is treated as the direct input. ｉ It can also be said that the predicted value is a value obtained by predicting the audio token under the condition ｉ ｖ is the audio token sequence a that is treated as the direct input. ｉThe generative model in this technology can be considered as a prediction value of an image token based on the condition. Therefore, the generative model in this technology is similar to the generative models in diffusion model-based conditional image generation technology and conditional audio generation technology. However, in these generative models, the modality conditioning is unidirectional, whereas in the generative model in this technology, the modality conditioning is symmetric. Therefore, the generative model in this technology can perform bidirectional conditional generation and generate both related image data and audio data without additional training. Furthermore, this technology is not limited to audio and image modalities, as it can be easily extended to two or more modalities.

[0044] In this way, the learning device 10 learns a generative model that generates non-input data from related image data and audio data. In the learning device 10, the training data is converted into tokens and input to the generative model, allowing for faster learning than when the training data is input directly.

[0045] <Data Flow in Learning Device> FIG. 2 is a diagram illustrating the data flow in the learning device 10 of FIG.

[0046] 2 , a mel spectrogram A of audio data among the training data is input to the audio VQGAN encoder 12 from the audio pre-processing unit 11. The audio VQGAN encoder 12 generates an audio token sequence a (Tokens) from this mel spectrogram A.

[0047] The audio tokens that make up this audio token sequence a are randomly masked at a predetermined mask rate by the mask adding unit 13. The audio token sequence after masking (Masked Tokens) is then randomly deleted at a predetermined rate by the mask adding unit 13.

[0048] On the other hand, one frame of image data V sampled from image data of multiple frames of the learning data is input to the image VQGAN encoder 15 from the image pre-processing unit 14. The image VQGAN encoder 15 generates an image token sequence v (Tokens) from this image data V.

[0049] The image tokens that make up this image token sequence v are randomly masked at a predetermined mask rate by the mask adding unit 16. The image token sequence after masking (Masked Tokens) is then randomly deleted at a predetermined rate by the mask adding unit 16.

[0050] The concatenated token embeddings (fa, fv) are generated by the concatenation unit 17 from the audio token sequence and image token sequence after deletion obtained as described above, and are input to the transformer encoder 18 .

[0051] The transformer encoder 18 encodes the input concatenated embedded representation (fa, fv). Then, the replacement unit 19 adds and replaces (padding and replace mask) the token embedded representation m to the encoded result. The replacement unit 19 generates a decoding target from the encoded result after embedding and replacing the token embedded representation m, and inputs the generated decoding target to the transformer decoder 20.

[0052] The transformer decoder 20 decodes the input decoding target and calculates prediction values ​​p and p (Token Indices Prediction). The loss calculation unit 21 trains a generative model based on the prediction values ​​p and p, as well as the audio token sequence a and the image token sequence v, so as to minimize the cross-entropy loss L of the masked tokens.

[0053] <Explanation of Learning Process> Fig. 3 is a flowchart illustrating the learning process performed by the learning device 10 of Fig. 1. This learning process starts when learning data is input to the learning device 10, for example.

[0054] In step S11 of FIG. 3 , the audio preprocessing unit 11 converts the audio data among the input learning data into a mel spectrogram A and supplies it to the audio VQGAN encoder 12.

[0055] In step S12, the audio VQGAN encoder 12 generates an audio token sequence a from the mel spectrogram obtained by the processing in step S11, and supplies the audio token sequence a to the mask adding unit 13.

[0056] In step S13, the mask adding unit 13 randomly adds masks at a predetermined mask rate to the audio tokens that make up the audio token sequence a generated in step S12.

[0057] In step S14, the mask adding unit 13 randomly deletes a predetermined proportion of the audio token sequence to which the mask was added in step S13, and supplies the deleted audio token sequence to the concatenation unit 17.

[0058] In step S15, the image preprocessing unit 14 samples image data from the input learning data and supplies the sampled image data V of any one frame to the image VQGAN encoder 15.

[0059] In step S16, the image VQGAN encoder 15 generates an image token sequence v from the image data V sampled in step S15 and supplies it to the mask adding unit 13.

[0060] In step S17, the mask adding unit 16 randomly adds masks at a predetermined mask rate to the image tokens that make up the image token sequence v generated in step S16.

[0061] In step S18 , the mask adding unit 16 randomly deletes a predetermined proportion of the image token sequence to which the mask was added in step S17 and supplies the deleted image token sequence to the concatenation unit 17 .

[0062] In step S19, the concatenation unit 17 generates a concatenated embedded representation (fa, fv) from the audio token sequence deleted in step S14 and the image token sequence deleted in step S18. The concatenation unit 17 supplies the concatenated embedded representation to the transformer encoder 18.

[0063] In step S20, the transformer encoder 18 encodes the concatenated embedded representation (fa, fv) generated in step S19 and supplies the encoded result to the permutation unit 19.

[0064] In step S21, the replacement unit 19 generates a decoding target from the encoding result obtained by the process of step S20, and supplies the generated decoding target to the transformer decoder 20.

[0065] In step S22, the transformer decoder 20 decodes the decoding target generated in step S21 to obtain the predicted values ​​pa and pv. The transformer decoder 20 supplies the predicted values ​​pa and pv to the loss calculation unit 21.

[0066] In step S23, the loss calculation unit 21 trains a generative model based on the predictions p a and p v obtained in step S22, as well as the audio token sequence a and the image token sequence v, so as to minimize the cross-entropy loss L. Then, the training process ends.

[0067] In this way, the learning device 10 learns a generative model with few parameters. Because the learning device 10 learns a generative model using tokens with masks added, it is possible to learn a generative model that has advantages such as compatibility with LLM and visual comprehension.

[0068] 2. First Embodiment FIG. 4 is a block diagram showing an example of the configuration of a first embodiment of a generating device to which the present technology is applied.

[0069] The generation device 100 in Figure 4 includes an audio preprocessing unit 108, an audio VQGAN encoder 109, a mask adding unit 110, an image preprocessing unit 111, an image VQGAN encoder 112, a mask adding unit 113, and a concatenation unit 114. The generation device 100 also includes, as generation units, a transformer encoder 115, a substitution unit 116, a transformer decoder 117, an audio VQGAN decoder 118, a vocoder 119, and an image VQGAN decoder 120. The generation device 100 uses a generative model trained by the training device 10 to generate non-input data of associated image data and audio data.

[0070] Specifically, when at least a portion of audio data is input, the audio pre-processing unit 108 divides the audio data into 10-second segments of audio data. Similar to the audio pre-processing unit 11 in FIG. 1 , the audio pre-processing unit 108 converts the divided 10-second segments of audio data into a mel spectrogram and supplies the mel spectrogram to the audio VQGAN encoder 109.

[0071] The audio VQGAN encoder 109 (tokenization unit), like the audio VQGAN encoder 12, converts the mel spectrogram supplied from the audio pre-processing unit 108 into discrete audio tokens and supplies them to the mask adding unit 110.

[0072] When all audio tokens of the audio data are supplied from the audio VQGAN encoder 109, the mask adding unit 110 reforms the audio tokens into a one-dimensional audio token sequence and supplies it to the concatenation unit 114. When only some audio tokens of the audio data are supplied from the audio VQGAN encoder 109, the mask adding unit 110 adds masked tokens to the audio tokens as audio tokens of the uninput portion. The mask adding unit 110 reforms the resulting audio tokens into a one-dimensional audio token sequence and supplies it to the concatenation unit 114. When no audio tokens are supplied from the audio VQGAN encoder 109, the mask adding unit 110 generates audio tokens in which all are masked. The mask adding unit 110 reforms the audio tokens into a one-dimensional audio token sequence and supplies it to the concatenation unit 114.

[0073] When at least a portion of image data of a moving image is input, the image pre-processing unit 111 supplies one frame of image data from that image data to the image VQGAN encoder 112, similar to the image pre-processing unit 14. When at least a portion of image data of a still image is input, the image pre-processing unit 111 supplies the image data to the image VQGAN encoder 112.

[0074] Similar to the image VQGAN encoder 15, the image VQGAN encoder 112 (tokenization unit) converts one frame of image data supplied from the image pre-processing unit 111 into discrete image tokens and supplies them to the mask addition unit 113.

[0075] When all image tokens of the image data are supplied from the image VQGAN encoder 112, the mask adding unit 113 reshapes the image tokens into a one-dimensional image token sequence and supplies it to the concatenation unit 114. When only some image tokens of the image data are supplied from the image VQGAN encoder 112, the mask adding unit 113 adds masked tokens to the image tokens as image tokens of the unused portion. The mask adding unit 113 reshapes the resulting image tokens into a one-dimensional image token sequence and supplies it to the concatenation unit 114. When no image tokens are supplied from the image VQGAN encoder 112, the mask adding unit 113 generates image tokens in which all are masked. The mask adding unit 113 reshapes the image tokens into a one-dimensional audio token sequence and supplies it to the concatenation unit 114.

[0076] Similar to the concatenation unit 17, the concatenation unit 114 generates an audio embedded representation from the audio token sequence supplied from the mask adding unit 110. Similar to the concatenation unit 17, the concatenation unit 114 generates an image embedded representation from the image token sequence supplied from the mask adding unit 113. The concatenation unit 114 concatenates the audio token embedded representation and the image embedded representation to generate a concatenated embedded representation (concatenated data), and supplies the concatenated embedded representation to the transformer encoder 115.

[0077] The transformer encoder 115 is an encoder of the generative model trained by the training device 10, encodes the concatenated embedded representation supplied from the concatenation unit 114, and supplies the encoded result to the substitution unit 116.

[0078] The replacement unit 116 replaces, with a predetermined token-embedded representation, the token-embedded representation of the masked token in the encoding result supplied from the transformer encoder 115. The replacement unit 116 adds a position-embedded representation and a modality-embedded representation for each modality to the encoding result after replacement, and supplies the result to the transformer decoder 117 as a decoding target.

[0079] The transformer decoder 117 is a decoder of the generative model trained by the training device 10, and similarly to the transformer decoder 20, it decodes the decoding target supplied from the permutation unit 116 and performs quantization and softmax operations. As a result, the transformer decoder 117 obtains a predicted value of the token to which the mask has been added.

[0080] Using this predicted value, the generating device 100 repeatedly generates predicted values, i.e., unmasks. Specifically, the generating device 100 sequentially updates masked tokens in the image token sequence and audio token sequence to (values ​​based on) predicted values ​​at a predetermined ratio, and repeats unmasking N times. As a result, the transformer decoder 117 eventually generates predicted values ​​for all masked tokens.

[0081] The transformer decoder 117 supplies the predicted value of the audio token with the mask added to the audio VQGAN decoder 118 among the generated predicted values, and supplies the predicted value of the image token with the mask added to the image VQGAN decoder 120.

[0082] The audio VQGAN decoder 118 is a decoder for the audio VQGAN. When a predicted value is supplied from the transformer decoder 117, the audio VQGAN decoder 118 converts the predicted value into a mel spectrogram of audio data. The audio VQGAN decoder 118 supplies this mel spectrogram to the vocoder 119. When a mel spectrogram is supplied from the audio VQGAN decoder 118, the vocoder 119 converts the mel spectrogram into audio data using a vocoder such as HiFi-GAN, and outputs the audio data.

[0083] The image VQGAN decoder 120 is a decoder for the image VQGAN. When a predicted value is supplied from the transformer decoder 117, the image VQGAN decoder 120 converts the predicted value into image data and outputs it.

[0084] As described above, the generating device 100 inputs a concatenated embedded representation of related image data and audio data, including the token-embedded representation of the input data as a ground truth token-embedded representation, into a generative model. Therefore, cross-modality attention is performed in the self-attention layer of the generative model. This generates uninput data conditioned on the token-embedded representation of the input data. That is, for example, the audio data and the image data are unmasked while taking the other into account.

[0085] <Data Flow of Generating Device> FIG. 5 is a diagram illustrating the data flow of generating device 100 when all image data among related image data and audio data is input and audio data, which is data that has not been input, is generated.

[0086] In this case, since nothing is input to the audio pre-processing unit 108, the mask adding unit 110 outputs an audio token sequence in which all audio tokens are masked (Initial Fully-Masked Audio Tokens).

[0087] The image VQGAN encoder 112 receives all of the image data for one frame from the image pre-processing unit 111. The image VQGAN encoder 112 converts the input image data into image tokens, and the mask adding unit 113 generates an image token sequence (Tokens) from the image tokens. Because the image VQGAN encoder 112 receives all of the image data for one frame, this image token sequence does not include image tokens with masks added.

[0088] The concatenated token embeddings are generated by the concatenation unit 114 from the audio token sequence in which masks are added to all audio tokens and the image token sequence in which no masks are added to all image tokens.

[0089] This concatenated embedded representation is input to the transformer encoder 115 and encoded. Then, the replacement unit 116 replaces the token-embedded representation of the masked token in the encoded result with a predetermined token-embedded representation (Replace Mask). The replacement unit 116 generates a decoding target from the encoded result after the replacement and inputs it to the transformer decoder 117.

[0090] This decoding target is decoded by a transformer decoder 117, and a predicted value (Token Indices Prediction) of the masked audio token is calculated. Using this predicted value, the masked tokens in the audio token sequence are sequentially updated to the predicted value at a predetermined ratio, and the masking is repeated N times. As a result, predicted values ​​of all masked audio tokens are finally generated. The predicted value is converted into a mel spectrogram by an audio VQGAN decoder 118, and then converted into audio data by a vocoder 119. As a result, audio data, which is not input data, is generated.

[0091] Although not shown in the figure, when the non-input data is image data, no masks are added to any audio tokens in the audio token sequence, and masks are added to all image tokens in the image token sequence. Then, using a generative model, predicted values ​​of all image tokens with masks added are generated from the audio tokens, thereby generating the non-input image data.

[0092] If the missing data is all of the associated image and audio data, a mask is applied to all tokens in the audio and image token sequences, and a generative model is used to generate predictions of all tokens with the mask applied, thereby simultaneously generating the missing associated audio and image data.

[0093] If the non-input data is a portion of one of the associated image data and audio data, a mask is added to some tokens in one of the token sequences, and no mask is added to the remaining tokens. Then, using the generative model, predicted values ​​of the masked tokens are generated from the tokens in the input data, thereby generating a portion of the non-input data, that is, one of the associated audio data and image data. In this way, for example, when audio data from which a portion of the guitar audio data has been deleted and image data of a trumpet are input, the generating device 100 can generate, as part of the audio data, trumpet audio data that smoothly connects with the existing guitar portion.

[0094] <Description of Generation Process> Fig. 6 is a flowchart illustrating the generation process by the generation device 100 of Fig. 4. This generation process is started when, for example, a user issues an instruction to start the generation process.

[0095] 6, the audio preprocessing unit 108 determines whether or not at least a portion of the audio data of the associated image data and audio data has been input. If it is determined in step S41 that at least a portion of the audio data has been input, the process proceeds to step S42.

[0096] In step S42, the audio pre-processing unit 108 divides the input audio data into 10-second segments of audio data and converts the divided 10-second segments of audio data into mel spectrograms. The audio pre-processing unit 108 supplies the mel spectrograms to the audio VQGAN encoder 109. Note that the processing from step S43 onward is performed for each mel spectrogram of the 10-second segments of audio data.

[0097] In step S43, the audio VQGAN encoder 109 converts the mel spectrogram obtained as a result of the processing in step S42 into audio tokens and supplies them to the mask adding unit 110.

[0098] In step S44, the mask adding unit 110 determines whether or not all audio tokens of the audio data have been input from the audio VQGAN encoder 109. If it is determined in step S44 that all audio tokens of the audio data have been input, the mask adding unit 110 generates an audio token sequence from the audio tokens and supplies the audio token sequence to the concatenation unit 114. The process then proceeds to step S47.

[0099] On the other hand, if it is determined in step S44 that not all audio tokens of the audio data have been input, i.e., if only some audio tokens of the audio data have been input, the process proceeds to step S45. In step S45, the mask adding unit 110 adds masked tokens of the uninputted portion to the audio tokens obtained as a result of the process in step S43. The mask adding unit 110 generates an audio token sequence from the resulting audio tokens and supplies it to the concatenation unit 114, and the process proceeds to step S47.

[0100] On the other hand, if it is determined in step S41 that no audio data has been input, the process proceeds to step S46. In step S46, the mask adding unit 110 generates an audio token sequence in which masks have been added to all audio tokens. The mask adding unit 110 supplies this audio token sequence to the concatenation unit 114, and the process proceeds to step S47.

[0101] In step S47, the image VQGAN encoder 112 determines whether or not at least a portion of one frame of image data has been input from the image pre-processing unit 111. If it is determined in step S47 that at least a portion of one frame of image data has been input, the process proceeds to step S48.

[0102] In step S48, the image VQGAN encoder 112 converts the input image data into image tokens and supplies them to the mask adding unit 113.

[0103] In step S49, the mask adding unit 113 determines whether or not all image tokens of one frame of image data have been input from the image VQGAN encoder 112. If it is determined in step S48 that all image tokens of the image data have been input, the mask adding unit 113 generates an image token sequence from the image tokens and supplies it to the concatenation unit 114, and the process proceeds to step S52.

[0104] On the other hand, if it is determined in step S49 that all image tokens of the image data have not been input, i.e., if some image tokens of the image data have not been input, the process proceeds to step S50. In step S50, the mask adding unit 113 adds masked tokens of the uninputted portion to the image tokens obtained as a result of the process of step S48. The mask adding unit 113 generates an image token sequence from the resulting image tokens and supplies it to the concatenation unit 114, and the process proceeds to step S52.

[0105] On the other hand, if it is determined in step S47 that no image data has been input, the process proceeds to step S51. In step S51, the mask adding unit 113 generates an image token sequence in which masks have been added to all image tokens. The mask adding unit 113 supplies this image token sequence to the concatenation unit 114, and the process proceeds to step S52.

[0106] In step S52, the concatenation unit 114 generates a concatenated embedded representation from the audio token sequence supplied from the mask addition unit 110 and the image token sequence supplied from the mask addition unit 113. The concatenation unit 114 supplies the concatenated embedded representation to the transformer encoder 115.

[0107] In step S53, the transformer encoder 115 encodes the concatenated embedded representation generated in step S52 and supplies the encoded result to the substitution unit 116.

[0108] In step S54, the replacement unit 116 generates a decoding target from the encoding result obtained by the process of step S53, and supplies the generated decoding target to the transformer decoder 117.

[0109] In step S55, the transformer decoder 117 decodes the decoding target generated in step S54 to obtain a predicted value of the token to which the mask has been added.

[0110] In step S56, the generating device 100 determines whether the processes of steps S52 to S55 have been repeated N times. If it is determined in step S56 that the processes have not been repeated N times, the process proceeds to step S57.

[0111] In step S57, the concatenation unit 114 updates a predetermined ratio of masked tokens from the audio token sequence and image token sequence used in the previous processing of steps S52 to S55 to the predicted values ​​obtained in the processing of step S55. Then, the processing returns to step S52, where the concatenation unit 114 generates concatenated embedded representations from the updated audio token sequence and image token sequence, and the subsequent processing is repeated.

[0112] On the other hand, if it is determined in step S56 that the process has been repeated N times, the process proceeds to step S58. In step S58, the transform decoder 117 determines whether or not the predicted value of the audio token to which a mask has been added is included in the predicted values ​​of all tokens to which a mask has been added obtained by the process of step S55. If it is determined in step S58 that the predicted value of the audio token to which a mask has been added is included, the transform decoder 117 supplies the predicted value to the audio VQGAN decoder 118, and the process proceeds to step S59.

[0113] In step S59, the audio VQGAN decoder 118 converts the predicted value supplied from the transformer decoder 117 into a mel spectrogram of audio data and supplies the mel spectrogram to the vocoder 119. In step S60, the vocoder 119 converts the mel spectrogram obtained by the processing of step S59 into audio data and outputs it. Then, the processing proceeds to step S61.

[0114] On the other hand, if it is determined in step S58 that the predicted value of the masked audio token is not included, the process proceeds to step S61.

[0115] In step S61, the transform decoder 117 determines whether the predicted value of the masked image token is included in the predicted values ​​of all the masked tokens obtained in the process of step S55. If it is determined in step S61 that the predicted value of the masked image token is included, the transform decoder 117 supplies the predicted value to the image VQGAN decoder 120, and the process proceeds to step S62.

[0116] In step S62, the image VQGAN decoder 120 converts the predicted value supplied from the transformer decoder 117 into image data and outputs it. Then, the generation process ends.

[0117] On the other hand, if it is determined in step S61 that the predicted value of the masked image token is not included, the generation process is terminated.

[0118] As described above, the generating device 100 can generate uninput data (uninput portions) of related image data and audio data at high speed using a simple and lightweight generative model.

[0119] 3. Second Embodiment Configuration Example of Generation Device FIG. 7 is a block diagram showing a configuration example of a generation device according to a second embodiment to which the present technology is applied.

[0120] In the generation device 210 in Fig. 7, the same reference numerals are used to designate parts that correspond to those in the generation device 100 in Fig. 4. Therefore, the description of those parts will be omitted as appropriate, and the description will focus on parts that are different from the generation device 100.

[0121] The generating device 210 differs from the generating device 100 in that the mask adding units 110 and 113 are replaced with mask adding units 211 and 212. The generating device 210 also differs from the generating device 100 in that the concatenation unit 114, the transformer encoder 115, and the substitution unit 116 are replaced with a concatenation unit 214, a transformer encoder 215, and a substitution unit 216. The generating device 210 further differs from the generating device 100 in that the transformer decoder 117 is replaced with a transformer decoder 217 and in that a decision unit 218 is newly provided. Other than these, the generating device 210 is configured similarly to the generating device 100. The generating device 210 generates non-input data of the associated image data and audio data by utilizing a CFG in the generative model trained by the training device 10.

[0122] Specifically, when at least some audio tokens of the audio data are supplied from the audio VQGAN encoder 109, the mask adding unit 211 generates an audio token sequence, similar to the mask adding unit 110, and supplies the audio token sequence to the concatenation unit 214. The mask adding unit 211 generates a one-dimensional audio token sequence of audio tokens that are all masked, and supplies the audio token sequence to the concatenation unit 214.

[0123] When at least some image tokens of image data are supplied from the image VQGAN encoder 112, the mask adding unit 212 generates an image token sequence, similar to the mask adding unit 113, and supplies it to the concatenation unit 214. The mask adding unit 212 generates a one-dimensional image token sequence of image tokens in which all are masked, and supplies it to the concatenation unit 214.

[0124] 4 , the concatenation unit 214 generates a concatenated embedded representation and supplies it to the transformer encoder 215. Similar to the audio embedded representation, the concatenation unit 214 also generates an all-masked audio embedded representation from the audio token sequence in which masks are added to all audio tokens supplied from the mask adding unit 211. Similar to the image embedded representation, the concatenation unit 214 generates an all-masked image embedded representation from the image token sequence in which masks are added to all image tokens supplied from the mask adding unit 212. Similar to the concatenated embedded representation, the concatenation unit 114 concatenates the all-masked audio token embedded representation and the all-masked image embedded representation to generate an all-masked concatenated embedded representation and supplies it to the transformer encoder 215.

[0125] The transformer encoder 215, the replacement unit 216, the transformer decoder 217, the decision unit 218, the audio VQGAN decoder 118, the vocoder 119, and the image VQGAN decoder 120 function as a generation unit.

[0126] More specifically, the transformer encoder 215 is an encoder of the generative model, encodes the concatenated embedded representation and the full mask concatenated embedded representation supplied from the concatenation unit 214 , and supplies the encoded results to the substitution unit 216 .

[0127] Similar to the permutation unit 116 , the permutation unit 216 generates a decoding target from the encoding results of the concatenated embedded representation and the all-mask concatenated embedded representation supplied from the transformer encoder 215 , and supplies the generated decoding target to the transformer decoder 217 .

[0128] The transformer decoder 217 is a decoder of the generative model, and decodes the decoding targets corresponding to the concatenated embedded representation and the all-masked concatenated embedded representation supplied from the replacement unit 216. As a result, the transformer decoder 217 obtains a conditional prediction value, which is a prediction value of a token to which a mask corresponding to the concatenated embedded representation is added, and an unconditional prediction value, which is a prediction value of all tokens to which a mask corresponding to the all-masked concatenated embedded representation is added. The transformer decoder 217 supplies the conditional prediction value and the unconditional prediction value to the determination unit 218.

[0129] The decision unit 218 calculates a final predicted value using the conditional predicted value and the unconditional predicted value supplied from the transformer decoder 217. For example, when the non-input data is audio data, the decision unit 218 calculates the final predicted value using the conditional predicted value p ａ (a ｉ ´,v ｉ ) and the unconditional predicted value p ａ (a ｉ ´, Mv ｉ ) and calculate the final predicted value p ａ Calculate ´.

[0130]

[0131] In equation (2), the hyperparameter s is the CFG guidance coefficient. ｉ ' is the audio token sequence used in this process, and is updated each time masking is repeated. ｉ is the image token sequence corresponding to the input image data, and Mv ｉ is the image token sequence with a mask added to every image token.

[0132] Similar to the generating device 100, the generating device 210 repeats the mask removal N times using the predicted values ​​determined by the determining unit 218. As a result, the determining unit 218 finally generates predicted values ​​of all masked tokens, and performs quantization and softmax operations on them.

[0133] The determination unit 218 supplies the predicted value of the audio token with the mask added to the audio VQGAN decoder 118, and supplies the predicted value of the image token with the mask added to the image VQGAN decoder 120, among the predicted values ​​obtained as a result.

[0134] CFGs are a powerful tool for achieving a trade-off between alignment and generation fidelity in a conditional generative model originally based on a diffusion model. In the generator 210, a CFG is utilized in the generative model by inputting the concatenated embedded representation and the full-mask concatenated embedded representation to the transformer encoder 215. Therefore, the generation capability for uninput data can be improved compared to the generator 100 in which a CFG is not utilized.

[0135] Unlike existing methods, the generative model of the present technology does not require the addition or modification of unconditional learning to utilize CFG. One reason for this is that the masking rate in the learning device 10 is set individually for each modality, so there is a possibility that masks will be added to all tokens of one modality. Another reason is that unconditional learning is performed by replacing the token-embedded representation m in the replacement unit 19.

[0136] <Data Flow of Generating Device> FIG. 8 is a diagram illustrating the data flow of generating device 210 when all image data among related image data and audio data is input and audio data, which is data that has not been input, is generated.

[0137] 5, the mask adding unit 211 outputs an audio token sequence in which all audio tokens are masked (Initial Fully-Masked Audio Tokens), and the mask adding unit 212 outputs an image token sequence in which all image tokens are not masked (Tokens) and an image token sequence in which all image tokens are masked (Fully-Masked Visual Tokens).

[0138] The concatenating unit 214 generates an all-masked concatenated embedded representation from the audio token sequence in which masks are added to all audio tokens and the image token sequence in which masks are added to all image tokens. Similarly to the case of Fig. 5, the concatenating unit 214 also generates concatenated embedded representations (Concatenated Token Embeddings) from the audio token sequence in which masks are added to all audio tokens and the image token sequence in which no masks are added to all image tokens. The concatenated embedded representations and all-masked concatenated embedded representations are then input to the transformer encoder 215, where they are each encoded.

[0139] The token-embedded representation of the masked token in each encoding result is replaced with a predetermined token-embedded representation by the replacement unit 116 (Replace Mask), and a position-embedded representation and a modality-specific embedded representation are added for each modality. The decoding targets corresponding to the resulting concatenated embedded representation and all-mask concatenated embedded representation are input to the transform decoder 217.

[0140] The decoding targets are decoded by the transformer decoder 217, and the conditional prediction value and unconditional prediction value (Token Indices Prediction) of the masked token are calculated.

[0141] Using these conditional predictions and unconditional predictions, the decision unit 218 calculates a final prediction (CFG Prediction). Using this final prediction, the masked tokens are sequentially updated to predictions at a predetermined ratio, and unmasking is repeated N times. This results in the generation of predictions for all masked audio tokens. The audio VQGAN decoder 118 converts these predictions into mel spectrograms, and the vocoder 119 converts these into audio data. As a result, audio data, which is not yet input data, is generated.

[0142] When the non-input data is image data, or when the non-input data is all or part of the related image data and audio data, the process is the same as in FIG. 8 except that the tokens to be masked are different.

[0143] As described above, the generating device 210 can generate non-input data among related image data and audio data at high speed by using a simple and lightweight generative model and utilizing CFG.

[0144] 4. Explanation of Effects FIGS. 9 and 10 are diagrams for explaining the effects of the generation devices 100 and 210. FIG.

[0145] Specifically, Fig. 9 is a diagram showing experimental results of the relationship between the mask rate in the learning device 10 and the ablation in the generating device 100. Fig. 10 is a diagram showing experimental results of the relationship between the CFG guidance coefficient s in the generating device 210 and the ablation.

[0146] 9 and 10, 182,563 pairs of video image data and audio data from the VGGSound dataset, which includes 309 classes of sound events, were used as training data. 15,290 pairs of video image data and audio data from the VGGSound dataset were used as test data for ablation.

[0147] During ablation, 10 frames of image data are uniformly sampled from the image data of the evaluation video, as in the image preprocessing unit 14, and the image data of the central frame of the 10 sampled frames is input to the image VQGAN encoder 112. The downsampling ratio in the image VQGAN encoders 15 and 112 is 8. The combinations (T, N) of the temperature T and the number of unmasking iterations N (= speed), which control the diversity and randomness of unmasking in the generation device 100 (210), are (4.5, 15), (4.5, 30), (9, 15), and (9, 30), respectively.

[0148] 9A to 9C, the horizontal axis represents the average mask rate in the learning device 10, and the vertical axis represents the Frechet Distance (FD), Frechet Audio Distance (FAD), and Inception Score (IS) in the generation device 100. FD and FAD are indices that represent the similarity between predicted values ​​and actual values, and IS is an index that represents quality and diversity.

[0149] The experimental results shown in Figures 9A to 9C show that FD, FAD, and IS improve as the hyperparameter μa increases until the average value of the mask rate, i.e., the hyperparameter μa, reaches approximately 0.7. On the other hand, when the hyperparameter μa exceeds 0.7, the FD, FAD, and IS decrease as the hyperparameter μa increases. The experimental results shown in Figures 9A to 9C show that the FD, FAD, and IS are best when the temperature T is 4.5 and the number of iterations N is 30.

[0150] 10A to 10C, the horizontal axis represents the CFG guidance coefficient s in the generating device 210, and the vertical axes represent the FD, FAD, and IS in the generating device 210. Note that a CFG guidance coefficient s of 1 means that CFG is not applied. In other words, when the CFG guidance coefficient s is 1, the FD, FAD, and IS in the generating device 210 are the same as the FD, FAD, and IS in the generating device 100.

[0151] The experimental results shown in Figures 10A and 10B show that when CFG is applied, the smaller the CFG guidance coefficient s, the smaller the FD and FAD. However, the experimental results shown in Figure 10C show that the larger the CFG guidance coefficient s, the higher the IS.

[0152] 9 and 10 show that the generation devices 100 and 210 can generate audio data that is not input data with high accuracy. Furthermore, as shown in Fig. 10, the generation device 210, which uses a CFG in its generative model, can improve the accuracy of generating audio data that is not input data compared to the generation device 100, which does not use a CFG.

[0153] Note that when the hyperparameters μ and μ are small, this additional training can prevent degradation of the FAD. The generative model may be further controlled by prompting input text data in addition to the concatenated embedded representations. Concatenation of embedded representations can be performed for any modality.

[0154] Although not shown in the figure, it has been found that the generating device 100 (210) can generate the non-input data with high accuracy even when the input data and non-input data are not video image data and audio data, respectively.

[0155] In the above description, the generating device 100 (210) includes a transformer encoder 115 (215) and a transformer decoder 117 (217), but it may also include only a transformer decoder 117 (217). If the generating device 100 (210) includes a transformer encoder 115 (215) and a transformer decoder 117 (217), it is possible to save computational resources.

[0156] 5. Description of a Computer to which the Present Technology is Applied The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, a program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0157] FIG. 11 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0158] In the computer, a processing circuit 901 , a ROM (Read Only Memory) 902 , and a RAM (Random Access Memory) 903 are interconnected by a bus 904 .

[0159] An input / output interface 905 is also connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0160] The input unit 906 may include physical or virtual operating means, such as a keyboard, mouse, or touch panel, that a user operates to input information, as well as means for the user to input information by voice, gaze, or the like. The input unit 906 may also include sensors for inputting various physical quantities to the computer. For example, the input unit 906 may include sensors, such as a camera or microphone, that acquire physical quantities such as light (including infrared light other than visible light) and sound. The input unit 906 may also include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, and distance. The output unit 907 may include means, such as a display, speaker, or haptic device, that present information to a user by stimulating the user's senses. The storage unit 908 may include a hard disk, non-volatile or volatile memory, or the like, and stores various information (including programs). The communication unit 909 may be a network interface or the like, and performs wired or wireless communication with the outside world. The drive 910 drives removable media 911, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0161] The processing circuit 901 includes a processor that executes a program, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). The processing circuit 901 (the processor) performs the above-described series of processes by loading a program stored in a storage unit 908 into a RAM 903 via an input / output interface 905 and a bus 904 and executing the program. The processing circuit 901 can output the processing results of the series of processes from an output unit 907, for example, via the bus 904 and the input / output interface 905, as necessary. The processing circuit 901 can also store the processing results in the storage unit 908, transmit them from a communication unit 909, etc.

[0162] The program executed by the computer (processing circuitry 901) can be provided by being recorded on removable media 911 such as package media, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0163] In a computer, the program can be installed in the storage unit 908 via the input / output interface 905 by inserting the removable medium 911 into the drive 910. The program can also be received by the communication unit 909 from another device such as a server via a wired or wireless transmission medium and installed in the storage unit 908. Alternatively, the program can be installed in the ROM 902 or the storage unit 908 in advance.

[0164] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0165] The processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).

[0166] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0167] When the above-described series of processes are performed by a computer executing a program, the processing circuit 901 (the processor thereof) executes the program to function as the learning device 10 or the generation device 100 (210).

[0168] The components (blocks) of the devices illustrated in the present specification are conceptual blocks of functions, and the actual device does not necessarily have to have the configuration shown in the drawings. In other words, the device may have any configuration in which the functions of the illustrated components are divided and / or integrated in any unit, for example, a configuration having a single block in which the functions of all the components are integrated.

[0169] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0170] For example, it is possible to adopt a configuration in which all or part of the above-described embodiments are combined.

[0171] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0172] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0173] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0174] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0175] The present technology may have the following configurations. (1) A generating device including a generating unit that generates non-input data, which is data that is not input, from data of multiple associated modalities using a generative model that is a transformer model for a single modality. (2) The generating device according to (1), configured such that the data of the multiple modalities include image data and audio data. (3) The generating device according to (1), in which the data of the multiple modalities are image data and audio data, and the non-input data is at least a part of the image data, and the generating unit uses the generative model to generate the non-input data related to the audio data from the input audio data and the image data other than the non-input data. (4) The generating device according to (1), in which the data of the multiple modalities are image data and audio data, and the non-input data is at least a part of the audio data, and the generating unit uses the generative model to generate the non-input data related to the image data from the input image data and the audio data other than the non-input data. (5) The generating device according to (1), wherein the data of the multiple modalities are image data and audio data, and the non-input data are the image data and the audio data, and the generating unit generates the related image data and the audio data using the generative model. (6) The generating device according to any of (1) to (5), wherein the generating unit generates the non-input data by inputting mask data and input data corresponding to the non-input data among the data of the multiple modalities into the generative model. (7) The generating device according to (6), wherein the generating unit generates the non-input data by inputting concatenated data, which is data in which the input data and the mask data are concatenated, into the generative model. (8) The generating device according to (6) or (7), wherein the generating unit generates the non-input data by inputting tokens of the input data and tokens of the mask data into the generative model.(9) The generating device according to (8), further comprising a tokenizer that converts the input data into tokens of the input data, wherein the generating unit generates the non-input data by using the tokens of the input data and the tokens of the mask data generated by the tokenizer as inputs to the generative model. (10) The generating device according to any of (1) to (9), wherein the generating unit is configured to have an encoder and a decoder. (11) The generating device according to any of (1) to (10), wherein the generative model is configured to use Classifier-Free Guidance. (12) A generating method, including: the generating device using a generative model that is a transformer model for a single modality to generate non-input data that is data that is not input among data of a plurality of associated modalities. (13) A program that causes a computer to execute processing including: using a generative model that is a transformer model for a single modality to generate non-input data that is data that is not input among data of a plurality of associated modalities.

[0176] 100 Generator, 109 Audio VQGAN Encoder, 112 Image VQGAN Encoder, 115 Transformer Encoder, 116 Substitution Unit, 117 Transformer Decoder, 118 Audio VQGAN Decoder, 119 Vocoder, 120 Image VQGAN Decoder, 210 Generator, 215 Transformer Encoder, 216 Substitution Unit, 217 Transformer Decoder, 218 Decision Unit

Claims

1. A generating device having a generating unit that generates non-input data, which is data that is not input, from data of multiple associated modalities using a generative model that is a transformer model for a single modality.

2. The generating device according to claim 1, wherein the data of the multiple modalities includes image data and audio data.

3. The generating device according to claim 1, wherein the data of the multiple modalities are image data and audio data, the non-input data is at least a part of the image data, and the generating unit uses the generative model to generate the non-input data related to the audio data from the input audio data and the image data other than the non-input data.

4. The generating device according to claim 1, wherein the data of the multiple modalities are image data and audio data, the non-input data is at least a part of the audio data, and the generating unit uses the generative model to generate the non-input data related to the image data from the input image data and the audio data other than the non-input data.

5. The generating device according to claim 1, wherein the data of the multiple modalities are image data and audio data, the non-input data are the image data and the audio data, and the generating unit uses the generating model to generate the related image data and audio data.

6. The generating device according to claim 1, wherein the generating unit generates the non-input data by inputting mask data and input data corresponding to the non-input data among the data of the plurality of modalities into the generating model.

7. The generating device according to claim 6, wherein the generating unit generates the non-input data by inputting concatenated data, which is data in which the input data and the mask data are concatenated, into the generating model.

8. The generating device according to claim 6, wherein the generating unit generates the non-input data by inputting the tokens of the input data and the tokens of the mask data to the generative model.

9. The generating device according to claim 8, further comprising a tokenization unit that converts the input data into tokens of the input data, wherein the generating unit generates the non-input data by inputting the tokens of the input data and the tokens of the mask data generated by the tokenization unit into the generative model.

10. The generating device according to claim 1, wherein the generating unit is configured to have an encoder and a decoder.

11. The generating device according to claim 1, configured to utilize classifier-free guidance in the generative model.

12. A generation method including a generation device generating non-input data, which is data that is not input, from data of multiple associated modalities using a generative model that is a transformer model for a single modality.

13. A program for causing a computer to execute a process including generating non-input data, which is data that is not input, from data of multiple related modalities using a generative model that is a transformer model for a single modality.

Images