Information processing device, method, and program

Abstract

The present disclosure relates to an information processing device, a method, and a program that enable accurate generating of audio of an unlearned length without additional learning. An information processing device according to the present invention includes a prompt information input unit that receives a constraint condition for audio to be generated as prompt information, a noise generation unit that generates noise for a duration of the audio to be generated, a position embedding unit that embeds position information in the generated noise by using RoPE for a position in a temporal axis direction of the audio and embeds position information by using an absolute position for a position in a frequency axis direction thereof, and a diffusion transformer module that performs denoising processing of an embedded vector in which the position information is embedded. The technology according to the present disclosure can be applied to, for example, a generation task for generating natural and accurate audio from text prompt input.

Description

Information processing device and method, and program

[0001] This disclosure relates to an information processing device and method, and a program, and more particularly to an information processing device and method, and a program that enables the accurate generation of unlearned audio of unlearned length without additional training.

[0002] Machine learning models that generate music from text are known (see, for example, Patent Document 1). Text-to-audio synthesis (TTA) is a generation task aimed at generating natural and accurate audio from text prompt input. In TTA, diffusion models have recently gained a strong presence. For example, text-audio diffusion models such as Tango2, AudioLDM2, and Make-An-Audio2 have shown remarkable performance in generating fixed-length audio, especially sound effects. These prior studies have achieved the remarkable result of low FAD scores for fixed-length audio (a lower FAD score indicates higher performance).

[0003] While TTA has been successful in generating fixed-length audio, real-world scenarios require the generation of audio of varying lengths. Sound lengths range from short, transient sounds like the snap of a twig or the sound of water droplets hitting the surface, to long, continuous sounds like the sound of ocean waves, the hustle and bustle of a city, or a sustained melody played on a musical instrument.

[0004] U.S. Patent Application Publication No. 2018 / 0190249

[0005] Diffusion models capable of generating audio of various lengths have not yet been sufficiently investigated or realized.

[0006] This disclosure is made in light of these circumstances and aims to enable the accurate generation of untrained audio of untrained length without additional training.

[0007] The first aspect of this technology is an information processing device comprising: a prompt information input unit that receives constraints for the audio to be generated as prompt information; a noise generation unit that generates noise for the duration of the generated audio; a position embedding unit that embeds position information for the generated noise using RoPE for the position in the time axis direction of the audio and using absolute position for the position in the frequency axis direction; and a diffusion transformer module that performs denoising processing on the embedding vector into which the position information is embedded.

[0008] The second aspect of this technology is an information processing method which includes an information processing device receiving constraints for the audio to be generated as prompt information, generating noise for the duration of the audio to be generated, embedding position information in the generated noise using RoPE for the time axis position of the audio and using absolute position for the frequency axis position, and performing denoising on the embedding vector into which the position information is embedded.

[0009] The program for the third aspect of this technology causes the computer to perform a process that includes receiving constraints for the audio to be generated as prompt information, generating noise for the duration of the audio to be generated, embedding position information in the generated noise using RoPE for the time axis position of the audio and using absolute position for the frequency axis position, and performing denoising on the embedding vector into which the position information is embedded.

[0010] In the first to third aspects of this technology, constraints on the audio to be generated are received as prompt information, noise is generated for the duration of the audio to be generated, position information is embedded in the generated noise using RoPE for the time axis position of the audio, position information is embedded using absolute position for the frequency axis position, and denoising is performed on the embedded vector into which the position information is embedded.

[0011] Furthermore, the information processing device of the first aspect of this disclosure can be realized by having a computer execute a program. The program to be executed by the computer can be provided by being transmitted via a transmission medium or by being recorded on a recording medium.

[0012] An information processing device may be an independent device or an internal block that constitutes a single device.

[0013] This is a diagram illustrating the TTA model of the information processing device of the present disclosure. This is a block diagram of the TTA model in learning mode. This is a diagram illustrating the tokenization of audio features in latent space. This is a diagram showing the relative positional relationship of each token. This is a diagram showing a detailed configuration example of the DiT block. This is a block diagram of the TTA model in inference mode. This is a flowchart illustrating the learning process of the TTA model. This is a flowchart illustrating the details of the denoising process in step S9 of Figure 7. This is a flowchart illustrating the inference process of the TTA model. This is a flowchart illustrating the details of the denoising process in step S48 of Figure 9. This is a diagram showing the results of benchmark tests between the present method and other prior models in the case of variable length generation. This is a diagram showing the comparison results between frequency-based RoPE and other PEs. This is a diagram showing the inpainting and outpainting performance of the present method. This is a block diagram showing an example of the hardware configuration of the computer of the present disclosure.

[0014] <Cross-reference to related applications> This application claims the benefit of U.S. Provisional Patent Application No. 63 / 698,142, filed on 24 September 2024, the entirety of which Provisional Application is incorporated herein by reference.

[0015] The embodiments for carrying out the technology of this disclosure (hereinafter referred to as "embodiments") will be described below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration will be denoted by the same reference numerals to avoid redundant explanations. The explanation will be given in the following order: 1. TTA model of the information processing device of this disclosure 2. TTA model in learning mode 3. Detailed configuration example of the DiT block 4. TTA model in inference mode 5. Audio generation of audio length exceeding the learning range 6. Learning process of the TTA model 7. Inference process of the TTA model 8. Evaluation of this method in comparison with other methods 9. Summary of this method 10. Description of a computer to which this technology is applied

[0016] <1. TTA Model of the Information Processing Device Disclosed> Figure 1 is a diagram illustrating the TTA model of the information processing device disclosed.

[0017] The information processing device 1 in Figure 1 has a TTA model 10. The TTA model 10 is an audio generation task that generates natural and accurate audio from text prompt input. In learning mode (during learning), the TTA model 10 generates its parameters (model parameters). Then, in inference mode (during inference), the TTA model 10 uses the parameters obtained through learning to generate audio according to the text prompt input. The TTA model 10 is capable of generating audio of a length that exceeds the range covered by learning.

[0018] Figure 1A shows the input and output data of the TTA model 10 in learning mode. During learning, a predetermined audio dataset is used to input audio (data) of a predetermined duration, along with its ground truth (true value), which is GT audio (data). In addition, text (data) describing the input audio is input as prompt information to instruct the generation process.

[0019] Figure 1B shows the input and output data of the TTA model 10 in inference mode. During inference, the duration of the audio to be generated is input. In addition, text describing the audio to be generated is input as prompt information to instruct generation.

[0020] The "duration" of audio refers to the time (length) during playback, and may also be referred to as audio length or audio duration below. The TTA model 10, which is an audio generation task, can be classified into the following two situations (scenarios) regarding the length of the audio it generates: ・TTCL (Train-Test-Consistent-Length) TTCL is audio generation where the length of the audio generated during inference is within the range covered by learning. For example, if the maximum audio length of the training data during learning is 10 seconds, the TTA model 10 will not only learn by inputting audio with an audio length of 10 seconds, as shown in Figure 1A, but will also use the 10-second audio length training data to generate training data of multiple durations less than or equal to the maximum length and learn from it. For example, the TTA model 10 will clip the 10-second audio length training data to generate training data of three different audio lengths: 2.5 seconds, 5 seconds, and 7.5 seconds. Therefore, in training, training data of four different audio lengths are used: 2.5 seconds, 5 seconds, 7.5 seconds, and 10 seconds. This enables the TTA model 10 to handle variable-length audio generation that is less than or equal to the maximum length of the training data. TSTL (Train-Short-Test-Long) TSTL is audio generation when the audio length to be generated during inference is longer than the range covered by training. For example, as shown in Figure 1B, the TTA model 10 can handle cases where the input audio duration is longer than the "10 seconds" covered by training, such as "20 seconds," and can generate "20 seconds" of audio. The TTA model 10 has achieved a remarkable result in audio generation with audio lengths longer than the range covered by training, which was a weakness of existing text-audio diffusion models such as Tango2, AudioLDM2, and Make-An-Audio2, with a low FAD score (a lower FAD score indicates higher performance).

[0021] The features of the TTA model 10 are as follows: 1. Flexibility The TTA model 10 can generate audio of various lengths without significantly sacrificing performance. In TSTL scenarios, additional training (fine-tuning) is not required to adapt to untrained long durations. 2. Performance The TTA model 10, with its novel architecture characterized by token modulation and replacing self-attention with RoPE attention, is effective not only for generating untrained, long-duration variable-length audio but also for generating audio within the range covered by training. The TTA model 10 has shown superior performance to state-of-the-art models (hereinafter also referred to as SOTA models) in benchmarks with a standard duration of "10 seconds". 3. Training Cost Because the TTA model 10 can train with shorter audio durations and handle untrained audio durations, it eliminates memory overhead and offers the advantage of requiring fewer computational resources.

[0022] <2. TTA Model in Learning Mode> Figure 2 is a block diagram of the TTA model 10 in learning mode.

[0023] The TTA model 10 in learning mode includes a prompt information input unit 21, a text ENC 22, a pooling layer 23, and an MLP 24. The TTA model 10 also includes an audio input unit 31, a VAE ENC 32, a patching unit 33, a noise generation unit 34, an addition unit 35, a position embedding unit 36, N (integer N>1) DiT blocks 37, a GT input unit 38, and a parameter update unit 39.

[0024] The prompt information input unit 21 receives text describing the audio, which is input by the user as prompt information, and outputs it to the text ENC 22. Prompt information is information that serves as a constraint condition when generating audio. For example, text describing audio such as "A train horn goes off loudly" is input to the audio input unit 31.

[0025] The text ENC (encoder) 22 converts the text supplied from the prompt information input unit 21 into a feature vector (hereinafter referred to as TEXT vector) that can be input to the DiT block 37 in the latent space, and outputs it to the pooling layer 23 and the DiT block 37. As the text ENC (encoder) 22, for example, FlanT5 or the like can be used.

[0026] The pooling layer 23 performs a pooling process on the TEXT vector from the text ENC 22, downsizes it to a size suitable for input to the MLP 24, and outputs it to the MLP 24.

[0027] The MLP (multi-layer perceptron) 24 consists of at least three node layers including a linear transformation and an activation function, and based on the downsized TEXT vector supplied from the pooling layer 23, generates coefficients used in the scale shift layer of the DiT block 37 and outputs them to the DiT block 37. The pooling layer 23 and the MLP 24 constitute a global coefficient generation unit 25 that generates global coefficients by further compressing the TEXT vector converted by the text ENC 22. The MLP 24 has scale coefficients α １ , α ２ , γ １ , and γ ２ , and generates and outputs the shift coefficients β １ and β ２ . The scale coefficients α １ , α ２ , γ １ , and γ ２ , and the shift coefficients β １ and β ２ correspond to global conditions (global conditions) that summarize the meaning and context of the entire input text. On the other hand, the TEXT vector directly input from the text ENC 22 to the DiT block 37 corresponds to fine conditions (local conditions) before summarizing the meaning and context of the entire input text.

[0028] In this embodiment, the prompt information is set to input text describing the audio, but it is also possible to configure the system to input audio instead of text. In this case, the text ENC 22 is changed to audio ENC.

[0029] The audio input unit 31 receives an audio u of a predetermined duration (e.g., 10 seconds) selected by the user from a predetermined audio dataset and outputs it to the VAE ENC 32. The audio u is, for example, in Mel Spectogram format.

[0030] VAE ENC32 is the audio u∈R' of the input space in Mel Spectogram format. ｃ×F’×T’ The audio features of the latent space x ０ ∈R D×F×T This is a variational autoencoder that converts to [a specific format]. Note that c and D are the channel dimensions of the input and latent spaces, respectively; F' and F are the frequency dimensions of the input and latent spaces, respectively; and T' and T are the time dimensions of the input and latent spaces, respectively. The latent space is a feature space where the features of the data to be learned (audio in this embodiment) are learned and the data representation is simplified.

[0031] The patching unit 33 processes the audio features x of the latent space. ０ Patching is performed by combining p (pxp) elements in the frequency axis and time axis directions into one patch. The patching unit 33 processes the audio feature quantity x of the latent space. ０ These are grouped into units of p in the frequency axis direction and p in the time axis direction, p 2 The result of downsampling by a factor of two is treated as a single token (feature vector). The audio features after downsampling are x. ０ However, suppose that the patching unit 33 tokenizes it into L tokens z.

[0032] Figure 3 shows the audio features x in the latent space. ０ This diagram shows an image of tokenizing into L tokens.

[0033] Audio features x in the potential space ０However, it is tokenized into a total of L tokens z. The L tokens z are assigned indices 0, 1, 2, ..., L-3, L-2, L-1 in the time axis, from smallest to largest. Also, the audio features x in the latent space ０ However, along the frequency axis, it is divided into three tokens, and the indices A, B, and C are assigned according to the position of the divided frequency. As a result, the L tokens z are indiced in ascending order along the time axis as follows: (0, A), (1, B), (2, C), (3, A), (4, B), (5, C), ..., (L-3, A), (L-2, B), (L-1, C).

[0034] Figure 4 shows the relative positions of each of the L tokens.

[0035] The vertical axis (Y-axis) in Figure 4 shows the token index, with the upper left corner as the origin. The tokens are arranged sequentially from the first token in the input sequence (0,A) to (1,B), (2,C), (3,A), (4,B), (5,C), and so on. Looking at the indices A, B, and C in the frequency axis direction of the tokens, we can identify tokens belonging to the same time bin frame. For token (1,B), we can see the previous token (0,A) and the RoPE computational token (0,B) which does not actually exist. For token (2,C), we can see the previous tokens (0,A), (1,B), (0,B) and the RoPE computational tokens (1,C) and (0,C) which do not actually exist. The horizontal axis (X-axis) represents unused tokens that are not introduced into the actual token sequence but exist theoretically in RoPE. In RoPE calculations, theoretical RoPE tokens exist in the lower triangular region based on the diagonal elements of the matrix.

[0036] Returning to Figure 2, the noise generation unit 34 generates noise following a Gaussian distribution (Gaussian noise) and outputs it to the summing unit 35. The summing unit 35 adds the noise supplied from the noise generation unit 34 to each of the L tokens z supplied from the patching unit 33. The L tokens z after noise addition are supplied to the position embedding unit 36.

[0037] The summing unit 35 adds the audio feature quantity x in the initial state (time T=0).０ By adding noise, the audio features x at time T=t are obtained by the following equation (1): ｔ = q(x) ｔ | x ０ ) generates. N(x) in equation (1) ｔ ;α ｔ x ０ , σ ２ ｔ I) is the mean α ｔ x ０ , covariance matrix σ ２ ｔ This represents the normal distribution (multivariate Gaussian distribution) of I. ｔ This is the original audio feature x ０ σ is a scaling factor that adjusts how much of the effect remains. ２ ｔ represents the noise variance, and I represents the identity matrix.

[0038] More generally, the audio features x from time T=s to time T=t (s<t) of the intermediate step from the initial state to the final state. ｔ = q(x) ｔ | x ｓ ) is expressed by the following equation (2). α in equation (2) ｔ / α ｓ σ is the scaling factor from time T=s to time T=t. ２ ｔ｜ｓ This represents the variance of noise as time progresses from T=s to T=t.

[0039] Audio feature x in equation (2) ｔ This can be replaced by equation (3) using the reparameterization trick to make it differentiable. ε in equation (3) t This represents the Gaussian noise at time T=t.

[0040] The position embedding unit 36 ​​generates and outputs a vector (embedding vector) for each of the L tokens z that takes into account the position information in the time axis direction and the position in the frequency axis direction. Specifically, the position embedding unit 36 ​​embeds the position information in the time axis direction using RoPE and the position information in the frequency axis direction using absolute position. In other words, the position embedding unit 36 ​​embeds the position information in the frequency axis direction for each of the L tokens z using absolute position embedding, and then generates and outputs a rotational embedding vector obtained by rotating the position information in the time axis direction using RoPE on the embedding vector into which the absolute position information has been embedded. The position embedding unit 36 ​​outputs the L embedding vectors (tokens) into which the position information has been embedded to the DiT block 37.

[0041] This section explains the embedding of positional information using absolute position embedding. Given the position m of a token in the input sequence 0 ≤ m ≤ L / 2, the i-dimensional embedding can be expressed by the following equation (4) using a sinusoidal function. In equation (4), D is the vector dimension after token embedding, and b is the basis, a hyperparameter that controls the scaling of angles, and plays a role in determining the rotation speed (angle) of the rotation matrix according to the position m of the tokens. The basis b is, for example, 10 ４ (b=10 ４ ) is set to ). Below, for simplicity, the dimensional index i in equation (4) is omitted, and P(m)∈R D If expressed in this way, P(m), which indicates the position in the frequency axis direction, is added to the token input to the position embedding unit 36.

[0042] Next, we will explain the embedding of location information using RoPE. The mth token z ｍ ∈R D When specified, the one-dimensional RoPE embedding vector f ｛ｑ，ｋ｝ This can be formulated as shown in equation (5) below. f in equation (5) ｛ｑ，ｋ｝ , W ｛ｑ，ｋ｝ is the mth token z ｍ In the case of a query vector, f ｑ , Wｑ And so the mth token z ｍ In the case of the key vector, f ｋ , W ｋ This indicates that it will be W ｛ｑ，ｋ｝ represents the weight matrix (learning parameters) of the query vector or key vector. h(θ) is the rotation frequency matrix of θ, and θ i = b （-2i / D） , i ∈ [1, 2, ..., D / 2], b=10 ４ It has.

[0043] The N DiT blocks 37 are connected in series such that the output of the preceding DiT block 37 is input to the subsequent DiT block 37. As will be described later with reference to Figure 5, the DiT blocks 37 are diffusion transformer modules equipped with scale shift layers, attention layers, etc., similar to the FiLM approach, and the N DiT blocks 37 as a whole function as a denoiser. For each of the L tokens z, the N DiT blocks 37 output a predicted value v_hat, which is a denoised feature vector, to the parameter update unit 39. The predicted value v_hat is defined as follows using v-prediction: v_hat = α ｔ ε t -σ ｔ x ０ ... (6)

[0044] The GT input unit 38 receives GT audio u_GT, which is the ground truth (true value) of audio u input to the audio input unit 31. The GT input unit 38 converts GT audio u_GT into a format v(x) corresponding to the predicted value v_hat output by the DiT block 37. t Convert to (t,y) and supply to the parameter update unit 39.

[0045] The parameter update unit 39 uses the predicted value v_hat, which is the output of the final stage (Nth) DiT block 37, and the ground truth v(x) supplied from the GT input unit 38. t Using t and y, the loss function L is given by equation (7) below. D The parameter update unit 39 calculates the loss function L. DWe find the learning parameters (weight coefficients W) of DiT block 37 that minimize the following.

[0046] <3. Detailed Configuration Example of a DiT Block> Figure 5 shows a detailed configuration example of a DiT block 37. Figure 5 is explained as the first (first stage) DiT block 37 of N DiT blocks 37 connected in series, but the second and subsequent DiT blocks 37 are configured similarly.

[0047] DiT block 37 is a Transformer-based Diffusion model.

[0048] The DiT block 37 is composed of a RoPE attention mechanism 61, a feedforward mechanism 63, a cross-attention mechanism 65, and residual coupling sections 62, 64, and 66. The RoPE attention mechanism 61 has a normalization layer 41, a scale-shift layer 42, a LlamaRMS normalization layer 43, a RoPE attention layer 44, and a scale layer 45. The feedforward mechanism 63 has a normalization layer 46, a scale-shift layer 47, a feedforward layer 48, and a scale layer 49. The cross-attention mechanism 65 has a cross-attention layer 51. The residual coupling section 62 is located between the RoPE attention mechanism 61 and the feedforward mechanism 63. The residual coupling section 64 is located between the feedforward mechanism 63 and the cross-attention mechanism 65. The residual coupling section 66 is located downstream of the cross-attention mechanism 65 and in the output stage of the DiT block 37.

[0049] The L embedding vectors (tokens) input from the position embedding unit 36 ​​to the DiT block 37 are supplied to the normalization layer 41 and the residual coupling unit 62.

[0050] The normalization layer 41 performs normalization on the input feature vector using the mean and variance. The scale shift layer 42 uses the scale coefficient γ supplied from the MLP 24. １ , shift coefficient β １Using this, scaling and shifting operations are performed on the feature vector that has been normalized by the normalization layer 41 and input. The Llama RMS normalization layer 43 performs a normalization operation using the root mean square value on the input feature vector. The Llama RMS normalization layer 43 is a normalization operation adopted in models such as Llama and Llama2.

[0051] The RoPE attention layer 44 embeds positional information into the query - Q and key - K by a rotation operation and calculates the attention score A. For example, for the token position at the first position, the RoPE attention layer 44 rotates the feature vector by the rotation angle θ, and for the second position, it rotates the feature vector by the rotation angle 2θ.

[0052] For the query vector q ｍ of the m - th token z ｍ = f ｑ (z ｍ , m, h(θ)), and for the key vector k ｎ of the n - th token z ｎ = f ｋ (z ｎ , n, h(θ)), then the attention score A of the RoPE attention layer 44 is calculated by the following formula (8).

[0053] The numerator <q ｍ， k ｎ > in formula (8) indicates the similarity between the key and the query and is calculated by the following formula (9).

[0054] φ() represents the mapping function of RoPE. As can be seen from formula (9), the positional information P(m), P(n) of the absolute position in the frequency axis direction is integrated into the formula for the similarity between the key and the query. For each dimension θ i , the wavelength λ i of RoPE is defined as the number of tokens (token length) required for one rotation (360°), and λ i = 2πb （2i / D）It is defined as follows. Smaller dimension i has a short wavelength and rotates quickly (high frequency), so it is sensitive to changes in position at short distances, while larger dimension i has a long wavelength and rotates slowly (low frequency), so it can also detect positions at long distances.

[0055] The scaling layer 45 is supplied with a scaling coefficient α from the MLP 24. １ The residual coupling unit 62 uses this to perform scaling on the feature vector output from the RoPE attention layer 44. The residual coupling unit 62 adds (residual coupling) the feature vector obtained by skipping the RoPE attention mechanism 61 with the feature vector output from the scaling layer 45 of the RoPE attention mechanism 61. The output of the residual coupling unit 62 is supplied to the normalization layer 46 of the feedforward mechanism 63 and to the residual coupling unit 64 located downstream of the feedforward mechanism 63.

[0056] The normalization layer 46 of the feedforward mechanism 63 performs normalization on the input feature vector using the mean and variance. The scale shift layer 47 uses the scale coefficient γ supplied from the MLP 24. ２ , shift coefficient β ２ Using this, scaling and shifting operations are performed on the input feature vector that has been normalized in the normalization layer 46. The feedforward layer 48 is a two-layer linear neural network with an activation function in between, and performs a nonlinear transformation on the input feature vector using a predetermined activation function. The feedforward layer 48 temporarily expands the dimensionality of the feature vector obtained by the RoPE attention mechanism 61, and then returns it to its original dimensionality, thereby further transforming the output of the RoPE attention mechanism 61 into an appropriate output. The scale layer 49 uses the scale coefficient α supplied from the MLP 24. ２ The residual coupling unit 64 uses this to perform scaling on the feature vector output from the feedforward layer 48. The residual coupling unit 64 adds (residual coupling) the feature vector obtained by skipping the feedforward mechanism 63 with the feature vector from the scaling layer 49 of the feedforward mechanism 63. The output of the residual coupling unit 64 is supplied to the cross-attention layer 51 of the cross-attention mechanism 65 and to the residual coupling unit 66 located downstream of the cross-attention mechanism 65.

[0057] The cross-attention layer 51 of the cross-attention mechanism 65 receives a TEXT vector from the text ENC 22. The cross-attention layer 51 generates a query Q using the output from the residual coupling unit 64 and generates a key K and a value V using the output from the text ENC 22. The cross-attention layer 51 then calculates an Attension (Q, K, V) using the query Q, key K, and value V. The residual coupling unit 66 adds (residual coupling) the feature vector obtained by skipping the cross-attention mechanism 65 with the feature vector from the cross-attention layer 51 of the cross-attention mechanism 65. The output of the residual coupling unit 66 is input to the normalization layer 41 of the next stage DiT block 37. The output of the residual coupling unit 66 of the Nth stage DiT block 37 becomes the final output for the entire N DiT blocks 37.

[0058] While the RoPE attention mechanism 61 is designed to capture the embedding of relative positions, the cross attention mechanism 65 is responsible for generating audio that aligns with semantic content.

[0059] The DiT block 37 is configured as described above.

[0060] <4. TTA Model in Inference Mode> Figure 6 is a block diagram of the TTA model 10 in inference mode. In Figure 6, parts common to the TTA model 10 in learning mode shown in Figure 2 are denoted by the same reference numerals, and redundant explanations are omitted as appropriate.

[0061] The inference mode TTA model 10 includes a prompt information input unit 21, a text ENC 22, a pooling layer 23, and an MLP 24. The TTA model 10 also includes a time input unit 81, a noise generation unit 34, a position embedding unit 36, N DiT blocks 37, an unpatch unit 82, a VAE DEC 83, a vocoder 84, and an output unit 85.

[0062] Compared to the learning mode, the inference mode TTA model 10 has a time input unit 81 instead of an audio input unit 31, and only a noise generation unit 34 is provided between the time input unit 81 and the position embedding unit 36. In addition, an unpatch unit 82, a VAE DEC 83, a vocoder 84, and an output unit 85 are newly provided after the N DiT blocks 37.

[0063] The time input unit 81 receives the audio length input by the user and outputs it to the noise generation unit 34. The user inputs the length of the audio to be generated (in seconds), for example, "20 (seconds)," for example, using a text prompt or the like. The user can input (specify) an audio length to the time input unit 81 that exceeds the maximum length of training data in learning mode (for example, 10 seconds).

[0064] The noise generation unit 34 generates Gaussian noise corresponding to the audio length specified by the user and outputs it to the position embedding unit 36. In other words, the noise generation unit 34 generates Gaussian noise for the duration supplied from the time input unit 81 and outputs it to the position embedding unit 36.

[0065] The position embedding unit 36 ​​converts the Gaussian noise supplied from the noise generation unit 34 into a number of tokens (feature vectors) corresponding to the audio length. For example, if L tokens are generated for a maximum training data length of 10 seconds in training mode, and the audio length is specified as "20 (seconds)" in the time input unit 81, the position embedding unit 36 ​​generates 2L tokens. If the audio length supplied from the time input unit 81 exceeds the range covered by training, in other words, if the number of tokens to be generated exceeds the range covered by training, the position embedding unit 36 ​​generates a number of tokens exceeding the range covered by training using RoPE extension with NTK-by-parts interpolation.

[0066] The N DiT blocks 37 perform the same processing as in the learning mode, except that they use the learning parameters (weights) learned in the learning mode. As the N DiT blocks 37 sequentially process using the learned learning parameters (weights), noise is removed from 2L tokens, and the denoised feature vector is output.

[0067] The unpatching unit 82 performs the reverse process of the patching unit 33, expanding each of the 2L tokens into p×p audio features in the frequency axis and time axis directions. The VAE DEC 83 is a decoder corresponding to the VAE ENC 32, converting the audio features in the latent space output from the unpatching unit 82 into the same output space as the input space, and converting the converted audio u in mel spectogram format. A Outputs.

[0068] The vocoder 84 is an audio u in Melspectogram format. A The audio signal is converted into an audio signal and output to the output unit 85. The output unit 85 outputs sound based on the audio signal from the vocoder 84. For example, a HiFiGAN can be used as the vocoder 84.

[0069] The TTA model 10 in inference mode is configured as described above.

[0070] <5. Audio generation for audio lengths exceeding the learning range> Next, we will explain audio generation when an audio length exceeding the learning range is specified during inference. Assume that the number of tokens corresponding to the audio length input in the time input unit 81 is L', and L' is greater than the number of tokens L of the maximum length of training data in the learning mode during learning (L' > L).

[0071] <RoPE Scaling> When the token sequence is extended from the maximum length L during learning to L', it is necessary to perform RoPE scaling to extend RoPE. To extend the token sequence from the maximum length L during learning to L', it is necessary to change the basis b of the rotation frequency matrix h(θ). The extension of the token sequence by changing the basis b of the rotation frequency matrix h(θ) is known as an interpolation technique for RoPE function extension. However, the TTA model 10 extends the token sequence by using NTK-by-parts interpolation for target interpolation of each dimension i of RoPE instead of changing the basis b. Specifically, the rotation frequency matrix h(θ) is changed by creating a piecewise linear function of the following equation (10) based on the i dimension. s in Equation (10) represents the scaling ratio between the maximum length L during learning and the generated audio length L', and is calculated as s = max(L' / L, 1.0). The scaling depends on the wavelength λ i that defines the rotation frequency in the i dimension. Therefore, γ() is an interpolation rate that linearly interpolates with the scaling ratio s, and a ramp-up function of the following equation (11) is adopted similar to the technology of YaRN. α and β in Equation (11) are hyperparameters for setting the threshold of the ratio r i . The ratio r i is calculated by Equation (12) according to the i dimension in order to linearly interpolate with the scaling ratio s.

[0072] According to Equation (11), when the ratio r i is smaller than α, γ(r ｉ ) = 0, and when the ratio r i is larger than β, γ(r ｉ ) = 1. When α ≤ γ(r ｉ ), γ(r ｉ ) linearly changes between 0 and 1. Therefore, in Equation (10), when γ(r ｉ ) = 0, h(θ ｉ ) = θ ｉ / s, that is, θ ｉ is scaled (stretched) to 1 / s, and when γ(r ｉIf ) = 1, then h(θ ｉ ) = θ ｉ , that is, θ ｉ We keep it as is, α≦γ(r ｉ ) ≤ β indicates that the two are mixed in a predetermined ratio.

[0073] Equation (12) shows the i-dimensional wavelength λ i This corresponds to how many of the tokens are collected within the token length L'. Wavelength λ i If it is long, ratio r i As becomes smaller, the scaling factor to 1 / s in equation (10) becomes stronger, and the wavelength λ i If the ratio r is short, i As this increases, the proportion of high-frequency components retained in equation (10) becomes stronger. As described above, the interpolation ratio of scaling γ(r ｉ ) with the i-dimensional wavelength λ i It can be changed accordingly.

[0074] NTK-by-parts interpolation uses wavelength λ i If the learning length is greater than or equal to the maximum length L (L ≤ λ) i This applies only if the position index m increases. As the position index m increases, a phase shift in the rotation angle occurs after rotation. To mitigate this phase shift, the wavelength λ i If it is smaller than the maximum length L during learning (λ i <L) applies Resonance YaRN technology, and according to equation (13), the rotation angle θ i It will be adjusted. λ in equation (13) i ' is, λ i The rounded integer value (λ i '=round(λ i )) and the rotation angle θ of RoPE in the i-th dimension i However, λ i θ using ' i It will be updated to '.

[0075] <Frequency-based attention scaling> When the token sequence is expanded from its maximum length L during training to L', and RoPE scaling is performed to extend RoPE, the query and key are also scaled using a scaling factor μ as shown in equation (14) to enhance the extrapolation function of length L'.

[0076] The calculation of the attention score A in equation (8) above also needs to be scaled to match the extension of RoPE. Specifically, the attention score A is scaled using the scaling coefficient μ by the following equation (15).

[0077] The scaling factor μ is a constant, and in YaRN technology, for example, it is calculated as "μ = 0.1 ln(s) + 1", where "s" represents the length of the token sequence. However, a constant scaling factor can impair the model's ability to scale to longer sequences.

[0078] Therefore, the TTA model 10 calculates the scaling coefficient μ using the following equation (16).

[0079] In equation (16), "F" represents the number of frequency divisions of the token sequence, and in the example in Figure 3, F = 3. "m" represents the position of the token, and "L'" represents the number of tokens in the generated audio.

[0080] Equation (16) is based on the observation that multiple attention maps corresponding to frequencies within a single time frame form multiple active blocks. F tokens within the same time frame are assigned the same scaling factor μ.

[0081] According to equation (16), the scaling coefficient μ can be dynamically changed depending on the token position m. Furthermore, the attention block can be preserved when extended to a longer token sequence by the number of divisions F in the frequency direction. The minimum value of the scaling coefficient μ can be set to (0.1 ln(s) + 1) or greater, i.e., "0.1 ln(s) + 1 ≤ μ", using YaRN's "μ = 0.1 ln(s) + 1".

[0082] <6. TTA Model Learning Process> Next, the learning process of the TTA model 10, which is performed in learning mode, will be explained with reference to the flowchart in Figure 7. This process starts, for example, when audio of a predetermined duration and its ground truth, GT audio, are selected from a predetermined audio dataset and input to the TTA model 10 as learning data, and when text as prompt information is input to the TTA model 10. The text input as prompt information is acquired by the prompt information input unit 21. The audio as learning data is acquired by the audio input unit 31. The GT audio is acquired by the GT input unit 38.

[0083] First, in step S1, the text ENC 22 converts the text supplied from the prompt information input unit 21 into a TEXT vector as a feature vector and outputs it to the pooling layer 23 and the DiT block 37. For example, FlanT5 can be used as the text ENC (encoder) 22.

[0084] In step S2, the pooling layer 23 performs pooling on the TEXT vector from the text ENC 22, downsizes it to a size suitable for input to the MLP 24, and outputs it to the MLP 24.

[0085] In step S3, the MLP 24 generates and outputs various coefficients as global conditions used in the DiT block 37 based on the downsized TEXT vector supplied from the pooling layer 23. Specifically, the MLP 24 generates the scale coefficient α １, α ２ gamma １ , and γ ２ and the shift coefficient β １ and β ２ It generates and outputs to DiT block 37.

[0086] In step S4, the VAE ENC 32 takes the audio u in mel-spectrogram format from the audio input unit 31 and the audio features x in the latent space. ０ It is converted and output to the patching unit 33.

[0087] In step S5, the patching unit 33 processes the audio features x of the latent space. ０ This is patched into one patch with p p×p patch sizes in both the frequency axis and time axis directions. This results in the latent space audio features x ０ ga p 2 It is downsampled by a factor of 2. Then, the patching unit 33 processes the audio features after downsampling x ０ This is tokenized into L tokens z.

[0088] In step S6, the noise generation unit 34 generates Gaussian noise and outputs it to the addition unit 35. The addition unit 35 adds the Gaussian noise generated by the noise generation unit 34 to each of the L tokens z and outputs it to the position embedding unit 36.

[0089] In step S7, the position embedding unit 36 ​​generates and outputs a vector (embedding vector) that takes into account position information in the time axis direction and the frequency axis direction. Specifically, the position embedding unit 36 ​​embeds position information using RoPE for the time axis direction and embeds position information using absolute position for the frequency axis direction.

[0090] In step S8, the GT input unit 38 outputs GT audio u_GT, which is the ground truth of audio u, in the format v(x) corresponding to the predicted value v_hat output by the DiT block 37. t Convert to (t,y) and output to parameter update unit 39.

[0091] The processes in steps S1 to S3, steps S4 to S7, and step S8 can all be executed in parallel.

[0092] In step S9, denoising is performed by N DiT blocks 37 connected in series, which repeat the process M times (M > 1). Through the denoising process performed by the N DiT blocks 37 M times, a predicted value v_hat, which is a denoised feature vector, is generated for each of the L tokens z, and is output to the parameter update unit 39.

[0093] In step S10, the parameter update unit 39 receives the predicted value v_hat, which is the output of the Nth stage DiT block 37, and v(x) supplied from the GT input unit 38 as the true value corresponding to the predicted value v_hat. t Using t and y, the loss function L is given by equation (7). D The parameter update unit 39 calculates the loss function L. D The learning parameters (weight coefficients W) for each of the N DiT blocks 37 are updated to minimize the value.

[0094] The denoising process in step S9 and the learning parameter update process in step S10 based on the denoised predicted value v_hat are repeated a predetermined number of times to obtain the final learning parameters.

[0095] Figure 8 is a flowchart illustrating the details of the denoising process performed in the N DiT blocks 37 in step S9 of Figure 7.

[0096] In step S21, the RoPE attention calculation process is performed by the RoPE attention mechanism 61. Specifically, the feature vector input to the RoPE attention mechanism 61 is subjected to normalization by the normalization layer 41, scaling and shifting by the scale shift layer 42, and normalization by the LlamaRMS normalization layer 43. Subsequently, the RoPE attention layer 44 rotates the query vector and key vector by a rotation angle θ corresponding to the token position m and calculates the attention score A in equation (8). Furthermore, the RoPE attention layer 44 outputs an embedding vector as a feature vector, which is obtained by calculating the dot product of the attention score A and the value vector. The output of the RoPE attention layer 44 is then processed by the scale layer 45, which calculates the scale coefficient α from the MLP 24. １ After scaling, it is supplied to the residual coupling section 62.

[0097] In step S22, the residual coupling unit 62 performs a residual coupling process that adds the feature vector output from the RoPE attention mechanism 61 with the feature vector obtained by skipping the RoPE attention mechanism 61, and outputs the result to the feedforward mechanism 63 and the residual coupling unit 64.

[0098] In step S23, a feedforward process is performed by the feedforward mechanism 63. More specifically, the feature vector input to the feedforward mechanism 63 is subjected to normalization by the normalization layer 46, and scaling and shifting by the scale shift layer 47. Subsequently, the feedforward layer 48 performs feature enhancement, transformation, etc., by expanding the dimension, performing a nonlinear transformation using a predetermined activation function, and compressing the dimension. The output of the feedforward layer 48 is processed by the scale layer 49, which receives the scale coefficient α from the MLP 24. ２ After scaling, it is supplied to the residual coupling section 64.

[0099] In step S24, the residual coupling unit 64 performs a residual coupling process that adds the feature vector output from the feedforward mechanism 63 and the feature vector obtained by skipping the feedforward mechanism 63, and outputs the execution result to the cross attention mechanism 65 and the residual coupling unit 66.

[0100] In step S25, the cross-attention calculation process is performed by the cross-attention layer 51 of the cross-attention mechanism 65. The cross-attention layer 51 generates a query Q using the output from the residual coupling unit 64 and generates a key K and a value V using the output from the text ENC 22. Then, the cross-attention layer 51 calculates Attension(Q,K,V) using the query Q, key K, and value V.

[0101] In step S26, the residual merging unit 66 performs a residual merging process that adds the feature vector output from the cross-attention mechanism 65 with the feature vector obtained by skipping the cross-attention mechanism 65.

[0102] Each of the N DiT blocks 37 sequentially executes the processes described in steps S21 to S26. After the series of denoising processes by the N DiT blocks 37 are repeated a predetermined number of times (for example, 50 or 100 times), the final output is output from the Nth DiT block 37 to the parameter update unit 39.

[0103] The above learning process is performed using multiple audio files selected from a predetermined audio dataset as training data. If the maximum length (duration) of the selected audio files is, for example, 10 seconds, the learning process is performed using audio files of multiple lengths by appropriately cropping the 10-second audio file. For example, audio files are generated by cropping the 10-second audio file to 2.5 seconds, 5.0 seconds, and 7.5 seconds, and the learning process is performed using audio files of four different lengths: 2.5 seconds, 5.0 seconds, 7.5 seconds, and 10 seconds. As a result, the TTA model 10 can be more appropriately generalized to various audio lengths, improving its performance in real-world scenarios where audio lengths can vary significantly.

[0104] <7. Inference Processing of the TTA Model> Next, the inference processing of the TTA model 10 performed in inference mode will be explained with reference to the flowchart in Figure 9. This processing starts, for example, when text describing the audio to be generated is input as prompt information to the prompt information input unit 21, and the length (duration) of the audio to be generated is input to the time input unit 81.

[0105] The processes from steps S41 to S43 are the same as steps S1 to S3 in the learning process, so their explanation will be omitted.

[0106] In step S44, the noise generation unit 34 generates Gaussian noise corresponding to the audio length specified by the user and outputs it to the position embedding unit 36. In other words, the noise generation unit 34 generates Gaussian noise for the length of the audio supplied from the time input unit 81 and outputs it to the position embedding unit 36.

[0107] In step S45, the position embedding unit 36 ​​determines whether the data length of the Gaussian noise exceeds the range covered by the learning process.

[0108] If, in step S45, it is determined that the data length of the Gaussian noise does not exceed the range covered by training, that is, if the data length of the Gaussian noise is less than or equal to the range covered by training, the process proceeds to step S46, where the position embedding unit 36 ​​generates a number of tokens corresponding to the data length by embedding RoPE and absolute position. If the data length of the Gaussian noise does not exceed the range during the training process, the token generation is performed in the same way as during the training process.

[0109] On the other hand, if in step S45 it is determined that the data length of the Gaussian noise exceeds the range covered by learning, the process proceeds to step S47, where the position embedding unit 36 ​​generates a number of tokens exceeding the range covered by learning, using RoPE extension with NTK-by-parts interpolation and dynamic attention scaling that takes frequency into account.

[0110] In step S48, denoising is performed by N DiT blocks 37 connected in series, which process M times (M > 1). This denoising process generates a denoised feature vector for each token z, which is output to the unpatch unit 82.

[0111] In step S49, the unpatching unit 82 expands each token supplied from the Nth DiT block 37 into p×p audio features in the frequency axis and time axis directions, and outputs them to the VAE DEC 83.

[0112] In step S50, the VAE DEC 83 converts the audio features of the latent space output from the unpatch unit 82 into audio u in the form of a Mel spectogram, which is the same output space format as the input space. A Convert to VAE DEC83. The converted audio is in Melspectogram format. A The output is sent to vocoder 84.

[0113] In step S51, the vocoder 84 outputs audio in Melspectogram format. A The audio signal is converted into an audio signal and output to the output unit 85. The output unit 85 outputs sound based on the audio signal from the vocoder 84 and terminates the inference process.

[0114] Figure 10 is a flowchart illustrating the details of the denoising process performed in the N DiT blocks 37 in step S48 of Figure 9.

[0115] In step S61, the RoPE attention mechanism 61 of the DiT block 37 determines whether the total number of tokens exceeds the length of the range covered by learning.

[0116] If it is determined in step S61 that the total number of tokens does not exceed the length covered by the learning process, the process proceeds to step S62, and the RoPE attention mechanism 61 performs the same RoPE attention calculation process as during the learning process. The RoPE attention layer 44 calculates the attention score A using equation (8).

[0117] On the other hand, if it is determined in step S61 that the total number of tokens exceeds the length covered by the learning, the process proceeds to step S63, where the RoPE attention mechanism 61 performs a RoPE attention calculation process by applying the scaling coefficient μ of equations (14) and (16) to the query and key. The RoPE attention layer 44 calculates the attention score A using equation (15).

[0118] The processes from step S64 to step S68 are the same as steps S22 to S26 in the learning process, so their explanation will be omitted.

[0119] Each of the N DiT blocks 37 sequentially executes the processes described in steps S61 to S68. After the series of denoising processes by the N DiT blocks 37 are repeated a predetermined number of times (for example, 50 or 100 times), the final output is output from the Nth DiT block 37 to the unpatch unit 82.

[0120] <8. Evaluation of this method in comparison with other methods> Next, we will explain the evaluation results of the TTA model 10 (hereinafter also referred to as this method) in comparison with state-of-the-art models (SOTA models).

[0121] The following evaluation experiments compare this method with recent TTA spread spectrum models such as "AudioLDM2," "Tango2," and "Make-an-Audio2." For completeness, "Stable Audio Open (SA-Open)" is also included in the evaluation experiments for comparison. However, since SA-Open focuses on 44.1kHz stereo audio, while this method and other prior models are related to 16kHz monaural audio, the comparison may not be entirely fair.

[0122] The model was trained and evaluated using AudioCaps, a standard TTA benchmark dataset. For evaluation using Clotho-eval, the model was trained using WavCaps and Clotho-train. All of these datasets were converted to 16kHz and 10-second clips.

[0123] As evaluation metrics, FAD (Frechet Audio Distance) scores and CLAP scores were used, in accordance with protocols from previous studies. A lower FAD score indicates better quality, while a higher CLAP score indicates better quality. In addition, human evaluations on a 5-point scale ("1": poor quality, "5": excellent quality) were also conducted for OVeralL impression (OVL) and text-audio RELevance (REL).

[0124] Figure 11 shows the results of benchmark tests of this method and other prior models using the AudioCaps dataset.

[0125] Figure 12 shows the results of benchmark tests of this method and other prior models using the Clotho dataset.

[0126] Figures 11 and 12 show the benchmark test results when the audio length is 10 seconds, the same as during training.

[0127] Referring to Figure 11, the FAD score of this method is lower than that of other prior models. Furthermore, the CLAP score, OVL, and REL scores of this method are higher than those of other prior models. Therefore, the test results in Figures 11 and 12 indicate that this method is effective compared to other prior models. "Param." in Figure 11 indicates the number of parameters, showing that this method requires fewer parameters for training compared to other prior models.

[0128] Figure 13 shows the results of benchmark tests comparing this method with other prior models in the case of variable-length generation.

[0129] Figure 13A shows the comparison results of FAD scores in variable length generation.

[0130] This method maintains an FAD score below 2.0 across different audio lengths (durations) from 5 to 30 seconds. Other prior models, when generating longer audio not covered in training, show a decline in performance as the audio length increases. U-Net-based models such as "AudioLDM2" and "Tango2" are trained only on 10-second audio, but their FAD scores worsen by more than 2 points when generating variable-length audio other than 10 seconds. Furthermore, "Make-an-Audio2" also shows a decline in performance when the audio length exceeds 20 seconds.

[0131] Figure 13B shows the results of a comparison with other RoPE extension technologies.

[0132] This experiment compares the present method with previous models for long-term generation of RoPE extensions. The present method is compared with other RoPE extension methods, including (1) no RoPE extension, (2) position interpolation, (3) NTK-aware interpolation, and (4) Resonance YaRN. In this experiment, training is performed with a 5-second audio length, and the generated audio length is set to exceed 5 seconds.

[0133] The evaluation results in Figure 13B show that methods that do not use attention scaling techniques, such as (2) position interpolation and (3) NTK-aware interpolation, are not effective for extending to unknown long sequences.

[0134] Figure 13C shows the results of comparing the cost efficiency of the TSTL scenario with that of "Make-an-Audio2". In Figure 13C, cost efficiency is compared in terms of the number of processing tokens and the GPU memory cost.

[0135] As shown in Figure 13C, "Make-an-Audio2" requires training on longer sequences to achieve effective performance. On the other hand, this method maintains the number of processing tokens, thus reducing the cost of long training sessions. Furthermore, while "Make-an-Audio2" requires approximately 10GB of GPU memory for a 10-second extension, this method avoids this extra memory usage by training on shorter audio sequences.

[0136] Figure 14 shows a comparison of frequency-based RoPE with other PEs. In Figure 14, the data was trained with a 10-second audio length and compared with lengths of 20 and 30 seconds.

[0137] In the comparison between RoPE and absolute PE in Figure 14, the FAD score of AudioCaps for 10 seconds decreased from 1.89 (absolute PE) to 3.20 (RoPE), showing that simply replacing absolute PE with RoPE in the DiT block degrades performance. In other words, implementing the DiT block by directly replacing absolute PE with RoPE is inappropriate. Therefore, we believe that, as in our method, tokens related to the frequency axis should be encoded into absolute position embeddings in order to preserve the structure of the mel spectogram. On the other hand, when using frequency-based RoPE, when trained with only 10 seconds of audio, a superior FAD score of 1.32 was achieved compared to the DiT block using absolute PE (FAD score = 1.89). As described above, the comparison of variable-length audio generation for 20 seconds and 30 seconds in Figure 14 demonstrates the superiority of our method (frequency-based RoPE) compared to absolute PE.

[0138] Figure 15 shows the inpainting and outpainting performance of this method for a masked audio input.

[0139] The experiment was conducted using audio clips of 10 and 20 seconds in length. In in-painting, the middle 15 seconds were masked, leaving 2.5 seconds at each end. In out-painting, 17.5 seconds were masked, starting from 2.5 seconds. The pre-trained model was used with the AudioCaps dataset without fine-tuning. As shown in Figure 15, the in-painting and out-painting reconstructions of this method are close to the upper limit scores based on VAE reconstruction from unmasked inputs.

[0140] <9. Summary of this method> The TTA model 10 can generate audio of any length without additional training or input conditions by using frequency-based RoPE, which is relative position embedding, thereby improving efficiency. This no-frills-up approach is superior to state-of-the-art preceding models in both standard audio generation and variable-length audio generation. Therefore, the TTA model 10 can accurately generate audio of previously untrained lengths without additional training. In addition, the model size is smaller compared to state-of-the-art preceding models, thus reducing resources.

[0141] <10. Description of a computer to which this technology is applied> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on the computer. Here, the computer includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0142] Figure 16 is a block diagram showing an example of the hardware configuration of the information processing device 1, where the information processing device 1 is composed of a computer that executes the series of processes described above by a program.

[0143] The information processing device 1 includes a processing circuit 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The processing circuit 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected by a bus 904.

[0144] An input / output interface 905 is further connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0145] The input unit 906 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 906 may include sensors for inputting various physical quantities to the computer. For example, the input unit 906 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 906 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, distance, etc. The output unit 907 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 908 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various types of information (including programs). The communication unit 909 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 910 drives removable media 911 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0146] The processing circuit 901 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 901 (its processor) performs the above-described series of processes by loading the program stored in the storage unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executing it. The processing circuit 901 can output the processing results of the series of processes from the output unit 907 via the bus 904 and the input / output interface 905 as needed. The processing circuit 901 can also store the processing results in the storage unit 908 or transmit them from the communication unit 909.

[0147] The program executed by the processing circuit 901 of the information processing device 1 can be provided by recording it on a removable medium 911, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0148] In the information processing device 1, which functions as a computer, programs can be installed in the storage unit 908 via the input / output interface 905 by inserting the removable media 911 into the drive 910. Alternatively, programs can be received by the communication unit 909 from other devices such as servers via wired or wireless transmission media and installed in the storage unit 908. Furthermore, programs can be pre-installed in the ROM 902 or the storage unit 908.

[0149] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0150] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0151] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0152] When the computer executes a program to perform the above-described series of processes, the input unit 906 of the information processing device 1 functions as a prompt information input unit 21, an audio input unit 31, a GT input unit 38, and a time input unit 81. When the processing circuit 901 (its processor) executes a program, it functions as a text ENC 22, a pooling layer 23, an MLP 24, a VAE ENC 32, a patching unit 33, a noise generation unit 34, an addition unit 35, a position embedding unit 36, a DiT block 37, a parameter update unit 39, an unpatching unit 82, a VAE DEC 83, and a vocoder 84. The output unit 907 functions as an output unit 85.

[0153] The components (blocks) of the information processing device 1 illustrated in this specification are functionally conceptual blocks, and the actual device does not need to have the illustrated configuration. That is, the device can have any configuration in which the functions of the illustrated components are divided into arbitrary units and / or integrated, for example, a configuration having one block in which the functions of all components are integrated.

[0154] The embodiments of this disclosure are not limited to those described above, and various modifications are possible without departing from the gist of the technology of this disclosure.

[0155] For example, a configuration can be adopted that arbitrarily combines all or part of the embodiments described above.

[0156] For example, the technology disclosed herein can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0157] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0158] The technology disclosed herein can take the following configurations: (1) An information processing device comprising: a prompt information input unit that receives constraints for the audio to be generated as prompt information; a noise generation unit that generates noise for the duration of the audio to be generated; a position embedding unit that embeds position information for the time axis position of the audio using RoPE and for the frequency axis position of the generated noise using absolute position; and a diffusion transformer module that performs denoising of the embedding vector into which the position information is embedded. (2) The information processing device according to (1), further comprising: an encoder that converts the prompt information into a feature vector that can be input to the diffusion transformer module; and a global coefficient generation unit that generates global coefficients by compressing the feature vector converted by the encoder, wherein the feature vector and the global coefficients are configured to be input to the diffusion transformer module. (3) The information processing device according to (2), wherein the feature vector is configured to be input to the cross-attention mechanism of the diffusion transformer module. (4) The information processing device according to any one of (1) to (3) above, wherein the diffusion transformer module includes a RoPE attention mechanism, a feedforward mechanism, and a cross attention mechanism. (5) The information processing device according to (4) above, wherein the attention layer of the RoPE attention mechanism executes a RoPE attention calculation process that applies a scaling factor to the query and key when the duration of the audio to be generated exceeds the maximum length during training. (6) The information processing device according to (5) above, wherein the scaling factor is configured to be calculated using the number of divisions in the frequency direction. (7) The information processing device according to any one of (1) to (6) above, wherein the position embedding unit expands the token sequence by using NTK-by-parts interpolation for target interpolation in each dimension of RoPE when the duration of the audio to be generated exceeds the maximum length during training.(8) The position embedding unit performs a process to reduce the phase shift of the rotation angle of each dimension of RoPE when the duration of the generated audio is less than the maximum length during training. (9) The information processing device according to (8), wherein in the process to reduce the phase shift of the rotation angle of each dimension, the rotation angle is calculated using an integer value obtained by rounding the wavelength of the dimension. (10) The information processing device according to (1) to (9), wherein the position embedding unit calculates a query and key by applying a scaling coefficient when the duration of the generated audio exceeds the maximum length during training. (11) The information processing device according to (1) to (10), further comprising: an audio input unit that accepts audio input as training data; a variational autoencoder that converts the audio input as training data into a latent space; and a patching unit that bundles multiple units in the frequency axis direction and time axis direction of the audio features converted into the latent space into a single patch, wherein the patched audio features are input to the position embedding unit. (12) An information processing device according to any one of (1) to (11), further comprising: an unpatching unit that unpatches the audio features output from the diffusion transformer module into multiple units in the frequency axis direction and the time axis direction; a decoder that converts the unpatched audio features into the same output space as the input space; and a vocoder that converts audio in Mel Spectogram format output from the decoder into an audio signal. (13) An information processing method comprising: an information processing device receiving constraints for the audio to be generated as prompt information; generating noise for the duration of the audio to be generated; embedding position information in the generated noise using RoPE for the position in the time axis direction of the audio, and embedding position information using absolute position for the position in the frequency axis direction; and performing denoising on the embedding vector in which the position information is embedded.(14) A program for causing a computer to perform a process that includes receiving constraints for the audio to be generated as prompt information; generating noise for the duration of the audio to be generated; embedding position information in the generated noise using RoPE for the position in the time axis direction of the audio, and embedding position information using absolute position for the position in the frequency axis direction; and performing denoising on the embedding vector into which the position information is embedded.

[0159] 1 Information processing device, 10 TTA model, 21 Prompt information input unit, 22 Text ENC, 23 Pooling layer, 24 MLP, 25 Global coefficient generation unit, 31 Audio input unit, 33 Patching unit, 34 Noise generation unit, 35 Addition unit, 36 Position embedding unit, 37 DiT block, 38 GT input unit, 39 Parameter update unit, 41 Normalization layer, 42 Scale shift layer, 43 Normalization layer, 44 RoPE attention layer, 45 Scale layer, 46 Normalization layer, 47 Scale shift layer, 48 Feedforward layer, 49 Scale layer, 51 Cross attention layer, 61 RoPE attention mechanism, 62 Residual coupling unit, 63 Feedforward mechanism, 64 Residual coupling unit, 65 Cross attention mechanism, 66 81 Residual coupling unit, 82 Time input unit, 83 Unpatch unit, 84 VAEDEC, 85 Vocoder, 901 Output unit, 901 Processing circuit, 902 ROM, 904 Bus, 905 Input / Output interface, 906 Input unit, 907 Output unit, 908 Storage unit, 909 Communication unit, 910 Drive

Claims

1. An information processing device comprising: a prompt information input unit that receives constraints for the audio to be generated as prompt information; a noise generation unit that generates noise for the duration of the audio to be generated; a position embedding unit that embeds position information for the generated noise using RoPE for the position in the time axis direction of the audio and using absolute position for the position in the frequency axis direction; and a diffusion transformer module that performs denoising processing on the embedding vector into which the position information is embedded.

2. The information processing apparatus according to claim 1, further comprising: an encoder that converts the prompt information into a feature vector that can be input to the diffusion transformer module; and a global coefficient generation unit that generates global coefficients by compressing the feature vector converted by the encoder, wherein the feature vector and the global coefficients are configured to be input to the diffusion transformer module.

3. The information processing device according to claim 2, configured such that the feature vector is input to the cross-attention mechanism of the diffusion transformer module.

4. The information processing apparatus according to claim 1, wherein the diffusion transformer module includes a RoPE attention mechanism, a feedforward mechanism, and a cross attention mechanism.

5. The information processing apparatus according to claim 4, wherein the attention layer of the RoPE attention mechanism performs a RoPE attention calculation process that applies a scaling factor to the query and key if the duration of the audio to be generated exceeds the maximum length during training.

6. The information processing apparatus according to claim 5, wherein the scaling coefficient is configured to be calculated using the number of divisions in the frequency direction.

7. The information processing apparatus according to claim 1, wherein the position embedding unit extends the token sequence by using NTK-by-parts interpolation for target interpolation of each dimension of RoPE when the duration of the generated audio exceeds the maximum length during training.

8. The information processing apparatus according to claim 1, wherein the position embedding unit performs a process to reduce the phase shift of the rotation angle of each dimension of RoPE when the duration of the generated audio is less than the maximum length during learning.

9. The information processing apparatus according to claim 8, wherein, in the process of reducing the phase shift of the rotation angle of each dimension, the rotation angle is calculated using an integer value obtained by rounding the wavelength of the dimension.

10. The information processing apparatus according to claim 1, wherein the position embedding unit applies a scaling factor to calculate a query and key if the duration of the generated audio exceeds the maximum length during training.

11. The information processing apparatus according to claim 1, further comprising: an audio input unit that accepts audio input as training data; a variational autoencoder that converts the audio input as training data into a latent space; and a patching unit that bundles multiple units of the audio features converted into the latent space in the frequency axis direction and the time axis direction into a single patch, wherein the patched audio features are configured to be input to the position embedding unit.

12. The information processing apparatus according to claim 1, further comprising: an unpatching unit that unpatches the audio features output from the diffusion transformer module into multiple units in the frequency axis direction and the time axis direction; a decoder that converts the unpatched audio features into the same output space as the input space; and a vocoder that converts audio in mel-spectogram format output from the decoder into an audio signal.

13. An information processing method comprising: an information processing device receiving constraints for audio to be generated as prompt information; generating noise for the duration of the audio to be generated; embedding position information in the generated noise using RoPE for the position in the time axis direction of the audio, and embedding position information using absolute position for the position in the frequency axis direction; and performing denoising on the embedding vector into which the position information is embedded.

14. A program for causing a computer to perform a process that includes: receiving constraints for the audio to be generated as prompt information; generating noise for the duration of the audio to be generated; embedding position information in the generated noise using RoPE for the position in the time axis direction of the audio, and embedding position information using absolute position for the position in the frequency axis direction; and performing denoising on the embedding vector into which the position information is embedded.

Images