A method and apparatus for generating audio tags

Abstract

This application provides a method and apparatus for generating audio tags. The generation method includes: acquiring audio track information of audio data to be processed; wherein the audio track information includes whether audio track separation processing has been performed, multiple audio track data to be processed after audio track separation processing, and the correlation between each audio track data to be processed; determining a target audio tag generation model from at least one candidate audio tag generation model according to the audio track information and a preset generation model matching rule; wherein the target audio tag generation model generates discrete audio tags based on the audio data; and inputting the audio data to be processed or the multiple audio track data to be processed into the target audio tag generation model to obtain the target audio tags. Thus, through the technical solution of this application, music tags suitable for generating multi-level music can be obtained, thereby improving the training effect of subsequent music models and increasing the quality of the output music.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a method and apparatus for generating audio tags. Background Technology

[0002] In existing audio tag generation models, the training process typically involves using an encoder and VQ (Vector quantization) to convert continuous audio data into discrete audio tokens. These tokens are then used by the language or music model in subsequent training steps. For a detailed description of the training process for existing audio tag generation models, please refer to [link to relevant documentation]. Figure 1 , Figure 1 A schematic diagram illustrating the training process of an existing audio token generation model, as shown below. Figure 1 In existing audio token generation models, the original audio data is input into the encoder, which then inputs the processed continuous vector into the VQ to generate discrete audio tokens. The generated discrete tokens are then input into the decoder to obtain the predicted audio. Finally, the model parameters are adjusted based on the loss function value between the original audio and the predicted audio to complete the training of the audio token generation model.

[0003] However, the audio token generation model trained using the above architecture tends to mix all music features together during the training of the encoder and VQ, which can lead to interference between different levels of audio features when processing multi-level training data. For example, it may mix vocals with accompaniment, multiple vocal parts, or multiple instruments. It may even interrupt the content of other levels when representing the target level, such as stopping the accompaniment when outputting vocals. If such music tokens are used directly to train the music generation model, the generated multi-level music may also have the same problems, resulting in poor output music quality. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method and apparatus for generating audio tags, which can obtain music tags suitable for generating multi-level music, thereby improving the training effect of subsequent music models and improving the quality of output music.

[0005] This application provides a method for generating audio tags, the method comprising:

[0006] Obtain the audio track information of the audio data to be processed; wherein, the audio track information includes whether audio track separation processing has been performed, multiple audio track data to be processed after audio track separation processing, and the correlation between each audio track data to be processed;

[0007] Based on the audio track information and preset generation model matching rules, a target audio tag generation model is determined from at least one candidate audio tag generation model; wherein, the target audio tag generation model generates discrete audio tags based on audio data;

[0008] The audio data to be processed or the multiple audio tracks to be processed are input into the target audio tag generation model to obtain the target audio tag.

[0009] This application embodiment also provides an audio tag generation apparatus, the generation apparatus comprising:

[0010] The acquisition module is used to acquire the audio track information of the audio data to be processed; wherein, the audio track information includes whether audio track separation processing has been performed, multiple audio track data to be processed after audio track separation processing, and the correlation between each audio track data to be processed;

[0011] The determining module is used to determine a target audio tag generation model from at least one candidate audio tag generation model based on the audio track information and a preset generation model matching rule; wherein the target audio tag generation model generates discrete audio tags based on audio data;

[0012] The generation module is used to input the audio data to be processed or the multiple audio tracks to be processed into the target audio tag generation model to obtain the target audio tag.

[0013] This application provides an audio tag generation method and apparatus. The generation method includes: acquiring audio track information of audio data to be processed; wherein the audio track information includes whether audio track separation processing has been performed, multiple audio track data to be processed after audio track separation processing, and the correlation between each audio track data to be processed; determining a target audio tag generation model from at least one candidate audio tag generation model according to the audio track information and a preset generation model matching rule; wherein the target audio tag generation model generates discrete audio tags based on audio data; and inputting the audio data to be processed or the multiple audio track data to be processed into the target audio tag generation model to obtain the target audio tag.

[0014] Thus, this solution provides multiple methods for generating music tokens required for multi-level music generation. By setting independent decoders or decoder groups and verification steps for the parts that need to be independent, it enhances the effect of each level in actual music token processing, avoiding interference and mixing between different levels. At the same time, the architecture of the designed generation model can also take into account scenarios that require both separation of levels and consideration of correlation. Using music tokens with clear levels to train the music generation model can enable the music generation model to generate more complex music, which can further improve the music generation effect.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram illustrating the training process of an existing audio tag generation model;

[0018] Figure 2 A flowchart illustrating an audio tag generation method provided in an embodiment of this application;

[0019] Figure 3 One of the schematic diagrams illustrating the principle of the training process of a target audio feature generation model provided in this application;

[0020] Figure 4 The second schematic diagram illustrating the principle of the training process of a target audio feature generation model provided in this application;

[0021] Figure 5 The third schematic diagram illustrating the principle of the training process of a target audio feature generation model provided in this application;

[0022] Figure 6 The fourth schematic diagram illustrating the principle of the training process of a target audio feature generation model provided in this application;

[0023] Figure 7 A schematic diagram of the structure of an audio tag generation device provided in an embodiment of this application;

[0024] Figure 8 A schematic diagram of another audio tag generation device provided in an embodiment of this application;

[0025] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0027] This application provides a method and apparatus for generating audio tokens, which can obtain music token representations suitable for generating and processing multi-level music. By training a music generation model using music tokens with clear hierarchical levels, the music generation model can generate more complex music, thereby improving the quality of the output music.

[0028] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for generating audio tags according to an embodiment of this application. Figure 2 As shown in the embodiments of this application, the generation method includes:

[0029] S101. Obtain the audio track information of the audio data to be processed.

[0030] S102. Based on the audio track information and the preset generation model matching rules, determine the target audio tag generation model from at least one candidate audio tag generation model.

[0031] S103. Input the audio data to be processed or the multiple audio track data to be processed into the target audio tag generation model to obtain the target audio tag.

[0032] The steps in this application example are explained below:

[0033] For step S101, the audio track information includes at least one of the following: whether audio track separation processing has been performed, multiple audio track data to be processed after audio track separation processing, and the correlation between each audio track data to be processed.

[0034] The relationship between the audio data and the track data is as follows: one audio data includes at least one track data. However, the audio data used in this solution typically includes multiple track data. When the audio data is music data, the included track data may include vocals and accompaniment, different instruments, different vocal parts (high, middle, low, etc.), different singers in a duet, etc. These various cases can also be combined; for example, the vocal part of a choral piece can be divided into high, middle, and low parts, and the musical part can be divided according to different instruments such as piano and violin, and so on.

[0035] The relationships between the various audio tracks to be processed can be determined based on pre-set data corresponding to strong relationships between audio tracks. For example, after audio track separation processing, the multiple audio tracks to be processed include audio track A, audio track B, and audio track C. The pre-set data corresponding to strong relationships between audio tracks specifies that audio track A and audio track B have a strong relationship, while the other audio tracks do not have a strong relationship with each other. Thus, it is determined that audio track A and audio track B have a strong relationship, while the other audio tracks do not have a strong relationship with each other.

[0036] Here, before obtaining the audio track information of the audio data to be processed (i.e., before executing step S101), the audio data to be processed, i.e., the data before separation processing, will be obtained.

[0037] Regarding step S102, the target audio tag generation model generates discrete audio tags based on audio data.

[0038] In this step, the preset generation model matching rules set the correspondence between audio track information and audio tag generation models.

[0039] For example, the determination of the target audio tag generation model is illustrated below: Assume that this application provides four candidate audio tag generation models, and the structures of the four candidate audio tag generation models are as follows:

[0040] Audio tag generation model 1: includes an encoder and a vector quantization layer.

[0041] Audio tag generation model 2: includes multiple encoders and multiple vector quantization layers, with the number of encoders and vector quantization layers being the same.

[0042] Audio tag generation model 3: includes an encoder group and multiple target vector quantization layers. The encoder group includes a first encoder and multiple second encoders, with the number of second encoders being the same as the number of vector quantization layers.

[0043] Audio tag generation model four includes multiple encoders and multiple vector quantization layers, with the number of encoders being less than the number of vector quantization layers.

[0044] When the audio track information is the audio data to be processed without track separation, the target audio tag generation model is determined to be either Audio Tag Generation Model 1 or Audio Tag Generation Model 3. The appropriate model can be selected adaptively based on factors such as the generation effect of the audio tag generation model and / or the type of music generated.

[0045] When the audio track information is audio data to be processed that has undergone audio track separation processing and includes multiple audio track data to be processed, the target audio tag generation model is determined to be audio tag generation model two.

[0046] When the audio track information is audio data to be processed that has undergone audio track separation processing, including audio track data A, audio track data B, and audio track data C, and audio track data A and audio track data B have a strong correlation, the target audio tag generation model determined at this time is audio tag generation model four.

[0047] Furthermore, this application provides the following methods for constructing the target audio tag generation model.

[0048] Example 1, for reference Figure 3 The target audio feature generation model is trained through the following steps:

[0049] S201. Obtain training audio data and an initial audio tag generation model.

[0050] S202. Input the training audio data and / or training track data into the encoder in the initial audio tag generation model to obtain the first audio vector data.

[0051] S203. Input the first audio vector data into the vector quantization layer in the initial audio tag generation model to obtain the first audio tag feature.

[0052] S204. Input the first audio tag features into each decoder in the initial audio tag generation model to obtain the predicted audio track data output by each decoder.

[0053] S205. Calculate the loss based on multiple predicted audio track data and multiple training audio track data, and update the parameters of the encoder, vector quantization layer, and each decoder based on the loss function results, until the output of the initial audio tag generation model satisfies the first preset condition, thereby obtaining the target audio tag generation model.

[0054] For step S201, the training audio data includes at least two training audio tracks obtained through track separation processing, and the initial audio tag generation model includes an encoder, a vector quantization layer, and multiple encoders.

[0055] Here, acquiring training audio data means acquiring multiple training audio data sets, and for each training audio data set, acquiring at least two training audio tracks obtained through track separation processing.

[0056] For step S202, this step includes: for each training audio data, inputting the training audio data and / or the training audio track data of the training audio data into the encoder of the initial audio tag generation model to obtain first audio vector data. More preferably, only the training audio data is input into the encoder of the initial audio tag generation model.

[0057] The first audio vector includes at least two levels of continuous audio vector features.

[0058] For step S203, the first audio tag feature is a discrete audio feature, and includes at least two levels of discrete audio features.

[0059] Regarding step S204, the predicted audio track data output by different decoders in this step are different.

[0060] Regarding step S205, when determining the loss function result based on multiple predicted audio track data and multiple training audio track data, this application provides an implementation method, specifically including:

[0061] S2051. For each predicted audio track data, determine the corresponding training audio track data.

[0062] S2052. Determine the loss function value at the track level based on the difference between the predicted track data and the corresponding training track data.

[0063] S2053. Merge the loss function values ​​of each audio track level to obtain the target loss function value.

[0064] For step S2051, this step may include: for each predicted audio track data, determining the corresponding training audio track data in the corresponding training audio data according to the type of the predicted audio track data.

[0065] For S2052, when there are multiple predicted audio track data, for each predicted audio track data in each training audio data, the loss function values ​​of multiple audio track levels of the training audio data are determined based on the difference between the predicted audio track data in the training audio data and the corresponding training audio track data in the training audio data.

[0066] For step S2053, this step includes: for each training audio data, fusing the loss function values ​​of multiple audio tracks of the training audio data to obtain the target loss function value of the training audio data.

[0067] Here, the loss function value at the audio track level is the loss function value at the single-layer level. The fusion process can be: addition, splicing, weighted addition, transform addition, etc.

[0068] In this process, after determining the target loss function value for each training audio data, the parameters of the encoder, vector quantization layer, and each decoder are updated based on the loss function result (target loss function value) until the output of the initial audio tag generation model satisfies the first preset condition, thus obtaining the target audio tag generation model.

[0069] The first preset condition may be that the difference between each predicted audio track data and the corresponding training audio track data in the output of the updated initial audio tag generation model is within a preset difference range for multiple consecutive times, the number of training times reaches the training threshold, and the model performance stops improving.

[0070] For an example, please refer to Figure 3 , Figure 3 This is one of the schematic diagrams illustrating the principle of the training process of a target audio feature generation model provided in this application. Through... Figure 3 The training process in steps S201-S205 is explained. For example... Figure 3 As shown, assuming the training audio data includes three training audio tracks (training track A, training track B, and training track C), the training audio data is input into an encoder, then passes through a vector quantization layer. After the vector quantization layer, the corresponding parts of each track are separated, and each part is input into the corresponding decoder for decoding, resulting in different decoded parts (i.e., predicted track A, predicted track B, and predicted track C). The loss of each decoded part is calculated separately from the corresponding part in the training audio data. Finally, the losses are fused to obtain the target loss function, thereby updating the model parameters.

[0071] Thus, through Figure 3 The training architecture shown is comparable to that of... Figure 1 The existing technology shown only requires optimization of the architecture of the decoding and verification parts. Each layer is separated in the decoding step, and the loss of each part is verified separately. The specific algorithms for the encoder, vector quantization layer, decoder, and loss verification step are the same as existing methods. The loss functions for audio tracks A, B, and C are determined based on the corresponding track types and are usually different, especially for vocal and instrumental tracks, where the corresponding loss functions are typically different.

[0072] Example 2: The initial audio tag generation model includes at least two encoders, vector quantization layers, and decoders, with the same number of encoders, vector quantization layers, and decoders. The target audio tag generation model is determined through the following steps:

[0073] S301. Obtain training audio data and the first audio track information of the training audio data.

[0074] S302. Input each training audio track data into the corresponding encoder in the initial audio tag generation model to obtain the second audio vector data output by each encoder.

[0075] S303. Input each second audio vector data into the corresponding vector quantization layer in the initial audio tag generation model to obtain the second audio tag data output by each vector quantization layer.

[0076] S304. Input each second audio tag data into the corresponding decoder in the initial audio tag generation model to obtain the predicted audio track data output by each decoder.

[0077] S305. Combine or fuse the multiple predicted audio track data to obtain predicted audio data;

[0078] S306. Calculate the loss based on the predicted audio data and the training audio data, and update the parameters of each encoder, each vector quantization layer and each decoder in the initial audio tag generation model according to the loss calculation results, until the output of the initial audio tag generation model satisfies the second preset condition, and obtain the target audio tag generation model.

[0079] For step S301, the first audio track information includes at least two training audio track data obtained through audio track separation processing.

[0080] For step S302, this step includes: for each training audio track data of each training audio data, inputting the corresponding training audio track data of the training audio data into the corresponding encoder in the initial audio tag generation model to obtain the second audio vector data of the training audio track data output by the encoder.

[0081] Here, the second audio vector data is a single-level continuous audio vector feature processed by the encoder, and the second audio vector data output by different encoders are different.

[0082] For step S303, this step includes: for the second audio vector data of each training audio data, inputting the second audio vector data of the training audio data into the corresponding vector quantization layer in the initial audio tag generation model, to obtain the second audio tag data of the second audio vector data output by the vector quantization layer.

[0083] Here, the second audio tag data output by different vector quantization layers is different.

[0084] For step S304, this step includes: for the second audio tag data of each training audio data, inputting the second audio tag data of the training audio data into the corresponding decoder in the initial audio tag generation model to obtain the predicted audio track data of the second audio tag data output by the decoder.

[0085] Here, the predicted audio track data output by different decoders are different.

[0086] Regarding step S305, before combining or fusing the multiple predicted audio track data, the multiple predicted audio track data are aligned. After alignment, the multiple predicted audio track data are combined or fused to obtain predicted audio data. The combination or fusion methods may include addition, splicing, weighted addition, transform addition, etc.

[0087] For step S306, this step may include: for each predicted audio data, performing loss calculation based on the difference between the predicted audio data and the corresponding training audio data, determining the target loss function value for this training, then updating the parameters of each encoder, each vector quantization layer, and each decoder in the initial audio tag generation model based on the target loss function value, and determining whether the output of the updated initial audio tag generation model satisfies the second preset condition. If it does, the update is stopped, and the target audio tag generation model is obtained.

[0088] Here, the second preset condition can be that the difference between the predicted audio data output by the updated initial audio tag generation model and the corresponding training audio data is within a preset difference range.

[0089] For an example, please refer to Figure 4 , Figure 4 This is the second schematic diagram illustrating the principle of the training process of a target audio feature generation model provided in this application. Figure 4 The training process in steps S301-S306 is explained. For example... Figure 4As shown, assuming the training audio data includes three training tracks, after separation processing, training tracks A, B, and C can be obtained. Training track A is input into encoder A to obtain audio vector data A; training track B is input into encoder B to obtain audio vector data B; and training track C is input into encoder C to obtain audio vector data C. Audio vector data A is input into vector quantization layer A to obtain audio tag data A; audio vector data B is input into vector quantization layer B to obtain audio tag data B; and audio vector data C is input into vector quantization layer C to obtain audio tag data C. Audio tag data A is input into decoder A to obtain predicted audio track data A; audio tag data B is input into decoder B to obtain predicted audio track data B; and audio tag data C is input into decoder C to obtain predicted audio track data C. The predicted audio track data A, predicted audio track data B, and predicted audio track data C are combined to obtain the predicted audio data. Based on the difference between the predicted audio data and the training audio data, the target loss function value is determined. Finally, the parameters of encoder A, encoder B, encoder C, vector quantization layer A, vector quantization layer B, vector quantization layer C, decoder A, decoder B, and decoder C are updated based on the determined target loss function value.

[0090] When there are multiple training audio data, multiple target loss function values ​​can be obtained, and multiple updates can be performed until the output of the initial audio tag generation model satisfies the second preset condition.

[0091] Example 3: The initial audio tag generation model includes an encoder group, at least two vector quantization layers, and a decoder. The encoder group includes a first encoder and at least two second encoders. The target audio tag generation model is determined through the following steps:

[0092] S401. Obtain training audio data and input the training audio data into the first encoder in the initial audio tag generation model to obtain third audio vector data.

[0093] S402. Input the third audio vector data into each of the second encoders in the initial audio tag generation model to obtain the fourth audio vector data output by each second encoder.

[0094] S403. Input the fourth audio vector data into the corresponding vector quantization layer in the initial audio tag generation model to obtain the third audio tag data output by each vector quantization layer.

[0095] S404. Input the third audio tag data into the corresponding decoder in the initial audio tag generation model to obtain the predicted audio track data output by each decoder.

[0096] S405. Merge or combine the multiple predicted audio track data to obtain predicted audio data.

[0097] S406. Calculate the loss based on the predicted audio data and the training audio data, and update the parameters of the first encoder, the second encoder, the vector quantization layer, and the decoder in the initial audio tag generation model according to the calculated loss value, until the output of the initial audio tag generation model satisfies the third preset condition, thereby obtaining the target audio tag generation model.

[0098] Regarding step S401, the third audio vector data is the continuous audio vector feature corresponding to the training audio data.

[0099] Regarding step S402, the fourth audio vector data represents the single-level continuous audio vector features corresponding to each audio track separated by the second encoder. The fourth audio vector data output by different second encoders are different.

[0100] For step S403, this step includes: for the fourth audio vector data of each training audio data, inputting the fourth audio vector data of the training audio data into the corresponding vector quantization layer in the initial audio tag generation model to obtain the third audio tag data of the fourth audio vector data output by the vector quantization layer.

[0101] For step S404, this step includes: for the third audio tag data of each training audio data, inputting the third audio tag data of the training audio data into the corresponding decoder in the initial audio tag generation model to obtain the predicted audio track data of the third audio tag data output by the decoder.

[0102] The description of S405 can be referred to that of S305, and the same technical effect can be achieved, so it will not be elaborated further.

[0103] Regarding step S406, the third preset condition may be the same as or different from the second preset condition. The description of S406 can be referred to the description of S306, and the processing logic is similar, so it will not be repeated here.

[0104] For an example, please refer to Figure 5 , Figure 5 This is the third schematic diagram illustrating the principle of the training process of a target audio feature generation model provided in this application. Figure 5 The training process in steps S401-S406 is explained. For example... Figure 5As shown, assuming the training audio data includes three training audio tracks, the specific steps in constructing the target audio tag generation model are as follows: The training audio data is input into the first encoder to obtain multi-level continuous audio vector features. Then, based on each track level, these multi-level continuous audio vector features are input into the second encoders A, B, and C respectively, yielding audio vector data A, B, and C. Audio vector data A is input into vector quantization layer A to obtain audio tag data A. Audio vector data B is input into vector quantization layer B to obtain audio tag data B. Audio vector data C is input into vector quantization layer C to obtain audio tag data C. Audio tag data A is input into decoder A to obtain predicted audio track data A. Audio tag data B is input into decoder B to obtain predicted audio track data B. Audio tag data C is input into decoder C to obtain predicted audio track data C. The predicted audio track data A, predicted audio track data B, and predicted audio track data C are combined to obtain the predicted audio data. Based on the difference between the predicted audio data and the training audio data, the target loss function value is determined. Finally, the parameters of the first encoder, second encoder A, second encoder B, second encoder C, vector quantization layer A, vector quantization layer B, vector quantization layer C, decoder A, decoder B, and decoder C are updated based on the determined target loss function value.

[0105] When there are multiple training audio data, multiple target loss function values ​​can be obtained, and multiple updates can be performed until the output of the initial audio tag generation model satisfies the third preset condition.

[0106] Example 4: The initial audio tag generation model includes at least one encoder, multiple vector quantization layers, and multiple decoders. The number of encoders is less than the number of decoders, and the number of encoders and vector quantization layers are the same. The target audio tag generation model is determined through the following steps:

[0107] S501. Obtain training audio data and the second track information of the training audio data.

[0108] S502. Input each training audio track data into the corresponding encoder in the initial audio tag generation model to obtain the fifth audio vector data output by each encoder.

[0109] S503. Input the single-level audio vector data and multi-level audio vector data into the corresponding vector quantization layer respectively to obtain the single-level audio tag data and multi-level audio tag data output by each vector quantization layer.

[0110] S504. Input the single-level audio tag data and the multi-level audio tag data into the corresponding decoders in the initial audio tag generation model to obtain the predicted audio track data output by each decoder.

[0111] S505. Combine or fuse the multiple predicted audio track data to obtain predicted audio data.

[0112] S506. Calculate the loss based on the predicted audio data and the training audio data, and update the parameters of the encoder, vector quantization layer and decoder in the initial audio tag generation model according to the calculated loss value, until the output of the initial audio tag generation model satisfies the fourth preset condition, and obtain the target audio tag generation model.

[0113] For step S501, the second audio track information includes at least two training audio track data obtained through audio track separation processing.

[0114] For step S502, the fifth audio vector data includes at least one single-level continuous audio vector data and at least one multi-level continuous audio vector data.

[0115] Here, if the correlation between at least two training audio tracks satisfies the correlation requirement, then multiple training audio tracks that satisfy the correlation requirement are input into the same encoder, which outputs multi-level continuous audio vector data. Training audio tracks whose correlation does not satisfy the correlation requirement are input into their respective encoders, which output single-level continuous audio vector data. That is, the initial audio tag generation model includes two types of encoders: one that outputs multi-level continuous audio vector data and the other that outputs single-level continuous audio vector data.

[0116] In step S504, the same multi-level audio tag data is input into multiple different decoders, and each decoder outputs different predicted audio track data.

[0117] The description of S505 can be referred to that of S305, and the same technical effect can be achieved, so it will not be elaborated further.

[0118] Regarding step S506, the fourth preset condition may be the same as or different from the second preset condition. The description of S506 can be referred to the description of S306, and the processing logic is similar, so it will not be repeated here.

[0119] For an example, please refer to Figure 6 , Figure 6 This is the fourth schematic diagram illustrating the principle of the training process of a target audio feature generation model provided in this application. Figure 6 The training process in steps S501-S506 is explained below. For example... Figure 6As shown, assuming the training audio data includes three training audio tracks, after separating the training audio data, we can obtain training audio track A, training audio track B, and training audio track C. The correlation between training audio track A and training audio track B meets the correlation requirement. The specific steps in constructing the target audio tag generation model are as follows: Input training audio track A and training audio track B into the same encoder 1 to obtain audio vector data AB; input training audio track C into encoder C to obtain audio vector data C. Input audio vector data AB into vector quantization layer 1 to obtain audio tag data AB; input audio vector data C into vector quantization layer C to obtain audio tag data C. Input audio tag data AB into decoder A to obtain predicted audio track data A; input audio tag data AB into decoder B to obtain predicted audio track data B; input audio tag data C into decoder C to obtain predicted audio track data C. The predicted audio track data A, predicted audio track data B, and predicted audio track data C are combined to obtain the predicted audio data. Based on the difference between the predicted audio data and the training audio data, the target loss function value is determined. Finally, the parameters of encoder 1, encoder C, vector quantization layer 1, vector quantization layer C, decoder A, decoder B, and decoder C are updated based on the determined target loss function value.

[0120] When there are multiple training audio data, multiple target loss function values ​​can be obtained, and multiple updates can be performed until the output of the initial audio tag generation model satisfies the fourth preset condition.

[0121] Regarding the above example, it should be noted that, in one embodiment provided in this application, obtaining the target audio tag generation model includes: after the initial audio tag generation model has been trained, fixing the parameters of each encoder and each vector quantization layer to obtain at least one target encoder and at least one target vector quantization layer, thereby completing the construction of the target audio tag generation model.

[0122] Here, since the ultimate goal is to generate high-quality, multi-level audio tags for subsequent music model training, the target audio tag generation model only needs to include an encoder and a vector quantization layer.

[0123] For the generation process examples of the above four target audio tag generation models, preferably, whether it is separating each audio track after the encoder or directly separating each audio track based on the audio file, the length and speed of each audio track file and the original audio file are the same, and the discrete frame rate of each vector quantization layer is also the same. Therefore, the length and frequency of the audio obtained by each decoder are also the same. In the final addition step, it is only necessary to embed / add each layer frame by frame to achieve fusion or combination.

[0124] Regarding step S103, in one embodiment provided in this application, inputting the audio data to be processed or the multiple audio track data to be processed into a target audio tag generation model to obtain a target audio tag includes: inputting the audio data to be processed or the multiple audio track data to be processed into at least one target encoder and at least one target vector quantization layer to obtain a target audio tag.

[0125] Here, the audio to be processed or the multiple audio tracks to be processed are first input into at least one target encoder, and then the output of the target encoder is input into the corresponding target vector quantization layer. Finally, the target audio tag is obtained based on the output of the target vector quantization layer.

[0126] Furthermore, as can be seen from the above examples, this application provides four different audio tag generation model architectures. Therefore, when obtaining the target audio tag, there are four processing methods.

[0127] Method 1: Input the audio data to be processed into the target audio tag generation model, and the unique encoder of the target audio tag generation model outputs multi-level target continuous vector data; input the multi-level target continuous vector data into the unique vector quantization layer of the target audio tag generation model to obtain multi-level target audio tags.

[0128] Here, multi-level target audio tags are generated, and the target audio tag generation model architecture used is as follows: Figure 3 As shown.

[0129] Method 2: Input each of the audio data to be processed into the corresponding second encoder of the target audio tag generation model to obtain single-level target continuous vector data output by each encoder; input each single-level target continuous vector data into the corresponding vector quantization layer of the target audio tag generation model to obtain single-level target audio tags output by each vector quantization layer.

[0130] Here, multiple single-level target audio tags are generated, and the target audio tag generation model architecture used is as follows: Figure 4 As shown.

[0131] Method 3: The audio track data to be processed is input into the target audio tag generation model, and the first encoder of the target audio tag generation model outputs multi-level target continuous vector data; the multi-level target continuous vector data is input into multiple second encoders of the target audio tag generation model respectively to obtain single-level target continuous vector data output by each second encoder; each single-level target continuous vector data is input into the corresponding vector quantization layer of the target audio tag generation model to obtain single-level target audio tags output by each vector quantization layer.

[0132] Here, multiple single-level target audio tags are generated, and the target audio tag generation model architecture used is as follows: Figure 5 As shown.

[0133] Method 4: Input multiple audio track data that meet the correlation requirements into the same encoder of the target audio tag generation model to obtain multi-level target continuous vector data output by the encoder; input each audio track data that does not meet the correlation requirements into the corresponding encoder in the target audio tag generation model to obtain single-level target continuous vector data output by the encoder; input the multi-level target continuous vector data and the single-level target continuous vector data into the corresponding vector quantization layer in the target audio tag generation model to obtain single-level target audio tags and multi-level target audio tags.

[0134] Here, multiple single-level target audio tags are generated, and the target audio tag generation model architecture used is as follows: Figure 6 As shown.

[0135] Thus, this solution provides multiple methods for generating music tokens required for multi-level music generation. By setting independent decoders and verification steps for the parts that need to be independent, it enhances the effect of each level when actually processing music tokens from different audio tracks, avoiding interference and mixing between different levels. At the same time, the architecture of the designed generation model can also take into account scenarios that require both separation of audio track levels and consideration of correlation. Using music tokens with clear levels to train the music generation model enables the music generation model to generate more complex music, which can further improve the music generation effect.

[0136] Based on the same inventive concept, this application also provides a generation device corresponding to the generation method. Since the principle of the device in this application is similar to the generation method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0137] Please see Figure 7 , Figure 8 , Figure 7 This is a schematic diagram of the structure of an audio tag generation device provided in an embodiment of this application. Figure 8 This is a schematic diagram of another audio tag generation device provided in an embodiment of this application. Figure 7 As shown, the generating apparatus 700 includes:

[0138] The acquisition module 710 is used to acquire the audio track information of the audio data to be processed; wherein, the audio track information includes whether audio track separation processing has been performed, multiple audio track data to be processed after audio track separation processing, and the correlation between each audio track data to be processed;

[0139] The determining module 720 is used to determine a target audio tag generation model from at least one candidate audio tag generation model based on the audio track information and a preset generation model matching rule; wherein the target audio tag generation model generates discrete audio tags based on audio data;

[0140] The generation module 730 is used to input the audio data to be processed or the multiple audio track data to be processed into the target audio tag generation model to obtain the target audio tag.

[0141] Optional, such as Figure 8 As shown, the generation device 700 further includes a model training module 740, which is used to train a target audio tag generation model through the following steps:

[0142] Acquire training audio data and an initial audio tag generation model, wherein the training audio data includes at least two training audio tracks obtained through track separation processing;

[0143] The training audio data and / or training audio track data are input into the encoder in the initial audio tag generation model to obtain the first audio vector data, wherein the first audio vector includes at least two levels of continuous audio vector features.

[0144] The first audio vector data is input into the vector quantization layer in the initial audio tag generation model to obtain the first audio tag feature. The first audio tag feature is a discrete audio feature and includes at least two levels of discrete audio features.

[0145] The first audio tag features are input into each decoder in the initial audio tag generation model to obtain the predicted audio track data output by each decoder;

[0146] Loss is calculated based on multiple predicted audio track data and multiple training audio track data, and the parameters of the encoder, vector quantization layer and each decoder are updated according to the loss function results until the output of the initial audio tag generation model meets the first preset condition, thus obtaining the target audio tag generation model.

[0147] Optionally, when the model training module 740 is used for loss calculation, it is specifically used for:

[0148] For each predicted audio track data, determine the corresponding training audio track data;

[0149] The loss function value at the track level is determined based on the difference between the predicted track data and the corresponding training track data.

[0150] The target loss function value is obtained by fusing the loss function values ​​of each audio track.

[0151] Optionally, the initial audio tag generation model includes at least two encoders, vector quantization layers, and decoders, with the same number of encoders, vector quantization layers, and decoders. The model training module 740 is further used to train the target audio tag generation model through the following steps:

[0152] Acquire training audio data and first audio track information of the training audio data, wherein the first audio track information includes at least two training audio track data obtained by audio track separation processing;

[0153] Each training audio track data is input into the corresponding encoder in the initial audio tag generation model to obtain the second audio vector data output by each encoder. The second audio vector data is a single-level continuous audio vector feature.

[0154] Each second audio vector data is input into the corresponding vector quantization layer in the initial audio tag generation model to obtain the second audio tag data output by each vector quantization layer;

[0155] Each second audio tag data is input into the corresponding decoder in the initial audio tag generation model to obtain the predicted audio track data output by each decoder;

[0156] Combine or fuse the multiple predicted audio track data to obtain predicted audio data;

[0157] Loss is calculated based on the predicted audio data and the training audio data. The parameters of each encoder, each vector quantization layer and each decoder in the initial audio tag generation model are updated based on the loss calculation results until the output of the initial audio tag generation model meets the second preset condition, thus obtaining the target audio tag generation model.

[0158] Optionally, the initial audio tag generation model includes an encoder group, at least two vector quantization layers, and a decoder. The model training module 740 is further used to train the target audio tag generation model through the following steps:

[0159] Acquire training audio data, wherein the encoder group includes a first encoder and at least two second encoders, and input the training audio data into the first encoder in the initial audio tag generation model to obtain third audio vector data, wherein the third audio vector data is a continuous audio vector feature corresponding to the training audio data;

[0160] The third audio vector data is input into each of the second encoders in the initial audio tag generation model to obtain the fourth audio vector data output by each second encoder. The fourth audio vector data represents the single-level continuous audio vector features corresponding to each audio track separated by the second encoder.

[0161] The fourth audio vector data is input into the corresponding vector quantization layer in the initial audio tag generation model to obtain the third audio tag data output by each vector quantization layer;

[0162] The third audio tag data is input into the corresponding decoder in the initial audio tag generation model to obtain the predicted audio track data output by each decoder;

[0163] By fusing or combining the multiple predicted audio track data, predicted audio data is obtained;

[0164] Loss is calculated based on the predicted audio data and the training audio data. The parameters of the first encoder, the second encoder, the vector quantization layer, and the decoder in the initial audio tag generation model are updated based on the calculated loss value until the output of the initial audio tag generation model meets the third preset condition, thus obtaining the target audio tag generation model.

[0165] Optionally, the initial audio tag generation model includes at least one encoder, multiple vector quantization layers, and multiple decoders. The number of encoders is less than the number of decoders, and the number of encoders and vector quantization layers is the same. The model training module 740 is further used to train the target audio tag generation model through the following steps:

[0166] Acquire training audio data and second audio track information of the training audio data, wherein the second audio track information includes at least two training audio track data obtained by audio track separation processing;

[0167] Each training audio track data is input into the corresponding encoder in the initial audio tag generation model to obtain the fifth audio vector data output by each encoder. The fifth audio vector data includes single-level continuous audio vector data and multi-level continuous audio vector data.

[0168] If the correlation between at least two training audio tracks satisfies the correlation requirement, then multiple training audio tracks that satisfy the correlation requirement are input into the same encoder, and the encoder outputs multi-level continuous audio vector data; training audio tracks whose correlation does not satisfy the correlation requirement are input into their respective encoders, and the encoder outputs single-level continuous audio vector data.

[0169] Single-level and multi-level audio vector data are input into the corresponding vector quantization layers to obtain single-level and multi-level audio tag data output by each vector quantization layer.

[0170] Single-level and multi-level audio tag data are input into the corresponding decoders in the initial audio tag generation model to obtain the predicted audio track data output by each decoder; wherein, the same multi-level audio tag data is input into multiple different decoders, and each decoder outputs different predicted audio track data;

[0171] Combine or fuse the multiple predicted audio track data to obtain predicted audio data;

[0172] Loss is calculated based on the predicted audio data and the training audio data. The parameters of the encoder, vector quantization layer and decoder in the initial audio tag generation model are updated according to the loss calculation value until the output of the initial audio tag generation model meets the fourth preset condition, thus obtaining the target audio tag generation model.

[0173] Optionally, the target audio tag generation model includes:

[0174] Once the initial audio tag generation model has been trained, the parameters of each encoder and each vector quantization layer are fixed to obtain at least one target encoder and at least one target vector quantization layer, thus completing the construction of the target audio tag generation model.

[0175] Optionally, the generation module 730 is further configured to:

[0176] The audio to be processed or the multiple audio tracks to be processed are input into at least one target encoder and at least one target vector quantization layer to obtain the target audio tag.

[0177] Optionally, the generation module 730 is further configured to:

[0178] The audio data to be processed is input into the target audio tag generation model, and the unique encoder of the target audio tag generation model outputs multi-level target continuous vector data;

[0179] The multi-level target continuous vector data is input into the unique vector quantization layer of the target audio tag generation model to obtain multi-level target audio tags;

[0180] Alternatively, the audio data to be processed can be input into the target audio tag generation model, and the first encoder of the target audio tag generation model can output multi-level target continuous vector data;

[0181] The multi-level target continuous vector data is input into multiple second encoders of the target audio tag generation model to obtain single-level target continuous vector data output by each second encoder.

[0182] Each single-level target continuous vector data is input into the corresponding vector quantization layer of the target audio tag generation model to obtain the single-level target audio tag output by each vector quantization layer;

[0183] Alternatively, each of the audio tracks to be processed can be input into the corresponding second encoder of the target audio tag generation model to obtain single-level target continuous vector data output by each encoder;

[0184] Each single-level target continuous vector data is input into the corresponding vector quantization layer of the target audio tag generation model to obtain the single-level target audio tag output by each vector quantization layer;

[0185] Alternatively, multiple audio tracks that meet the correlation requirements can be input into the same encoder of the target audio tag generation model to obtain multi-level target continuous vector data output by the encoder.

[0186] Each audio track data that does not meet the correlation requirement is input into the corresponding encoder in the target audio tag generation model to obtain the single-level target continuous vector data output by the encoder.

[0187] The multi-level target continuous vector data and the single-level target continuous vector data are respectively input into the corresponding vector quantization layer in the target audio tag generation model to obtain single-level target audio tags and multi-level target audio tags.

[0188] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 900 includes a processor 910, a memory 920, and a bus 930.

[0189] The memory 920 stores machine-readable instructions executable by the processor 910. When the electronic device 900 is running, the processor 910 and the memory 920 communicate via the bus 930. When the machine-readable instructions are executed by the processor 910, they can perform the operations described above. Figures 2 to 6 The steps in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0190] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figures 2 to 6 The steps in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0191] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0195] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating audio tags, characterized in that, The generation method includes: Obtain the audio track information of the audio data to be processed; wherein, the audio track information includes whether audio track separation processing has been performed, multiple audio track data to be processed after audio track separation processing, and the correlation between each audio track data to be processed; Based on the audio track information and preset generation model matching rules, a target audio tag generation model is determined from at least one candidate audio tag generation model; wherein, the target audio tag generation model generates discrete audio tags based on audio data; The candidate audio tag generation model includes: a first audio tag generation model, a second audio tag generation model, a third audio tag generation model, and a fourth audio tag generation model; wherein, the first audio tag generation model includes an encoder and a vector quantization layer; the second audio tag generation model includes multiple encoders and multiple vector quantization layers, with the number of encoders and vector quantization layers being the same; the third audio tag generation model includes an encoder group and multiple target vector quantization layers, wherein the encoder group includes a first encoder and multiple second encoders, with the number of second encoders and vector quantization layers being the same; and the fourth audio tag generation model includes multiple encoders and multiple vector quantization layers, with the number of encoders being less than the number of decoders. The step of determining the target audio tag generation model from at least one candidate audio tag generation model includes: when the audio track information is audio data to be processed that has not undergone audio track separation processing, determining either the first audio tag generation model or the third audio tag generation model as the target audio tag generation model; when the audio track information is audio data to be processed that has undergone audio track separation processing and includes multiple audio track data to be processed, determining the second audio tag generation model as the target audio tag generation model; when the audio track information is audio data to be processed that has undergone audio track separation processing and includes the first audio track data to be processed, the second audio track data to be processed, and the third audio track data to be processed, and the first audio track data to be processed and the second audio track data to be processed have a strong correlation, determining the fourth audio tag generation model as the target audio tag generation model. The audio data to be processed or the multiple audio tracks to be processed are input into the target audio tag generation model to obtain the target audio tag.

2. The generation method according to claim 1, characterized in that, When the target audio tag generation model is the first audio tag generation model, the target audio tag generation model is trained through the following steps: Acquire training audio data and an initial audio tag generation model. The training audio data includes at least two training audio tracks obtained through track separation processing. The initial audio tag generation model includes an encoder, a vector quantization layer, and multiple decoders. The training audio data and / or training audio track data are input into the encoder in the initial audio tag generation model to obtain the first audio vector data, wherein the first audio vector includes at least two levels of continuous audio vector features. The first audio vector data is input into the vector quantization layer in the initial audio tag generation model to obtain the first audio tag feature. The first audio tag feature is a discrete audio feature and includes at least two levels of discrete audio features. The first audio tag features are input into each decoder in the initial audio tag generation model to obtain the predicted audio track data output by each decoder; Loss is calculated based on multiple predicted audio track data and multiple training audio track data, and the parameters of the encoder, vector quantization layer and each decoder are updated according to the loss function results until the output of the initial audio tag generation model meets the first preset condition, thus obtaining the target audio tag generation model.

3. The generation method according to claim 2, characterized in that, Loss calculation includes: For each predicted audio track data, determine the corresponding training audio track data; The loss function value at the track level is determined based on the difference between the predicted track data and the corresponding training track data. The target loss function value is obtained by fusing the loss function values ​​of each audio track.

4. The generation method according to claim 1, characterized in that, The initial audio tag generation model includes at least two encoders, vector quantization layers, and decoders, with the same number of encoders, vector quantization layers, and decoders. When the target audio tag generation model is the second audio tag generation model, the target audio tag generation model is determined through the following steps: Acquire training audio data and first audio track information of the training audio data, wherein the first audio track information includes at least two training audio track data obtained by audio track separation processing; Each training audio track data is input into the corresponding encoder in the initial audio tag generation model to obtain the second audio vector data output by each encoder. The second audio vector data is a single-level continuous audio vector feature. Each second audio vector data is input into the corresponding vector quantization layer in the initial audio tag generation model to obtain the second audio tag data output by each vector quantization layer; Each second audio tag data is input into the corresponding decoder in the initial audio tag generation model to obtain the predicted audio track data output by each decoder; Combine or fuse the multiple predicted audio track data to obtain predicted audio data; Loss is calculated based on the predicted audio data and the training audio data. The parameters of each encoder, each vector quantization layer, and each decoder in the initial audio tag generation model are updated based on the loss calculation results until the output of the initial audio tag generation model satisfies the second preset condition, thus obtaining the target audio tag generation model.

5. The generation method according to claim 1, characterized in that, The initial audio tag generation model includes an encoder group, at least two vector quantization layers, and a decoder. The encoder group includes a first encoder and at least two second encoders. When the target audio tag generation model is a third audio tag generation model, the target audio tag generation model is determined through the following steps: Acquire training audio data and input the training audio data into the first encoder in the initial audio tag generation model to obtain third audio vector data, wherein the third audio vector data is the continuous audio vector feature corresponding to the training audio data; The third audio vector data is input into each of the second encoders in the initial audio tag generation model to obtain the fourth audio vector data output by each second encoder. The fourth audio vector data represents the single-level continuous audio vector features corresponding to each audio track separated by the second encoder. The fourth audio vector data is input into the corresponding vector quantization layer in the initial audio tag generation model to obtain the third audio tag data output by each vector quantization layer; The third audio tag data is input into the corresponding decoder in the initial audio tag generation model to obtain the predicted audio track data output by each decoder; By fusing or combining the multiple predicted audio track data, predicted audio data is obtained; Loss is calculated based on the predicted audio data and the training audio data. The parameters of the first encoder, the second encoder, the vector quantization layer, and the decoder in the initial audio tag generation model are updated based on the calculated loss value until the output of the initial audio tag generation model meets the third preset condition, thus obtaining the target audio tag generation model.

6. The generation method according to claim 1, characterized in that, The initial audio tag generation model includes multiple encoders, multiple vector quantization layers, and multiple decoders. The number of encoders is less than the number of decoders, and the number of encoders and vector quantization layers are the same. When the target audio tag generation model is the fourth audio tag generation model, the target audio tag generation model is determined through the following steps: Acquire training audio data and second audio track information of the training audio data, wherein the second audio track information includes at least two training audio track data obtained by audio track separation processing; Each training audio track data is input into the corresponding encoder in the initial audio tag generation model to obtain the fifth audio vector data output by each encoder. The fifth audio vector data includes single-level continuous audio vector data and multi-level continuous audio vector data. If the correlation between at least two training audio tracks satisfies the correlation requirement, then multiple training audio tracks that satisfy the correlation requirement are input into the same encoder, and the encoder outputs multi-level continuous audio vector data; training audio tracks whose correlation does not satisfy the correlation requirement are input into their respective encoders, and the encoder outputs single-level continuous audio vector data. Single-level and multi-level audio vector data are input into the corresponding vector quantization layers to obtain single-level and multi-level audio tag data output by each vector quantization layer. Single-level and multi-level audio tag data are respectively input into the corresponding decoders in the initial audio tag generation model to obtain the predicted audio track data output by each decoder; wherein, the same multi-level audio tag data is input into multiple different decoders, and each decoder outputs different predicted audio track data; Combine or fuse the multiple predicted audio track data to obtain predicted audio data; Loss is calculated based on the predicted audio data and the training audio data. The parameters of the encoder, vector quantization layer and decoder in the initial audio tag generation model are updated based on the calculated loss value until the output of the initial audio tag generation model meets the fourth preset condition, thus obtaining the target audio tag generation model.

7. The generation method according to any one of claims 2, 4-6, characterized in that, The target audio tag generation model includes: Once the initial audio tag generation model has been trained, the parameters of each encoder and each vector quantization layer are fixed to obtain at least one target encoder and at least one target vector quantization layer, thus completing the construction of the target audio tag generation model.

8. The generation method according to claim 7, characterized in that, The audio data to be processed or the multiple audio tracks to be processed are input into the target audio tag generation model to obtain target audio tags, including: The audio to be processed or the multiple audio tracks to be processed are input into at least one target encoder and at least one target vector quantization layer to obtain the target audio tag.

9. The generation method according to claim 8, characterized in that, The process of obtaining the target audio tag includes: The audio data to be processed is input into the target audio tag generation model, and the unique encoder of the target audio tag generation model outputs multi-level target continuous vector data; The multi-level target continuous vector data is input into the unique vector quantization layer of the target audio tag generation model to obtain multi-level target audio tags; Alternatively, the audio data to be processed can be input into the target audio tag generation model, and the first encoder of the target audio tag generation model can output multi-level target continuous vector data; The multi-level target continuous vector data is input into multiple second encoders of the target audio tag generation model to obtain single-level target continuous vector data output by each second encoder. Each single-level target continuous vector data is input into the corresponding vector quantization layer of the target audio tag generation model to obtain the single-level target audio tag output by each vector quantization layer; Alternatively, each of the audio tracks to be processed can be input into the corresponding second encoder of the target audio tag generation model to obtain single-level target continuous vector data output by each encoder; Each single-level target continuous vector data is input into the corresponding vector quantization layer of the target audio tag generation model to obtain the single-level target audio tag output by each vector quantization layer; Alternatively, multiple audio tracks that meet the correlation requirements can be input into the same encoder of the target audio tag generation model to obtain multi-level target continuous vector data output by the encoder. Each audio track data that does not meet the correlation requirement is input into the corresponding encoder in the target audio tag generation model to obtain the single-level target continuous vector data output by the encoder. The multi-level target continuous vector data and the single-level target continuous vector data are respectively input into the corresponding vector quantization layer in the target audio tag generation model to obtain single-level target audio tags and multi-level target audio tags.

10. An audio tag generation apparatus, characterized in that, The generating apparatus includes: The acquisition module is used to acquire the audio track information of the audio data to be processed; wherein, the audio track information includes whether audio track separation processing has been performed, multiple audio track data to be processed after audio track separation processing, and the correlation between each audio track data to be processed; The determining module is used to determine a target audio tag generation model from at least one candidate audio tag generation model based on the audio track information and a preset generation model matching rule; wherein the target audio tag generation model generates discrete audio tags based on audio data; The candidate audio tag generation model includes: a first audio tag generation model, a second audio tag generation model, a third audio tag generation model, and a fourth audio tag generation model; wherein, the first audio tag generation model includes an encoder and a vector quantization layer; the second audio tag generation model includes multiple encoders and multiple vector quantization layers, with the number of encoders and vector quantization layers being the same; the third audio tag generation model includes an encoder group and multiple target vector quantization layers, wherein the encoder group includes a first encoder and multiple second encoders, with the number of second encoders and vector quantization layers being the same; and the fourth audio tag generation model includes multiple encoders and multiple vector quantization layers, with the number of encoders being less than the number of decoders. When determining a target audio tag generation model from at least one candidate audio tag generation model, the determining module is configured to: determine either the first or third audio tag generation model as the target audio tag generation model when the audio track information is audio data to be processed that has not undergone audio track separation processing; determine the second audio tag generation model as the target audio tag generation model when the audio track information is audio data to be processed that has undergone audio track separation processing and includes multiple audio track data to be processed; and determine the fourth audio tag generation model as the target audio tag generation model when the audio track information is audio data to be processed that has undergone audio track separation processing and includes the first, second, and third audio track data to be processed, and the first and second audio track data to be processed have a strong correlation. The generation module is used to input the audio data to be processed or the multiple audio tracks to be processed into the target audio tag generation model to obtain the target audio tag.

Images