Encoder training method, audio generation method, and audio retrieval method

Abstract

The application provides an encoder training method, an audio generation method and an audio retrieval method, and relates to the technical field of audio processing. The encoder training method comprises: adjusting an audio quantization encoder to obtain a hierarchical encoder; wherein the hierarchical encoder is provided with an associated semantic codebook and an audio codebook; and jointly training the hierarchical encoder and a text encoder based on a joint loss to obtain an optimized encoder; wherein the hierarchical encoder is used for quantization compression processing of audio data to obtain fusion features, the fusion features comprising semantic features and audio features; the text encoder is used for processing text information associated with the audio data to obtain a text vector; and the fusion features and the text vector have relevance. The audio generation method comprises: determining a generation requirement; wherein the generation requirement comprises a semantic requirement and an audio requirement; and processing based on the generation requirement by using the optimized encoder to obtain requirement audio data.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and more specifically, to an encoder training method, an audio generation method, and an audio retrieval method. Background Technology

[0002] To generate audio, compose music, and compress audio, audio discretization methods based on VQ-VAE (Vector Quantization Variational Autoencoder) are commonly used. The encoder maps the audio waveform or spectrum into continuous latent vectors using a convolutional neural network. A predefined or end-to-end learned codebook replaces the continuous latent vectors with the most recent discrete entries in the codebook. Finally, the decoder reconstructs the original audio based on the discrete codebook entries.

[0003] However, current audio discretization methods lack the ability to align codebook entries with semantic information such as text and labels because they are learned solely through reconstruction loss. This results in discrete tokens being unable to express high-level semantics. Furthermore, codebook entries are abstract symbols learned without supervision, making it difficult to establish a mapping with human-understandable semantic units. The audio discretization results are unrelated to modalities such as text and images, and cannot support cross-modal retrieval or generation. Consequently, the current audio processing performance is poor and cannot meet the needs of various application scenarios. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an encoder training method, an audio generation method, and an audio retrieval method to improve the problem of poor audio data processing effect in the prior art.

[0005] To address the aforementioned problems, in a first aspect, embodiments of this application provide an encoder training method, the method comprising:

[0006] The audio quantization encoder is adjusted to obtain a layered encoder; wherein, the layered encoder is equipped with an associated semantic codebook and an audio codebook;

[0007] The hierarchical encoder and the text encoder are jointly trained based on the joint loss to obtain an optimized encoder;

[0008] The layered encoder is used to quantize and compress audio data to obtain fusion features, which include semantic features and audio features; the text encoder is used to process the text information associated with the audio data to obtain a text vector; the fusion features are related to the text vector.

[0009] In the above implementation process, to enable the encoder to understand and represent the corresponding semantic meaning, the original audio quantization encoder, which can only quantize audio features, can be adjusted to obtain a hierarchical encoder with associated semantic and audio codebooks, capable of simultaneously expressing low-level acoustic features and high-level semantic information. This hierarchical encoder then performs quantization compression on the audio data to obtain fused features containing both semantic and audio features. Furthermore, considering the high correlation between text information and audio data in various application scenarios, the text encoder can process the text information associated with the audio data to obtain text vectors. That is, there is a correlation between the text vectors and the fused features. To improve the tightness and effectiveness of the correlation between the text vectors and the fused features, a joint loss can be set. Based on the joint loss, the hierarchical encoder and the text encoder are jointly trained to obtain an optimized encoder that understands semantics and can process audio data based on semantics. By designing hierarchical semantic and audio codebooks and jointly training multiple encoders, the performance of the optimized encoder in processing audio data based on semantic information is effectively improved, meeting the needs of various application scenarios.

[0010] Optionally, adjusting the audio quantization encoder to obtain a layered encoder includes:

[0011] Determine the semantic codebook for quantizing the semantic information of the audio data;

[0012] Determine the audio codebook associated with the semantic codebook for quantizing the audio data;

[0013] Based on the semantic codebook and the audio codebook, the audio quantization encoder is adjusted to obtain the layered encoder.

[0014] In the above implementation process, when adjusting the audio quantization encoder, a semantic codebook for quantizing semantic information related to the audio data and an audio codebook for quantizing audio data that is related to the semantic codebook can be determined first. Then, based on the semantic codebook and the audio codebook, the audio quantization encoder is adjusted to obtain the corresponding hierarchical encoder. Different levels of related semantic codebooks and audio codebooks can be set so that the resulting hierarchical encoder can not only quantize audio but also quantize related semantic information, thereby improving the semantic understanding capability of the hierarchical encoder during quantization compression processing.

[0015] Optionally, the process of semantically quantizing the audio data in the hierarchical encoder includes:

[0016] ;

[0017] in, The semantic weight matrix is ​​used, and Attention is a multi-head attention mechanism. For the semantic features, For the audio data, For semantic feature quantization algorithms;

[0018] In the layered encoder, the process of audio quantization of the audio data includes:

[0019] ;

[0020] Where α is the hierarchical fusion coefficient, and Upsample is the upsampling operation. For the aforementioned audio features, For audio feature quantization algorithms;

[0021] The fusion feature output by the hierarchical encoder is:

[0022] ;

[0023] in, β represents the fusion feature, and β is the fusion coefficient.

[0024] In the above implementation process, a semantic weight matrix and a multi-head attention mechanism can be introduced. Through the interaction of the semantic weight matrix and the multi-head attention mechanism, semantic information can be quantized to obtain corresponding semantic features. Based on the obtained semantic features, hierarchical fusion coefficients and upsampling operations can be combined. Upsampling operations fuse high-level semantic features into low-level features to obtain corresponding audio features, thereby quantizing and representing the details of the audio. The audio features and semantic features are synthesized using fusion coefficients to create the final fused features, which serve as the output of the hierarchical encoder. This allows the hierarchical encoder to simultaneously output high-level semantics and low-level audio details, effectively improving the semantic understanding capability of the hierarchical encoder during quantization compression processing.

[0025] Optionally, the step of jointly training the hierarchical encoder and the text encoder based on joint loss to obtain an optimized encoder includes:

[0026] Determine the joint loss associated with the hierarchical encoder and the text encoder;

[0027] Based on the joint loss, the hierarchical encoder and the text encoder are jointly trained to obtain the optimized encoder.

[0028] In the above implementation process, in order to enable the hierarchical encoder and text encoder to output associated encoding results, the joint loss of the hierarchical encoder and text encoder can be determined. Based on the joint loss, the hierarchical encoder and text encoder are jointly trained so that the optimized encoder obtained by the final training can not only ensure the high-fidelity reconstruction of the audio signal, but also improve the semantic feature sequence in the output fusion feature to conform to semantic logic and temporal dependency relationship. This upgrades the audio discretization process from the existing "signal reconstruction" to "semantic drive", providing core technical support for application scenarios such as intelligent voice interaction and cross-modal content generation.

[0029] Optionally, determining the joint loss associated with the hierarchical encoder and the text encoder includes:

[0030] Determine the composite loss of the hierarchical encoder;

[0031] Determine the language loss of the language model in the text encoder;

[0032] The joint loss is determined based on the composite loss, the language loss, and the weight balancing coefficient.

[0033] In the above implementation process, considering the different types of losses in different encoders during training, the composite loss in the hierarchical encoder and the language loss of the language model in the text encoder can be determined. The final joint loss is then determined based on the composite loss, the language loss, and weight balancing coefficients adjusted for different types of losses. This approach, combining multiple losses representing different time periods in different encoders to determine the joint loss, effectively improves the effectiveness of joint training based on the joint loss, thereby enhancing the encoder's performance in processing audio data.

[0034] Optionally, the joint loss includes:

[0035] ;

[0036] in, For the aforementioned joint loss, For the composite loss, The weighting balance coefficient is... For the language loss;

[0037] ;

[0038] in, For audio reconstruction loss, The coefficient of variation, To constrain the difference loss between continuous coding features and discrete codebooks, For comparison coefficients, The contrast loss is used to semantically align the audio data with the text information;

[0039] ;

[0040] in, Let t be the semantic token in the semantic features. The length of the token sequence. The conditional probability distribution output by the language model.

[0041] In the above implementation process, the composite loss of the hierarchical encoder can be determined based on the audio reconstruction loss, difference coefficient, difference loss between constrained continuous coding features and discrete codebook, contrast coefficient, and contrast loss for semantic alignment of audio data and text information in the hierarchical encoder. The language loss of the language model can also be determined based on multiple semantic tokens in the semantic features, token sequence length, and conditional probability distribution of the language model output. The joint loss used during training can be determined by combining the composite loss, weight balance coefficient, and language loss.

[0042] Optionally, the contrast loss includes:

[0043] ;

[0044] in, The semantic feature of the i-th audio sample in the audio data. The text vector associated with the i-th audio sample. The text vector is the text vector of all text samples in the text information. For similarity calculation, This refers to the temperature parameter.

[0045] In the above implementation process, to improve the consistency between semantic features and audio features, the distance between the audio and text representations can be expressed based on contrastive loss. This enables the optimized encoding model to generate audio that matches the text description or to determine a text description that matches the audio. Therefore, the corresponding contrastive loss can be determined based on semantic features, text vectors, temperature parameters, and a similarity calculation method.

[0046] Secondly, embodiments of this application also provide an audio generation method, the method comprising:

[0047] Determine the generation requirements; wherein, the generation requirements include semantic requirements and / or audio requirements;

[0048] By optimizing the encoder and processing it based on the generated requirements, the required audio data is obtained; wherein the optimized encoder is trained by any of the methods described in the first aspect above.

[0049] In the above implementation process, in audio generation application scenarios, the actual generation requirements can be determined first. These requirements may include corresponding semantic and / or audio requirements. The optimization encoder processes these requirements and outputs the audio data that meets them as the generation result. The optimization encoder's ability to understand and process the semantic and audio information in the generation requirements effectively improves the accuracy of the required audio data and meets various audio generation needs of users.

[0050] Optionally, the step of optimizing the encoder and processing the generated audio data based on the requirements to obtain the required audio data includes:

[0051] If the generated requirement is the semantic requirement, then by optimizing the encoder, a requirement text vector is determined based on the semantic requirement; based on the requirement text vector, an associated first requirement fusion feature is determined; and based on the first requirement fusion feature, the requirement audio data is generated.

[0052] If the generated requirement is the audio requirement, then the optimized encoder determines the input fusion feature based on the audio requirement; determines the second requirement fusion feature associated with the input fusion feature; and generates the requirement audio data based on the second requirement fusion feature.

[0053] In the above implementation process, when the generated requirement is a semantic requirement, the encoder can be optimized to determine the corresponding requirement text vector based on the semantic requirement, and then determine the first requirement fusion feature associated with it based on the requirement text vector, so as to generate the corresponding requirement audio data according to the first requirement fusion feature. When the generated requirement is an audio requirement, the encoder can be optimized to determine the input fusion feature based on the audio requirement, and then determine the second requirement fusion feature associated with the input fusion feature, that is, determine the second requirement feature that has musical relevance to the input fusion feature, so as to generate the corresponding requirement audio data according to the second requirement fusion feature. The optimized encoder can perform different processing based on different types of requirements to generate requirement audio data that meets the corresponding requirements.

[0054] Thirdly, embodiments of this application also provide an audio retrieval method, the method comprising:

[0055] By optimizing the encoder, retrieval fusion features are determined according to retrieval requirements; wherein the optimized encoder is trained by any one of the methods described in the first aspect above;

[0056] The optimized encoder determines similar fusion features whose similarity to the retrieved fusion features is greater than or equal to a similarity threshold.

[0057] The target audio is generated using the optimized encoder based on the similarity fusion features.

[0058] In the above implementation process, in audio retrieval applications, the encoder can be optimized to determine corresponding retrieval fusion features based on various types of input retrieval needs. Then, similarity analysis is performed on these fusion features to identify similar fusion features with a similarity greater than or equal to a preset similarity threshold. Based on these similar fusion features, the corresponding target audio is generated as the audio retrieval result. The ability of the optimized encoder to determine similar fusion features based on retrieval needs effectively improves the correspondence between target audio and retrieval requirements, satisfying diverse audio retrieval needs of users.

[0059] Fourthly, embodiments of this application also provide an electronic device, which includes a memory and a processor. The memory stores program instructions, and when the processor reads and runs the program instructions, it executes the steps in any of the above-described implementations of the encoder training method, audio generation method, or audio retrieval method.

[0060] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, it implements the steps of any one of the encoder training method, audio generation method, or audio retrieval method described above.

[0061] In summary, the embodiments of this application provide an encoder training method, an audio generation method, and an audio retrieval method. By designing a semantic codebook and an audio codebook in a hierarchical manner, and by jointly training multiple encoders, the method effectively improves the effect of optimizing the encoder to process audio data based on semantic information, and meets the usage requirements of various application scenarios. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 A block diagram illustrating an electronic device provided in an embodiment of this application;

[0064] Figure 2 A flowchart illustrating an encoder training method provided in an embodiment of this application;

[0065] Figure 3 A detailed flowchart of step S210 provided for an embodiment of this application;

[0066] Figure 4 A detailed flowchart of step S220 provided for an embodiment of this application;

[0067] Figure 5 A detailed flowchart of step S221 provided for an embodiment of this application;

[0068] Figure 6 A flowchart illustrating an audio generation method provided in an embodiment of this application;

[0069] Figure 7 A detailed flowchart of step S320 provided for an embodiment of this application;

[0070] Figure 8 This is a flowchart illustrating an audio retrieval method provided in an embodiment of this application.

[0071] Icons: 100 - Electronic device; 111 - Memory; 112 - Memory controller; 113 - Processor; 114 - Peripheral interface; 115 - Input / output unit; 116 - Display unit. Detailed Implementation

[0072] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0073] Current audio discretization methods suffer from several drawbacks. First, codebook entries learn solely through reconstruction loss, lacking alignment with semantic information such as text and tags. This results in discretized tokens failing to express high-level semantics, such as "phonemes," "emotions," or "instrument types" in audio. Second, codebook entries are abstract symbols learned through unsupervised learning, making it difficult to map them to human-understandable semantic units, such as lyrics or scene descriptions. Third, audio discretization results are unrelated to text or image modalities, hindering cross-modal retrieval and generation, such as the inability to generate corresponding audio based on text descriptions. Therefore, existing audio discretization methods suffer from semantic gaps, poor interpretability, and cross-modal fragmentation, leading to suboptimal audio processing performance and failing to meet the needs of various application scenarios.

[0074] To address the aforementioned issues, this application provides an encoder training method applicable to electronic devices, such as servers, personal computers (PCs), and cloud devices with logical computing capabilities. The electronic devices are equipped with various types of encoders. By designing semantic and audio codebooks in a layered manner and jointly training multiple encoders, the method effectively improves the optimization of the encoder's processing of audio data based on semantic information, meeting the needs of various application scenarios.

[0075] Optionally, please refer to Figure 1 , Figure 1 This is a block diagram illustrating an electronic device according to an embodiment of this application. The electronic device 100 may include a memory 111, a memory controller 112, a processor 113, a peripheral interface 114, an input / output unit 115, and a display unit 116. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device 100. For example, the electronic device 100 may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0076] The aforementioned memory 111, memory controller 112, processor 113, peripheral interface 114, input / output unit 115, and display unit 116 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The aforementioned processor 113 is used to execute executable modules stored in the memory.

[0077] The memory 111 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 111 stores programs. After receiving execution instructions, the processor 113 executes the programs. The methods executed by the electronic device 100 as defined in any embodiment of this application can be applied to the processor 113, or implemented by the processor 113.

[0078] The aforementioned processor 113 may be an integrated circuit chip with signal processing capabilities. The processor 113 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0079] The peripheral interface 114 described above couples various input / output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113, and the memory controller 112 may be implemented on a single chip. In other instances, they may be implemented on separate chips.

[0080] The input / output unit 115 described above is used to provide user input data. The input / output unit 115 can be, but is not limited to, a mouse and keyboard, etc.

[0081] The aforementioned display unit 116 provides an interactive interface (e.g., a user interface) between the electronic device 100 and the user, or displays image data for the user's reference. In this embodiment, the display unit can be a liquid crystal display (LCD) or a touch display. If it is a touch display, it can be a capacitive touchscreen or a resistive touchscreen that supports single-point and multi-point touch operations. Supporting single-point and multi-point touch operations means that the touch display can sense touch operations generated simultaneously from one or more locations on the touch display and pass the sensed touch operations to the processor for calculation and processing. In this embodiment, the display unit 116 can display various types of information such as text information and audio sheet music.

[0082] The electronic device in this embodiment can be used to execute various steps in the encoder training methods provided in the embodiments of this application. The implementation process of the encoder training method is described in detail below through several embodiments.

[0083] Please see Figure 2 , Figure 2 This is a flowchart illustrating an encoder training method provided in an embodiment of this application. The method may include steps S210-S220.

[0084] Step S210: Adjust the audio quantization encoder to obtain a layered encoder.

[0085] The hierarchical encoder includes associated semantic and audio codebooks. To enable the encoder to understand and represent the corresponding semantic meanings, the existing audio quantization encoder, which can only quantize audio features, can be modified to create a hierarchical encoder with associated semantic and audio codebooks, capable of simultaneously expressing low-level acoustic features and high-level semantic information. This allows for the quantization and compression of audio data using the hierarchical encoder, resulting in a fused feature that includes both semantic and audio features.

[0086] Optionally, the audio quantization encoder can be a model of various types such as VQ-VAE (vector quantization variational autoencoder). For example, a Semantic-VQVAE model can be selected as the audio quantization encoder. During processing, the audio waveform or spectrum of the audio signal of the audio data can be mapped into continuous latent vectors. For example, 32 one-dimensional vectors can be transformed into 1024 vectors. Then, according to the set codebook, multiple vectors are compressed into a small number of discrete entries in the codebook, thereby realizing quantization compression processing.

[0087] It should be noted that, in actual use, the semantic codebook and audio codebook can also be updated in real time.

[0088] Step S220: Jointly train the hierarchical encoder and the text encoder based on the joint loss to obtain the optimized encoder.

[0089] Considering the high correlation between text information and audio data in various application scenarios, the text encoder can process the text information associated with the audio data to obtain text vectors. That is, there is a correlation between the text vectors and the fused features. In order to improve the tightness and effectiveness of the correlation between the text vectors and the fused features, a corresponding joint loss can be set. Based on the joint loss, the hierarchical encoder and the text encoder are jointly trained to obtain an optimized encoder that can understand semantics and process audio data based on semantics.

[0090] It should be noted that the layered encoder is used to perform quantization compression processing on audio data. Since the layered encoder has codebooks at different levels, the resulting fusion features can include semantic features and audio features. Semantic features are data obtained by extracting features from the semantic information contained in the audio data, and audio features are data obtained by extracting features from the audio signal of the audio data. The text encoder is used to process the text information associated with the audio data to obtain text vectors. For example, the audio description content in the text information can be converted into corresponding word vectors as text vectors. Furthermore, since the audio data and text information are related, the fusion features obtained after processing are also related to the text vectors.

[0091] For example, a text encoder can include models that can process textual information, such as language models, RNN (Recurrent Neural Network) / LSTM (Long Short-Term Memory Network), CNN (Convolutional Neural Network), and other types of models.

[0092] Optionally, the text information can be tag information of associated text classes output along with the audio data, or text information extracted from associated voice information, image information, etc., output along with the audio data, enabling audio generation, retrieval, and other processing based on data of different modalities.

[0093] Optionally, the audio data and associated text information can be highly correlated data selected from historical data. For example, if the audio data is a soothing piano piece, the corresponding text information could be content that accurately describes the audio data, such as "piano, soothing, instrumental music." Multiple sets of associated audio data and text information can be divided into corresponding training and validation sets to iteratively train the hierarchical encoder and text encoder, further optimizing the performance of the final optimized encoder.

[0094] exist Figure 2In the illustrated embodiment, by designing a hierarchical semantic codebook and an audio codebook, and by jointly training multiple encoders, the effect of optimizing the encoder to process audio data based on semantic information is effectively improved, meeting the usage requirements of various application scenarios.

[0095] Optionally, please refer to Figure 3 , Figure 3 The following is a detailed flowchart of step S210 provided in an embodiment of this application. Step S210 may include steps S211-S213.

[0096] Step S211: Determine the semantic codebook for quantizing the semantic information of the audio data.

[0097] Step S212: Determine the audio codebook associated with the semantic codebook that quantizes the audio data.

[0098] When adjusting the audio quantization encoder, one can first determine the semantic codebook that quantizes the semantic information related to the audio data, and the audio codebook that quantizes the audio signals in the audio data and is related to the semantic codebook.

[0099] Optionally, the semantic codebook may include high-level semantic information from the audio data, such as phonemes, musical segments, and emotion tags. For example, it may encode information such as instrument type (piano, violin, drums), musical segment structure (intro, verse, chorus), key information (C major, A minor), and rhythmic pattern (4 / 4 time, waltz). The semantic codebook contains a small number of entries (e.g., 256), each corresponding to an understandable semantic concept and aligned with the text vector.

[0100] Optionally, the audio codebook may include low-level features of the audio data, such as timbre and spectrum. For example, in piano performance audio, the audio codebook is responsible for encoding detailed information such as the specific timbre (bright or warm), spectral energy distribution (high-frequency overtones), and amplitude envelope of the time-domain waveform. The audio codebook contains a large number of codebook entries (e.g., 1024), and each codebook entry corresponds to a specific acoustic feature vector, maintaining a high level of sound quality reconstruction capability.

[0101] It should be noted that the number of semantic codebooks and audio codebooks can be set according to the actual situation and needs. In scenarios where high audio accuracy is required, multiple semantic codebooks and audio codebooks can be set, for example, 5 semantic codebooks and 5 audio codebooks can be set respectively to further optimize the processing effect of the hierarchical encoder.

[0102] Existing audio quantization encoders process audio signals using only a single audio codebook, making it difficult to simultaneously express low-level acoustic features and high-level semantic information. The audio codebook defined in this application can be denoted as a Low-Level Codebook, and the semantic codebook as a High-Level Semantic Codebook. Furthermore, the layered codebook of this application reduces redundancy while preserving details. The high-level semantic codebook is used for abstract semantic modeling, while the low-level audio codebook only encodes local acoustic differences, avoiding repeated encoding of the same semantic meaning. This effectively improves codebook utilization and quantization efficiency, significantly increasing compression ratio compared to a single codebook while maintaining the same reconstruction quality. Moreover, by pre-associating semantic tags in the semantic codebook, high-precision semantic control (such as dialect speech synthesis) can be achieved with limited labeled data.

[0103] Step S213: Based on the semantic codebook and the audio codebook, adjust the audio quantization encoder to obtain the layered encoder.

[0104] Among them, the model algorithm inside the audio quantization encoder can be adjusted according to the semantic codebook and the audio codebook to obtain the corresponding layered encoder.

[0105] Optionally, let the input audio data be... , representing the time-frequency feature matrix extracted from the audio signal of the audio data. Audio data can be acquired through microphones, recording devices, audio files, or network audio streams, and converted into time-frequency features using methods such as Short-Time Fourier Transform (STFT) and Mel-frequency cepstral coefficients (MFCC). The corresponding text information (such as "piano performance" or "cheerful music") can be encoded into a text vector using a corresponding text encoder, denoted as . The audio features Z are simultaneously input into the overall model to achieve semantic alignment training.

[0106] Optionally, a hierarchical encoder can map continuous audio data into a discrete token sequence, ensuring reconstruction quality while giving each token an interpretable semantic meaning. The input consists of audio data Z and the corresponding text vector. Audio data After processing by a hierarchical encoder, a latent audio representation is obtained, followed by hierarchical quantization to obtain semantic features. and audio features The final output is the fused feature. .

[0107] Optionally, in a layered encoder, the process of semantically quantizing audio data may include:

[0108] ;

[0109] in, The semantic weight matrix is ​​used, and Attention is a multi-head attention mechanism. For semantic features, For audio data. Multi-head attention is a mechanism that uses multiple independent attention functions in parallel, allowing a hierarchical encoder to simultaneously extract information from different representation subspaces (from different perspectives) of the input sequence, and then integrates all the information. The semantic weight matrix can be a semantic attention weight matrix, which can quantify and visualize the correlation strength between all elements in a sequence under the semantic quantization task. This allows the hierarchical encoder to flexibly and efficiently integrate global information, thereby generating high-quality sequence representations as semantic features. Semantic features are the result data of quantizing the high-level semantic information contained in the audio data.

[0110] Optionally, in a layered encoder, the process of audio quantizing the audio data may include:

[0111] ;

[0112] Where α is the layer fusion coefficient, with a value ranging from 0.5 to 0.8, which can be adjusted according to the actual encoding effect of the layer encoder; Upsample is the upsampling operation. Audio features are the result of quantizing the detailed parts of the audio signal contained in the audio data, such as timbre and spectrum. Upsampling operations can increase the sampling rate of the audio signal, that is, insert more sample points per unit time, so that the audio becomes longer and denser on the time axis. For example, upsampling operations can include interpolation, transposed convolution / deconvolution and other methods. Audio features are the quantized result of quantizing the detailed parts of the audio signal contained in the audio data, such as timbre and spectrum.

[0113] It should be noted that, The corresponding semantic feature quantization algorithm is an algorithm that maps high-level semantic information to the semantic codebook. This refers to the corresponding audio feature quantization algorithm, which is an algorithm that maps the underlying audio details to the audio codebook.

[0114] It should be noted that the fused features output by the hierarchical encoder are as follows:

[0115] ;

[0116] in, β is the fusion feature, and β is the fusion coefficient, which ranges from 0.2 to 0.4 and can be adjusted according to the actual coding effect of the layer encoder.

[0117] Therefore, in the processing of the hierarchical encoder, a semantic weight matrix and a multi-head attention mechanism can be introduced. Through the interaction of these two mechanisms, semantic information can be quantized to obtain corresponding semantic features. Based on these semantic features, hierarchical fusion coefficients and upsampling operations can be combined. Upsampling integrates high-level semantic features into low-level features to obtain corresponding audio features, which quantize and represent the details of the audio. The audio features and semantic features are then synthesized using fusion coefficients to produce the final fused features, which serve as the output of the hierarchical encoder. This allows the hierarchical encoder to simultaneously output high-level semantics and low-level audio details, effectively improving its semantic understanding capability during quantization compression processing.

[0118] exist Figure 3 In the illustrated embodiment, semantic codebooks and audio codebooks with different levels and correlation can be set so that the layered encoder can not only quantize audio, but also quantize the associated semantic information, thereby improving the semantic understanding ability of the layered encoder when performing quantization compression processing.

[0119] Optionally, please refer to Figure 4 , Figure 4 The following is a detailed flowchart of step S220 provided in an embodiment of this application. Step S220 may include steps S221-S222.

[0120] Step S221: Determine the joint loss associated with the hierarchical encoder and the text encoder.

[0121] In order for the hierarchical encoder and the text encoder to output associated encoding results, the joint loss associated with the hierarchical encoder and the text encoder can be determined.

[0122] Optionally, the joint loss may include multiple losses representing different situations from two different types of hierarchical encoders.

[0123] Step S222: Based on the joint loss, the hierarchical encoder and the text encoder are jointly trained to obtain the optimized encoder.

[0124] In this study, the hierarchical encoder and the text encoder are jointly trained based on the joint loss. This results in an optimized encoder that not only ensures high-fidelity reconstruction of audio signals but also improves the semantic feature sequence in the output fusion features to conform to semantic logic and temporal dependencies. This upgrades the audio discretization process from the existing "signal reconstruction" to "semantic drive," providing core technical support for application scenarios such as intelligent voice interaction and cross-modal content generation.

[0125] It should be noted that during joint training, the text encoder can learn the distribution patterns of semantic features output by the hierarchical encoder, ensuring that the generated semantic features conform to semantic logic and temporal dependencies. For example, in music: chord progressions should conform to music theory (the I-IV-VI progression is common in C major); for ambient sounds: scene transitions should be smooth (it shouldn't abruptly change from "rain" to "car horn"). The input is the semantic features output by the hierarchical encoder, which can employ an autoregressive Transformer (such as the GPT architecture) to predict the next associated token after inputting a prefix token in the semantic features.

[0126] Optionally, during joint training, the optimization of the text encoder and the hierarchical encoder is performed alternately. Typically, the hierarchical encoder is trained first in the first 10-20 epochs (training cycles) to ensure that it can extract the necessary semantic information from real audio data; as training progresses, the text encoder is gradually added to the optimization process, so that the semantic features can not only accurately represent the audio content, but also conform to the generation rules of natural language.

[0127] exist Figure 4 In the illustrated embodiment, joint loss is used to improve the correlation between the outputs of the hierarchical encoder and the text encoder.

[0128] Optionally, please refer to Figure 5 , Figure 5 The following is a detailed flowchart of step S221 provided in an embodiment of this application. Step S221 may include steps S2211-S2213.

[0129] Step S2211: Determine the composite loss of the layered encoder.

[0130] Step S2212: Determine the language loss of the language model in the text encoder.

[0131] In particular, considering the different types of losses in different encoders during training, the composite loss in the hierarchical encoder and the language loss of the language model in the text encoder can be determined.

[0132] Step S2213: Determine the joint loss based on the composite loss, language loss, and weight balancing coefficient.

[0133] The final joint loss can be determined based on the composite loss, the language loss, and the weight balancing coefficients that are adjusted for different types of loss.

[0134] Optionally, the joint loss may include:

[0135] ;

[0136] in, For joint losses, For the composite loss of the layered encoder, This is the weighting balance coefficient. This represents the language loss of the text encoder.

[0137] Optionally, the weight balancing coefficient can adjust the importance of the hierarchical encoder and the text encoder at different training stages. Specific settings for the weight balancing coefficient can include:

[0138] ;

[0139] In the early stages of training, the focus is on optimizing audio reconstruction loss and contrast loss. Once the semantic features are stable, the weight of language loss is gradually increased, so that the generation process transitions from "imitating signals" to "understanding semantics and making reasonable predictions".

[0140] Optionally, ;

[0141] in, For audio reconstruction loss, The coefficient of variation, To constrain the difference loss between continuous coding features and discrete codebooks, For comparison coefficients, This refers to the contrastive loss used for semantic alignment of audio data and text information. The composite loss of the hierarchical encoder can be determined based on the audio reconstruction loss, difference coefficient, difference loss between constrained continuous encoded features and discrete codebooks, contrast coefficient, and the contrastive loss for semantic alignment of audio data and text information. The audio reconstruction loss measures the difference between the audio reconstructed by the subsequent decoder and the original input audio, thus measuring the fidelity of the reconstructed audio. The difference loss ensures that the continuous features output by the hierarchical encoder can be well represented by a codebook entry, ensuring quantization stability. The contrastive loss characterizes the similarity between the semantic information of the audio data and the text vectors of the text information, achieving semantic alignment.

[0142] Optionally, ;

[0143] in, Let t be the semantic token in the semantic features. The length of the token sequence. The language loss, representing the conditional probability distribution output by the language model, enables the text encoder to generate text vectors that conform to temporal dependencies and semantic consistency. The language loss of the language model can be determined based on multiple semantic tokens in the semantic features, the token sequence length, and the conditional probability distribution output by the language model. This, combined with the composite loss, weight balancing coefficients, and language loss, determines the joint loss used during training.

[0144] Optionally, the contrast loss may include:

[0145] ;

[0146] in, Let be the semantic features of the i-th audio sample in the audio data. Let be the text vector associated with the i-th audio sample. This represents the text vectors of all text samples in the text information. For similarity calculation, The temperature parameter is used. The text encoder is responsible for converting text information into text vectors, while the hierarchical encoder converts audio data into semantic features. To achieve semantic consistency between the semantic features and text vectors, a contrastive loss is introduced. Through the contrastive loss, the semantics of the audio and the representation of the text are brought closer, enabling the final optimized encoder to generate audio features that match the input text description. The output text vector is a 512-dimensional text embedding vector naturally aligned with the audio semantic space. This represents the embedding vectors (including positive and negative samples) for all text samples in the batch. Contrastive loss enables associated audio data and text information to be closer in a shared semantic space; for example, when the text information is "piano performance," semantic features... The semantic features should be close to the text vectors related to "piano"; when the text information is of a "cheerful mood", the semantic features should be close to the text vectors related to "piano". The system should capture emotion-related features such as rhythm and pitch in the audio. By jointly optimizing the hierarchical encoder and the text encoder, a contrastive loss is used to establish a mapping relationship between semantic features and text vectors. This allows semantically similar features and text vectors to cluster in the feature space. Each entry in the high-level semantic codebook maps to a set of related text concepts and supports subsequent cross-modal retrieval (searching for similar audio from input text) and generation (generating audio from text).

[0147] exist Figure 5 In the illustrated embodiment, the joint loss can be determined by combining multiple losses representing different time conditions in different encoders, which effectively improves the effectiveness of joint training based on joint loss, thereby improving the effect of optimizing the encoder in processing audio data.

[0148] Please see Figure 6 , Figure 6 This is a flowchart illustrating an audio generation method provided in an embodiment of this application. The method may include steps S310-S320.

[0149] Step S310: Determine the generation requirements.

[0150] In audio generation applications, the actual generation requirements can be determined first. These requirements include semantic and / or audio requirements. For example, when a user directly inputs a sound source, that sound source is the corresponding audio requirement. When a user inputs text, speech, or an image, the user's semantic requirements can be extracted from the text, speech, or image. For example, if a user wants to generate an elegant violin piece, they can also input a sound source and add corresponding semantic requirements, such as replacing the piano accompaniment in the original sound source with a flute accompaniment.

[0151] Step S320: By optimizing the encoder and processing based on the generation requirements, the required audio data is obtained.

[0152] The optimized encoder is trained using the encoder training method in any of the above embodiments. It can be processed by the optimized encoder based on the actual generation requirements and output the required audio data that meets the generation requirements as the generation result.

[0153] Optionally, the optimized encoder can process text information with semantic requirements as input and audio data with audio requirements as input accordingly.

[0154] exist Figure 6 In the illustrated embodiment, the optimized encoder can understand and process the semantic and audio information in the generated requirements, effectively improving the accuracy of the required audio data and meeting the diverse audio generation needs of users.

[0155] Optionally, please refer to Figure 7 , Figure 7 The following is a detailed flowchart of step S320 provided in an embodiment of this application. Step S320 may include steps S321-S322.

[0156] Step S321: If the generated requirement is a semantic requirement, then the encoder is optimized to determine the requirement text vector based on the semantic requirement; based on the requirement text vector, the associated first requirement fusion feature is determined; and the requirement audio data is generated based on the first requirement fusion feature.

[0157] In the case of generating semantic requirements, the encoder can be optimized to determine the corresponding requirement text vector based on the semantic requirements, and the first requirement fusion feature associated with the requirement text vector can be determined based on the requirement text vector, so as to generate the corresponding requirement audio data according to the first requirement fusion feature.

[0158] Optionally, the optimized encoder can process the semantic requirements based on the text encoder set therein to obtain the corresponding requirement text vector, and the hierarchical encoder jointly trained with the text encoder can determine the first semantic feature and the first audio feature that are semantically aligned with the requirement text vector as the first requirement fusion feature, so as to reconstruct the corresponding requirement audio data based on the first requirement fusion feature.

[0159] Step S322: If the generated requirement is an audio requirement, then by optimizing the encoder, the input fusion feature is determined based on the audio requirement; the second requirement fusion feature associated with the input fusion feature is determined; and the requirement audio data is generated based on the second requirement fusion feature.

[0160] In the case of generating audio requirements, the encoder can be optimized to determine the input fusion features based on the audio requirements, and to determine the second requirement fusion features associated with the input fusion features. That is, the second requirement features that are musically related to the input fusion features can be determined, so as to generate the corresponding requirement audio data according to the second requirement fusion features.

[0161] Optionally, the optimized encoder can process the audio requirements based on the hierarchical encoder set therein, and obtain the corresponding input semantic features and input audio features as input fusion features. Since the hierarchical encoder has been jointly trained with the text encoder, the hierarchical encoder can determine the second semantic features and the second audio features that conform to semantic logic and temporal dependency and have a distributional correlation with the input fusion features as the second requirement fusion features, so as to reconstruct the corresponding requirement audio data based on the second requirement fusion features.

[0162] exist Figure 7 In the illustrated embodiment, the optimized encoder can perform different processing based on different types of needs, generating required audio data that meets the corresponding requirements.

[0163] Please see Figure 8 , Figure 8 This is a flowchart illustrating an audio retrieval method provided in an embodiment of this application. The method may include steps S410-S430.

[0164] Step S410: By optimizing the encoder, the retrieval fusion features are determined according to the retrieval requirements.

[0165] The optimized encoder is trained using the encoder training method in any of the above embodiments. In audio retrieval applications, the optimized encoder can determine the corresponding retrieval fusion features based on various types of input retrieval needs.

[0166] Optionally, the retrieval requirement can be the user's need for similar music to a specific audio source, such as finding music similar to "Ode to Joy." The optimized encoder can process the retrieval requirement based on the hierarchical encoder set therein, obtaining the retrieval semantic features and retrieval audio features of the input audio source as retrieval fusion features.

[0167] Step S420: By optimizing the encoder, similar fusion features with a similarity greater than or equal to the similarity threshold with the retrieved fusion features are determined.

[0168] Step S430: Determine the target audio by optimizing the encoder based on similarity fusion features.

[0169] Among them, similarity analysis can be performed based on the retrieval fusion features to determine similar fusion features whose similarity to the retrieval fusion features is greater than or equal to a preset similarity threshold. Based on the similar fusion features, the corresponding target audio can be generated as the result of audio retrieval.

[0170] For example, the similarity threshold can be set based on actual needs, such as setting the similarity threshold to 90%. In cases where multiple fusion features have similarities exceeding the similarity threshold, the feature with the highest similarity can be selected as the similarity fusion feature.

[0171] Optionally, since the hierarchical encoder has been jointly trained with the text encoder, the hierarchical encoder can determine the third semantic feature and the third audio feature with the highest similarity to the retrieved fusion feature as similar fusion features based on the retrieved fusion feature. Multiple historical audios stored in the audio database have been processed by the optimized encoder and have corresponding historical fusion features. Therefore, one or more audios with the same historical fusion feature as the similar fusion feature or the highest similarity can be selected as the corresponding target audio.

[0172] exist Figure 8 In the illustrated embodiment, the optimized encoder can determine similar fusion features based on retrieval requirements, which effectively improves the correspondence between the target audio and the retrieval requirements and meets the diverse audio retrieval needs of users.

[0173] In the embodiments provided in this application, the high-level semantic codebook entries in the optimized encoder directly correspond to text-describable semantic features (such as "C major" and "dog barking"), support semantic-level audio editing and retrieval, have high semantic interpretability, and can generate semantically aligned audio token sequences based on multiple inputs (such as inputting "rain sound" to generate corresponding ambient sound), and can achieve cross-modal audio generation.

[0174] Since the principles of the audio generation method and audio retrieval method in this embodiment are similar to those in the aforementioned encoder training method embodiment, the implementation of the audio generation method and audio retrieval method in this embodiment can refer to the description in the aforementioned encoder training method embodiment, and repeated details will not be repeated.

[0175] This application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the steps of any one of the encoder training method, audio generation method, or audio retrieval method described above.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed device can also be implemented in other ways. The device embodiments described above are merely illustrative; for example, the block diagrams in the accompanying drawings illustrate the possible architecture, functions, and operations of the device according to various embodiments of this application. In this regard, each block in the block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram, and combinations of block diagrams, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0177] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0178] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0180] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

[0181] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. An encoder training method, characterized in that, The method includes: The audio quantization encoder is adjusted to obtain a layered encoder; wherein, the layered encoder is equipped with an associated semantic codebook and an audio codebook; The hierarchical encoder and the text encoder are jointly trained based on the joint loss to obtain an optimized encoder; The layered encoder is used to quantize and compress audio data to obtain fusion features, which include semantic features and audio features; the text encoder is used to process the text information associated with the audio data to obtain a text vector; the fusion features are related to the text vector.

2. The method according to claim 1, characterized in that, The adjustment of the audio quantization encoder to obtain a layered encoder includes: Determine the semantic codebook for quantizing the semantic information of the audio data; Determine the audio codebook associated with the semantic codebook for quantizing the audio data; Based on the semantic codebook and the audio codebook, the audio quantization encoder is adjusted to obtain the layered encoder.

3. The method according to claim 2, characterized in that, in, In the layered encoder, the process of semantically quantizing the audio data includes: ； in, The semantic weight matrix is ​​used, and Attention is a multi-head attention mechanism. For the semantic features, For the audio data, For semantic feature quantization algorithms; In the layered encoder, the process of audio quantization of the audio data includes: ； Where α is the hierarchical fusion coefficient, and Upsample is the upsampling operation. For the aforementioned audio features, For audio feature quantization algorithms; The fusion feature output by the hierarchical encoder is: ； in, β represents the fusion feature, and β is the fusion coefficient.

4. The method according to claim 1, characterized in that, The step of jointly training the hierarchical encoder and the text encoder based on joint loss to obtain an optimized encoder includes: Determine the joint loss associated with the hierarchical encoder and the text encoder; Based on the joint loss, the hierarchical encoder and the text encoder are jointly trained to obtain the optimized encoder.

5. The method according to claim 4, characterized in that, Determining the joint loss associated with the hierarchical encoder and the text encoder includes: Determine the composite loss of the hierarchical encoder; Determine the language loss of the language model in the text encoder; The joint loss is determined based on the composite loss, the language loss, and the weight balancing coefficient.

6. The method according to claim 5, characterized in that, in, The joint loss includes: ； in, For the aforementioned joint loss, For the composite loss, The weighting balance coefficient is... For the language loss; ； in, For audio reconstruction loss, The coefficient of variation, To constrain the difference loss between continuous coding features and discrete codebooks, For comparison coefficients, The contrast loss is used to semantically align the audio data with the text information; ； in, Let t be the semantic token in the semantic features. The length of the token sequence. The conditional probability distribution output by the language model.

7. The method according to claim 6, characterized in that, in, The comparison loss includes: ； in, The semantic feature of the i-th audio sample in the audio data. The text vector associated with the i-th audio sample. The text vector is the text vector of all text samples in the text information. For similarity calculation, This refers to the temperature parameter.

8. An audio generation method, characterized in that, The method includes: Determine the generation requirements; wherein, the generation requirements include semantic requirements and / or audio requirements; By optimizing the encoder and processing it based on the generated requirements, the required audio data is obtained; wherein the optimized encoder is trained by the method described in any one of claims 1-7.

9. The method according to claim 8, characterized in that, The process of optimizing the encoder and processing the generated requirements to obtain the required audio data includes: If the generated requirement is the semantic requirement, then by optimizing the encoder, a requirement text vector is determined based on the semantic requirement; based on the requirement text vector, an associated first requirement fusion feature is determined; and based on the first requirement fusion feature, the requirement audio data is generated. If the generated requirement is the audio requirement, then the optimized encoder determines the input fusion feature based on the audio requirement; determines the second requirement fusion feature associated with the input fusion feature; and generates the requirement audio data based on the second requirement fusion feature.

10. An audio retrieval method, characterized in that, The method includes: By optimizing the encoder, retrieval fusion features are determined according to retrieval requirements; wherein the optimized encoder is trained by the method described in any one of claims 1-7; The optimized encoder determines similar fusion features whose similarity to the retrieved fusion features is greater than or equal to a similarity threshold. The target audio is determined using the optimized encoder based on the similarity fusion features.

Images