Music understanding model training method, audio processing method, equipment and medium

Abstract

The invention discloses a music understanding model training method, an audio processing method, equipment and a medium, which are applied to the technical field of computers, and comprise the following steps: compressing a song audio through an initial music understanding model to obtain an audio compression representation; the audio compression representation comprises any one or more of timbre information, emotion information and semantic information; performing coding processing on the pitch sequence of the song audio through the initial music understanding model to obtain melody compression representation containing melody information; constructing a fusion compression representation based on the audio compression representation and the melody compression representation; and optimizing the initial music understanding model based on the fusion compression representation and the audio compression representation to obtain a music understanding model. According to the method, the pitch track of the melody is reserved through the pitch sequence, redundant information is directly stripped, semantic focusing of melody characteristics is achieved, melody compression representation is obtained based on the pitch sequence, the characteristic of strong association of the music before and after is reserved, and the method is more suitable for characteristic learning of the music melody.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a music comprehension model training method, an audio processing method, a device and a medium. Background Art

[0002] With the rapid rise and development of artificial intelligence (AI) technology, the bridge between machines and humans is becoming increasingly clear, and human-machine dialogue has reached an unprecedented level of naturalness. The smooth communication between AI models and humans is due to the models' ability to understand data. The same is true for music understanding models. By learning and understanding songs, they convert the song audio into an abstract vector representation, which can then be used for downstream tasks related to the song. Existing music understanding models have various structures and schemes, but none of them explicitly emphasize the musicality of the audio in their audio processing.

[0003] Therefore, how to provide a music understanding model with more efficient music representation is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a music understanding model training method, audio processing method, device and medium to solve the problem in the prior art that the music understanding model is insufficient in learning the music representation of song audio.

[0005] To solve the above technical problems, the present invention provides a music comprehension model training method, comprising:

[0006] Compressing the song audio using the initial music understanding model to obtain an audio compression representation; the audio compression representation includes any one or more of timbre information, emotional information, and semantic information;

[0007] encoding the pitch sequence of the song audio using the initial music understanding model to obtain a melody compression representation; the melody compression representation includes melody information;

[0008] Constructing a corresponding fused compression representation based on the audio compression representation and the melody compression representation;

[0009] The initial music understanding model is optimized based on the fused compression representation and the audio compression representation to obtain a music understanding model.

[0010] Optionally, encoding the pitch sequence of the song audio using the initial music understanding model to obtain a melody compression representation includes:

[0011] Extracting a pitch sequence from the song audio using the initial music understanding model to obtain an initial sequence;

[0012] The initial sequence is encoded using the initial music understanding model to obtain the melody compression representation.

[0013] Optionally, extracting a pitch sequence from the song audio using the initial music understanding model to obtain an initial sequence includes:

[0014] Segment and pre-process the song audio to obtain a preliminary processing result;

[0015] Using a multi-branch structure to extract features from the preliminary processing results to obtain fusion features; the fusion features include time domain waveform features and frequency domain Mel spectrum features;

[0016] Mapping the fused features into a fundamental frequency probability distribution through a fully connected layer, and performing peak detection on the fused features based on the fundamental frequency probability distribution to generate a candidate pitch sequence;

[0017] The candidate pitch sequence is optimized by using dynamic programming and filtering, and the optimized sequence is quantized to obtain the initial sequence including the start time, end time and pitch value of the note.

[0018] Optionally, encoding the initial sequence using the initial music understanding model to obtain the melody compression representation includes:

[0019] Sort and concatenate the start time, end time, and pitch value of each note in chronological order to obtain a one-dimensional pitch vector;

[0020] The initial time and the end time in the one-dimensional pitch vector are uniformly normalized, and the pitch value in the one-dimensional pitch vector is normalized to obtain the melody compression representation.

[0021] Optionally, compressing the song audio using the initial music understanding model to obtain an audio compression representation includes:

[0022] Sampling the song audio to obtain various sampling points;

[0023] Input each sampling point into a one-dimensional convolutional layer to perform feature extraction and dimensionality reduction to obtain output features;

[0024] The output features are input into a Transformer encoder for encoding to obtain the audio compression representation.

[0025] Optionally, constructing a corresponding fused compression representation based on the audio compression representation and the melody compression representation includes:

[0026] splicing the audio compression representation and the melody compression representation to obtain a splicing feature;

[0027] Quantizing and encoding the splicing features to obtain encoding features;

[0028] The encoded features are input into a fully connected layer for compression to obtain the fused compressed representation.

[0029] Optionally, optimizing the initial music understanding model based on the fused compression representation and the audio compression representation to obtain a music understanding model includes:

[0030] Calculating the square error between the fused compressed representation and the audio compressed representation and using it as the reconstruction loss;

[0031] The initial music understanding model is optimized and trained according to the reconstruction loss until a preset convergence condition is met, thereby obtaining the music understanding model.

[0032] In one aspect, the present invention further provides an audio processing method, comprising:

[0033] Inputting the song audio to be processed into a music understanding model, compressing and encoding the song audio through the music understanding model to obtain a compressed representation containing melody information and audio information; the audio information includes at least one or more of timbre information, emotional information, and semantic information; the music understanding model is obtained using the above-mentioned music understanding model training method;

[0034] Processing the compressed representation based on a music processing task to obtain a processing result;

[0035] The music processing task includes any one of a song synthesis task, a song classification task or a song recognition task.

[0036] In one aspect, the present invention further provides a music comprehension model training device, comprising:

[0037] An audio compression representation acquisition module is used to compress the song audio using the initial music understanding model to obtain an audio compression representation; the audio compression representation includes any one or more of timbre information, emotional information, and semantic information;

[0038] A melody compression representation acquisition module is used to encode the pitch sequence of the song audio through the initial music understanding model to obtain a melody compression representation; the melody compression representation includes melody information;

[0039] A fusion module, configured to construct a corresponding fused compression representation based on the audio compression representation and the melody compression representation;

[0040] A training module is used to optimize the initial music understanding model based on the fused compression representation and the audio compression representation to obtain a music understanding model.

[0041] In one aspect, the present invention further provides an audio processing device, comprising:

[0042] An input module is configured to input the song audio to be processed into a music understanding model, so as to compress and encode the song audio through the music understanding model to obtain a compressed representation containing melody information and audio information; the audio information includes at least one or more of timbre information, emotional information, and semantic information; and the music understanding model is obtained using the above-mentioned music understanding model training method;

[0043] A processing module processes the compressed representation based on a music processing task to obtain a processing result; wherein the music processing task includes any one of a song synthesis task, a song classification task or a song recognition task.

[0044] In one aspect, the present invention further provides an electronic device, comprising:

[0045] Memory, used to store computer programs;

[0046] A processor is configured to execute the computer program to implement the above method.

[0047] In one aspect, the present invention further provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0048] In one aspect, the present invention further provides a computer program product, comprising a computer program / instruction, which implements the above method when executed by a processor.

[0049] It can be seen that the present invention compresses the song audio through the initial music understanding model to obtain an audio compression representation; the audio compression representation includes any one or more of timbre information, emotional information and semantic information; the pitch sequence of the song audio is encoded and processed by the initial music understanding model to obtain a melody compression representation; the melody compression representation includes melody information; a corresponding fusion compression representation is constructed based on the audio compression representation and the melody compression representation; the initial music understanding model is optimized based on the fusion compression representation and the audio compression representation to obtain a music understanding model. It can be seen that the present invention directly strips off redundant information by processing the pitch sequence of the song audio, and obtains a melody compression representation that only retains the pitch trajectory of the melody, achieves semantic focus on the melody features, and improves the representation of the music melody by the music understanding model; and the pitch sequence includes the two-dimensional trajectory of the pitch-time of the melody, and the natural melody compression representation also retains the strong correlation between the music itself, so the music understanding model is more suitable for learning the characteristics of music melody. The music understanding model trained in this way can output a more musical representation and can better adapt to downstream music tasks.

[0050] In addition, the present invention also provides an audio processing method, device and medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0052] Figure 1 A system framework diagram of a music comprehension model training method provided by an embodiment of the present invention;

[0053] Figure 2 A flowchart of a music understanding model training method provided by an embodiment of the present invention;

[0054] Figure 3 A flowchart of a specific music comprehension model training method provided by an embodiment of the present invention;

[0055] Figure 4 A flowchart illustrating a music comprehension model training method according to an embodiment of the present invention;

[0056] Figure 5 A flowchart of an audio processing method provided by an embodiment of the present invention;

[0057] Figure 6A schematic structural diagram of a music comprehension model training device provided by an embodiment of the present invention;

[0058] Figure 7 A schematic structural diagram of an audio processing device provided by an embodiment of the present invention;

[0059] Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0061] With the rapid rise and development of artificial intelligence (AI) technology, the bridge between machines and humans is becoming increasingly clear, and human-machine dialogue has reached an unprecedented level of naturalness. We can now pose questions to large language models, and AI models can quickly return high-quality answers, eliminating the need to painstakingly dig through search results for valid information. This direct communication between models and humans is facilitated by their ability to understand data, and this is also true for music understanding models. Music understanding models are networks trained through self-supervision using large amounts of music data, which can be used to extract musical representations and train downstream tasks. By learning and understanding songs, music understanding models can convert song audio into an abstract vector representation, which can then be used for downstream tasks that require the musical information contained in the representation.

[0062] Existing music understanding models have a variety of structures and schemes, but none explicitly emphasize the musicality of song audio. For example, MERT (Music Understanding Model with Large-Scale Self-supervised Training) uses CQT features (Constant Q Transform) as a "teacher" to learn the relationship between frequency bands and pitch from the spectrum. Because CQT features are extracted from the local time-frequency characteristics of a single frame, and music is a strongly correlated signal, they do not adequately learn the melodic characteristics of music. Furthermore, CQT features contain a large amount of redundant information unrelated to the semantics of music, making it impossible for the model's self-supervised learning approach to completely eliminate this redundant information.

[0063] Therefore, current music understanding models do not clearly address the musicality of song audio, which is a key difference between songs and other audio. Furthermore, this paper proposes a method for training a music understanding model with melodic characteristics, enabling the model to not only compress representations but also learn the melodic characteristics of songs.

[0064] The system framework used in the music comprehension model training method of the present invention can be found in Figure 1 shown. Figure 1 This is a system framework diagram of a music understanding model training method provided by an embodiment of the present invention, which may specifically include: a backend server 01 and a user terminal 02 connected to the backend server 01.

[0065] In the present invention, the user terminal 02 is used to obtain song audio samples for training the music understanding model and song audio to be processed for audio processing. The user terminal 02 can be a client, such as a mobile phone, tablet, computer, etc., or a web page, which is not limited here.

[0066] Furthermore, the backend server 01 is used to perform the steps of the music understanding model training method and / or the audio processing method, including:

[0067] Obtaining song audio for training and / or song audio to be processed sent by the user terminal 02;

[0068] Among them, for the trained song audio, the song audio is compressed by the initial music understanding model to obtain an audio compression representation; wherein, the audio compression representation includes any one or more of timbre information, emotional information and semantic information; the pitch sequence of the song audio is encoded by the initial music understanding model to obtain a melody compression representation; the melody compression representation includes melody information; further, a corresponding fused compression representation is constructed based on the audio compression representation and the melody compression representation; the initial music understanding model is optimized based on the fused compression representation and the audio compression representation to obtain a music understanding model.

[0069] Accordingly, the song audio to be processed is input into the trained music understanding model, which compresses and encodes the song audio and outputs a compressed representation containing melody and audio information. The compressed representation is then processed based on the music processing task to obtain a processing result. The music processing task can include any of song synthesis, song classification, or song recognition tasks.

[0070] See also Figure 2 As shown, Figure 2 A flowchart of a music understanding model training method provided by an embodiment of the present invention includes:

[0071] S101: Compress the song audio through the initial music understanding model to obtain an audio compression representation; the audio compression representation includes any one or more of timbre information, emotional information and semantic information.

[0072] It should be noted that the song audio in step S101 is a training sample for training the music understanding model, and the number can generally be multiple. The song audio in this embodiment serves as an input source, providing the original carrier of the music signal, including all acoustic information such as melody, harmony, and timbre. This embodiment compresses the song audio through the initial music understanding model to obtain an audio compression representation containing various basic audio information. Specifically, the audio compression representation may include but is not limited to timbre information, emotional information, and voice information. It can be understood that the initial music understanding model in this embodiment can be a large model, or other types of algorithmic models. The initial music understanding model in this embodiment may include at least a convolutional layer, a coding layer, and a fully connected layer.

[0073] In a specific embodiment, song audio is compressed using an initial music understanding model to obtain a compressed audio representation. This may include: sampling the song audio to obtain individual sampling points; inputting each sampling point into a one-dimensional convolutional layer for feature extraction and dimensionality reduction to obtain output features; and inputting the output features into a Transformer encoder (a deep learning model for natural language processing tasks) for encoding to obtain the compressed audio representation. Specifically, a one-dimensional convolution operation performs preliminary extraction and dimensionality reduction on each sampled audio sampling point. A sliding convolution kernel calculation is used to convert the music signal into a low-dimensional audio feature sequence, filtering out some redundant acoustic details (such as high-frequency noise) while retaining the underlying spectral profile. This provides a more concise input for the Transformer encoder, addressing the high dimensionality and information complexity of the original audio, which makes direct temporal semantic modeling difficult. The Transformer encoder then performs temporal dynamic modeling on the features output by the one-dimensional convolution. A self-attention mechanism is used to capture long-range dependencies in the audio feature sequence (such as the correlation between spectral changes at different times and the echo of rhythmic patterns), integrating the scattered frame-level features into an abstract representation that incorporates temporal logic, namely the compressed audio representation.

[0074] S102: Encoding the pitch sequence of the song audio through the initial music understanding model to obtain a melody compression representation; the melody compression representation contains melody information.

[0075] This embodiment does not limit the execution order of step S101 and step S102. It should be noted that the initial music understanding model in this embodiment also includes a pitch extraction module for separately extracting the pitch sequence of the song audio. This embodiment does not limit the specific method of extracting the pitch sequence (pitch sequence). For example, the PYIN (an improved fundamental frequency estimation algorithm) algorithm can be used to extract the pitch sequence; or the CREPE (a monophonic pitch tracker based on a deep convolutional neural network) neural network can be used to extract the pitch sequence. After obtaining the pitch sequence, it can be further encoded to obtain a melody compression representation. This embodiment does not limit the encoding method. For example, it can be a method of automatically learning encoding by an autoencoder network; or it can also be a normalized encoding method.

[0076] It should be noted that the pitch sequence, as the "skeleton" of the melody, carries the trajectory of pitch changes over time. It includes the melody's basic timing and pitch logic (such as interval jumps and modal direction), and is the core semantic material for subsequent encoding. Furthermore, since the pitch sequence is global, the compressed representation of the melody here naturally also includes global melodic information. Therefore, this embodiment processes the pitch sequence separately to directly capture the melody skeleton. The resulting pitch sequence (measured in Hz (Hertz) or MIDI (Musical Instrument Digital Interface) pitch) naturally corresponds to the melody's two-dimensional "pitch-time" trajectory, and directly removes the overtone components, retaining only the melody's pitch trajectory. This eliminates the difficulty of fully extracting melodic features from mixed spectra in self-supervised learning of music understanding models, and eliminates the problem of single-frame feature matching computational losses that fragment the melody's temporal semantics.

[0077] S103: Constructing a corresponding fused compression representation based on the audio compression representation and the melody compression representation.

[0078] It is understandable that this embodiment cross-modally fuses the audio compression representation and the melody compression representation to obtain a fused compression representation. This allows the music understanding model to simultaneously utilize the global acoustic context and precise melodic semantics to make up for the information loss of a single modal representation and enhance the joint modeling capability of temporal dynamics and melodic semantics. This embodiment does not limit the specific construction method. For example, it can be an element-level weighted summation fusion method; or it can be an element-level multiplication fusion method.

[0079] In a specific embodiment, constructing a corresponding fused compressed representation based on an audio compression representation and a melody compression representation may include: concatenating the audio compression representation and the melody compression representation to obtain a concatenated feature; quantizing and encoding the concatenated feature to obtain an encoded feature; and inputting the encoded feature into a fully connected layer for compression to obtain a fused compressed representation. Specifically, the audio compression representation and the melody compression representation are concatenated to obtain the concatenated feature. The concatenated feature not only contains the compression characteristics of the original song audio, namely, the timbre, emotional information, and semantic information contained in the audio compression representation, but also the melodic information highlighted by the melody compression representation. This embodiment does not specifically limit the quantization encoding method. For example, RVQ (Residual Vector Quantization) or other quantization methods may be used. For example, this embodiment utilizes RVQ to efficiently compress and discretize the concatenated features. Through quantization encoding, key information is retained while reducing feature dimensionality, enabling the model to learn a more compact and discriminative representation. This addresses the issue of redundant high-dimensional features that hinder efficient storage and computation, and provides concise input for subsequent reconstruction loss optimization. The discretized compressed representation (i.e., the encoded features) after quantization encoding retains the core correlation information between the audio and melody while also possessing low dimensionality and high semantic density. A fully connected layer then performs nonlinear transformation and dimensionality adaptation on the encoded features, mapping them back to a dimension matching the original audio features to generate a fused compressed representation, preparing for the subsequent calculation of the reconstruction loss.

[0080] S104: Optimizing the initial music understanding model based on the fused compression representation and the audio compression representation to obtain a music understanding model.

[0081] This example calculates the difference between the fused compressed representation and the audio compressed representation, and reversely optimizes the initial music understanding model. In other words, using the fused compressed representation as a "teacher," the initial music understanding model is forced to learn what information (such as melodic timing and key spectral features) should be retained in the output compressed representation. This ensures that the output compressed representation still accurately restores the core audio content, verifying the effectiveness of temporal modeling and semantic decoupling.

[0082] In a specific embodiment, the initial music understanding model is optimized based on the fused compression representation and the audio compression representation to obtain the music understanding model, which may specifically include: calculating the square error of the fused compression representation and the audio compression representation and using it as the reconstruction loss; optimizing the initial music understanding model according to the reconstruction loss until the preset convergence condition is met to obtain the music understanding model. and audio compression characterization Do the reconstruction loss L1, the specific formula is as follows:

[0083] .

[0084] Where M represents the total number.

[0085] The initial music understanding model is optimized and trained according to the calculated reconstruction loss L1 until the network converges to obtain an optimized and trained music understanding model.

[0086] The music understanding model training method provided by the embodiment of the present invention is applied to compress the song audio through the initial music understanding model to obtain an audio compression representation; the audio compression representation includes any one or more of timbre information, emotional information and semantic information; the pitch sequence of the song audio is encoded and processed by the initial music understanding model to obtain a melody compression representation; the melody compression representation includes melody information; a corresponding fusion compression representation is constructed based on the audio compression representation and the melody compression representation; the initial music understanding model is optimized based on the fusion compression representation and the audio compression representation to obtain a music understanding model. It can be seen that the present invention directly strips off redundant information by processing the pitch sequence of the song audio, and obtains a melody compression representation that only retains the pitch trajectory of the melody, achieves semantic focus on the melody features, and improves the music understanding model's representation of the music melody; and the pitch sequence includes the two-dimensional trajectory of the melody's pitch-time, and the natural melody compression representation also retains the strong correlation between the music itself, so the music understanding model is more suitable for learning the characteristics of music melody. The music understanding model trained in this way can output a more musical representation and can better adapt to downstream music tasks.

[0087] See also Figure 3 As shown, Figure 3 A flowchart of a specific music comprehension model training method provided by an embodiment of the present invention may include:

[0088] S201: Compress the song audio through the initial music understanding model to obtain an audio compression representation; the audio compression representation includes any one or more of timbre information, emotional information and semantic information.

[0089] In this embodiment, regarding the specific process of the above-mentioned step S201, reference may be made to the corresponding content disclosed in the above-mentioned embodiment, and no further details will be given here.

[0090] S202: Extracting a pitch sequence from the song audio using an initial music understanding model to obtain an initial sequence.

[0091] This example first extracts a pitch sequence from the song audio to obtain an initial pitch sequence (i.e., the initial sequence). This example directly isolates the pitch information directly related to the melody from the song audio, filters out redundant acoustic information such as harmony and accompaniment, and decouples the melodic semantics from the mixed audio, further facilitating the self-supervised training of the music understanding model.

[0092] In a specific embodiment, extracting a pitch sequence from song audio to obtain an initial sequence can include: segmenting the song audio to preprocess it and obtain preliminary processing results; extracting features from the preliminary processing results using a multi-branch structure to obtain fused features; the fused features include time-domain waveform features and frequency-domain Mel-spectrogram features; mapping the fused features to a fundamental frequency probability distribution using a fully connected layer, and performing peak detection on the fused features based on the fundamental frequency probability distribution to generate a candidate pitch sequence; optimizing the candidate pitch sequence using dynamic programming and filtering, and quantizing the optimized sequence to obtain an initial sequence containing the start time, end time, and pitch value of the notes. Specifically, the song audio (in the form of a signal) is segmented into overlapping frames, a Hanning window is applied to reduce spectral leakage, and amplitude normalization is performed on each frame to eliminate volume differences caused by the environment or device. The time-domain features of the waveform are extracted using one-dimensional convolution, and a spectrogram is calculated from the short-time Fourier transform of each frame. Logarithmic compression and Mel filtering are applied to obtain a Mel-spectrogram. The time-domain and frequency-domain features at different scales are then fused at multiple scales to obtain the fused features. The fused features are mapped to a fundamental frequency probability distribution through a fully connected layer. Peak detection is performed based on this fundamental frequency probability distribution to produce a candidate pitch sequence. This sequence essentially represents the possible fundamental frequency values ​​for each frame of audio (possibly multiple candidates or noise points). These candidate values ​​may exhibit discontinuities, jumps, or miscalculations in the temporal dimension (for example, a noise frame may be mistakenly identified as a valid pitch). Temporal continuity constraints on pitch are then leveraged (e.g., pitches cannot change suddenly between adjacent frames) to find the optimal path within the candidate pitch sequence. This process then plans the processed sequence. A median filter or Kalman filter is applied to the initially estimated processed sequence to eliminate jump noise, resulting in a smoothed sequence. Finally, quantization is performed to convert the smoothed sequence into musically meaningful discrete pitches (e.g., MIDI note numbers). The final output is an initial sequence in the format [(t0, t1, p0), (t2, t3, p1), …, (tm, tn, pk)]. Among them, t0, t2 and tm represent the start time of each note; t1, t3 and tn represent the end time of each note; p0, p1 and pk represent the pitch value of each note.

[0093] S203: Encoding the initial sequence through the initial music understanding model to obtain a melody compression representation.

[0094] Based on the initial sequence obtained in the above steps, further encoding is performed to obtain a compressed melody representation of the training information. This processing method reduces computational complexity, simplifies implementation, and uniformly scales features, preserving the explicit associations of the original features without relying on neural network training.

[0095] In a specific embodiment, the initial sequence is encoded to obtain a melody compression representation, which may include: sorting and splicing the start time, end time and pitch value of each note in chronological order to obtain a one-dimensional pitch vector; uniformly normalizing the initial time and end time in the one-dimensional pitch vector, and normalizing the pitch value in the one-dimensional pitch vector to obtain a melody compression representation. Specifically, the encoding here is to convert the initial sequence of triplets into a sequence of one-dimensional arrays, that is, in the form of [t0, t1, p0, t2, t3, p1, ..., tm, tn, pk]. Considering that the time value and the pitch value do not belong to variables of the same dimension. Therefore, normalization is performed separately. For t0, t1, ..., tm, this embodiment adopts unified normalization; for the pitch value p, it is normalized separately. The normalization formula is:

[0096] .

[0097] in, represents the i-th feature point (i.e., note); min(X) represents the minimum value of the time dimension / pitch dimension in the initial sequence; max(X) represents the maximum value of the time dimension / pitch dimension in the initial sequence; are the normalized feature points.

[0098] S204: Constructing a corresponding fused compression representation based on the audio compression representation and the melody compression representation.

[0099] In this embodiment, regarding the specific process of the above-mentioned step S204, reference may be made to the corresponding content disclosed in the above-mentioned embodiment, and no further details will be given here.

[0100] S205: Optimizing the initial music understanding model based on the fused compression representation and the audio compression representation to obtain a music understanding model.

[0101] In this embodiment, regarding the specific process of the above-mentioned step S205, reference may be made to the corresponding content disclosed in the above-mentioned embodiment, and no further details will be given here.

[0102] The music understanding model training method provided by the embodiment of the present invention is applied to compress the song audio through the initial music understanding model to obtain an audio compression representation; the pitch sequence in the song audio is extracted through the initial music understanding model to obtain an initial sequence; the initial sequence is encoded and processed by the initial music understanding model to obtain a melody compression representation; a corresponding fusion compression representation is constructed based on the audio compression representation and the melody compression representation; the initial music understanding model is optimized based on the fusion compression representation and the audio compression representation to obtain a music understanding model. It can be seen that the present invention directly strips off redundant information by processing the pitch sequence of the song audio, and obtains a melody compression representation that only retains the pitch trajectory of the melody, achieves semantic focus on the melody features, and improves the representation of the music melody by the music understanding model; and the pitch sequence contains the two-dimensional trajectory of the pitch-time of the melody, and the natural melody compression representation also retains the strong correlation between the front and back of the music itself, so the music understanding model is more suitable for learning the characteristics of music melody. The music understanding model trained in this way can output a more musical representation and can better adapt to downstream music tasks.

[0103] In order to better understand the present invention, please refer to Figure 4 , Figure 4 A flowchart illustrating a method for training a music understanding model provided by an embodiment of the present invention. The song audio is input into the one-dimensional convolution module of the initial music understanding model to perform preliminary compression on the features. The compressed features are then input into the encoder of the initial music understanding model for encoding to obtain an audio compression representation (embedding 1). The song audio is input into the pitch extraction module of the initial music understanding model to obtain a pitch sequence, which is then encoded and normalized to obtain a melody compression representation. The melody compression representation and embedding 1 are feature concatenated, and the concatenation result is input into the quantization encoding module to obtain the encoding features. The encoding features are input into the fully connected layer for compression to obtain embedding 2 (fused compression representation). Reconstruction loss is applied to embedding 1 and embedding 2, and the initial music understanding model is trained and optimized based on the reconstruction loss until the network meets the preset convergence conditions, thereby obtaining a trained music understanding model.

[0104] See also Figure 5 As shown, Figure 5 A flowchart of an audio processing method provided by an embodiment of the present invention may include:

[0105] S301: Input the song audio to be processed into the music understanding model, so as to compress and encode the song audio through the music understanding model to obtain a compressed representation containing melody information and audio information; the audio information includes at least any one or more of timbre information, emotional information and semantic information; the music understanding model is obtained using the above-mentioned music understanding model training method.

[0106] S302: Processing the compressed representation based on a music processing task to obtain a processing result; wherein the music processing task includes any one of a song synthesis task, a song classification task, or a song recognition task.

[0107] This embodiment provides a specific application scenario for a music understanding model. For example, it can be a song synthesis task, a song classification task, or a song recognition task. In these downstream tasks, the music understanding model needs to fully understand the melody to help improve the accuracy of the downstream tasks.

[0108] The audio processing method provided by the embodiment of the present invention is applied, and the song audio to be processed is input into the music understanding model, so that the song audio is compressed and encoded by the music understanding model to obtain a compressed representation containing melody information and audio information; the compressed representation is processed based on the music processing task to obtain a processing result. It can be seen that music understanding is an important part of music downstream tasks. Therefore, the music understanding model trained by the above-mentioned music understanding model training method is used to process the song audio to be processed to obtain a compressed representation containing melody information and audio information. In this way, the music understanding model can fully understand the melody characteristics and is more helpful for downstream tasks. The present invention makes the music understanding model more musical by improving the learning of the melody characteristics of the music understanding model. When used on downstream tasks, the pitch can be more accurate and the song has more rhythm, which can improve the user experience of singing synthesis, music synthesis, etc.

[0109] The following is an introduction to a music comprehension model training device provided by an embodiment of the present invention. The music comprehension model training device described below and the music comprehension model training method described above can be referenced to each other.

[0110] Please refer to Figure 6 , Figure 6 A schematic structural diagram of a music comprehension model training device provided by an embodiment of the present invention may include:

[0111] The audio compression representation acquisition module 100 is used to compress the song audio using the initial music understanding model to obtain an audio compression representation; the audio compression representation includes any one or more of timbre information, emotional information, and semantic information;

[0112] A melody compression representation acquisition module 200 is configured to encode the pitch sequence of the song audio using the initial music understanding model to obtain a melody compression representation; the melody compression representation includes melody information;

[0113] A fusion module 300, configured to construct a corresponding fused compression representation based on the audio compression representation and the melody compression representation;

[0114] The training module 400 is used to optimize the initial music understanding model based on the fused compression representation and the audio compression representation to obtain a music understanding model.

[0115] Based on the above embodiment, the melody compression representation acquisition module 200 may include:

[0116] an initial sequence acquisition unit, configured to extract a pitch sequence from the song audio using the initial music understanding model to obtain an initial sequence;

[0117] The encoding processing unit is used to encode the initial sequence through the initial music understanding model to obtain the melody compression representation.

[0118] Based on the above embodiment, the initial sequence acquisition unit may include:

[0119] A pre-processing sub-unit, configured to segment and pre-process the song audio to obtain a preliminary processing result;

[0120] A feature extraction subunit is used to extract features from the preliminary processing results using a multi-branch structure to obtain fused features; the fused features include time domain waveform features and frequency domain Mel spectrum features;

[0121] a mapping detection subunit, configured to map the fused features into a fundamental frequency probability distribution through a fully connected layer, and perform peak detection on the fused features based on the fundamental frequency probability distribution to generate a candidate pitch sequence;

[0122] The optimization subunit is used to optimize the candidate pitch sequence by using dynamic programming and filtering, and quantize the optimized sequence to obtain the initial sequence including the start time, end time and pitch value of the note.

[0123] Based on the above embodiment, the encoding processing unit may include:

[0124] a concatenation subunit, configured to sort and concatenate the start time, end time, and pitch value of each note in chronological order to obtain a one-dimensional pitch vector;

[0125] The processing subunit is used to perform uniform normalization processing on the initial time and the end time in the one-dimensional pitch vector, and to perform normalization processing on the pitch value in the one-dimensional pitch vector to obtain the melody compression representation.

[0126] Based on the above embodiment, the audio compression representation acquisition module 100 may include:

[0127] A sampling subunit, configured to sample the song audio to obtain various sampling points;

[0128] A convolution subunit, configured to input each sampling point into a one-dimensional convolution layer to perform feature extraction and dimensionality reduction to obtain output features;

[0129] The encoding subunit is used to input the output features into the Transformer encoder for encoding to obtain the audio compression representation.

[0130] Based on any of the above embodiments, the fusion module 300 may include:

[0131] a splicing unit, configured to splice the audio compression representation and the melody compression representation to obtain a splicing feature;

[0132] a quantization encoding unit, configured to perform quantization encoding on the splicing features to obtain encoding features;

[0133] A compression unit is used to input the encoded features into a fully connected layer for compression to obtain the fused compressed representation.

[0134] Based on the above embodiment, the training module 400 may include:

[0135] an error calculation unit, configured to calculate a square error between the fused compressed representation and the audio compressed representation and use the square error as a reconstruction loss;

[0136] An optimization training unit is used to optimize the training of the initial music understanding model according to the reconstruction loss until a preset convergence condition is met to obtain the music understanding model.

[0137] It should be noted that the order of the modules and units in the above-mentioned music comprehension model training device can be changed without affecting the logic.

[0138] The music comprehension model training device provided by the embodiment of the present invention is applied, and the audio compression representation acquisition module 100 is used to compress the song audio through the initial music comprehension model to obtain an audio compression representation; the audio compression representation includes any one or more of timbre information, emotional information and semantic information; the melody compression representation acquisition module 200 is used to encode the pitch sequence of the song audio through the initial music comprehension model to obtain a melody compression representation; the melody compression representation includes melody information; the fusion module 300 is used to construct a corresponding fused compression representation based on the audio compression representation and the melody compression representation; the training module 400 is used to optimize the initial music comprehension model based on the fused compression representation and the audio compression representation to obtain a music comprehension model. This device processes song audio in pitch sequences, directly stripping away redundant information. The resulting compressed melody representation retains only the melody's pitch trajectory, achieving semantic focus on melodic features and improving the music understanding model's representation of musical melodies. Furthermore, the pitch sequence contains the melody's two-dimensional pitch-time trajectory, and the naturally compressed melody representation also retains the music's inherently strong contextual correlations. Therefore, this music understanding model is more suitable for learning the characteristics of musical melodies. The resulting music understanding model can output more musical representations and better adapt to downstream music tasks.

[0139] The following is an introduction to an audio processing device provided by an embodiment of the present invention. The audio processing device described below and the audio processing method described above can be referenced to each other.

[0140] Please refer to Figure 7 , Figure 7 A schematic structural diagram of an audio processing device provided in an embodiment of the present invention may include:

[0141] Input module 500 is used to input the song audio to be processed into the music understanding model, so that the song audio is compressed and encoded by the music understanding model to obtain a compressed representation containing melody information and audio information; the audio information includes at least one or more of timbre information, emotional information, and semantic information; the music understanding model is obtained using the above-mentioned music understanding model training method;

[0142] The processing module 600 processes the compressed representation based on a music processing task to obtain a processing result; wherein the music processing task includes any one of a song synthesis task, a song classification task or a song recognition task.

[0143] It should be noted that the order of the modules and units in the above audio processing device can be changed without affecting the logic.

[0144] The music understanding model training device provided by an embodiment of the present invention uses an input module 500 to input the song audio to be processed into the music understanding model, so that the music understanding model compresses and encodes the song audio to obtain a compressed representation containing melodic and audio information. The audio information includes at least one or more of timbre, emotion, and semantic information. The music understanding model is obtained using the above-mentioned music understanding model training method. A processing module 600 processes the compressed representation based on a music processing task to obtain a processing result. The music processing task includes any of song synthesis, song classification, or song recognition. As can be seen, music understanding is an important part of downstream music tasks. Therefore, the music understanding model trained using the above-mentioned music understanding model training method processes the song audio to be processed to obtain a compressed representation containing melodic and audio information. This allows the music understanding model to fully understand the melodic characteristics and is more helpful for downstream tasks. By improving the learning of the melodic characteristics of the music understanding model, the present invention makes the music understanding model more musical. When used in downstream tasks, the pitch can be more accurate and the song has a better sense of rhythm, which can improve the user experience of singing synthesis, music synthesis, etc.

[0145] The electronic device provided by the embodiment of the present invention is introduced below. The electronic device described below and the music understanding model training method and / or audio processing method described above can be referenced to each other.

[0146] Please refer to Figure 8 , Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present invention may include:

[0147] Memory 10, for storing computer programs;

[0148] The processor 20 is used to execute a computer program to implement the above-mentioned music understanding model training method and / or audio processing method.

[0149] The memory 10 , the processor 20 , and the communication interface 31 all communicate with each other via the communication bus 32 .

[0150] In the embodiment of the present invention, the memory 10 is used to store one or more programs. The program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:

[0151] The song audio is compressed using the initial music understanding model to obtain an audio compression representation; the audio compression representation includes any one or more of timbre information, emotional information, and semantic information; the pitch sequence of the song audio is encoded using the initial music understanding model to obtain a melody compression representation; the melody compression representation includes melody information; a corresponding fused compression representation is constructed based on the audio compression representation and the melody compression representation; the initial music understanding model is optimized based on the fused compression representation and the audio compression representation to obtain a music understanding model;

[0152] and / or,

[0153] The song audio to be processed is input into the music understanding model, so that the song audio is compressed and encoded by the music understanding model to obtain a compressed representation containing melody information and audio information; the audio information includes at least any one or more of timbre information, emotional information and semantic information; the music understanding model is obtained using the above-mentioned music understanding model training method; the compressed representation is processed based on the music processing task to obtain a processing result; wherein the music processing task includes any one of a song synthesis task, a song classification task or a song recognition task.

[0154] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function, etc.; the data storage area may store data created during use.

[0155] In addition, the memory 10 may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset or an extended set thereof. The operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0156] The processor 20 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic device. The processor 20 may be a microprocessor or any conventional processor. The processor 20 may call a program stored in the memory 10 .

[0157] The communication interface 31 may be an interface of a communication module, used for connecting to other devices or systems.

[0158] Of course, it needs to be explained that Figure 8The structure shown does not constitute a limitation on the electronic device in the embodiment of the present invention. In actual applications, the electronic device may include Figure 8 More or fewer components than shown, or combinations of certain components.

[0159] The computer-readable storage medium provided in an embodiment of the present invention is introduced below. The computer-readable storage medium described below and the music understanding model training method and / or audio processing method described above can be referenced to each other.

[0160] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned music understanding model training method and / or audio processing method.

[0161] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0162] A computer program product provided in an embodiment of the present application is introduced below. The computer program product described below can be referenced with other embodiments described herein.

[0163] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned disclosed music understanding model training method and / or audio processing method.

[0164] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0165] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0166] Finally, it should be noted that, in this document, relationships such as first and second, etc., are used solely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0167] The above is a detailed introduction to a music understanding model training method, audio processing method, music understanding model training device, audio processing device, electronic device, computer-readable storage medium and computer program product provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A music comprehension model training method, characterized in that: include: Compress the song audio using the initial music understanding model to obtain an audio compression representation; The audio compression representation includes any one or more of timbre information, emotional information and semantic information; Encoding the pitch sequence of the song audio using the initial music understanding model to obtain a melody compression representation; The melody compression representation includes melody information; Constructing a corresponding fused compression representation based on the audio compression representation and the melody compression representation; The initial music understanding model is optimized based on the fused compression representation and the audio compression representation to obtain a music understanding model.

2. The music comprehension model training method according to claim 1, wherein The pitch sequence of the song audio is encoded by the initial music understanding model to obtain a melody compression representation, including: Extracting a pitch sequence from the song audio using the initial music understanding model to obtain an initial sequence; The initial sequence is encoded using the initial music understanding model to obtain the melody compression representation.

3. The music comprehension model training method according to claim 2, wherein: Extracting a pitch sequence from the song audio using the initial music understanding model to obtain an initial sequence includes: Segment and pre-process the song audio to obtain a preliminary processing result; Using a multi-branch structure to extract features from the preliminary processing results to obtain fusion features; the fusion features include time domain waveform features and frequency domain Mel spectrum features; Mapping the fused features into a fundamental frequency probability distribution through a fully connected layer, and performing peak detection on the fused features based on the fundamental frequency probability distribution to generate a candidate pitch sequence; The candidate pitch sequence is optimized by using dynamic programming and filtering, and the optimized sequence is quantized to obtain the initial sequence including the start time, end time and pitch value of the note.

4. The music comprehension model training method according to claim 3, wherein: The initial sequence is encoded by the initial music understanding model to obtain the melody compression representation, including: Sort and concatenate the start time, end time, and pitch value of each note in chronological order to obtain a one-dimensional pitch vector; The initial time and the end time in the one-dimensional pitch vector are uniformly normalized, and the pitch value in the one-dimensional pitch vector is normalized to obtain the melody compression representation.

5. The music comprehension model training method according to claim 1, wherein The song audio is compressed using the initial music understanding model to obtain an audio compression representation, including: Sampling the song audio to obtain various sampling points; Input each sampling point into a one-dimensional convolutional layer to perform feature extraction and dimensionality reduction to obtain output features; The output features are input into a Transformer encoder for encoding to obtain the audio compression representation.

6. The music comprehension model training method according to any one of claims 1 to 5, characterized in that: Constructing a corresponding fused compression representation based on the audio compression representation and the melody compression representation, including: splicing the audio compression representation and the melody compression representation to obtain a splicing feature; Quantizing and encoding the splicing features to obtain encoding features; The encoded features are input into a fully connected layer for compression to obtain the fused compressed representation.

7. The music comprehension model training method according to claim 1, wherein The initial music understanding model is optimized based on the fused compression representation and the audio compression representation to obtain a music understanding model, including: Calculating the square error between the fused compressed representation and the audio compressed representation and using it as the reconstruction loss; The initial music understanding model is optimized and trained according to the reconstruction loss until a preset convergence condition is met, thereby obtaining the music understanding model.

8. An audio processing method, characterized in that: include: Inputting the song audio to be processed into a music understanding model, compressing and encoding the song audio through the music understanding model to obtain a compressed representation containing melody information and audio information; the audio information includes at least one or more of timbre information, emotional information, and semantic information; the music understanding model is obtained using the music understanding model optimization method according to any one of claims 1 to 7; Processing the compressed representation based on a music processing task to obtain a processing result; The music processing task includes any one of a song synthesis task, a song classification task or a song recognition task.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which implements the method according to any one of claims 1 to 8 when executed by a processor.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images