Self-supervised music representation extraction method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a self-supervised music representation extraction method, device, equipment and medium, the method comprises the following steps: carrying out residual vector quantization on a first Mel spectrum to obtain a plurality of discrete tokens, optimizing the initial word segmentation device by utilizing a combined target loss function in a discrete token reconstruction process to obtain a target word segmentation device, performing word segmentation on a second Mel frequency spectrum of the music data set to obtain a plurality of target token sequences, and performing multi-head self-attention and convolution processing on the randomly shielded second Mel frequency spectrum to obtain a plurality of target token sequences; and obtaining a prediction token sequence corresponding to the shielding position, determining cross entropy loss between the prediction token sequence and a target token sequence, optimizing the representation learning model to obtain a target music representation learning model, and extracting music vector representation of the music data set. According to the method, the extraction efficiency and reliability of the music representation are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a self-supervised music representation extraction method, device, equipment and medium. BACKGROUND

[0002] In the field of artificial intelligence-assisted music understanding and analysis, the key challenge lies in how to enable computer models to learn a universal "music language" that contains both semantic information and acoustic features of music. However, the existing mainstream methods have significant shortcomings: on the one hand, methods based on heavy-weight encoders and decoders, although they generate stable tokens, have a huge amount of computation and low efficiency; on the other hand, lightweight methods based on random projection, although they are faster, are unstable in performance and highly dependent on the initial state of random parameters, lacking reliability. These problems limit the learning ability and practical application effect of the model in the unlabeled data environment. Therefore, there is an urgent need for a new method that can efficiently and stably generate high-quality learning targets and effectively learn the universal representation of music.

[0003] In the field of medical health, music is used for emotion regulation, stress relief, and auxiliary treatment of mental health. For example, by analyzing the physiological data of patients and the emotional characteristics of music, technology can automatically recommend music that meets the emotional needs of patients, helping them to alleviate anxiety or depression symptoms. However, the current technology has certain defects: systems based on traditional music analysis methods require a large amount of manually annotated data to effectively learn the influence of different types of music on user emotions, which limits their applicability in the unlabeled data environment. While self-supervised learning methods can make music recommendations without labeled data, due to their large amount of computation and high requirements for model stability and real-time performance in medical applications, the technology often performs unstably in practical applications.

[0004] In the field of financial technology business, music is applied in customer experience optimization and emotion analysis, such as evaluating users' emotional responses during the use of financial products through music analysis, or improving user stickiness through emotion-driven music recommendations. Self-supervised learning technology is used to extract emotional and behavioral features from a large amount of unlabeled user interaction data to achieve more personalized services. However, this application also faces challenges: models based on complex encoders and decoders can ensure a certain degree of accuracy in emotion recognition, but their large amount of computation makes them unable to meet the real-time and fast feedback needs in financial products. On the other hand, lightweight models based on random projection are relatively efficient, but their unstable performance makes it difficult for the model to provide consistent and accurate services in a changing user emotional and financial environment, affecting the overall user experience.

[0005] Therefore, in the current technology, there are problems of low extraction efficiency and poor stability of music representation. SUMMARY

[0006] The application provides a self-supervised music representation extraction method, device, equipment and medium, which mainly aims to solve the problems of low extraction efficiency and poor stability of music representation.

[0007] In a first aspect, to achieve the above object, the application provides a self-supervised music representation extraction method, comprising: obtaining a first mel-frequency spectrum of target music, performing residual vector quantization on the first mel-frequency spectrum by using a preset initial tokenizer to obtain a plurality of discrete tokens; reconstructing the discrete tokens, and optimizing the initial tokenizer by using a combination target loss function in the reconstruction process to obtain a target tokenizer; obtaining a music data set, extracting a second mel-frequency spectrum of the music data set, and tokenizing the second mel-frequency spectrum by using the target tokenizer to obtain a plurality of target token sequences; randomly masking the second mel-frequency spectrum to obtain a masked mel-frequency spectrum; performing multi-head self-attention and convolution processing on the masked mel-frequency spectrum by using a preset representation learning model to obtain a predicted token sequence corresponding to a masking position of the masked mel-frequency spectrum; determining a cross-entropy loss between the predicted token sequence and the target token sequence, optimizing the representation learning model by using the cross-entropy loss to obtain a target music representation learning model, and extracting a music vector representation of the music data set by using the target music representation learning model.

[0008] In a second aspect, the application further provides a self-supervised music representation extraction device, comprising: a residual vector quantization module configured to obtain a first mel-frequency spectrum of target music, perform residual vector quantization on the first mel-frequency spectrum by using a preset initial tokenizer to obtain a plurality of discrete tokens; a tokenizer optimization module configured to reconstruct the discrete tokens, and optimize the initial tokenizer by using a combination target loss function in the reconstruction process to obtain a target tokenizer; a data set tokenization module configured to obtain a music data set, extract a second mel-frequency spectrum of the music data set, and tokenize the second mel-frequency spectrum by using the target tokenizer to obtain a plurality of target token sequences; a spectrum random masking module configured to randomly mask the second mel-frequency spectrum to obtain a masked mel-frequency spectrum; The learning module prediction module is configured to perform multi-head self-attention and convolution processing on the masked mel-spectrogram by using a preset representation learning model to obtain a predicted token sequence corresponding to a masking position of the masked mel-spectrogram. The music representation extraction module is configured to determine a cross-entropy loss between the predicted token sequence and the target token sequence, optimize the representation learning model by using the cross-entropy loss to obtain a target music representation learning model, and extract a music vector representation of the music data set by using the target music representation learning model.

[0009] In a third aspect, the present application further provides an electronic device, which comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the self-supervised music representation extraction method described above.

[0010] In a fourth aspect, the present application further provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is executed by a processor in an electronic device to implement the self-supervised music representation extraction method described above.

[0011] The application can accurately capture key acoustic features in the spectrum by obtaining the first mel spectrum of the target music, using a preset initial tokenizer to perform residual vector quantization on the first mel spectrum to obtain a plurality of discrete tokens, determining the distance between the low-dimensional spectrum feature and the codebook vector, and screening the target embedding vector with the smallest distance, reconstructing the discrete tokens, and using a combination target loss function in the reconstruction process to optimize the initial tokenizer to obtain a target tokenizer. The combination target loss function organically combines the reconstruction accuracy, the stability of the embedding vector and the effectiveness of feature learning to provide a comprehensive optimization direction for the tokenizer, so that the target tokenizer is more efficient and accurate when processing spectrum data. Obtain a music data set, extract a second mel spectrum of the music data set, and use the target tokenizer to tokenize the second mel spectrum to obtain a plurality of target token sequences. The high-dimensional features are converted into discrete target token sequences through residual vector quantization, ensuring compact representation of information, simplifying the data structure, and enhancing the processing efficiency of the model. The second mel spectrum is randomly masked to obtain a masked mel spectrum, which improves the generalization ability and robustness of the model. A preset representation learning model is used to perform multi-head self-attention and convolution processing on the masked mel spectrum to obtain a predicted token sequence corresponding to the masking position of the masked mel spectrum. The cross-entropy loss between the predicted token sequence and the target token sequence is determined, and the representation learning model is optimized using the cross-entropy loss to obtain a target music representation learning model. The cross-entropy loss updates the model parameters through back propagation, optimizing the prediction ability of the model, thereby improving the learning efficiency and accuracy of music information. The target music representation learning model is used to extract music vector representation of the music data set, improving the extraction efficiency and reliability of music representation. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0013] Figure 1 An application environment diagram of a self-supervised music representation extraction method in an embodiment of the present application; Figure 2 A flowchart of a self-supervised music representation extraction method provided by an embodiment of the present application; Figure 3 A flowchart of a residual vector quantization process in a self-supervised music representation extraction method provided by an embodiment of the present application; Figure 4A module schematic diagram of a self-supervised music representation extraction device provided for an embodiment of the present application is provided. Figure 5 A structure schematic diagram of an electronic device for implementing a self-supervised music representation extraction method provided for an embodiment of the present application is provided. Figure 6 Another structure schematic diagram of an electronic device for implementing a self-supervised music representation extraction method provided for an embodiment of the present application is provided.

[0014] The purposes, functional characteristics and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0015] In order to make the person skilled in the art better understand the technical solutions of the present disclosure, and to fully understand and implement the implementation process of the present disclosure how to apply technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all embodiments. The embodiments of the present disclosure and each feature in the embodiments can be combined with each other without conflict, and the technical solutions formed thereby are all within the protection scope of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor shall fall within the scope of protection of the present disclosure.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or equipment including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or equipment.

[0017] The self-supervised music representation extraction method provided in the embodiments of the present application can be executed by software or hardware installed in a terminal device or a server device. The server includes but is not limited to a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be a stand-alone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0018] The self-supervised music representation extraction method provided in the embodiments of the present application can be applied to, for example, Figure 1The application is applied to the application environment of the client and the server. The client communicates with the server through the network. The server can obtain the first mel spectrum of the target music through the client, use the preset initial tokenizer to perform residual vector quantization on the first mel spectrum to obtain a plurality of discrete tokens, determine the distance between the low-dimensional spectral feature and the codebook vector, and filter the target embedding vector with the minimum distance, so as to accurately capture the key acoustic features in the spectrum, reconstruct the discrete tokens, and optimize the initial tokenizer by using the combination target loss function in the reconstruction process to obtain a target tokenizer. The combination target loss function organically combines the reconstruction accuracy, the stability of the embedding vector and the effectiveness of feature learning, provides a comprehensive optimization direction for the tokenizer, makes the target tokenizer more efficient and accurate when processing spectral data, obtains a music data set, extracts the second mel spectrum of the music data set, and uses the target tokenizer to tokenize the second mel spectrum to obtain a plurality of target token sequences. The residual vector quantization converts high-dimensional features into discrete target token sequences, ensures compact representation of information, simplifies the data structure, and enhances the processing efficiency of the model. The second mel spectrum is randomly masked to obtain a masked mel spectrum, improve the generalization ability and robustness of the model, and use a preset representation learning model to perform multi-head self-attention and convolution processing on the masked mel spectrum to obtain a predicted token sequence corresponding to the masking position of the masked mel spectrum. The cross-entropy loss between the predicted token sequence and the target token sequence is determined, and the representation learning model is optimized by using the cross-entropy loss to obtain a target music representation learning model. The cross-entropy loss updates the model parameters through back propagation, optimizes the prediction ability of the model, thereby improving the learning efficiency and accuracy of the music information, and extracts the music vector representation of the music data set by using the target music representation learning model, improves the extraction efficiency and reliability of the music representation. Finally, the music vector representation is output and fed back to the user client. The client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The application will be described in detail below through specific embodiments.

[0019] The following is explained in the specification of the application. The application generates discrete tokens by extracting mel spectrum and performing residual vector quantization, optimizes the tokenizer to reconstruct the token sequence, uses the target tokenizer to process music data and performs random masking, uses the representation learning model to predict the token of the masking position through self-attention and convolution, optimizes the representation learning model, and finally extracts the music vector representation of the music data set.

[0020] Reference Figure 2Fig. 1 is a flowchart of a self-supervised music representation extraction method according to an embodiment of the present application. In this embodiment, the self-supervised music representation extraction method comprises the following steps: S1, obtaining a first mel-frequency spectrum of target music, using a preset initial tokenizer to perform residual vector quantization on the first mel-frequency spectrum to obtain a plurality of discrete tokens.

[0021] In this embodiment of the present application, the initial tokenizer is used to convert the first mel-frequency spectrum into low-dimensional spectral features, all initial embedding vectors are extracted from the codebook of the initial tokenizer, the target embedding vector with the minimum distance between the low-dimensional spectral features and each embedding vector is screened out, and the screened target embedding vector is mapped to the corresponding discrete token, thereby completing the quantization process.

[0022] In the medical health specific scenario, self-supervised learning and tokenization technology can be used to analyze and process patient medical image data, genetic data or electronic health records (EHR). By converting these complex medical data into low-dimensional representations, potential patterns of diseases can be learned from massive unlabeled data efficiently. For example, using the tokenization technology of music signals, electrocardiogram (ECG) signals, MRI images, etc. of patients can be converted into low-dimensional features, and disease prediction and classification can be performed without manual annotation, helping doctors to achieve early disease detection and personalized treatment recommendations.

[0023] In the financial technology specific scenario, self-supervised learning and tokenization technology can be applied to the analysis of financial transaction data and the construction of credit scoring models. For example, by tokenizing and feature learning on large-scale unlabeled transaction data, financial institutions can extract potential credit risk patterns or fraudulent behaviors from users' transaction behaviors, and optimize the risk control model. By converting users' transaction behaviors into discrete "tokens", accurate credit assessment, anomaly detection and risk prediction can be performed on users, thereby improving the accuracy and security of financial products.

[0024] Figure 3 Fig. 2 is a flowchart of the residual vector quantization process in a self-supervised music representation extraction method according to an embodiment of the present application.

[0025] In this embodiment of the present application, the residual vector quantization of the first mel-frequency spectrum using the preset initial tokenizer to obtain a plurality of discrete tokens comprises the following steps: converting the first mel-frequency spectrum into low-dimensional spectral features using the preset initial tokenizer; obtaining a codebook of the initial tokenizer and extracting initial embedding vectors of the codebook; determining the distance between the low-dimensional spectral features and each initial embedding vector, and screening out a target embedding vector corresponding to the minimum distance; Map the selected target embedding vector to a corresponding discrete token.

[0026] In detail, the Mel-spectrum is a sound representation method that simulates the auditory characteristics of the human ear, converting complex sound wave signals into a "spectrum graph" similar to an image, with higher resolution in the low frequency region.

[0027] The initial Mel-RVQ tokenizer maps the original high-dimensional Mel-spectrum data to a lower-dimensional space through linear projection or similar dimension reduction methods, effectively compressing the spectrum information while preserving important acoustic features, providing a more simplified and easily processed feature representation for subsequent quantization steps.

[0028] After obtaining the codebook of the initial tokenizer, extract each embedding vector from it, which represents the high-dimensional feature representation of each discrete symbol in the codebook. The codebook contains a set of pre-defined vectors that have been trained to capture the underlying patterns and regularities in the Mel-spectrum. The initial embedding vectors serve as the basis for the tokenizer and are directly used in subsequent quantization operations, helping to map complex features in the Mel-spectrum to discrete token space.

[0029] Calculate the distance between the low-dimensional spectrum features and each initial embedding vector, usually using Euclidean distance or cosine similarity to measure their similarity. From all embedding vectors, select the target embedding vector with the smallest distance to the low-dimensional spectrum feature. The embedding vector corresponding to this minimum distance is considered to be the vector that best represents the spectrum feature, and is selected as the discrete representation of the feature, preparing for subsequent quantization process.

[0030] Through the lookup table or mapping rules, match the target embedding vector corresponding to the minimum distance with the index in the codebook. The mapping process converts the target embedding vector into a discrete identifier, i.e. token, representing the position of the spectrum feature in the quantization space. These discrete tokens form a series of discrete symbols as the basis for subsequent model learning and prediction.

[0031] By converting high-dimensional spectrum to low-dimensional features and using embedding vectors in the codebook for matching, it helps to significantly reduce the complexity of data while preserving its key information. By determining the distance between low-dimensional spectrum features and codebook vectors and selecting the target embedding vector with the smallest distance, it can accurately capture the key acoustic features in the spectrum, generating discrete tokens, effectively compressing the representation of music data, and providing a more concise and easily processed data representation for subsequent model training, improving learning efficiency and prediction accuracy.

[0032] S2, reconstruct the discrete tokens and optimize the initial tokenizer using the combined target loss function in the reconstruction process to obtain the target tokenizer.

[0033] In the embodiment of the present application, the discrete tokens are decoded, the reconstruction loss, the codebook loss and the commitment loss in the reconstruction process are calculated, and are combined by weighting to obtain a combined target loss function. The parameters of the initial tokenizer are back propagated and optimized using the combined target loss function, the tokenizer parameters are updated, the initial tokenizer is updated based on the optimized parameters to obtain a target tokenizer, and the training and optimization process of the tokenizer is completed.

[0034] In the medical health specific scenario, self-supervised learning and tokenization technology can be applied to the processing of medical images (such as X-ray, MRI or CT scans) and genetic data. By converting these complex medical data into low-dimensional features, potential disease patterns can be automatically learned from massive unlabeled data. For example, by converting a patient's MRI scan into discrete tokens, the type, size and distribution pattern of brain tumors can be identified, and automated disease prediction can be performed to assist doctors in making quick decisions and improve the accuracy and efficiency of early diagnosis.

[0035] In the financial technology specific scenario, self-supervised learning and tokenization technology can be used to analyze customers' transaction behavior, credit card consumption patterns and financial market data. By converting transaction data into discrete tokens, users' consumption preferences, financial status and other potential features can be learned from a large amount of unannotated data. For example, by performing self-supervised learning on customers' bank transaction history, financial institutions can identify potential credit risks, optimize credit review processes, and implement intelligent anti-fraud detection, which helps financial companies improve risk control efficiency, reduce human intervention, and reduce default and fraud risks.

[0036] In the embodiment of the present application, the discrete tokens are decoded, the reconstruction loss, the codebook loss and the commitment loss in the reconstruction process are calculated, and are combined by weighting to obtain a combined target loss function. The parameters of the initial tokenizer are back propagated and optimized using the combined target loss function, the tokenizer parameters are updated, the initial tokenizer is updated based on the optimized parameters to obtain a target tokenizer, and the training and optimization process of the tokenizer is completed. The discrete tokens are decoded to obtain a reconstructed spectrum; The reconstruction loss between the reconstructed spectrum and the first mel spectrum is determined; The codebook loss and the commitment loss are determined according to the target embedding vector; The reconstruction loss, the codebook loss and the commitment loss are summed by weighting to obtain a combined target loss function; The parameters of the initial tokenizer are back propagated and optimized using the combined target loss function to obtain updated tokenizer parameters; The initial tokenizer is updated using the updated tokenizer parameters to obtain a target tokenizer.

[0037] In detail, each discrete token is mapped back to an embedding vector in the codebook, representing a discrete representation of low-dimensional spectral features. The decoder converts the vector representation of discrete tokens back to the spectral space through an inverse transformation, generating a reconstructed spectrum close to the original input. This ensures that key information in the spectrum is preserved as much as possible through discretization and quantization, ensuring similarity between the reconstructed spectrum and the original spectrum.

[0038] The difference between the reconstructed spectrum and the original first mel-spectrum is calculated, and the mean square error (MSE) or other appropriate loss function is used to measure their similarity. By comparing the spectral features of the two, the loss function quantifies the error between the reconstructed spectrum and the original spectrum, reflecting the degree of information loss in the quantization and decoding process, and providing feedback for optimizing the tokenizer, prompting the model to preserve as much key information as possible in the original spectrum during reconstruction. The calculation formula of the reconstruction loss is as follows:

[0039] wherein, denotes the decoder, denotes the target embedding vector, denotes the first mel-spectrum.

[0040] The codebook loss is measured by calculating the difference between the target embedding vector and the corresponding embedding vector in the codebook, ensuring that the target embedding vector matches the vector in the codebook as accurately as possible, thereby optimizing the representation ability of the codebook. The calculation formula is as follows:

[0041] wherein, denotes the normalization function, denotes the embedding vector of the codebook, denotes the low-dimensional spectral feature, () denotes the "stop gradient" operation.

[0042] The commitment loss constrains the behavior of the encoder by measuring the distance between the low-dimensional spectral features output by the encoder and the target embedding vector, prompting the encoder to learn to generate more suitable feature representations for the current codebook. The calculation formula is as follows:

[0043] wherein, denotes the normalization function, denotes the low-dimensional spectral feature, () denotes the "stop gradient" operation, denotes the target embedding vector.

[0044] By minimizing these two losses, the model can optimize the tokenizer's performance while preserving spectral information, improving the accuracy of quantization and reconstruction.

[0045] A suitable weight is assigned to the reconstruction loss, codebook loss, and commitment loss to balance their importance in the optimization process. The reconstruction loss mainly focuses on the similarity between the reconstructed spectrum and the original spectrum, the codebook loss optimizes the quality of the embedding vector, and the commitment loss ensures that the encoder learns a suitable feature representation. By weighting and summing these loss terms, a combined target loss function is obtained, as shown in the following formula:

[0046] wherein, represents the codebook loss, represents the commitment loss, represents the reconstruction loss, represents the weight. Considering the reconstruction accuracy, codebook stability, and feature learning effect of the encoder, the combined target loss function is used to optimize the tokenizer, balancing and improving in various aspects to achieve the best performance.

[0047] The loss between the output of the current model and the target value is calculated, and the gradients of the parameters are calculated according to the loss using the backpropagation algorithm. The initial tokenizer parameters are adjusted according to the calculated gradients using the gradient descent optimization method (such as the Adam optimizer). This optimization process is iterated continuously to gradually reduce the value of the combined target loss function, so that the tokenizer can better fit the training data. The updated tokenizer parameters can effectively improve the quantization accuracy, reconstruction quality, and feature learning effect, thereby improving the overall performance of the tokenizer.

[0048] The parameters obtained through backpropagation and optimization are applied to the weights and biases of the initial tokenizer. After adjusting the parameters of the initial tokenizer, it can better adapt to the feature representation and quantization requirements in the data. The updated tokenizer will more accurately perform Mel spectrum quantization and decoding after optimization, generating higher quality discrete tokens. Through this parameter update, the target tokenizer is obtained, which can more efficiently process spectral data and improve the overall performance and reconstruction accuracy of the model.

[0049] By reconstructing the discrete tokens and optimizing the initial tokenizer with a combined target loss function, the performance of the tokenizer can be significantly improved. By decoding the discrete tokens to generate the reconstructed spectrum and calculating the reconstruction loss between the original mel spectrum, the accuracy of the decoding process is ensured. By combining the codebook loss and commitment loss, the quality of the embedding vector and the matching degree of the encoder feature are further optimized. The combined target loss function with weights organically combines the reconstruction accuracy, the stability of the embedding vector and the effectiveness of feature learning, providing a comprehensive optimization direction for the tokenizer. By backpropagation and optimization of tokenizer parameters, the quantization accuracy and reconstruction quality of the model can be improved, making the target tokenizer more efficient and accurate in processing spectral data, significantly improving the performance of the tokenizer in practical applications and better supporting subsequent model training and task execution.

[0050] S3, obtaining a music data set, extracting a second mel spectrum of the music data set, and tokenizing the second mel spectrum using the target tokenizer to obtain a plurality of target token sequences.

[0051] In the embodiment of the present application, valid segments are selected from the music data set, and segment filtering is used to ensure that only good quality data is used. For each valid music segment, short-time Fourier transform, filtering and dimension reduction are performed to obtain encoded spectral features, reducing the dimension of the spectral data and retaining key information. The encoded spectral features are subjected to residual vector quantization to map to a plurality of target token sequences to generate discrete representations.

[0052] In the specific scenario of medical health, by converting these data into low-dimensional feature representations, disease patterns and prediction markers can be learned from massive unlabeled data. For example, by analyzing a patient's electrocardiogram (ECG) or blood glucose monitoring data, potential health risks such as heart attack or early signs of diabetes can be automatically identified, helping doctors make early diagnosis and personalized treatment decisions, improving the efficiency and accuracy of medical services.

[0053] In the specific scenario of financial technology, for example, by processing a customer's transaction history, credit card consumption data and loan records, the user's financial behavior features can be extracted from massive unlabeled transaction data and risk prediction can be performed. By converting the user's transaction data into discrete "tokens", potential fraudulent behavior or credit risk can be discovered, providing real-time risk assessment and automated credit approval for banks and financial institutions, thereby improving the level of business intelligence and reducing the cost of manual review.

[0054] In the embodiment of the present application, the extraction of the second mel spectrum of the music data set and the tokenization of the second mel spectrum using the target tokenizer to obtain a plurality of target token sequences comprises: performing segment filtering on the music dataset to obtain valid music segments; performing short-time Fourier transform and filtering processing on the valid music segments to obtain second mel-frequency spectrum; performing dimension reduction processing on the second mel-frequency spectrum by using the target segmenter to obtain encoded spectrum features; performing residual vector quantization on the encoded spectrum features to obtain a plurality of target token sequences.

[0055] In detail, the standard for defining valid music segments can include factors such as duration of audio, signal intensity, tone quality, etc. By preset rules or algorithms to scan the entire dataset, the segments that do not meet the standard are removed. For example, audio segments can be filtered according to length, and segments that are too short or too long are removed, or audio segments with too much noise or poor quality are removed, and the remaining valid music segments can be used for subsequent analysis and processing to ensure the quality of the input data and improve the effect of subsequent tasks.

[0056] Performing short-time Fourier transform on the audio segments to convert time-domain signals into frequency-domain signals. The audio signal is decomposed into frequency components in multiple time periods, capturing the spectral changes in the audio, and the obtained spectrum is filtered, usually using a mel filter bank to simulate the frequency response of human hearing, thereby obtaining a mel spectrum. The mel spectrum enhances important frequency information related to audio perception through nonlinear mapping of the frequency axis, and the filtered mel spectrum effectively represents the time-frequency features of the audio signal for subsequent segmentation and analysis.

[0057] The second mel-frequency spectrum is input into the target segmenter, which usually includes an encoder part for mapping the input high-dimensional spectrum data to a lower-dimensional feature space. In the dimension reduction process, the target segmenter retains important information in the mel-frequency spectrum and removes redundant parts by learning embedding vectors and encoding rules, compressing the spectrum data from high-dimensional space into more compact encoded spectrum features, which can more efficiently represent the core information of the audio, providing a simplified but effective data representation for subsequent quantization and modeling.

[0058] The dimension-reduced encoded spectrum features are matched with the embedding vectors in the codebook, and the closest embedding vector is found according to the distance between each encoded spectrum feature and the codebook vector, and the closest embedding vector is used as a representative. The encoded spectrum features are quantized into discrete tokens, which correspond to different embedding vectors in the codebook. Residual vector quantization minimizes the difference between features and codebook vectors to ensure that each quantized feature retains as much key information as possible from the original spectrum, and the resulting plurality of target token sequences effectively represent the spectrum features of the audio.

[0059] The fragment filtering ensures that only valid music fragments are selected, improves the quality of data, avoids noise interference, extracts the time-frequency features of the audio by combining the short-time Fourier transform and the Mel filter, obtains the second Mel spectrum, thereby retaining the frequency information with the most perceptual significance in the audio signal, reduces the dimension of data by using a target word segmenter to perform dimension reduction processing on the Mel spectrum, and converts the high-dimensional features into discrete target token sequences by residual vector quantization, ensuring compact representation of information, simplifying the data structure, and enhancing the processing efficiency of the model, so that subsequent music analysis or generation tasks are more efficient and accurate.

[0060] S4, randomly masking the second Mel spectrum to obtain a masked Mel spectrum.

[0061] In the embodiment of the present application, the masking ratio determines the proportion of the spectrum part to be masked, and the masking method determines the specific operation of masking (such as zero padding or mean padding, etc.). Based on the set masking ratio, the corresponding masking index is generated to indicate which part of the spectrum needs to be masked, and the spectrum region to be masked is selected according to the masking index. According to the masking method, the corresponding replacement processing is performed on the to-be-masked region, and the replaced masking spectrum region and the corresponding masking index are combined to generate a complete masked Mel spectrum.

[0062] In the specific scenario of medical health, the word segmentation and masking technology in audio data can be applied to the processing of patient health data, such as ECG (electrocardiogram) signals or electroencephalogram (EEG) data. By performing time-frequency analysis on the long-term health data of patients, key physiological features (e.g., heart rate changes, brain wave patterns, etc.) are extracted, and these signals are segmented and filtered to generate specific health spectra. Using a self-supervised learning method, the recovery of health data can be simulated in the case of missing data, helping doctors identify early signs of disease or predict disease progression, thereby realizing personalized treatment and intervention.

[0063] In the specific scenario of financial technology, the word segmentation and masking technology in audio data can be used to process a large amount of transaction data or market price data. For example, by performing spectral analysis on user transaction behavior, the user's consumption patterns and financial behavior features are extracted, and the masking technology is used to simulate missing transaction information or historical data, and then through self-supervised learning, future transaction behavior is predicted, which not only improves the accuracy of fraud detection, but also helps financial institutions accurately assess user credit, optimize credit risk control, and adjust investment portfolios or insurance strategies based on real-time data, thereby improving business efficiency and customer experience.

[0064] In the embodiment of the present application, the random masking of the second Mel spectrum to obtain a masked Mel spectrum comprises: obtaining a masking proportion and a masking manner, generating a masking index according to the masking proportion; performing masking region screening on the second mel-spectrum according to the masking index, to obtain a to-be-masked spectral region; performing replacement on the to-be-masked spectral region according to the masking manner, to obtain a masked spectral region; generating a masked mel-spectrum according to the masked spectral region and the corresponding masking index.

[0065] In detail, the proportion of data to be masked is determined, i.e., how much proportion of data needs to be masked. The masking proportion is usually a number less than 1, for example, 0.2, indicating that 20% of the data will be masked. According to the masking manner, how to mask is selected, common manners include replacing data values with zero, mean filling or replacing with random values, etc. According to the determined masking proportion, a masking index is generated, which indicates which data points or regions need to be masked. The masking index can be a binary array, where elements marked as 1 represent regions that need to be masked, and elements marked as 0 represent regions that do not need to be masked.

[0066] Each element in the masking index matches a corresponding position in the mel-spectrum. Where the index value is 1, it indicates that the spectrum at that position needs to be masked, and the to-be-masked region in the mel-spectrum can be located, i.e., those spectrum points or frequency ranges marked as 1 in the masking index. These to-be-masked spectral regions will be extracted and prepared for subsequent replacement operations to complete the simulation of missing information of the audio signal.

[0067] According to the masking manner, appropriate replacement values are selected for filling. For example, if zero filling is used, all values of the to-be-masked spectral region are replaced with zero; if mean filling is used, the mean value of the unmasked part of the region is used to replace the values of the to-be-masked region; if random value filling is used, a random value is assigned to each to-be-masked spectrum point. The replacement process can simulate the loss or absence of spectral data, providing challenges for subsequent self-supervised learning tasks, enabling the model to effectively recover features in the absence of information.

[0068] The masking index is used to determine which spectral regions need to be masked, and these regions are replaced with the specified filling values (such as zero, mean or random value). The masked spectral region is spliced with other unmasked parts of the original mel-spectrum according to the masking index, forming the final masked mel-spectrum, which contains part of the missing information and retains part of the original signal structure, for model training and learning.

[0069] By randomly masking the second mel-frequency spectrum, the information loss in the real world can be effectively simulated, thereby improving the generalization ability and robustness of the model. By determining the masking proportion and masking method, the spectrum data is partially masked, so that the model needs to be trained on incomplete input data, which not only helps the model learn how to recover the missing information, but also forces the model to extract more representative and robust features, avoiding over-reliance on a single frequency pattern, and improving the model's ability to handle unknown data, especially in handling complex and missing information in real environments, and performing more stably and accurately.

[0070] S5, utilizing a preset feature learning model to perform multi-head self-attention and convolution processing on the masked mel-frequency spectrum to obtain a predicted token sequence corresponding to a masking position of the masked mel-frequency spectrum.

[0071] In the embodiment of the present application, the global structure in the mel-frequency spectrum is identified by the feature learning model, the relationship dependency between different parts of the audio signal is captured, the local feature of the mel-frequency spectrum is extracted, and more fine-grained acoustic information is captured. By combining the global semantic, relationship dependency and local acoustic features of the spectrum, a context-aware feature vector is generated, and based on the feature vector, each masking position is predicted by a multi-linear prediction head analysis, and finally the token corresponding to the maximum probability is selected as the prediction result. All predicted tokens are summarized to obtain a complete predicted token sequence, which is used to fill in the masked mel-frequency spectrum region and restore the structure and semantics of the original signal.

[0072] In the medical health specific scenario, it can be applied to the processing and analysis of pathological data or biological signals, such as ECG (electrocardiogram) or EEG (electroencephalogram) signals. By converting the patient's electrocardiogram signal into a spectrum representation, the missing physiological data is simulated using the masking technique to help the model learn to recover and predict the patient's health status in the case of missing data. By analyzing global and local features, the dependency relationship between different electrocardio patterns and brain activity is captured, thereby achieving early warning of arrhythmia, epilepsy and other diseases. In addition, the combination of convolution and self-attention mechanism enables the model to extract local physiological features from time series and identify global patterns, thereby improving the accuracy and robustness of disease prediction.

[0073] In the specific scenario of financial technology, it can be used to process user transaction behavior and financial market data. By converting financial transaction data into spectral features, masking techniques can be used to simulate missing or abnormal transaction data, training models to predict future market trends or user behavior in incomplete or distorted data. By analyzing the global dependencies and local fluctuations of transaction data, the model can capture market trends and investment risks. The self-attention mechanism helps the model extract useful global information from historical data, while the convolution layer is good at identifying local transaction patterns or abnormal behavior, not only improving the accuracy of market prediction, but also improving the ability of the risk control system to identify abnormal transactions and fraudulent behavior.

[0074] In the embodiment of the present application, the multi-head self-attention and convolution processing of the masking mel spectrum by the preset feature learning model obtains the predicted token sequence corresponding to the masking position of the masking mel spectrum, which comprises: performing semantic analysis on the masking mel spectrum by using a preset feature learning model to obtain a global spectrum semantic; capture the relationship dependency of the global spectrum semantic; performing convolution processing on the masking mel spectrum to obtain local acoustic features; combining the global spectrum semantic, the relationship dependency and the local acoustic features to obtain a context-aware feature vector; According to the context-aware feature vector, the multi-linear prediction head analysis is performed on each masking position of the masking mel spectrum to generate a plurality of token probabilities of the masking position; selecting the token corresponding to the maximum token probability as the predicted token of each masking position; The predicted tokens are summarized to obtain a predicted token sequence corresponding to the masking position of the masking mel spectrum.

[0075] In detail, the feature learning model comprises a Conformer encoder architecture and a linear prediction head architecture, wherein the Conformer encoder architecture comprises a feedforward network module, a multi-head self-attention module, a convolution module, and a second feedforward network module; the linear prediction head architecture is composed of N completely independent linear layers.

[0076] The masking mel spectrum is input into the feature learning model, the feedforward network module captures the long-distance dependency between each frequency part in the spectrum, extracts high-level semantic information of the audio signal such as melody, harmony, rhythm pattern, etc., extracts global audio patterns or emotional features from local frequency changes, and understands the overall semantics of the entire music segment or signal. The generated global spectrum semantic features can provide deep context information support for subsequent tasks such as prediction or missing information recovery.

[0077] By characterizing the multi-head self-attention mechanism in the learning model, the learning model can identify the interdependence between different spectral parts in the entire mel-spectrum. This means that even if certain regions of the audio signal are masked or missing, the model can still understand how different frequency components interact based on known contextual information, especially on a long time scale. For example, in a piece of music, the melody at the beginning may have a close relationship with the melody at the end. Through the self-attention mechanism, the model can capture these global structural relationships, helping it accurately infer and fill in the missing parts, and better understand the overall emotion, rhythm, and structural hierarchy of the music.

[0078] By using a convolutional neural network (CNN) to progressively scan local regions in the mel-spectrum, the convolutional layer extracts features from adjacent frequency and time regions in the spectrum using sliding filters, which can capture detailed information in the audio signal, such as short-term rhythm, pitch changes, and local timbre characteristics. For example, in a piece of music, fast drumbeats or specific melody progressions may exhibit significant patterns within local regions. The convolutional layer can identify and enhance these local features, enabling not only an understanding of the overall structure of the audio signal but also precise capture of subtle changes in local audio.

[0079] By fusing the global spectral semantics, relationship dependencies, and local acoustic features together through the second feedforward network module, a comprehensive context-aware feature vector is formed, which can better infer and recover in the presence of missing data or masking, fully understand the content of the audio signal, and improve performance in complex audio tasks.

[0080] The learning model passes the context-aware feature vector to the multi-linear prediction head, which, through processing of the context-aware feature vector, predicts multiple token classes that the position may correspond to. At each masked position, the probability distribution of multiple potential tokens is calculated, resulting in a number of tokens corresponding to each position. Multiple possible predictions about the masked position are generated based on information from the surrounding unmasked regions. This multi-head analysis allows the model to consider multiple possibilities simultaneously, enhancing the diversity and accuracy of the predictions.

[0081] For each masked position, the token with the highest probability is selected as the prediction result for that position. All of these predicted tokens are aggregated to form a complete predicted token sequence, representing the recovery result of the masked portion in the masked mel-spectrum. Not only can the missing spectral information be effectively filled in, but the recovered spectrum can also be ensured to be consistent and accurate in terms of audio structure, enhancing the model's understanding and reconstruction capabilities for audio signals.

[0082] The representation learning model captures long-term structure and patterns in the audio signal through spectral global semantic analysis and relational dependency capture. Local acoustic features such as rhythm and timbre changes are extracted through convolutional processing, allowing local and global information to complement each other. Combining spectral global semantics, relational dependencies, and local features, a context-aware feature vector is generated, providing rich context support for each masking position prediction. Under the analysis of the multi-linear prediction head, the probability of multiple tokens is generated for each masking position, and the token with the maximum probability is selected to ensure the accuracy of the prediction. All predicted tokens are aggregated to obtain the complete recovery result of the masked mel-spectrum, improving the reconstruction ability of the audio signal and ensuring the coherence of the recovered spectrum in terms of quality and structure with the original audio.

[0083] S6, determine the cross-entropy loss between the predicted token sequence and the target token sequence, optimize the representation learning model using the cross-entropy loss to obtain a target music representation learning model, and extract music vector representations of the music data set using the target music representation learning model.

[0084] In the embodiments of the present application, the representation learning model is backpropagated using the cross-entropy loss function to calculate the error between the model prediction and the actual label, and then the model parameters are adjusted. Through multiple iterations of optimization, the model continuously adjusts its internal parameters to minimize the loss function, thereby gradually obtaining an optimized target music representation learning model. Using this model to extract the context representation vector sequence of the second mel-spectrum of the music data set, applying maximum pooling operations to these context representation vector sequences can retain the most significant feature information, thereby generating representative music vector representations.

[0085] In the medical health specific scenario, by extracting the context features of medical images such as X-ray films and CT scan images, the system can identify abnormal areas in the image such as tumors or lesions, thereby assisting doctors in diagnosis. By training a deep learning model and using labeled image data for backpropagation optimization, a medical image representation that can accurately identify lesion areas is obtained. Through maximum pooling operations, key features in the image are extracted, and finally a patient's medical representation is generated for early warning, diagnosis, and personalized treatment recommendations for diseases.

[0086] In the specific scenario of financial technology, by analyzing the user's financial behavior data (such as transaction history, account activity, etc.), the representation learning model can extract the user's behavior characteristics and optimize them to accurately predict the credit risk. By converting user data into vector representation, the potential risk characteristics of each user are captured, and the model is continuously optimized through cross-entropy loss function to improve the accuracy of credit scoring. Using the extracted user behavior vector, financial institutions can provide personalized credit evaluation reports for each user, helping to improve the efficiency and accuracy of lending decisions while reducing the risk of default.

[0087] In the embodiment of the present application, the representation learning model is optimized using the cross-entropy loss to obtain a target music representation learning model, and the music vector representation of the music data set is extracted using the target music representation learning model. The representation learning model is updated using the cross-entropy loss. The representation learning model is iteratively optimized using the updated model parameters to obtain a target music representation learning model. The context representation vector sequence of the second mel-frequency spectrum of the music data set is extracted using the target music representation learning model. The context representation vector sequence is maximally pooled to obtain a music vector representation.

[0088] In detail, the cross-entropy loss is used to evaluate the matching degree between the predicted token sequence and the target token sequence. The cross-entropy loss function formula is as follows:

[0089] Wherein, represents the true probability distribution of the th token in the target token sequence, represents the predicted probability distribution of the th token in the predicted token sequence.

[0090] The difference between the predicted output of the representation learning model and the true label is calculated using the cross-entropy loss function. The difference is passed back to each level of the network through the backpropagation algorithm, and the contribution of each parameter (such as weight and bias) to the loss is calculated step by step, effectively updating the parameters of the model and adjusting the weights of the network, so that the output of the model is closer to the true label. After each backpropagation, the parameters of the model are adjusted according to the gradient descent method to minimize the cross-entropy loss, thereby improving the prediction accuracy of the model. After multiple iterations, the model continuously optimizes the parameters and performs more and more accurately on specific tasks, and finally obtains a target music representation learning model that can effectively extract music features, accurately represent the semantic information of music data, and be applied to music classification, recommendation, etc.

[0091] The second mel-frequency spectrum is input into the target music representation learning model as input, and the model extracts the context features of the music signal according to the internal structure. Through multi-layer convolution and self-attention mechanism, the relationship information in different dimensions such as frequency and time in the mel-frequency spectrum is captured. By gradually analyzing the mel-frequency spectrum, the model can extract the context representation at each time, that is, the vector representation of the context information corresponding to each spectrum frame, which reflects the global features and details of the music signal, and provides accurate input for subsequent tasks such as emotion analysis and music recommendation.

[0092] Max-pooling is to reduce the dimension of features and extract the most important information by selecting the most representative maximum value in a certain window. In the process of extracting music vector representation, max-pooling helps to retain the most significant features in each time period or frequency band, while ignoring the details of fluctuations. The model slides the pooling window in the context representation vector sequence and selects the maximum value for each window, finally obtaining a lower-dimensional vector containing key music features, effectively compressing the information in the original mel-frequency spectrum while retaining the global features of the music.

[0093] Cross-entropy loss updates the model parameters through backpropagation, optimizing the prediction ability of the model, thereby improving the learning efficiency and accuracy of music information. After iterative optimization, not only the understanding ability of music signal is improved, but also the generalization ability in more complex tasks is shown. The context representation vector sequence extracted by the optimized model is further processed by max-pooling to extract the most important music features, reducing redundant information and ensuring that the global features of the music signal are accurately captured, thereby providing high-quality music vector representation for subsequent music classification, recommendation or emotion analysis tasks.

[0094] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0095] As shown in Figure 4 , it is a functional module diagram of a self-supervised music representation extraction device provided by an embodiment of the present application.

[0096] In the embodiment of the present disclosure, a self-supervised music representation extraction device is provided, which corresponds one-to-one with the self-supervised music representation extraction method of the above embodiment. As shown in Figure 4As shown, the self-supervised music representation extraction device 100 can be installed in an electronic device, and according to the implemented functions, the self-supervised music representation extraction device 100 includes a residual vector quantization module 101, a tokenizer optimization module 102, a data set tokenization module 103, a spectrum random masking module 104, a learning module prediction module 105, and a music representation extraction module 106. The detailed description of each functional module is as follows: The residual vector quantization module 101 is configured to obtain a first mel spectrum of target music, perform residual vector quantization on the first mel spectrum by using a preset initial tokenizer, and obtain a plurality of discrete tokens. The tokenizer optimization module 102 is configured to reconstruct the discrete tokens and optimize the initial tokenizer by using a combination target loss function in the reconstruction process to obtain a target tokenizer. The data set tokenization module 103 is configured to obtain a music data set, extract a second mel spectrum of the music data set, and tokenize the second mel spectrum by using the target tokenizer to obtain a plurality of target token sequences. The spectrum random masking module 104 is configured to randomly mask the second mel spectrum to obtain a masked mel spectrum. The learning module prediction module 105 is configured to perform multi-head self-attention and convolution processing on the masked mel spectrum by using a preset representation learning model to obtain a predicted token sequence corresponding to a masking position of the masked mel spectrum. The music representation extraction module 106 is configured to determine a cross-entropy loss between the predicted token sequence and the target token sequence, optimize the representation learning model by using the cross-entropy loss to obtain a target music representation learning model, and extract a music vector representation of the music data set by using the target music representation learning model.

[0097] In an embodiment, the residual vector quantization module 101 performs residual vector quantization on the first mel spectrum by using a preset initial tokenizer to obtain a plurality of discrete tokens, including: Converting the first mel spectrum into a low-dimensional spectrum feature by using the preset initial tokenizer; Obtaining a codebook of the initial tokenizer and extracting an initial embedding vector of the codebook; Determining the distance between the low-dimensional spectrum feature and each initial embedding vector, and screening a target embedding vector corresponding to the minimum distance; Mapping the screened target embedding vector to a corresponding discrete token.

[0098] In an embodiment, the wordpiece optimizer module 102 performs reconstruction on the discrete tokens, and optimizes the initial wordpiece model using a combined target loss function in the reconstruction process to obtain a target wordpiece model, including: decoding the discrete tokens to obtain a reconstructed spectrum; determining a reconstruction loss between the reconstructed spectrum and the first mel spectrum; determining a codebook loss and a commitment loss according to the target embedding vector; performing weighted summation on the reconstruction loss, the codebook loss, and the commitment loss to obtain a combined target loss function; performing backpropagation and optimization on the parameters of the initial wordpiece model using the combined target loss function to obtain updated wordpiece parameters; updating the initial wordpiece model using the updated wordpiece parameters to obtain a target wordpiece model.

[0099] In an embodiment, the dataset wordpiece module 103 performs extraction of the second mel spectrum of the music dataset, and performs wordpiece segmentation on the second mel spectrum using the target wordpiece model to obtain a plurality of target token sequences, including: performing segment filtering on the music dataset to obtain valid music segments; performing short-time Fourier transform and filtering processing on the valid music segments to obtain a second mel spectrum; performing dimensionality reduction processing on the second mel spectrum using the target wordpiece model to obtain encoded spectral features; performing residual vector quantization on the encoded spectral features to obtain a plurality of target token sequences.

[0100] In an embodiment, the spectrum random masking module 104 performs random masking on the second mel spectrum to obtain a masked mel spectrum, including: obtaining a masking ratio and a masking method, and generating a masking index according to the masking ratio; performing masking region screening on the second mel spectrum according to the masking index to obtain a to-be-masked spectral region; replacing the to-be-masked spectral region according to the masking method to obtain a masked spectral region; generating a masked mel spectrum according to the masked spectral region and the corresponding masking index.

[0101] In an embodiment, the learning module prediction module 105 performs multi-head self-attention and convolution processing on the masked mel spectrum using a preset representation learning model to obtain a predicted token sequence corresponding to the masking position of the masked mel spectrum, including: perform semantic analysis on the masked mel-spectrogram by using a preset representation learning model to obtain a global semantic of the spectrum; capture a relationship dependency of the global semantic of the spectrum; perform convolution processing on the masked mel-spectrogram to obtain a local acoustic feature; combine the global semantic of the spectrum, the relationship dependency, and the local acoustic feature to obtain a context-aware feature vector; perform multi-linear prediction head analysis on each masked position of the masked mel-spectrogram according to the context-aware feature vector to generate a plurality of token probabilities of the masked position; select a token corresponding to the maximum token probability as a predicted token of each masked position; aggregate the predicted tokens to obtain a predicted token sequence corresponding to the masked positions of the masked mel-spectrogram.

[0102] In an embodiment, the music representation extraction module 106 performs optimization of the representation learning model by using the cross-entropy loss to obtain a target music representation learning model, and extracts a music vector representation of the music data set by using the target music representation learning model, including: perform back propagation of the representation learning model by using the cross-entropy loss to obtain updated model parameters; perform iterative optimization of the representation learning model by using the updated model parameters to obtain a target music representation learning model; extract a context representation vector sequence of a second mel-spectrogram of the music data set by using the target music representation learning model; perform max-pooling on the context representation vector sequence to obtain a music vector representation.

[0103] In the present application, for a self-supervised music representation extraction device, first, the present application obtains the first mel spectrum of the target music, uses the preset initial tokenizer to perform residual vector quantization on the first mel spectrum, obtains a plurality of discrete tokens, determines the distance between the low-dimensional spectral feature and the codebook vector, and selects the target embedding vector with the smallest distance, which can accurately capture the key acoustic features in the spectrum, reconstructs the discrete tokens, and uses the combination target loss function in the reconstruction process to optimize the initial tokenizer to obtain the target tokenizer. The combination target loss function organically combines the reconstruction accuracy, the stability of the embedding vector and the effectiveness of feature learning, provides a comprehensive optimization direction for the tokenizer, so that the target tokenizer is more efficient and accurate when processing spectral data. Obtain a music data set, extract the second mel spectrum of the music data set, and use the target tokenizer to tokenize the second mel spectrum to obtain a plurality of target token sequences. The residual vector quantization converts high-dimensional features into discrete target token sequences, ensuring compact representation of information, simplifying the data structure, and enhancing model processing efficiency. Then, the second mel spectrum is randomly masked to obtain a masked mel spectrum, which improves the generalization ability and robustness of the model. The preset representation learning model is used to perform multi-head self-attention and convolution processing on the masked mel spectrum to obtain a predicted token sequence corresponding to the masking position of the masked mel spectrum. Determine the cross-entropy loss between the predicted token sequence and the target token sequence. Finally, the representation learning model is optimized using the cross-entropy loss to obtain a target music representation learning model. The cross-entropy loss updates the model parameters through backpropagation, optimizing the prediction ability of the model, thereby improving the learning efficiency and accuracy of music information. The target music representation learning model is used to extract music vector representation of the music data set, improving the extraction efficiency and reliability of music representation. For specific limitations of a self-supervised music representation extraction device, see the limitations of a self-supervised music representation extraction method in the above text, which will not be repeated here. Each module in the above self-supervised music representation extraction device can be realized by software, hardware and their combination. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls and executes the operations corresponding to each module.

[0104] In one embodiment, a computer device can be a server, and its internal structure diagram can be as follows Figure 5As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media, internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external client through the network connection. The computer program is executed by the processor to realize the functions or steps of the self-supervised music representation extraction method server side.

[0105] In one embodiment, a computer device is provided, which can be a client, and its internal structure diagram can be as shown in the figure. Figure 6 As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media, internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external server through the network connection. The computer program is executed by the processor to realize the functions or steps of the self-supervised music representation extraction method client side.

[0106] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the following steps: Obtaining a first mel-frequency spectrum of target music, using a preset initial tokenizer to perform residual vector quantization on the first mel-frequency spectrum to obtain a plurality of discrete tokens; Reconstructing the discrete tokens and optimizing the initial tokenizer using a combination target loss function in the reconstruction process to obtain a target tokenizer; Obtaining a music data set, extracting a second mel-frequency spectrum of the music data set, and using the target tokenizer to tokenize the second mel-frequency spectrum to obtain a plurality of target token sequences; Randomly masking the second mel-frequency spectrum to obtain a masked mel-frequency spectrum; Using a preset representation learning model to perform multi-head self-attention and convolution processing on the masked mel-frequency spectrum to obtain a predicted token sequence corresponding to the masking position of the masked mel-frequency spectrum; Determine a cross-entropy loss between the predicted token sequence and the target token sequence, optimize the representation learning model using the cross-entropy loss to obtain a target music representation learning model, and extract music vector representations of the music data set using the target music representation learning model.

[0107] In several embodiments provided by the present application, it should be understood that the disclosed devices and apparatuses can be implemented in other manners. For example, the above-described system embodiments are merely illustrative. For example, the division of the modules is merely logical function division. In actual implementation, other division manners can be adopted.

[0108] In addition, each function module in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware, or in the form of hardware plus software function modules.

[0109] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims to which they relate.

[0110] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0111] In some embodiments of the present embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium, wherein the computer program is executed by a processor to implement the steps of the method described in the above embodiments.

[0112] The readable storage medium of the present application stores a computer program, and the computer program can realize the following when executed by a processor of an electronic device: Obtain a first mel-frequency spectrum of target music, perform residual vector quantization on the first mel-frequency spectrum using a preset initial tokenizer to obtain a plurality of discrete tokens; Reconstruct the discrete tokens, and optimize the initial tokenizer using a combination target loss function in the reconstruction process to obtain a target tokenizer; Obtain a music data set, extract a second mel-frequency spectrum of the music data set, and tokenize the second mel-frequency spectrum using the target tokenizer to obtain a plurality of target token sequences; randomly mask the second mel-spectrum to obtain a masked mel-spectrum; perform multi-head self-attention and convolution processing on the masked mel-spectrum by using a preset representation learning model to obtain a predicted token sequence corresponding to a masking position of the masked mel-spectrum; determine a cross-entropy loss between the predicted token sequence and the target token sequence, optimize the representation learning model by using the cross-entropy loss to obtain a target music representation learning model, and extract a music vector representation of the music data set by using the target music representation learning model.

[0113] It should be noted that the functions or steps described above with respect to the computer-readable storage medium or the computer device can correspond to the related descriptions of the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0114] The computer-readable storage medium can also store at least one computer executable program / instruction, such as computer readable instructions. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory, etc. The computer-readable storage medium may, for example, include read-only memory (ROM), hard disk, flash memory, etc. For example, the non-transitory computer-readable storage medium can be connected to a computing device such as a computer, and then when the computing device runs the computer readable instructions stored on the computer readable storage medium, the various methods described above can be performed.

[0115] In addition, the computer device can also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and an input / output device (for example, a keyboard, a mouse, a speaker, etc.), etc.

[0116] The processor can communicate with external devices through the I / O bus via wired or wireless networks.

[0117] In one embodiment, the at least one computer executable instruction can also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by the processor to perform the steps of various functions and / or methods described in the embodiments of the present technology.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0119] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0120] In the embodiments provided by the present disclosure, it should be understood that the disclosed apparatus and method can also be implemented in other manners. The embodiments described above are merely exemplary for describing the present disclosure. For example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operation of the apparatus, method and computer program product according to the embodiments of the present disclosure. In this regard, each block in the flowcharts and block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that, in some alternative implementations, the functions noted in the blocks can occur in different orders from those noted in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a special-purpose hardware-based system for implementing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0121] It should be noted that, in the present disclosure, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. Without more limitations, the element limited by the statement "including a" does not exclude the presence of additional same elements in the process, method, article or device including the element.

[0122] The above-described embodiments are merely used to illustrate the technical solutions of the present disclosure, rather than limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure.

[0123] It should be noted that, in the embodiments of the present disclosure, if non-company software tools or components appear, they are only used for example introduction, and do not represent actual use.

Claims

1. A self-supervised music representation extraction method, characterized in that: The method comprises: Obtain a first Mel spectrum of the target music, and perform residual vector quantization on the first Mel spectrum using a preset initial word segmenter to obtain a plurality of discrete tokens; Reconstructing the discrete tokens and optimizing the initial word segmenter using a combined objective loss function in the reconstruction process to obtain a target word segmenter; Acquire a music data set, extract a second mel spectrum of the music data set, and segment the second mel spectrum using the target word segmenter to obtain a plurality of target token sequences; Randomly masking the second Mel spectrum to obtain a masked Mel spectrum; Using a preset representation learning model to perform multi-head self-attention and convolution processing on the masked Mel spectrum, to obtain a prediction token sequence corresponding to the masked position of the masked Mel spectrum; Determine the cross entropy loss between the predicted token sequence and the target token sequence, optimize the representation learning model using the cross entropy loss to obtain a target music representation learning model, and extract the music vector representation of the music data set using the target music representation learning model.

2. The self-supervised music representation extraction method according to claim 1, wherein The first Mel spectrum is subjected to residual vector quantization by using a preset initial word segmenter to obtain a plurality of discrete tokens, including: Using a preset initial word segmenter to convert the first Mel spectrum into a low-dimensional spectrum feature; Obtaining a codebook of the initial word segmenter and extracting an initial embedding vector of the codebook; Determining the distance between the low-dimensional spectral feature and each of the initial embedding vectors, and selecting the target embedding vector corresponding to the minimum distance; The filtered target embedding vector is mapped to the corresponding discrete token.

3. The self-supervised music representation extraction method according to claim 2, wherein: The step of reconstructing the discrete tokens and optimizing the initial word segmenter using a combined objective loss function in the reconstruction process to obtain a target word segmenter includes: Decoding the discrete tokens to obtain a reconstructed spectrum; determining a reconstruction loss between the reconstructed spectrum and the first mel spectrum; determining a codebook loss and a commitment loss according to the target embedding vector; Performing a weighted summation on the reconstruction loss, the codebook loss, and the commitment loss to obtain a combined objective loss function; Backpropagation and optimization of the parameters of the initial word segmenter are performed using the combined objective loss function to obtain updated word segmenter parameters; The initial word segmenter is updated using the updated word segmenter parameters to obtain a target word segmenter.

4. The self-supervised music representation extraction method according to claim 1, wherein: The extracting the second Mel spectrum of the music data set and segmenting the second Mel spectrum using the target word segmenter to obtain a plurality of target token sequences includes: Performing segment filtering on the music data set to obtain valid music segments; Performing short-time Fourier transform and filtering on the valid music clip to obtain a second Mel spectrum; Using the target word segmenter to perform dimensionality reduction processing on the second Mel spectrum to obtain a coded spectrum feature; Perform residual vector quantization on the coding spectrum features to obtain several target token sequences.

5. The self-supervised music representation extraction method according to claim 1, wherein: The randomly masking the second Mel spectrum to obtain a masked Mel spectrum includes: Obtaining a masking ratio and a masking mode, and generating a masking index according to the masking ratio; Perform masking area screening on the second Mel spectrum according to the masking index to obtain a spectrum area to be masked; replacing the to-be-masked spectrum region according to the masking method to obtain a masked spectrum region; A masked Mel spectrum is generated according to the masked spectrum region and the corresponding masking index.

6. The self-supervised music representation extraction method according to claim 1, wherein: The method of performing multi-head self-attention and convolution processing on the masked Mel spectrum using a preset representation learning model to obtain a prediction token sequence corresponding to the masked position of the masked Mel spectrum includes: Performing semantic analysis on the masked Mel spectrum using a preset representation learning model to obtain global semantics of the spectrum; capturing relational dependencies of the global semantics of the spectrum; Performing convolution processing on the masked Mel spectrum to obtain local acoustic features; Combining the spectrum global semantics, the relational dependency, and the local acoustic features to obtain a context-aware feature vector; Performing a multi-linear prediction head analysis on each masked position of the masked Mel spectrum according to the context-aware feature vector to generate a plurality of token probabilities of the masked position; Filtering out a token corresponding to the maximum token probability as a predicted token for each of the masked positions; The prediction tokens are aggregated to obtain a prediction token sequence corresponding to the masked position of the masked Mel spectrum.

7. The self-supervised music representation extraction method according to claim 1, wherein: The method of optimizing the representation learning model by using the cross entropy loss to obtain a target music representation learning model, and extracting a music vector representation of the music data set by using the target music representation learning model, includes: Backpropagating the representation learning model using the cross entropy loss to obtain updated model parameters; Iteratively optimizing the representation learning model using the updated model parameters to obtain a target music representation learning model; Extracting a context representation vector sequence of a second mel-spectrogram of the music dataset using the target music representation learning model; Max pooling is performed on the context representation vector sequence to obtain a music vector representation.

8. A self-supervised music representation extraction device, characterized in that The device comprises: A residual vector quantization module is used to obtain a first Mel spectrum of the target music, and perform residual vector quantization on the first Mel spectrum using a preset initial word segmenter to obtain a plurality of discrete tokens; A word segmenter optimization module is used to reconstruct the discrete tokens and optimize the initial word segmenter using a combined objective loss function in the reconstruction process to obtain a target word segmenter; a data set word segmentation module, configured to obtain a music data set, extract a second mel spectrum of the music data set, and segment the second mel spectrum using the target word segmentor to obtain a plurality of target token sequences; a spectrum random masking module, configured to randomly mask the second Mel spectrum to obtain a masked Mel spectrum; A learning module prediction module is used to perform multi-head self-attention and convolution processing on the masked Mel spectrum using a preset representation learning model to obtain a prediction token sequence corresponding to the masked position of the masked Mel spectrum; A music representation extraction module is used to determine the cross-entropy loss between the predicted token sequence and the target token sequence, optimize the representation learning model using the cross-entropy loss to obtain a target music representation learning model, and extract the music vector representation of the music data set using the target music representation learning model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the self-supervised music representation extraction method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the self-supervised music representation extraction method according to any one of claims 1 to 7 is implemented.

Images