A metadata determination method, device and equipment of a music file and a medium

Abstract

The application discloses a metadata determination method, device and equipment of a music file and a medium. The method comprises the following steps: receiving an uploaded music file and extracting an audio signal of the music file; generating audio attribute information of the music file according to the audio signal; wherein the audio attribute information comprises a number of beats per minute and a tonality parameter; determining an audio mutation point according to the audio signal, dividing the audio signal into a plurality of audio signal segments according to the audio mutation point, and generating a music feature evolution vector according to each audio signal segment; determining label attribute information corresponding to the music feature evolution vector, integrating the audio attribute information and the label attribute information, and obtaining metadata of the music file; wherein the label attribute information comprises emotion information, style information and scene information. The technical scheme realizes automatic and standardized extraction of basic audio attributes and deep semantic attributes of the music file, thereby meeting the demand for large-scale extraction of metadata of a large number of music files.

Description

Technical Field

[0001] This application belongs to the field of digital signal processing technology, specifically relating to a method, apparatus, device, and medium for determining metadata of music files. Background Technology

[0002] With the rapid development of digital music technology and streaming platforms, massive amounts of music works are constantly circulating and stored on various platforms. Accurate and complete music metadata can not only improve the management efficiency of music works, but also form an important foundation for realizing intelligent retrieval, classification recommendation, and scene adaptation.

[0003] Currently, mainstream music metadata extraction solutions primarily focus on collecting basic audio parameters such as tempo and tonality. However, deeper semantic information implied in music, such as emotion, style, and applicable scenarios, is difficult to extract automatically using traditional signal processing methods, thus still heavily relying on manual annotation. However, manual annotation has significant drawbacks: firstly, the annotation results are highly subjective, and consistency between different annotators is difficult to guarantee; secondly, the annotation efficiency is extremely low, and the cost is prohibitively high when dealing with massive amounts of music files, making large-scale processing difficult. Summary of the Invention

[0004] This application provides a method, apparatus, device, and medium for determining metadata of music files, with the aim of automating and standardizing the extraction of basic audio attributes and deep semantic attributes of music files, thereby meeting the needs of large-scale extraction of metadata from massive amounts of music files.

[0005] In a first aspect, embodiments of this application provide a method for determining the metadata of a music file, the method comprising: Receive uploaded music files and extract the audio signals from the music files; The audio attribute information of the music file is generated based on the audio signal; wherein, the audio attribute information includes beats per minute and key parameters; Based on the audio signal, determine the audio abrupt change point and divide the audio signal into multiple audio signal segments according to the audio abrupt change point; generate a music feature evolution vector according to each audio signal segment. The label attribute information corresponding to the music feature evolution vector is determined, and the audio attribute information and the label attribute information are integrated to obtain the metadata of the music file; wherein, the label attribute information includes emotion information, style information and scene information.

[0006] Furthermore, the step of generating music feature evolution vectors based on each audio signal segment includes: Extract the music feature vectors corresponding to each audio signal segment; wherein the dimensions of the music feature vectors include beat intensity, short-time energy, and spectral centroid. Calculate the change vector of music features between adjacent audio signal segments; Construct a music feature evolution vector based on the music feature change vector.

[0007] Furthermore, the step of constructing a music feature evolution vector based on the music feature change vector includes: The structural function type of each audio signal segment is determined based on the music feature vector corresponding to each audio signal segment, and the confidence level of each audio signal segment is determined based on the structural function type of each audio signal segment. The structural migration confidence between adjacent audio signal segments is calculated based on the confidence of each audio signal segment, and the structural migration confidence is concatenated with the corresponding music feature change vector to obtain the enhanced music feature change vector. The enhanced music feature change vectors are stacked to obtain the music feature evolution vector.

[0008] Furthermore, determining the confidence level of each audio signal segment based on its structural function type includes: The basic confidence level of each audio signal segment is determined based on the structural function type of each audio signal segment and the pre-constructed mapping relationship between the structural function type and the confidence level. Calculate the feature stability coefficient of each audio signal segment based on the music feature vector corresponding to each audio signal segment; The confidence level of each audio signal segment is obtained by multiplying its base confidence level by its corresponding characteristic stability coefficient.

[0009] Furthermore, the step of calculating the feature stability coefficient of each audio signal segment based on the music feature vector corresponding to each audio signal segment includes: For each of the audio signal segments, calculate the average music feature vector between the music feature vectors corresponding to two adjacent audio signal segments of the audio signal segment; Calculate the Euclidean distance between the average music feature vector and the music feature vector corresponding to the audio signal segment, and determine the feature stability coefficient of the audio signal segment based on the Euclidean distance.

[0010] Furthermore, determining the audio abrupt change point based on the audio signal includes: The audio signal is segmented into frames to obtain multiple audio frames, and the short-time energy change rate and spectral flux of each audio frame are calculated. Audio frames whose short-time energy change rate exceeds a first threshold and whose spectral flux exceeds a second threshold are identified as target audio frames. The starting position of the target audio frame is determined as the audio mutation point.

[0011] Furthermore, determining the label attribute information corresponding to the music feature evolution vector includes: Receive the attached information corresponding to the uploaded music file; wherein, the attached information includes lyrics text and / or album cover image; The music feature evolution vector and the associated information are input into a pre-trained multimodal prediction model to obtain the label attribute information output by the multimodal prediction model.

[0012] Secondly, embodiments of this application provide a metadata determination device for music files, the device comprising: An audio signal extraction module is used to receive uploaded music files and extract the audio signals from the music files. An audio attribute generation module is used to generate audio attribute information of the music file based on the audio signal; wherein, the audio attribute information includes beats per minute and key parameters; An evolution vector generation module is used to determine audio abrupt change points based on the audio signal and divide the audio signal into multiple audio signal segments based on the audio abrupt change points, and generate music feature evolution vectors based on each audio signal segment; The tag attribute generation module is used to determine the tag attribute information corresponding to the music feature evolution vector, integrate the audio attribute information and the tag attribute information to obtain the metadata of the music file; wherein, the tag attribute information includes emotion information, style information and scene information.

[0013] Furthermore, the evolution vector generation module is specifically used for: Extract the music feature vectors corresponding to each audio signal segment; wherein the dimensions of the music feature vectors include beat intensity, short-time energy, and spectral centroid. Calculate the change vector of music features between adjacent audio signal segments; Construct a music feature evolution vector based on the music feature change vector.

[0014] Furthermore, the evolution vector generation module is specifically used for: The structural function type of each audio signal segment is determined based on the music feature vector corresponding to each audio signal segment, and the confidence level of each audio signal segment is determined based on the structural function type of each audio signal segment. The structural migration confidence between adjacent audio signal segments is calculated based on the confidence of each audio signal segment, and the structural migration confidence is concatenated with the corresponding music feature change vector to obtain the enhanced music feature change vector. The enhanced music feature change vectors are stacked to obtain the music feature evolution vector.

[0015] Furthermore, the evolution vector generation module is specifically used for: The basic confidence level of each audio signal segment is determined based on the structural function type of each audio signal segment and the pre-constructed mapping relationship between the structural function type and the confidence level. Calculate the feature stability coefficient of each audio signal segment based on the music feature vector corresponding to each audio signal segment; The confidence level of each audio signal segment is obtained by multiplying its base confidence level by its corresponding characteristic stability coefficient.

[0016] Furthermore, the evolution vector generation module is specifically used for: For each of the audio signal segments, calculate the average music feature vector between the music feature vectors corresponding to two adjacent audio signal segments of the audio signal segment; Calculate the Euclidean distance between the average music feature vector and the music feature vector corresponding to the audio signal segment, and determine the feature stability coefficient of the audio signal segment based on the Euclidean distance.

[0017] Furthermore, the evolution vector generation module is specifically used for: The audio signal is segmented into frames to obtain multiple audio frames, and the short-time energy change rate and spectral flux of each audio frame are calculated. Audio frames whose short-time energy change rate exceeds a first threshold and whose spectral flux exceeds a second threshold are identified as target audio frames. The starting position of the target audio frame is determined as the audio mutation point.

[0018] Furthermore, the tag attribute generation module is specifically used for: Receive the attached information corresponding to the uploaded music file; wherein, the attached information includes lyrics text and / or album cover image; The music feature evolution vector and the associated information are input into a pre-trained multimodal prediction model to obtain the label attribute information output by the multimodal prediction model.

[0019] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method described in the first aspect.

[0020] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the method described in the first aspect.

[0021] In this embodiment, an uploaded music file is received, and its audio signal is extracted. Audio attribute information of the music file is generated based on the audio signal, including beats per minute and tonality parameters. Audio abrupt changes are determined based on the audio signal, and the audio signal is divided into multiple audio signal segments based on these abrupt changes. A music feature evolution vector is generated based on each audio signal segment. Tag attribute information corresponding to the music feature evolution vector is determined, and the audio attribute information and the tag attribute information are integrated to obtain the metadata of the music file. The tag attribute information includes emotion information, style information, and scene information. This method for determining the metadata of a music file automates and standardizes the extraction of basic audio attributes and deep semantic attributes of music files, thereby meeting the large-scale extraction needs of metadata from massive amounts of music files. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a method for determining metadata of a music file provided in an embodiment of this application; Figure 2 This is a flowchart illustrating another method for determining the metadata of a music file provided in an embodiment of this application; Figure 3 This is a flowchart illustrating another method for determining the metadata of a music file provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a metadata determination device for music files provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0024] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0026] The following description, in conjunction with the accompanying drawings, details the method, apparatus, device, and medium for determining the metadata of music files provided in this application, through specific embodiments and application scenarios.

[0027] First, this application applies to scenarios involving the annotation of music metadata during the music storage process. Based on the above usage scenario, it can be understood that the executing entity of this application can be a terminal device with audio processing capabilities.

[0028] Music import refers to the process of importing and storing music files into a music library system. Specifically, a music library system can be a software platform used to store, manage, and retrieve music files, such as a short video soundtrack library, a music copyright material library, and a car music library.

[0029] Figure 1 This is a flowchart illustrating a method for determining metadata of a music file provided in an embodiment of this application. Figure 1 As shown, the specific steps include the following: S101, Receive the uploaded music file and extract the audio signal of the music file.

[0030] Among them, music files can be music content carriers stored in digital formats, including formats such as MP3, FLAC, WAV, AAC, and OGG.

[0031] In one embodiment, the method of receiving uploaded music files can be either receiving music files uploaded by the client through a network interface, or selecting music files stored locally on the terminal device.

[0032] The audio signal can be a discrete digital electrical signal obtained by sampling, quantizing, and encoding continuous analog sound waves.

[0033] In one embodiment, the method for extracting the audio signal from a music file can be to read the music file encapsulation protocol, remove the non-audio data stream from the music file, and then call the decoder to decompress and resample it to restore it to a pulse code modulation signal, which is the audio signal.

[0034] S102, Generate audio attribute information of the music file based on the audio signal; wherein, the audio attribute information includes beats per minute and key parameters.

[0035] The audio attribute information can be a set of parameters describing the underlying acoustic characteristics, which can be directly calculated from the audio signal. Audio attribute information may include beats per minute and tonality parameters.

[0036] Specifically, beats per minute can be the number of beats contained in a minute, used to measure the tempo of music, such as 120 BPM; the tonality parameter can be the tonic center and mode type of music, used to indicate the pitch organization relationship, such as C major and A minor.

[0037] In one embodiment, the method for generating audio attribute information of a music file based on an audio signal can employ a dynamic programming algorithm to extract the beat cycle from the audio signal and convert the beat cycle into beats per minute, and then use the Krumhansl-Schmuckler algorithm to parse the audio signal to obtain tonality parameters. The dynamic programming algorithm can be an algorithm that estimates the tempo and beat position by calculating the optimal path between the beat phase and beat cycle at different time points in the audio signal; the Krumhansl-Schmuckler algorithm can be an algorithm that estimates the tonality center of the music based on the correlation matching between a set of pitch levels and a tonality template.

[0038] The audio attribute information may also include waveforms. Specifically, a waveform can be a two-dimensional graph showing how the amplitude of the audio signal changes over time.

[0039] Accordingly, in one embodiment, the waveform of a music file can be generated by extracting the amplitude value of the audio signal and normalizing it, and then plotting the waveform with time as the horizontal axis and amplitude as the vertical axis.

[0040] S103, determine audio mutation points based on the audio signal and divide the audio signal into multiple audio signal segments based on the audio mutation points, and generate music feature evolution vectors based on each audio signal segment.

[0041] Among them, the audio mutation point can be the location where the acoustic characteristics of the audio signal change significantly on the time axis.

[0042] In one embodiment, the method for determining audio abrupt change points based on audio signals can be to perform frame-by-frame processing on the audio signal, extract the Mel frequency cepstral coefficient features of each audio frame, calculate the cosine distance between the Mel frequency cepstral coefficient features of adjacent audio frames, and determine the starting position of audio frames whose cosine distance exceeds a preset distance threshold as audio abrupt change points.

[0043] In one embodiment, determining the audio abrupt change point based on the audio signal includes: performing frame segmentation processing on the audio signal to obtain multiple audio frames, and calculating the short-time energy change rate and spectral flux of each audio frame; determining the audio frame whose short-time energy change rate exceeds a first threshold and whose spectral flux exceeds a second threshold as the target audio frame; and determining the starting position of the target audio frame as the audio abrupt change point.

[0044] An audio frame can be the smallest processing unit obtained by dividing a continuous audio signal into segments with a fixed time length or number of sampling points.

[0045] In one embodiment, the method of performing frame segmentation on the audio signal to obtain multiple audio frames can be as follows: the audio signal is divided into multiple basic audio frames according to a preset single frame length and a preset frame overlap rate, and a Hanning window is used to perform windowing processing on each basic audio frame to obtain the final multiple audio frames.

[0046] Among them, the short-time energy change rate can be the ratio of the short-time energy of the current audio frame to the short-time energy of the previous audio frame minus 1, which is used to measure the drasticness of energy change; the spectral flux can be the sum of the amplitude spectrum differences between adjacent audio frames in the frequency domain, which is used to measure the degree of change in the spectral shape.

[0047] In one embodiment, the method for calculating the short-time energy change rate and spectral flux of each audio frame can be as follows: calculate the short-time energy of each audio frame, divide the short-time energy of the current audio frame by the short-time energy of the previous audio frame, subtract 1 from the resulting ratio to obtain the short-time energy change rate, perform a fast Fourier transform on each audio frame to obtain the amplitude spectrum, and calculate the sum of the absolute values ​​of the amplitude differences at each frequency point between the current audio frame and the previous audio frame to obtain the spectral flux.

[0048] The target audio frame can be an audio frame in which both the short-time energy change rate and the spectral flux exceed their respective thresholds, indicating that the audio frame may have undergone a sudden change in musical structure or timbre.

[0049] The first threshold can be a lower limit of the short-time energy change rate used to determine whether the short-time energy change is significant, such as 2.0; the second threshold can be a lower limit of the spectral flux used to determine whether the spectral change is significant, such as 0.5.

[0050] If the short-time energy change rate of an audio frame exceeds the first threshold and the spectral flux exceeds the second threshold, it indicates that the audio frame has undergone drastic changes in both energy and spectrum compared to the previous audio frame. It is very likely to be the boundary of a music segment, and therefore the audio frame can be identified as the target audio frame.

[0051] The starting position of the audio frame can be the time point corresponding to the first sampling point of the audio frame. Therefore, the starting position of the target audio frame can be determined as the audio mutation point.

[0052] The advantage of this approach is that by combining abrupt changes in both the energy and frequency domains, the detection of abrupt changes has higher robustness and accuracy compared to using a single feature. It can effectively filter out false detections of musical abrupt changes caused by a single energy change or a single spectral fluctuation, thereby more accurately locating the boundaries of the musical structure.

[0053] Among them, audio signal segmentation can be a continuous audio signal segment between adjacent audio abrupt change points.

[0054] In one embodiment, the method of dividing the audio signal into multiple audio signal segments based on audio abrupt change points can be achieved by using all audio abrupt change points as dividing boundaries and cutting the audio signal sequentially in chronological order, with each cutting result being an audio signal segment.

[0055] Among them, the music feature evolution vector can be a serialized vector composed of the differences or rates of change of music features in time sequence after extracting music features from each audio signal segment.

[0056] In one embodiment, the method of generating music feature evolution vectors based on each audio signal segment can be achieved by extracting the music feature vectors corresponding to each audio signal segment, calculating the music feature change vectors between the music feature vectors corresponding to adjacent audio signal segments, and constructing the music feature evolution vector based on the music feature change vectors.

[0057] S104, determine the tag attribute information corresponding to the music feature evolution vector, integrate the audio attribute information and the tag attribute information to obtain the metadata of the music file; wherein, the tag attribute information includes emotion information, style information and scene information.

[0058] The tag attribute information can be high-level annotation information that describes the semantic attributes of the music content. Tag attribute information can include emotion information, style information, and scene information.

[0059] Specifically, emotional information can be the category of emotions expressed or evoked by the music, such as joy, sadness, excitement, and tranquility; style information can be the genre or art form to which the music belongs, such as rock, classical, electronic, and jazz; and scene information can be the typical environment or activity in which the music is suitable for use or playback, such as sports, studying, parties, and meditation.

[0060] In one embodiment, the label attribute information corresponding to the music feature evolution vector can be determined by inputting the music feature evolution vector into a pre-trained label attribute prediction model to obtain the label attribute information output by the label attribute prediction model. The label attribute prediction model can be a supervised learning model that takes the music feature evolution vector as input and the label attribute information as output; the label attribute prediction model uses the music feature evolution vector with manually labeled label attribute information as input training samples, and uses the cross-entropy loss function between the model's output label attribute information and the manually labeled label attribute information as the optimization objective.

[0061] In one embodiment, determining the tag attribute information corresponding to the music feature evolution vector includes: receiving the ancillary information corresponding to the uploaded music file; wherein the ancillary information includes lyrics text and / or album cover image; inputting the music feature evolution vector and the ancillary information into a pre-trained multimodal prediction model to obtain the tag attribute information output by the multimodal prediction model.

[0062] Additional information can be supplementary descriptive information related to the music content, in addition to the music file itself. This additional information may include lyrics and / or album art.

[0063] Specifically, lyrics can be textual records of the musical performance or descriptions of passages; album cover images can be digital images used for visual presentation when the music is released.

[0064] In one embodiment, the method for receiving the ancillary information corresponding to the uploaded music file can be either receiving the ancillary information uploaded by the client through a network interface, or selecting the ancillary information stored locally on the terminal device.

[0065] The multimodal prediction model can be a deep learning model capable of simultaneously processing audio features (music feature evolution vectors) and non-audio data (lyrics text and / or album cover images, etc.) and outputting semantic labels. Specifically, the multimodal prediction model consists of an audio feature encoder, a text encoder, an image encoder, and a fusion prediction layer. Each encoder extracts deep features of its corresponding modality, and after fusion, the features are output through a fully connected layer to show the probability distribution of labels such as emotion, style, and scene.

[0066] In one embodiment, the method of inputting the music feature evolution vector and its associated information into a pre-trained multimodal prediction model to obtain the label attribute information output by the multimodal prediction model can be as follows: the music feature evolution vector is used as the audio modal input, the lyrics text is used as the text modal input after word segmentation and word embedding, the album cover image is used as the image modal input after scaling and normalization, and the multimodal prediction model outputs the predicted result of the label attribute information after forward inference.

[0067] The advantage of this approach is that by integrating information from audio, text, and image modalities, it can more comprehensively capture the semantic features of music, compensate for the lack of lyric meaning and visual atmosphere in single audio analysis, and improve the prediction accuracy and robustness of emotion, style, and scene labels.

[0068] Metadata can be structured data describing file attributes. In this scheme, the metadata of a music file consists of audio attribute information and tag attribute information.

[0069] In one embodiment, the way to integrate audio attribute information and tag attribute information to obtain the metadata of a music file is to merge the audio attribute information and tag attribute information into JSON format, which is the metadata.

[0070] In this embodiment, an uploaded music file is received, and its audio signal is extracted. Audio attribute information of the music file is generated based on the audio signal, including beats per minute and tonality parameters. Audio abrupt changes are determined based on the audio signal, and the audio signal is divided into multiple audio signal segments based on these abrupt changes. A music feature evolution vector is generated based on each audio signal segment. Tag attribute information corresponding to the music feature evolution vector is determined, and the audio attribute information and the tag attribute information are integrated to obtain the metadata of the music file. The tag attribute information includes emotion information, style information, and scene information. This method for determining the metadata of a music file automates and standardizes the extraction of basic audio attributes and deep semantic attributes of music files, thereby meeting the large-scale extraction needs of metadata from massive amounts of music files.

[0071] Figure 2 This is a flowchart illustrating another method for determining metadata of music files provided in an embodiment of this application. Figure 2 As shown, the specific steps include the following: S201, Receive the uploaded music file and extract the audio signal of the music file.

[0072] S202, Generate audio attribute information of the music file based on the audio signal; wherein, the audio attribute information includes beats per minute and key parameters.

[0073] S203, determine the audio abrupt change point based on the audio signal and divide the audio signal into multiple audio signal segments based on the audio abrupt change point.

[0074] S204, extract the music feature vector corresponding to each audio signal segment; wherein, the dimensions of the music feature vector include beat intensity, short-time energy, and spectral centroid.

[0075] In this context, a music feature vector can be a multidimensional set of numerical values ​​used to quantify the acoustic properties of segments of an audio signal. The dimensions of a music feature vector include beat intensity, short-time energy, and spectral centroid.

[0076] Specifically, beat intensity can be the degree of energy concentration related to rhythm within an audio signal segment, used to indicate the prominence of the beat; short-time energy can be the sum of the squares of the amplitudes of each sampling point within an audio signal segment, used to indicate the loudness energy of the audio signal segment; and spectral centroid can be the centroid frequency position of the spectral energy of an audio signal segment, used to indicate the brightness of the timbre.

[0077] In one embodiment, the method for extracting the music feature vector corresponding to each audio signal segment can be as follows: for each audio signal segment, calculate the spectral flux of that audio signal segment, select the mean of the peak values ​​of the spectral flux as the beat intensity, calculate the sum of squared amplitudes of all sampling points within that audio signal segment as the short-time energy, perform a fast Fourier transform on that audio signal segment to obtain the amplitude spectrum, calculate the amplitude of each frequency point multiplied by the weighted sum of frequencies, and then divide the weighted sum by the sum of amplitudes to obtain the spectral centroid.

[0078] S205, calculate the music feature change vector between the music feature vectors corresponding to adjacent audio signal segments.

[0079] Among them, the music feature change vector can be the difference vector between the music feature vectors of two adjacent audio signal segments.

[0080] In one embodiment, the method for calculating the music feature change vector between music feature vectors corresponding to adjacent audio signal segments can be to subtract the music feature vector of the previous audio signal segment from the music feature vector of the subsequent audio signal segment to obtain the music feature change vector.

[0081] S206, construct a music feature evolution vector based on the music feature change vector.

[0082] In one embodiment, the method for constructing a music feature evolution vector based on the music feature change vector can be by stacking the music feature change vectors to obtain the music feature evolution vector.

[0083] In one embodiment, constructing a music feature evolution vector based on the music feature change vector includes: determining the structural function type of each audio signal segment based on the music feature vector corresponding to each audio signal segment, and determining the confidence level of each audio signal segment based on the structural function type of each audio signal segment; calculating the structural migration confidence level between adjacent audio signal segments based on the confidence level of each audio signal segment, and concatenating the structural migration confidence level with the corresponding music feature change vector to obtain an enhanced music feature change vector; and stacking the enhanced music feature change vectors to obtain a music feature evolution vector.

[0084] Among them, the structural function type can be the role category of the musical structural unit in the whole work, such as verse, chorus, transition section, intro and coda, etc.

[0085] In one embodiment, determining the structural function type of each audio signal segment based on its corresponding music feature vector can be achieved by inputting the music feature vector corresponding to each audio signal segment into a pre-trained structural classifier (e.g., a support vector machine or a lightweight neural network), which then outputs the structural function type of that audio signal segment. For example, a three-layer perceptron can be pre-trained with beat intensity, short-time energy, and spectral centroid as inputs, outputting a probability distribution of five categories (verse, chorus, transition, intro, and outro), and the category with the highest probability is taken as the structural function type of that audio signal segment.

[0086] The confidence level can be the influence weight of the audio signal segment on the final label attribute information prediction result, which is used to represent the importance of the audio signal segment in the semantic inference of the whole music.

[0087] In one embodiment, the confidence level of each audio signal segment based on its structure-function type can be determined by pre-constructing a mapping table between structure-function types and confidence levels. The confidence level corresponding to the current structure-function type can be retrieved from this mapping table. As an example, the pre-constructed mapping table could include: a confidence level of 0.7 for the structure-function type (verse); 0.9 for the structure-function type (chorus); 0.5 for the structure-function type (transition); 0.3 for the structure-function type (intro); and 0.2 for the structure-function type (outro).

[0088] Among them, the structural transfer confidence can be the overall influence weight of the structural change process from the current audio signal segment to the next audio signal segment on the final label attribute information prediction result.

[0089] In one embodiment, the method for calculating the structural migration confidence between adjacent audio signal segments based on the confidence of each audio signal segment can be to use the average confidence of adjacent audio signal segments as the structural migration confidence.

[0090] Among them, the enhanced music feature change vector can be a new vector obtained by splicing the structure transfer confidence as an additional dimension on the basis of the original music feature change vector.

[0091] In one embodiment, the method of concatenating the structural transfer confidence score with the corresponding music feature change vector to obtain the enhanced music feature change vector can be achieved by appending the structural transfer confidence score as the last dimension to the end of the original music feature change vector.

[0092] In one embodiment, the method of stacking enhanced music feature change vectors to obtain music feature evolution vectors can be achieved by arranging the enhanced music feature change vectors corresponding to all adjacent audio signal segments in chronological order to form a two-dimensional matrix, storing the two-dimensional matrix in vector format to obtain the music feature evolution vector.

[0093] The advantage of this approach is that by introducing confidence and structural transfer confidence, the uneven contribution of different musical segments to the semantics of the whole piece, as well as the semantic enhancement or attenuation effect of segment transitions, can be quantified into feature values. This makes the final musical feature evolution vector not only include the changing trend of acoustic features, but also incorporate semantic importance information at the structural level.

[0094] S207, determine the tag attribute information corresponding to the music feature evolution vector, integrate the audio attribute information and the tag attribute information to obtain the metadata of the music file; wherein, the tag attribute information includes emotion information, style information and scene information.

[0095] The advantage of this approach is that by explicitly calculating the music feature vectors and their change vectors in audio signal segments, the dynamic evolution of music is quantified into music feature evolution vectors that can be input into the model. This allows the model to capture the changing patterns of music structure over time, thereby more accurately inferring semantic tags such as emotional progression, style shifts, and scene adaptation.

[0096] Figure 3 This is a flowchart illustrating another method for determining metadata of music files provided in an embodiment of this application. For example... Figure 3 As shown, the specific steps include the following: S301, Receive the uploaded music file and extract the audio signal of the music file.

[0097] S302, Generate audio attribute information of the music file based on the audio signal; wherein, the audio attribute information includes beats per minute and key parameters.

[0098] S303, determine the audio abrupt change point based on the audio signal and divide the audio signal into multiple audio signal segments based on the audio abrupt change point.

[0099] S304, extract the music feature vector corresponding to each audio signal segment; wherein, the dimensions of the music feature vector include beat intensity, short-time energy, and spectral centroid.

[0100] S305, calculate the music feature change vector between the music feature vectors corresponding to adjacent audio signal segments.

[0101] S306, determine the structural function type of each audio signal segment based on the music feature vector corresponding to each audio signal segment.

[0102] S307, determine the basic confidence level of each audio signal segment based on the structural function type of each audio signal segment and the pre-built mapping relationship between the structural function type and the confidence level.

[0103] The mapping relationship between the pre-built structural function types and their confidence levels can be a predefined lookup table of structural function types and their corresponding confidence levels. Specifically, the mapping relationship between the pre-built structural function types and their confidence levels can include: if the structural function type is the verse, the corresponding confidence level is 0.7; if the structural function type is the chorus, the corresponding confidence level is 0.9; if the structural function type is a transition, the corresponding confidence level is 0.5; if the structural function type is an intro, the corresponding confidence level is 0.3; and if the structural function type is an outro, the corresponding confidence level is 0.2.

[0104] The basic confidence level of the audio signal segmentation is the initial confidence level determined solely based on the structural function type.

[0105] In one embodiment, the method for determining the basic confidence level of each audio signal segment based on the structural function type of each audio signal segment and the pre-constructed mapping relationship between structural function type and confidence level can be as follows: using the current structural function type as the query condition, querying within the pre-constructed mapping relationship between structural function type and confidence level, and determining the confidence level in the query result as the basic confidence level.

[0106] S308, calculate the feature stability coefficient of each audio signal segment based on the music feature vector corresponding to each audio signal segment.

[0107] Among them, the feature stability coefficient can be used as an indicator to quantify the consistency of acoustic features within segments of an audio signal. The higher the coefficient, the smaller the feature fluctuations within segments of the audio signal, and the more stable and reliable the information.

[0108] In one embodiment, the characteristic stability coefficient of each audio signal segment can be calculated based on the music feature vector corresponding to each audio signal segment by calculating the standard deviation of the beat intensity, short-time energy, and spectral centroid of all audio frames within the audio signal segment, then taking the average of the three standard deviations, and subtracting the average from 1 to obtain the characteristic stability coefficient.

[0109] In one embodiment, calculating the feature stability coefficient of each audio signal segment based on the music feature vector corresponding to each audio signal segment includes: for each audio signal segment, calculating the average music feature vector between the music feature vectors corresponding to two adjacent audio signal segments of the audio signal segment; calculating the Euclidean distance between the average music feature vector and the music feature vector corresponding to the audio signal segment; and determining the feature stability coefficient of the audio signal segment based on the Euclidean distance.

[0110] The average music feature vector can be obtained by adding the music feature vectors of the previous and next adjacent audio signal segments of the current audio signal segment dimension by dimension and then dividing by 2.

[0111] The Euclidean distance between the average music feature vector and the music feature vector corresponding to the audio signal segment can be considered as the straight-line distance between the average music feature vector and the music feature vector corresponding to the audio signal segment in the feature space.

[0112] In one embodiment, the Euclidean distance between the average music feature vector and the music feature vector corresponding to the audio signal segment can be calculated by subtracting the average music feature vector from the music feature vector corresponding to the audio signal segment, summing the squares of the difference, and then taking the square root, which is the Euclidean distance.

[0113] In one embodiment, the characteristic stability coefficient of an audio signal segment can be determined by mapping the Euclidean distance to the (0,1) interval using the Sigmoid function, and then subtracting the mapped value from 1 to obtain the characteristic stability coefficient.

[0114] The advantage of this approach is that it assesses stability by comparing the consistency of acoustic features between the current audio signal segment and its preceding and following neighbors. This utilizes contextual information, which, compared to relying solely on the independent features of the audio signal segment itself, better reflects the relative smoothness and transition rationality of the audio signal segment in the music, thus providing a more globally consistent basis for confidence adjustment.

[0115] S309, the basic confidence level of each audio signal segment is multiplied by the corresponding characteristic stability coefficient to obtain the confidence level of each audio signal segment.

[0116] S310, calculate the structural migration confidence between adjacent audio signal segments based on the confidence of each audio signal segment, and concatenate the structural migration confidence with the corresponding music feature change vector to obtain the enhanced music feature change vector.

[0117] S311, stack the enhanced music feature change vectors to obtain the music feature evolution vector.

[0118] S312, determine the tag attribute information corresponding to the music feature evolution vector, integrate the audio attribute information and the tag attribute information to obtain the metadata of the music file; wherein, the tag attribute information includes emotion information, style information and scene information.

[0119] The advantage of this scheme is that by dynamically weighting the basic confidence score through feature stability coefficients, the confidence score can reflect both the prior semantic importance of the structural functional type and the consistency of acoustic features within the audio signal segments. This effectively suppresses the interference of noisy audio signal segments on the subsequent structural transfer confidence score and evolution vector construction, thereby improving the robustness and accuracy of the final label attribute prediction.

[0120] Figure 4 This is a schematic diagram of the structure of a metadata determination device for music files provided in an embodiment of this application. Figure 4 As shown, the device includes: The audio signal extraction module 410 is used to receive the uploaded music file and extract the audio signal of the music file; The audio attribute generation module 420 is used to generate audio attribute information of the music file based on the audio signal; wherein, the audio attribute information includes beats per minute and key parameters; The evolution vector generation module 430 is used to determine audio abrupt change points based on the audio signal and divide the audio signal into multiple audio signal segments based on the audio abrupt change points, and generate music feature evolution vectors based on each audio signal segment; The tag attribute generation module 440 is used to determine the tag attribute information corresponding to the music feature evolution vector, integrate the audio attribute information and the tag attribute information to obtain the metadata of the music file; wherein, the tag attribute information includes emotion information, style information and scene information.

[0121] Furthermore, the evolution vector generation module 430 is specifically used for: Extract the music feature vector corresponding to each audio signal segment; wherein the dimensions of the music feature vector include beat intensity, short-time energy, and spectral centroid. Calculate the change vector of music features between adjacent audio signal segments; Construct a music feature evolution vector based on the music feature change vector.

[0122] Furthermore, the evolution vector generation module 430 is specifically used for: The structural function type of each audio signal segment is determined based on the music feature vector corresponding to each audio signal segment, and the confidence level of each audio signal segment is determined based on the structural function type of each audio signal segment. The structural migration confidence between adjacent audio signal segments is calculated based on the confidence of each audio signal segment, and the structural migration confidence is concatenated with the corresponding music feature change vector to obtain the enhanced music feature change vector. The enhanced music feature change vectors are stacked to obtain the music feature evolution vector.

[0123] Furthermore, the evolution vector generation module 430 is specifically used for: The basic confidence level of each audio signal segment is determined based on the structural function type of each audio signal segment and the pre-constructed mapping relationship between the structural function type and the confidence level. Calculate the feature stability coefficient of each audio signal segment based on the music feature vector corresponding to each audio signal segment; The confidence level of each audio signal segment is obtained by multiplying its base confidence level by its corresponding characteristic stability coefficient.

[0124] Furthermore, the evolution vector generation module 430 is specifically used for: For each of the audio signal segments, calculate the average music feature vector between the music feature vectors corresponding to two adjacent audio signal segments of the audio signal segment; Calculate the Euclidean distance between the average music feature vector and the music feature vector corresponding to the audio signal segment, and determine the feature stability coefficient of the audio signal segment based on the Euclidean distance.

[0125] Furthermore, the evolution vector generation module 430 is specifically used for: The audio signal is segmented into frames to obtain multiple audio frames, and the short-time energy change rate and spectral flux of each audio frame are calculated. Audio frames whose short-time energy change rate exceeds a first threshold and whose spectral flux exceeds a second threshold are identified as target audio frames. The starting position of the target audio frame is determined as the audio mutation point.

[0126] Furthermore, the tag attribute generation module 440 is specifically used for: Receive the attached information corresponding to the uploaded music file; wherein, the attached information includes lyrics text and / or album cover image; The music feature evolution vector and the associated information are input into a pre-trained multimodal prediction model to obtain the label attribute information output by the multimodal prediction model.

[0127] In this embodiment, an audio signal extraction module is used to receive an uploaded music file and extract the audio signal from the music file; an audio attribute generation module is used to generate audio attribute information of the music file based on the audio signal; wherein, the audio attribute information includes beats per minute and tonality parameters; an evolution vector generation module is used to determine audio mutation points based on the audio signal and divide the audio signal into multiple audio signal segments based on the audio mutation points, and generate music feature evolution vectors based on each audio signal segment; a tag attribute generation module is used to determine the tag attribute information corresponding to the music feature evolution vectors, integrate the audio attribute information and the tag attribute information to obtain the metadata of the music file; wherein, the tag attribute information includes emotion information, style information, and scene information. The above-mentioned music file metadata determination device realizes the automated and standardized extraction of basic audio attributes and deep semantic attributes of music files, thereby meeting the large-scale extraction needs of metadata from massive music files.

[0128] The metadata determination device for music files in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0129] The metadata determination device for music files in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit the specific operating system.

[0130] The metadata determination device for music files provided in this application embodiment can implement the various processes implemented in the above embodiments. To avoid repetition, it will not be described again here.

[0131] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501, a memory 502, and a program or instructions stored in the memory 502 and executable on the processor 501. When the program or instructions are executed by the processor 501, they implement the various processes of the above-mentioned music file metadata determination embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0132] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0133] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-mentioned music file metadata determination embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0134] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0135] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0137] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0138] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A method for determining the metadata of a music file, characterized in that, The method includes: Receive uploaded music files and extract the audio signals from the music files; The audio attribute information of the music file is generated based on the audio signal; wherein, the audio attribute information includes beats per minute and key parameters; Based on the audio signal, determine the audio abrupt change point and divide the audio signal into multiple audio signal segments according to the audio abrupt change point; generate a music feature evolution vector according to each audio signal segment. The label attribute information corresponding to the music feature evolution vector is determined, and the audio attribute information and the label attribute information are integrated to obtain the metadata of the music file; wherein, the label attribute information includes emotion information, style information and scene information.

2. The method for determining the metadata of a music file according to claim 1, characterized in that, The step of generating music feature evolution vectors based on each audio signal segment includes: Extract the music feature vectors corresponding to each audio signal segment; wherein the dimensions of the music feature vectors include beat intensity, short-time energy, and spectral centroid. Calculate the change vector of music features between adjacent audio signal segments; Construct a music feature evolution vector based on the music feature change vector.

3. The method for determining the metadata of a music file according to claim 2, characterized in that, The step of constructing a music feature evolution vector based on the music feature change vector includes: The structural function type of each audio signal segment is determined based on the music feature vector corresponding to each audio signal segment, and the confidence level of each audio signal segment is determined based on the structural function type of each audio signal segment. The structural migration confidence between adjacent audio signal segments is calculated based on the confidence of each audio signal segment, and the structural migration confidence is concatenated with the corresponding music feature change vector to obtain the enhanced music feature change vector. The enhanced music feature change vectors are stacked to obtain the music feature evolution vector.

4. The method for determining the metadata of a music file according to claim 3, characterized in that, The determination of the confidence level of each audio signal segment based on the structure-function type of each audio signal segment includes: The basic confidence level of each audio signal segment is determined based on the structural function type of each audio signal segment and the pre-constructed mapping relationship between the structural function type and the confidence level. Calculate the feature stability coefficient of each audio signal segment based on the music feature vector corresponding to each audio signal segment; The confidence level of each audio signal segment is obtained by multiplying its base confidence level by its corresponding characteristic stability coefficient.

5. The method for determining the metadata of a music file according to claim 4, characterized in that, The step of calculating the feature stability coefficient of each audio signal segment based on the music feature vector corresponding to each audio signal segment includes: For each of the audio signal segments, calculate the average music feature vector between the music feature vectors corresponding to two adjacent audio signal segments of the audio signal segment; Calculate the Euclidean distance between the average music feature vector and the music feature vector corresponding to the audio signal segment, and determine the feature stability coefficient of the audio signal segment based on the Euclidean distance.

6. The method for determining the metadata of a music file according to any one of claims 1-5, characterized in that, The step of determining the audio abrupt change point based on the audio signal includes: The audio signal is segmented into frames to obtain multiple audio frames, and the short-time energy change rate and spectral flux of each audio frame are calculated. Audio frames whose short-time energy change rate exceeds a first threshold and whose spectral flux exceeds a second threshold are identified as target audio frames. The starting position of the target audio frame is determined as the audio mutation point.

7. The method for determining the metadata of a music file according to any one of claims 1-5, characterized in that, The step of determining the label attribute information corresponding to the music feature evolution vector includes: Receive the attached information corresponding to the uploaded music file; wherein, the attached information includes lyrics text and / or album cover image; The music feature evolution vector and the associated information are input into a pre-trained multimodal prediction model to obtain the label attribute information output by the multimodal prediction model.

8. A device for determining the metadata of a music file, characterized in that, The device includes: An audio signal extraction module is used to receive uploaded music files and extract the audio signals from the music files. An audio attribute generation module is used to generate audio attribute information of the music file based on the audio signal; wherein, the audio attribute information includes beats per minute and key parameters; An evolution vector generation module is used to determine audio abrupt change points based on the audio signal and divide the audio signal into multiple audio signal segments based on the audio abrupt change points, and generate music feature evolution vectors based on each audio signal segment; The tag attribute generation module is used to determine the tag attribute information corresponding to the music feature evolution vector, integrate the audio attribute information and the tag attribute information to obtain the metadata of the music file; wherein, the tag attribute information includes emotion information, style information and scene information.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method for determining the metadata of a music file as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the method for determining the metadata of a music file as described in any one of claims 1-7.

Images