Music structure analysis method, terminal equipment and computer readable storage medium

Abstract

The invention is suitable for the technical field of audio processing, and particularly relates to a music structure analysis method, terminal equipment and a computer readable storage medium. The method comprises the following steps: acquiring first multi-dimensional feature information of a first audio signal corresponding to a first time frame; wherein the first multi-dimensional feature information comprises feature information of different dimensions, and the feature information of different dimensions represents different audio semantic features; performing feature fusion processing according to the first multi-dimensional feature information to obtain first fused feature information; detecting an audio attribute of the first audio signal according to the first fusion feature information; wherein the audio attribute is used for representing that the first audio signal is a refrain or a non-refrain. Through the method, the semantic features of the audio signal can be deeply and fully learned, the audio attribute of the audio signal is detected by using the fused feature information, and the detection precision of the audio attribute can be effectively improved without depending on a lyric text.

Description

Technical Field

[0001] The present application belongs to the field of audio processing technology, and in particular, relates to a music structure analysis method, a terminal device, and a computer-readable storage medium. Background Art

[0002] With the development of artificial intelligence technology, its application scope is becoming more and more extensive. For example, in the field of music, artificial intelligence technology can be used to automatically detect the chorus part of music. Among them, the chorus is usually the most prominent part of the music, which often has high repetitiveness, emotional intensity and auditory appeal. Chorus detection involves both low-level signal processing and high-level pattern recognition technology.

[0003] Current chorus detection technology often relies on the lyrics text information corresponding to the audio, that is, using the lyrics text information as feature information to assist in detecting the audio attributes (chorus or non-chorus). However, for some music without lyrics, the existing technology cannot accurately detect the audio attributes of the music. Summary of the invention

[0004] The embodiments of the present application provide a music structure analysis method, a terminal device, and a computer-readable storage medium, which can effectively improve the detection accuracy of audio attributes without relying on lyrics text.

[0005] In a first aspect, an embodiment of the present application provides a music structure analysis method, comprising:

[0006] Acquire first multidimensional feature information of a first audio signal corresponding to a first time frame; wherein the first multidimensional feature information includes feature information of different dimensions, and the feature information of different dimensions represents different audio semantic features;

[0007] Performing feature fusion processing according to the first multi-dimensional feature information to obtain first fused feature information;

[0008] An audio attribute of the first audio signal is detected according to the first fusion feature information; wherein the audio attribute is used to indicate whether the first audio signal is a chorus or a non-chorus.

[0009] In an embodiment of the present application, by fusing the multi-dimensional feature information of the audio signal, the algorithm can learn the semantic features of the audio signal in depth and more fully; using the fused feature information to detect the audio properties of the audio signal can effectively improve the detection accuracy of the audio properties without relying on the lyrics text.

[0010] In a possible implementation manner of the first aspect, the first multidimensional feature information includes first feature information, second feature information, and third feature information; the first feature information is a rhythm feature of the first audio signal, the second feature information is an emotional feature of the first audio signal, and the third feature information is a frequency domain feature of the first audio signal;

[0011] The performing feature fusion processing according to the first multi-dimensional feature information to obtain first fused feature information includes:

[0012] Performing feature fusion processing on the second feature information and the third feature information to obtain fourth feature information;

[0013] Feature fusion processing is performed according to the fourth feature information and the first feature information to obtain the first fused feature information.

[0014] In an embodiment of the present application, the third feature information is the frequency domain feature of the audio signal, which is equivalent to taking the frequency domain feature of the audio signal as a reference and superimposing the rhythm feature and emotion feature of the audio signal on the frequency domain feature. In this way, different audio semantic features such as audio spectrum, audio rhythm and audio emotion can be integrated to obtain deep audio feature information, thereby providing a reliable data basis for subsequent detection process.

[0015] In a possible implementation manner of the first aspect, performing feature fusion processing according to the second feature information and the third feature information to obtain fourth feature information includes:

[0016] Performing linear transformation processing according to the second feature information to obtain a first vector;

[0017] Performing linear transformation processing according to the third feature information to obtain a second vector;

[0018] The fourth feature information is obtained by performing probability distribution mapping on the first vector and the second vector.

[0019] Through the above method, multi-dimensional feature information can be linearly combined and mapped into probability distribution, providing a reliable data basis for subsequent detection tasks.

[0020] In a possible implementation manner of the first aspect, obtaining first multidimensional feature information of a first audio signal corresponding to a first time frame includes:

[0021] Inputting the first audio signal into a trained first model and outputting first feature information; wherein the first model is used to identify the rhythm feature of the input audio signal;

[0022] Inputting the first audio signal into a trained second model and outputting second feature information; wherein the second model is used to identify the emotional features of the input audio signal;

[0023] The frequency domain features of the first audio signal are identified to obtain third feature information.

[0024] In the embodiment of the present application, the first model and the second model are pre-trained. Since the detection accuracy of the trained first model and the second model meets the standard, the characteristic information of the audio signal can be accurately detected in the subsequent detection process, which helps to improve the accuracy and efficiency of the detection.

[0025] In a possible implementation manner of the first aspect, detecting the audio attribute of the first audio signal according to the first fused feature information includes:

[0026] Performing feature extraction processing according to the first fused feature information to obtain first target feature information;

[0027] Detecting the paragraph category of the first audio signal according to the first target feature information to obtain a first detection value; wherein the first detection value is used to indicate the paragraph category to which the first audio signal belongs;

[0028] Detecting a paragraph boundary of the first audio signal according to the first target feature information to obtain a second detection value; wherein the second detection value is used to indicate a probability of paragraph change of the first audio signal;

[0029] An audio property of the first audio signal is detected according to the first target feature information, the first detection value, and the second detection value.

[0030] Through the above method, the model can accurately identify paragraph categories and paragraph dividing points, and detect the audio properties of the audio signal in combination with the detection values ​​of different detection tasks, which is equivalent to considering the different semantic features in the audio signal, thereby helping to improve the model's detection accuracy for the chorus part.

[0031] In a possible implementation manner of the first aspect, detecting the audio attribute of the first audio signal according to the first target feature information, the first detection value, and the second detection value includes:

[0032] generating an observation sequence according to the first target feature information, the first detection value, and the second detection value;

[0033] Calculating prediction probabilities corresponding to each audio attribute according to the observation sequence; wherein the prediction probability represents a probability of a state of a second audio signal changing to a state of a first audio signal belonging to the audio attribute, the second audio signal comprising an audio signal corresponding to a second time frame, the second time frame being at least one time frame within a preset time window before the first time frame;

[0034] An audio attribute corresponding to a maximum value among the calculated prediction probabilities is determined as the audio attribute of the first audio signal.

[0035] In the above implementation, the audio attribute of the audio signal of the current time frame is detected according to the state change of the audio signal between the previous and next time frames, and the characteristic information of the audio signal of the historical time frame is combined, which is conducive to mining the semantic relationship between the audio signals of the previous and next time frames, thereby improving the detection accuracy. In addition, the preset time window can be set according to the needs, which increases the adaptability of the music structure analysis method.

[0036] In a possible implementation manner of the first aspect, detecting the audio attribute of the first audio signal according to the first fused feature information includes:

[0037] Acquire a trained detection model, where the detection model is used to detect audio properties of the first audio signal;

[0038] The first fusion feature information is input into the trained detection model to obtain the audio attribute of the first audio signal.

[0039] In the above implementation, the trained detection model is used to detect the audio properties of the audio signal, which is conducive to improving the detection efficiency; in addition, since the detection accuracy of the trained detection model meets the standard, detection through the trained detection model is conducive to improving the detection accuracy.

[0040] In a possible implementation manner of the first aspect, obtaining the trained detection model includes:

[0041] Acquiring second multi-dimensional feature information of a sample signal, wherein the audio attribute of the sample signal is known;

[0042] The second multidimensional feature information is input into the detection model, and the audio attribute of the sample signal is output; wherein the detection model includes a feature fusion module and a first network; the feature fusion module is used to perform feature fusion processing according to the input multidimensional feature information to obtain fused feature information; the first network is used to perform feature extraction processing according to the fused feature information output by the feature fusion module to obtain target feature information;

[0043] Calculating the loss value of the detection model according to the audio properties of the sample signal and a multi-task loss function; wherein the multi-task loss function includes a plurality of different loss functions;

[0044] Updating the model parameters of the detection model according to the loss value to obtain the updated detection model;

[0045] Continue to train the updated detection model until the detection accuracy of the detection model reaches a preset threshold, thereby obtaining the trained detection model.

[0046] In the embodiment of the present application, multi-task loss functions share model parameters, which can reduce the number of parameters required for training and reduce computational costs, thereby helping to improve training efficiency and the generalization ability of the model; in addition, since multiple different loss functions are included, the model can simultaneously optimize multiple detection tasks, which is beneficial to the training efficiency of the model; and, by learning multiple detection tasks, the model can better understand the intrinsic structure of the data, which is beneficial to enhancing the detection accuracy of the model.

[0047] In a possible implementation manner of the first aspect, the multi-task loss function includes a first function, a second function, and a third function;

[0048] The first function is used to calculate the degree of difference between the audio attribute output by the detection model and the real audio attribute of the sample signal;

[0049] The second function is used to calculate the probability of paragraph switching of the sample signal;

[0050] The third function is used to calculate the similarity between the feature representations of the sample signal at different time resolutions.

[0051] Through the first function, the model can learn the difference between the chorus and non-chorus, so that the model has the ability to distinguish between the chorus and non-chorus. Through the second function, the model can learn the difference between the two situations where the paragraph is transformed and not transformed, so that the model has the ability to distinguish whether the paragraph is transformed. Since the chorus part usually has multi-time scale repeatability, through the third function, the model can capture feature representations at different time resolutions, such as low-frequency features capture long-term dependencies and high-frequency features capture short-term changes, which is conducive to improving the model's accuracy in detecting the chorus. The above-mentioned multi-task loss function combines paragraph classification, paragraph transformation and feature representations at different time resolutions, which enables the model to accurately identify paragraph categories and paragraph demarcation points, thereby helping to improve the model's detection accuracy for the chorus part.

[0052] In a second aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a music structure analysis method as described in any one of the first aspects above.

[0053] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the music structure analysis method as described in any one of the first aspects above is implemented.

[0054] In a fourth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the music structure analysis method described in any one of the first aspects above.

[0055] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0057] Figure 1 is a schematic diagram of a detection system provided in an embodiment of the present application;

[0058] Figure 2 is a schematic diagram of a detection system provided by another embodiment of the present application;

[0059] Figure 3 is a flowchart of a music structure analysis method provided in an embodiment of the present application;

[0060] Figure 4 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0062] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0063] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0064] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0065] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0066] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0067] With the development of artificial intelligence technology, its application scope is becoming more and more extensive. For example, in the field of music, artificial intelligence technology can be used to automatically detect the chorus part of music. Among them, the chorus is usually the most prominent part of the music, which often has high repetitiveness, emotional intensity and auditory appeal. Chorus detection involves both low-level signal processing and high-level pattern recognition technology.

[0068] Current chorus detection technology often relies on the lyrics text information corresponding to the audio, that is, using the lyrics text information as feature information to assist in detecting the audio attributes (chorus or non-chorus). However, for some music without lyrics, the existing technology cannot accurately detect the audio attributes of the music.

[0069] Based on this, the embodiment of the present application provides a music structure analysis method. In the embodiment of the present application, by fusing the multi-dimensional feature information of the audio signal, the algorithm can learn the semantic features of the audio signal in depth and more fully; the audio attributes of the audio signal are detected by using the fused feature information, which can effectively improve the detection accuracy of the audio attributes without relying on the lyrics text.

[0070] First, the detection system involved in the embodiment of the present application is introduced. Figure 1 , is a schematic diagram of the detection system provided in an embodiment of the present application.

[0071] By way of example and not limitation, Figure 1 As shown, the detection system may include a first model 11 , a second model 12 , an audio processing module 13 , a detection model 14 and a post-processing module 15 .

[0072] The first model 11 is used to identify the rhythm features of the input audio signal. Optionally, the first model 11 may be a neural network model.

[0073] The second model 12 is used to identify the emotional features of the input audio signal. Optionally, the second model 12 may be a neural network model.

[0074] In one embodiment, the first model 11 and the second model 12 may be pre-trained.

[0075] Taking the training of the first model 11 as an example, sample signals are obtained, each sample signal corresponds to real rhythm information; the sample signal is input into the first model, and the predicted rhythm information of the sample signal is output; the similarity between the predicted rhythm information output by the model and the real rhythm information is calculated as the loss value; the model parameters of the first model are adjusted according to the loss value; the cycle is repeated until the first model reaches the preset accuracy, and the trained first model is obtained. In practical applications, the rhythm features of the audio signal can be detected by using the partial network used to extract features in the trained first model.

[0076] Taking the training of the second model 12 as an example, sample signals are obtained, each sample signal corresponds to a real emotion label; the sample signal is input into the second model, and the predicted emotion label of the sample signal is output; the similarity between the predicted emotion label output by the model and the real emotion label is calculated as the loss value; the model parameters of the second model are adjusted according to the loss value; the cycle is repeated until the second model reaches the preset accuracy, and the trained second model is obtained. In practical applications, the emotional features of the audio signal can be detected using the partial network used to extract features in the trained second model.

[0077] In the embodiment of the present application, the first model and the second model are pre-trained. Since the detection accuracy of the trained first model and the second model meets the standard, the characteristic information of the audio signal can be accurately detected in the subsequent detection process, which helps to improve the accuracy and efficiency of the detection.

[0078] The audio processing module 13 is used to identify the frequency domain features of the input audio signal. Optionally, the audio processing module 13 may be encapsulated with an audio processing algorithm, such as a time-frequency conversion algorithm, a Mel filter algorithm, etc., and perform audio processing on the input audio signal through the encapsulated audio processing algorithm.

[0079] The detection model 14 is used to detect the audio properties of the audio signal according to the feature information output by the first model 11 , the second model 12 and the audio processing module 13 .

[0080] In one embodiment, see Figure 2 , is a schematic diagram of a detection system provided in another embodiment of the present application. As an example and not a limitation, Figure 2 As shown, the detection model 14 may include a feature fusion module 141 , a first network 142 , a first detection head 143 and a second detection head 144 .

[0081] The feature fusion module 141 is used to perform feature fusion processing on the feature information output by the first model 11, the second model 12 and the audio processing module 13 to obtain fused feature information. Optionally, the feature fusion module 14 may encapsulate a feature fusion algorithm based on a cross attention mechanism.

[0082] The first network 142 is used to perform feature extraction processing according to the fused feature information output by the feature fusion module 141 to obtain target feature information.

[0083] The first detection head 143 is used to detect the paragraph category of the audio signal according to the target feature information output by the first network 142 .

[0084] The second detection head 144 is used to detect paragraph boundaries of the audio signal according to the target feature information output by the first network 142 .

[0085] In one embodiment, the detection model 14 may be pre-trained. The training process may include:

[0086] Acquire second multi-dimensional feature information of the sample signal; the audio attribute of the sample signal is known;

[0087] Inputting the second multi-dimensional feature information into the detection model and outputting the audio attribute of the sample signal;

[0088] Calculating a loss value of the detection model according to the audio properties of the sample signal and a multi-task loss function; wherein the multi-task loss function includes a plurality of different loss functions;

[0089] Update the model parameters of the detection model according to the loss value to obtain an updated detection model;

[0090] Continue to train the updated detection model until the detection accuracy of the detection model reaches a preset threshold, thereby obtaining a trained detection model.

[0091] In the embodiment of the present application, multi-task loss functions share model parameters, which can reduce the number of parameters required for training and reduce computational costs, thereby helping to improve training efficiency and the generalization ability of the model; in addition, since multiple different loss functions are included, the model can simultaneously optimize multiple detection tasks, which is beneficial to the training efficiency of the model; and, by learning multiple detection tasks, the model can better understand the intrinsic structure of the data, which is beneficial to enhancing the detection accuracy of the model.

[0092] In one implementation, the multi-task loss function includes a first function, a second function, and a third function.

[0093] The first function is used to calculate the degree of difference between the audio attribute output by the detection model and the real audio attribute of the sample signal.

[0094] Optionally, the first function may use a cross entropy loss function to constrain the chorus and non-chorus.

[0095] For example, the cross entropy loss function as the first function can be

[0096]

[0097] Among them, y i and Represent the true label and the predicted label output by the detection model respectively, N represents the number of categories of audio attributes, and in the embodiment of the present application, N=2 (including chorus and non-chorus).

[0098] Through the first function, the model can learn the difference between the chorus and non-chorus, so that the model has the ability to distinguish between the chorus and non-chorus.

[0099] The second function is used to calculate the probability of paragraph switching occurring in the sample signal.

[0100] Optionally, the second function may use a cross entropy loss function to identify the probability of paragraph switching occurring in each time frame.

[0101] For example, the cross entropy loss function as the second function can be:

[0102]

[0103] Among them, a i and They represent the true label and the predicted label output by the detection model respectively, N represents the transformation category, and in the embodiment of the present application, N=2 (including transformation and non-transformation) can be used.

[0104] Through the second function, the model can learn the difference between the cases where a paragraph is transformed and when it is not transformed, thereby enabling the model to distinguish whether a paragraph has been transformed.

[0105] The third function is used to calculate the similarity between the feature representations of the sample signal at different time resolutions.

[0106] Optionally, the third function may use a cosine similarity function to identify the similarity between feature representations at different time resolutions.

[0107] For example, the cosine similarity function as the third function may be:

[0108]

[0109] Among them, U i and V i are the two feature vectors corresponding to different time frames at the i-th resolution (such as the target feature information output by the first network), K represents the number of resolutions used, y represents whether the two feature vectors belong to the same paragraph (1 for yes, 0 for no), cos() is the cosine similarity function, and margin is the control threshold.

[0110] Since the chorus part usually has multi-time scale repetitiveness, the third function enables the model to capture feature representations at different time resolutions. For example, low-frequency features capture long-term dependencies, and high-frequency features capture short-term changes, which helps to improve the accuracy of the model in detecting the chorus.

[0111] The above-mentioned multi-task loss function combines paragraph classification, paragraph transformation and feature representation at different time resolutions, which enables the model to accurately identify paragraph categories and paragraph dividing points, thereby helping to improve the model's detection accuracy for the chorus part.

[0112] The post-processing module 15 is used to detect the audio attribute of the audio signal according to the detection value output by the first detection head 143, the detection value output by the second detection head 144 and the target feature information output by the first network 142. Optionally, the post-processing module 15 may encapsulate a post-processing method based on a hidden Markov model.

[0113] It should be noted that the various models or modules in the above detection system are only examples, and more or fewer models / modules may be provided in actual applications to implement the same technical solution. The embodiments of the present application do not specifically limit this.

[0114] The method provided in the embodiment of the present application can be applied to an intelligent device, which may include a terminal device and a driving device, a smart car, a robot and other devices. The driving device may be, for example, a vehicle. The terminal device may be a server, a computer, a mobile phone, a tablet and other devices capable of audio attribute analysis.

[0115] The following describes the music structure analysis method provided by the present application. Figure 3 , is a flowchart of a music structure analysis method provided in an embodiment of the present application. As an example but not a limitation, the method may include the following steps:

[0116] S301: Acquire first multi-dimensional feature information of a first audio signal corresponding to a first time frame.

[0117] Among them, feature information of different dimensions in the first multi-dimensional feature information represents different audio semantic features.

[0118] It should be noted that the first audio signal in the embodiment of the present application may refer to a signal corresponding to a time frame in a certain music. The first audio signal may be a discrete signal or may include multiple continuous signals.

[0119] In one embodiment, the first multidimensional feature information includes first feature information, second feature information and third feature information, wherein the first feature information is a rhythm feature of the first audio signal, the second feature information is an emotional feature of the first audio signal, and the third feature information is a frequency domain feature of the first audio signal.

[0120] based on Figure 1 In the detection system shown, in one implementation, an implementation of S301 may include:

[0121] Inputting the first audio signal into the first model and outputting first feature information;

[0122] Inputting the first audio signal into the second model and outputting second feature information;

[0123] The first audio signal is input into the audio processing module to identify the frequency domain characteristics of the first audio signal to obtain third characteristic information.

[0124] In the embodiment of the present application, the first model and the second model are pre-trained. Since the detection accuracy of the trained first model and the second model meets the standard, the characteristic information of the audio signal can be accurately detected in the subsequent detection process, which helps to improve the accuracy and efficiency of the detection.

[0125] In the embodiment of the present application, the first model and the second model may be obtained by the smart device from other devices (such as the cloud), or the smart device may have the first model and the second model. Alternatively, the smart device sends the first audio signal to the device having the first model and / or the second model respectively, and then the device having the first model and / or the second model obtains the first feature information and / or the second feature information using the first model and / or the second model and sends it to the smart device, which is not limited in the embodiment of the present application.

[0126] Optionally, the process of the audio processing module identifying the frequency domain feature of the first audio signal may include:

[0127] Performing time-frequency transformation processing on the first audio signal to obtain a first frequency domain signal corresponding to the first audio signal;

[0128] Mel filtering is performed on the first frequency domain signal to obtain third feature information.

[0129] In the Mel filter processing, the Mel filter bank is applied to the power spectrum of the signal, the energy in each filter is added, and the logarithm of the energy of all filter banks is discrete cosine transformed (DCT), and finally the parameters reflecting the speech characteristics are extracted.

[0130] The signal processed by Mel filtering can better reflect the characteristics of the human auditory system, thereby improving the efficiency and accuracy of audio detection.

[0131] S302: Perform feature fusion processing according to the first multi-dimensional feature information to obtain first fused feature information.

[0132] In one embodiment, S302 may include:

[0133] Performing feature fusion processing on the second feature information and the third feature information to obtain fourth feature information;

[0134] The fourth feature information and the first feature information are subjected to feature fusion processing to obtain first fused feature information.

[0135] It is understandable that in actual applications, another embodiment is to first perform feature fusion processing on the first feature information and the third feature information to obtain the fifth feature information; and then perform feature fusion processing on the fifth feature information and the first feature information to obtain the first fused feature information.

[0136] In an embodiment of the present application, the third feature information is the frequency domain feature of the audio signal, which is equivalent to taking the frequency domain feature of the audio signal as a reference and superimposing the rhythm feature and emotion feature of the audio signal on the frequency domain feature. In this way, different audio semantic features such as audio spectrum, audio rhythm and audio emotion can be integrated to obtain deep audio feature information, thereby providing a reliable data basis for subsequent detection process.

[0137] Taking the fusion of the second feature information and the third feature information as an example, in one implementation, the feature fusion processing method may include: concatenating the second feature information and the third feature information into a feature vector.

[0138] In another implementation, the feature fusion processing method may further include: performing weighted summation of corresponding elements in the second feature information and the third feature information.

[0139] In another implementation, the feature fusion processing method may include:

[0140] Performing linear transformation processing according to the second feature information to obtain a first vector;

[0141] Performing linear transformation processing according to the third feature information to obtain a second vector;

[0142] The fourth feature information is obtained by mapping the probability distribution of the first vector and the second vector.

[0143] Exemplarily, feature fusion processing can be performed using the following formula:

[0144]

[0145] in, is the second characteristic information, is the third feature information, W Q and W K is a linear transformation parameter, Q is a first vector, K and V are second vectors, and softmax() is a function used to map probability distribution.

[0146] It should be noted that, in practical applications, the sigmoid function may be used instead of the sofmax function, and the embodiment of the present application does not specifically limit the function used to implement the mapping of the probability distribution.

[0147] It can be understood that the process of performing feature fusion processing on the fourth feature information and the first feature information is based on the same principle as the process of performing feature fusion processing on the second feature information and the third feature information. For details, please refer to the process of performing feature fusion processing on the second feature information and the third feature information, which will not be repeated here.

[0148] based on Figure 2 In the detection system shown, one implementation of S302 is to input the first multi-dimensional feature information into the feature fusion module 141 and output the first fused feature information. It can be understood that in this implementation, the feature fusion module 141 encapsulates an algorithm for implementing the above feature fusion processing method.

[0149] Through the above method, multi-dimensional feature information can be linearly combined and mapped into probability distribution. Compared with the simple superposition fusion method, this method can better reflect the relationship between feature information and provide a reliable data basis for subsequent detection tasks.

[0150] S303: Detect an audio attribute of the first audio signal according to the first fused feature information.

[0151] Among them, the audio attributes in the embodiment of the present application may include two categories, which are respectively used to indicate that the first audio signal is a chorus or a non-chorus.

[0152] In one embodiment, S303 may include: detecting an audio attribute of the first audio signal according to the first fused feature information.

[0153] In another embodiment, S303 may include:

[0154] Performing feature extraction processing according to the first fused feature information to obtain first target feature information;

[0155] Detecting the paragraph category of the first audio signal according to the first target feature information to obtain a first detection value; wherein the first detection value is used to indicate the paragraph category to which the first audio signal belongs;

[0156] Detecting the paragraph boundary of the first audio signal according to the first target feature information to obtain a second detection value; wherein the second detection value is used to indicate the probability of paragraph change of the first audio signal;

[0157] An audio property of the first audio signal is detected according to the first target feature information, the first detection value, and the second detection value.

[0158] Compared with the method of directly detecting audio attributes based on the first fusion information, the above method enables the model to accurately identify paragraph categories and paragraph dividing points, and detect the audio attributes of the audio signal in combination with the detection values ​​of different detection tasks, which is equivalent to considering different semantic features in the audio signal, thereby helping to improve the model's detection accuracy for the chorus part.

[0159] based on Figure 2The detection system shown in the figure can be implemented by inputting the first fusion feature information into the first network 142 to output the first target feature information; inputting the first target feature information into the first detection head 143 to output the first detection value; inputting the first target feature information into the second detection head 144 to output the second detection value; inputting the first target feature information, the first detection value and the second detection value into the post-processing module 15 to output the audio attributes of the first audio signal.

[0160] Optionally, the first network 142 may use an embedding layer. In practical applications, the structure of the embedding layer may be set according to requirements to achieve different functions.

[0161] For example, the embedding layer can convert high-dimensional sparse data into a low-dimensional dense vector representation through matrix multiplication. By performing feature extraction processing on the first fused feature information through such an embedding layer, the dimension of the obtained first target feature information is reduced, which enables the detection head to better understand the input data and helps reduce the subsequent calculation amount.

[0162] For another example, the embedding layer can extract high-dimensional features based on low-dimensional data. By performing feature extraction processing on the first fused feature information through such an embedding layer, the obtained first target feature information can express richer audio features, so that the detection head can better capture the semantic information of the audio, which is conducive to improving the detection accuracy.

[0163] In one embodiment, the processing steps of the post-processing module 15 may include:

[0164] generating an observation sequence according to the first target feature information, the first detection value, and the second detection value;

[0165] Calculating the prediction probability corresponding to each audio attribute according to the observation sequence; wherein the prediction probability represents the probability of a state of the second audio signal changing to a state of the first audio signal belonging to the audio attribute, the second audio signal includes an audio signal corresponding to a second time frame, and the second time frame is at least one time frame within a preset time window before the first time frame;

[0166] The audio attribute corresponding to the maximum value among the calculated prediction probabilities is determined as the audio attribute of the first audio signal.

[0167] In the above implementation, the audio attribute of the audio signal of the current time frame is detected according to the state change of the audio signal between the previous and next time frames, and the characteristic information of the audio signal of the historical time frame is combined, which is conducive to mining the semantic relationship between the audio signals of the previous and next time frames, thereby improving the detection accuracy. In addition, the preset time window can be set according to the needs, which increases the adaptability of the music structure analysis method.

[0168] Optionally, the post-processing module 15 may detect the audio attribute of the first audio signal by a post-processing method based on a hidden Markov model.

[0169] Exemplarily, the recursive formula of the forward algorithm of the hidden Markov model is:

[0170]

[0171] Among them, α t (i)=P(o1,o2,…,p t ,q t =S i |λ) is the hidden layer state at time t, S i , and observe the characteristic sequence o1,o2,…,o t The probability of t represents the observation sequence corresponding to time t. The preset parameters of the hidden Markov model include A and B, A={a ij} is the state transition probability matrix, B={b j (o t )} is the observation probability matrix.

[0172] In the embodiment of the present application, six hidden layer states can be set, including non-pre-chorus, non-chorus, non-post-chorus, pre-chorus, chorus, post-chorus, M=6 in the above formula.

[0173] The above formula can be used to calculate the probability value corresponding to each hidden layer state, that is, the predicted probability corresponding to each audio attribute.

[0174] Since the hidden Markov model can abstract the decision-making problem into a series of state and action transfer processes, and can simplify complex decision-making problems by establishing a state transfer probability matrix, the post-processing algorithm based on the hidden Markov model is conducive to simplifying the complexity of the music structure analysis method. In addition, since the hidden Markov model focuses on the change process of the previous and next states, the post-processing algorithm based on the hidden Markov model is conducive to mining the semantic relationship between the audio signals of the previous and next time frames, thereby improving the detection accuracy.

[0175] In an embodiment of the present application, by fusing the multi-dimensional feature information of the audio signal, the algorithm can learn the semantic features of the audio signal in depth and more fully; using the fused feature information to detect the audio properties of the audio signal can effectively improve the detection accuracy of the audio properties without relying on the lyrics text.

[0176] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0177] Figure 4 Schematic diagram of the structure of the terminal device provided in the embodiment of the present application. Figure 4 As shown, the terminal device 4 of this embodiment includes: at least one processor 40 ( Figure 4 Only one is shown in the figure) a processor, a memory 41, and a computer program 42 stored in the memory 41 and executable on the at least one processor 40, wherein the processor 40 implements the steps in any of the above-mentioned music structure analysis method embodiments when executing the computer program 42.

[0178] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 4 It is only an example of the terminal device 4 and does not constitute a limitation on the terminal device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0179] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0180] In some embodiments, the memory 41 may be an internal storage unit of the terminal device 4, such as a hard disk or memory of the terminal device 4. In other embodiments, the memory 41 may also be an external storage device of the terminal device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the terminal device 4. The memory 41 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 41 may also be used to temporarily store data that has been output or is to be output.

[0181] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0182] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0184] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0185] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0186] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A music structure analysis method, characterized in that: include: Acquire first multidimensional feature information of a first audio signal corresponding to a first time frame; wherein the first multidimensional feature information includes feature information of different dimensions, and the feature information of different dimensions represents different audio semantic features; Performing feature fusion processing according to the first multi-dimensional feature information to obtain first fused feature information; An audio attribute of the first audio signal is detected according to the first fusion feature information; wherein the audio attribute is used to indicate whether the first audio signal is a chorus or a non-chorus.

2. The music structure analysis method according to claim 1, characterized in that: The first multi-dimensional feature information includes first feature information, second feature information and third feature information; the first feature information is a rhythm feature of the first audio signal, the second feature information is an emotional feature of the first audio signal, and the third feature information is a frequency domain feature of the first audio signal; The performing feature fusion processing according to the first multi-dimensional feature information to obtain first fused feature information includes: Performing feature fusion processing on the second feature information and the third feature information to obtain fourth feature information; Feature fusion processing is performed according to the fourth feature information and the first feature information to obtain the first fused feature information.

3. The music structure analysis method according to claim 2, characterized in that: The performing feature fusion processing according to the second feature information and the third feature information to obtain fourth feature information includes: Performing linear transformation processing according to the second feature information to obtain a first vector; Performing linear transformation processing according to the third feature information to obtain a second vector; The fourth feature information is obtained by performing probability distribution mapping on the first vector and the second vector.

4. The music structure analysis method according to claim 2 or 3, characterized in that: The obtaining first multi-dimensional feature information of the first audio signal corresponding to the first time frame includes: Inputting the first audio signal into a trained first model and outputting first feature information; wherein the first model is used to identify the rhythm feature of the input audio signal; Inputting the first audio signal into a trained second model and outputting second feature information; wherein the second model is used to identify the emotional features of the input audio signal; The frequency domain features of the first audio signal are identified to obtain third feature information.

5. The music structure analysis method according to claim 1, characterized in that: The detecting the audio attribute of the first audio signal according to the first fusion feature information includes: Performing feature extraction processing according to the first fused feature information to obtain first target feature information; Detecting the paragraph category of the first audio signal according to the first target feature information to obtain a first detection value; wherein the first detection value is used to indicate the paragraph category to which the first audio signal belongs; Detecting a paragraph boundary of the first audio signal according to the first target feature information to obtain a second detection value; wherein the second detection value is used to indicate a probability of paragraph change of the first audio signal; An audio property of the first audio signal is detected according to the first target feature information, the first detection value, and the second detection value.

6. The music structure analysis method according to claim 5, characterized in that: The detecting the audio attribute of the first audio signal according to the first target feature information, the first detection value, and the second detection value includes: generating an observation sequence according to the first target feature information, the first detection value, and the second detection value; Calculating prediction probabilities corresponding to each audio attribute according to the observation sequence; wherein the prediction probability represents a probability of a state of a second audio signal changing to a state of a first audio signal belonging to the audio attribute, the second audio signal comprising an audio signal corresponding to a second time frame, the second time frame being at least one time frame within a preset time window before the first time frame; An audio attribute corresponding to a maximum value among the calculated prediction probabilities is determined as the audio attribute of the first audio signal.

7. The music structure analysis method according to any one of claims 1 to 6, characterized in that: The detecting the audio attribute of the first audio signal according to the first fusion feature information includes: Acquire a trained detection model, where the detection model is used to detect audio properties of the first audio signal; The first fusion feature information is input into the trained detection model to obtain the audio attribute of the first audio signal.

8. The music structure analysis method according to claim 7, characterized in that: The step of obtaining the trained detection model includes: Acquiring second multi-dimensional feature information of a sample signal, wherein the audio attribute of the sample signal is known; The second multidimensional feature information is input into the detection model, and the audio attribute of the sample signal is output; wherein the detection model includes a feature fusion module and a first network; the feature fusion module is used to perform feature fusion processing according to the input multidimensional feature information to obtain fused feature information; the first network is used to perform feature extraction processing according to the fused feature information output by the feature fusion module to obtain target feature information; Calculating the loss value of the detection model according to the audio properties of the sample signal and a multi-task loss function; wherein the multi-task loss function includes a plurality of different loss functions; Updating the model parameters of the detection model according to the loss value to obtain the updated detection model; Continue to train the updated detection model until the detection accuracy of the detection model reaches a preset threshold, thereby obtaining the trained detection model.

9. The music structure analysis method according to claim 8, characterized in that: The multi-task loss function includes a first function, a second function and a third function; The first function is used to calculate the degree of difference between the audio attribute output by the detection model and the real audio attribute of the sample signal; The second function is used to calculate the probability of paragraph switching of the sample signal; The third function is used to calculate the similarity between the feature representations of the sample signal at different time resolutions.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images