A method for intelligent analysis and arrangement of accompaniment music

Abstract

The application relates to the technical field of audio signal processing, and discloses a music accompaniment intelligent analysis and arrangement method, which comprises the following steps: converting an audio signal into a mel-frequency spectrum feature map; obtaining beat point probability values and heavy beat point probability values, a BPM (beats per minute) range, frame-level segmentation types and chord category probability values through different pre-training neural network models; inputting the beat point probability values, the heavy beat point probability values and the BPM range into a dynamic Bayesian network to obtain accurate measure time points and a final BPM value; obtaining complete segmentation information by combining the segmentation types with the accurate measure time points; obtaining a chord sequence by decoding the chord category probability values through a conditional random field; and finally integrating analysis results and outputting a data packet. Through unified analysis and joint decoding of multidimensional music features, the accurate alignment of beat, paragraph and chord information is realized, and high-quality structured data support is provided for intelligent accompaniment arrangement.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, and in particular to a method for intelligent analysis and arrangement of music accompaniment. Background Technology

[0002] Currently, intelligent analysis and arrangement technology for music accompaniment mainly relies on the extraction of single-dimensional features from audio signals. Traditional methods usually treat beat tracking, segmentation, and chord recognition as independent sub-tasks, resulting in a lack of inherent consistency between analysis results. For example, they cannot align beat boundaries with segment boundaries, and there are deviations between chord switching points and beat points, making it difficult to directly apply the analysis results to actual music arrangement.

[0003] In terms of beat and tempo analysis, existing technologies mostly employ autocorrelation methods based on audio energy envelopes or beat tracking algorithms based on dynamic programming. When faced with music that has tempo fluctuations, complex rhythms, or rich percussion, these methods are prone to problems such as beat phase drift and incorrect downbeat positioning. Especially in the absence of a global tempo prior, traditional beat tracking algorithms are often computationally inefficient. In addition, the beat timing points output by existing methods lack semantic alignment with higher-level musical structures such as chords and sections.

[0004] In the field of music structure analysis, existing studies have used recurrent neural networks or hidden Markov models to segment and identify music. However, these methods usually only output segment labels at the frame level or at a fixed time granularity, ignoring the natural boundary information of the music. At the same time, the segmentation results are generated independently of BPM and beat position, which may lead to illogical results in the final analysis.

[0005] In chord recognition, existing technologies mostly use convolutional neural networks or recurrent neural networks to classify each frame independently, or smooth the prediction results through simple post-processing. On the one hand, these methods are difficult to effectively model the long-range dependencies of chords in time. On the other hand, they are prone to producing chord abrupt changes that do not conform to music theory. Therefore, there is an urgent need for an intelligent analysis method for music accompaniment that can uniformly extract beat, measure, paragraph and chord information and maintain internal consistency. Summary of the Invention

[0006] The purpose of this invention is to propose a method for intelligent analysis and arrangement of music accompaniment in order to solve the problems in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for intelligent analysis and arrangement of music accompaniment, comprising the following steps:

[0008] S1: Acquire the audio signal of the input music and convert it into a Mel-spectrum feature map through preprocessing;

[0009] S2, input the Mel spectrum feature map into a pre-trained recurrent neural network to identify the beat probability value and downbeat probability value of each frame of audio signal;

[0010] S3, input the Mel spectrum feature map into a pre-trained convolutional neural network, output an initial BPM value, and offset the BPM range up and down with this value as the center. Input the beat probability value, the downbeat probability value and the BPM range into a dynamic Bayesian network for post-processing to obtain the precise measure time point and the final BPM value of the input music.

[0011] S4, input the Mel spectrum feature map into the pre-trained self-attention neural network, output the segment type of each frame of audio signal, and further combine it with the precise measure time point to obtain the complete segment information of the input music;

[0012] S5, input the Mel spectrum feature map into the pre-trained deep auditory model, identify the chord type probability value of each frame of audio, decode the chord type probability value through a linear chain conditional random field, and obtain the chord sequence of the input music, including start time, end time and chord type;

[0013] S6. The precise measure timing points, the final BPM value of the input music, the complete segmentation information of the input music, and the chord sequence of the input music are integrated into a unified music accompaniment analysis result data package and output.

[0014] The beneficial effects of the technical solution provided by this invention include at least the following:

[0015] This invention constructs a unified analysis framework for multi-dimensional music features, using the Mel spectrum feature map as the unified input for all sub-models, and integrating precise measure time points, complete segmentation information and chord sequences into a single data packet output. This end-to-end framework design ensures the natural alignment of the analysis results of each dimension on the time axis, fundamentally avoiding the boundary misalignment problem caused by independent analysis.

[0016] This invention transforms the initial BPM value output by the convolutional neural network into a BPM range and uses it as a priori velocity constraint for the dynamic Bayesian network. This effectively reduces the search space of the hidden state of the beat cycle, accelerating the decoding efficiency of the Viterbi algorithm and avoiding global decoding from falling into ambiguous solutions. At the same time, the dynamic Bayesian network models three hidden state variables simultaneously: beat cycle, beat phase, and downbeat marker. It adaptively tracks the velocity fluctuations in the music and outputs beat points, downbeat points, and bar lines accurate to the frame level. Compared with traditional methods, this is more in line with the practical needs of music arrangement.

[0017] This invention proposes a measure-level segment merging and CRF chord decoding scheme, which forcibly aligns the segment boundaries with the measure boundaries. The output complete segment information naturally conforms to the organizational logic of music. At the same time, this invention outputs frame-by-frame chord probabilities through a fully convolutional deep auditory model and further uses a linear chain conditional random field for global decoding. It utilizes the state transition feature function to learn the chord progression rules that conform to music theory, effectively suppressing unreasonable chord abrupt changes. The combination of these two methods ensures that the output segment information and chord sequence have inherent consistency in rhythm and harmony.

[0018] In summary, this invention solves the problem of inconsistent results caused by the isolated operation of each analysis module in the prior art, and realizes intelligent and high-precision conversion from raw audio to structured accompaniment analysis data. The final output analysis results can be directly used in application scenarios such as automatic accompaniment generation, intelligent composition, and music education, which significantly improves the automation level of the music arrangement system and the professionalism of the output results. Attached Figure Description

[0019] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a method provided in an embodiment of the present invention. Detailed Implementation

[0021] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a music accompaniment intelligent analysis and arrangement method proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0024] The following description, in conjunction with the accompanying drawings, details a specific scheme for an intelligent analysis and arrangement method for music accompaniment provided by this invention.

[0025] Please see Figure 1 The diagram illustrates a flowchart of a music accompaniment intelligent analysis and arrangement method according to an embodiment of the present invention. The method includes the following steps:

[0026] S1: Acquire the audio signal of the input music and convert it into a Mel-spectrum feature map through preprocessing;

[0027] S2, input the Mel spectrum feature map into the pre-trained recurrent neural network to identify the beat probability value and downbeat probability value of each frame of audio signal;

[0028] S3. Input the Mel spectrum feature map into the pre-trained convolutional neural network and output the initial BPM value. Shift the BPM range up and down from this value. Input the beat probability value, downbeat probability value, and BPM range into the dynamic Bayesian network for post-processing to obtain the precise measure time point and the final BPM value of the input music.

[0029] S4. Input the Mel spectrum feature map into the pre-trained self-attention neural network, output the segment type of each frame of audio signal, and further combine it with the precise measure time point to obtain the complete segment information of the input music;

[0030] S5. Input the Mel spectrum feature map into the pre-trained deep auditory model to identify the chord type probability value of each frame of audio. Decode the chord type probability value through a linear chain conditional random field to obtain the chord sequence of the input music, including the start time, end time and chord type.

[0031] S6 integrates the precise measure timings, the final BPM value of the input music, the complete segmentation information of the input music, and the chord sequence of the input music into a unified music accompaniment analysis result data package and outputs it.

[0032] In one embodiment of the present invention, the step of acquiring the audio signal of the input music and converting it into a Mel-spectrum feature map through preprocessing includes:

[0033] The audio signal of the input music is acquired and resampled. The resampled audio signal is then processed into frames. A Hanning window is applied to each frame of the audio signal. The windowed signal is then subjected to a short-time Fourier transform to obtain a linear spectrum.

[0034] The linear spectrogram is mapped through a set of Mel filters, where the center frequencies of each Mel filter are equally spaced on the Mel scale, to obtain the Mel spectral characteristics of each frame of audio signal.

[0035] Logarithmic energy compression is performed on the Mel-spectral features of all frames of audio signals to generate the final Mel-spectral feature map.

[0036] It should be noted that in this embodiment, the goal of resampling is to unify the input audio to a preset target sampling rate in order to avoid the influence of different original sampling rates on feature extraction. The preset target sampling rate is preferably 22050Hz. If the input audio sampling rate is lower than the target sampling rate, upsampling interpolation is performed; if it is higher than the target sampling rate, downsampling low-pass filtering is performed.

[0037] In this embodiment, a sliding window method is used for frame division. When the sampling rate is 22050Hz, the frame length (window size) is set to 2048 sampling points, approximately 92.9ms, and the frame shift is set to 512 sampling points, approximately 23.2ms, to ensure that there is sufficient overlap between adjacent frames to meet the short-time stationary assumption.

[0038] In this embodiment, a Hanning window is applied to each frame, and an N-point short-time Fourier transform is performed on each windowed frame. Preferably, N=2048 (i.e., the same as the frame length) is taken to obtain a linear spectrum containing complex results, the dimension of which is the number of frequency points × the number of frames, where the number of frequency points = N / 2 + 1 = 1025.

[0039] In this embodiment, the Mel filter bank consists of 128 triangular bandpass filters. The center frequencies of each filter are evenly distributed on the Mel scale. The energy spectrum of the linear spectrum is multiplied by the frequency response of each Mel filter and summed to obtain the 128-dimensional Mel spectrum feature of each frame. The natural logarithm or common logarithm is taken for all values, and the final generated dimension is: (number of Mel filters, number of time frames), that is, (128, number of time frames) two-dimensional feature matrix, which serves as the Mel spectrum feature map.

[0040] In one embodiment of the present invention, the step of inputting the Mel spectrum feature map into a pre-trained recurrent neural network to identify the beat probability value and downbeat probability value of each frame of audio signal includes:

[0041] Local features are aggregated from the Mel spectral feature map by temporal convolutional blocks to obtain aggregated local features. The temporal convolutional block contains two convolutional layers, each followed by a batch normalization layer and a ReLU activation function.

[0042] The local features are expanded along the time axis and sequentially input into a recurrent neural network based on a two-layer bidirectional long short-term memory network structure. A fully connected layer with an output dimension of 2 is connected after the recurrent network to output the activation values ​​of the beat point and the downbeat point of each frame of audio signal.

[0043] The activation value is mapped to the [0,1] interval by using the Sigmoid activation function, and serves as the beat probability value and downbeat probability value of each frame of audio signal.

[0044] It should be noted that in this embodiment, the temporal convolutional block is composed of two stacked one-dimensional convolutional layers, used to aggregate local time-frequency features of adjacent time frames to enhance the perception of short-term rhythmic patterns. The kernel size of each convolutional layer is preferably set to 3 or 5, with a stride of 1, and a same padding mode (i.e., the output time length is the same as the input). The number of input channels is the number of frequency bands in the Mel spectrum, i.e., the number of Mel filters in the aforementioned preprocessing steps, which is 128 in this embodiment. The number of intermediate layer channels is preferably set to 64 or 128, while the number of output channels remains unchanged. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function (f(x)=max(0,x)). The dimension of the output local features is:

[0045] (batch_size, time_steps, feature_dim);

[0046] Where batch_size is the batch size, which is the number of audio samples processed in parallel at one time. It can usually be set to 1, which means processing one piece of music at a time. time_steps is the number of frames, which is the total number of frames into which the audio signal of the input music is divided. It is consistent with the time axis length of the Mel-spectrum feature map obtained by preprocessing in the previous steps. feature_dim is the feature dimension, which is the length of the feature vector corresponding to each frame. After aggregation by temporal convolutional blocks, this dimension is equal to the number of output channels of the last convolutional layer.

[0047] The local features are input sequentially into a two-layer bidirectional LSTM network to capture the contextual dependencies between each frame. The hidden_size of each LSTM layer is preferably set to 128 or 256. The final output feature dimension is:

[0048] (time_steps, hidden_size×2).

[0049] A fully connected layer with an output dimension of 2 is connected after the bidirectional LSTM, corresponding to the original activation values ​​of the "beat point" and the "re-beat point" respectively. Then, the Sigmoid function is applied to these two activation values ​​to independently map them to the (0,1) interval, which are the probabilities of the frame belonging to the beat point and the re-beat point respectively.

[0050] It is important to note that the aforementioned recurrent neural network needs to be pre-trained on a labeled dataset, such as a music database with beat and downbeat position labels. The preferred training loss function is binary cross-entropy loss, which is calculated separately for beat prediction and downbeat prediction and then averaged. After training, the recurrent neural network takes the Mel-spectrum feature map as input and outputs two probability sequences of equal length. The length of these probability sequences is equal to the total number of audio frames, corresponding to the beat probability value and downbeat probability value of each frame, respectively, for use by the subsequent dynamic Bayesian network.

[0051] The parameters mentioned in the above steps, such as kernel size, number of hidden units, and number of fully connected layers, are all conventional choices in this field and can be adjusted according to specific datasets and tasks without affecting the feasibility of the overall solution.

[0052] In one embodiment of the present invention, after identifying the beat probability value and downbeat probability value of each frame of audio signal, the method further includes the following steps:

[0053] Gaussian filtering is applied to smooth the beat point probability value and the repeat point probability value respectively. Frames with smoothed beat point probability values ​​higher than a preset beat threshold are marked as candidate beat frames, and frames with smoothed repeat point probability values ​​higher than a preset repeat threshold are marked as candidate repeat frames.

[0054] Time interval constraints are applied to candidate beat frames and candidate downbeat frames, and frames with time intervals less than the minimum beat period are removed. The minimum beat period is calculated based on the initial BPM value of the input music.

[0055] The beat point probability value and replay point probability value of the remaining frames after the removal process are output as the input parameters of the dynamic Bayesian network in S3.

[0056] It should be noted that in this embodiment, the beat probability sequence and the repeat probability sequence are convolved along the time axis using a one-dimensional Gaussian kernel. The standard deviation σ of the Gaussian kernel is preferably set to 3 to 7 frames, the kernel width is 6σ+1, and the preset beat threshold and preset repeat threshold are preferably set to between 0.3 and 0.6. The same frame can be marked as a candidate beat frame and a candidate repeat frame at the same time.

[0057] In real music, the beat cycle, i.e., the time interval between two adjacent beats, is not infinitely small. Therefore, the minimum beat cycle is used to filter candidate frames, eliminating those with too close intervals. The minimum beat cycle is calculated as follows:

[0058] T min =60 / (BPM) initial ×ɑ);

[0059] The specific value of the coefficient α can be adjusted according to the music style, preferably 2 or 1.5, to ensure that overly dense predictions that are significantly beyond the reasonable range are eliminated.

[0060] Traverse the candidate beat frame sequence and the candidate replay frame sequence. If the time interval between the current frame and the previous retained frame is less than T... min If the current frame is not selected, the time interval should be converted to absolute time (seconds) on the time axis instead of frame index.

[0061] After the above elimination process is completed, the remaining candidate beat frames and candidate repeat frames, along with their corresponding smoothed probability values, are output to the dynamic Bayesian network in the subsequent steps for global optimization decoding, so as to finally determine the precise beat position, repeat position, and bar line.

[0062] In one embodiment of the present invention, the step of inputting the Mel-frequency feature map into a pre-trained convolutional neural network, outputting an initial BPM value, and obtaining the BPM range by shifting the value vertically and vertically around this value includes:

[0063] The Mel spectrum feature map is input into a pre-trained convolutional neural network based on a residual convolutional network structure, and its output is mapped through a global pooling layer and a fully connected layer to obtain the initial BPM value of the input music.

[0064] Centered on the initial BPM value, offset upwards and downwards by 20% respectively to generate a BPM range including the lower and upper limits of BPM, and output this BPM range as the prior velocity constraint of the dynamic Bayesian network in S3.

[0065] It should be noted that in this embodiment, a residual convolutional network is used as the initial BPM estimation model, and a preferred network structure is as follows:

[0066] Input layer: used to receive Mel spectrum feature maps, its dimension is (number of time frames, number of Mel bands), where the number of Mel bands is known to be 128 in this embodiment. To adapt to the convolution operation, an additional dimension is added as the channel dimension, becoming (number of time frames, number of Mel bands, 1).

[0067] The main body of the convolution is composed of multiple residual blocks stacked together. Each residual block contains two 3×3 convolutional layers. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The input is directly added to the output through skip connections.

[0068] Time-frequency aggregation: After the main convolution, average pooling or max pooling is first performed along the frequency dimension to compress the Mel frequency band information and obtain a feature vector with dimension (time frame number, 1). Then, global average pooling is performed along the time dimension to compress the entire time series into a single scalar vector, which is used to comprehensively represent the rhythm and speed information of the entire input music piece.

[0069] Output layer: The pooled single scalar vector is mapped through one or more fully connected layers to finally output a scalar value as the initial BPM value. The output layer can use a linear activation function or ReLU to ensure that the output is non-negative.

[0070] It is important to note that the above convolutional neural network needs to be trained in advance on a large-scale music dataset with labeled BPM values. The training loss function adopts mean squared error loss or mean absolute error loss. After training, the convolutional neural network can directly regress the initial BPM estimate of the input music from the Mel spectrogram feature map end-to-end.

[0071] In this embodiment, a BPM interval is generated by shifting the initial BPM value output by the convolutional neural network to the center and then shifting it upwards and downwards by 20% respectively. This interval serves as a velocity prior constraint for the dynamic Bayesian network. A specific example is as follows:

[0072] Let the initial BPM value be B0, then:

[0073] BPM lower limit: B min =B0×(1-20%)=B0×0.8;

[0074] BPM limit: B max =B0×(1+20%)=B0×1.2;

[0075] The final generated BPM range is [B min B max This range will be used in subsequent steps to constrain the value range of the beat cycle hidden state variable of the dynamic Bayesian network, limiting it to a reasonable range, thereby accelerating the decoding process and reducing the ambiguity of the global optimal solution search; among them, the 20% offset is an empirical value set based on the general variation law of music tempo. The tempo variation of most musical works, or the tempo difference between different sections of the same work, usually does not exceed ±20% of the average value.

[0076] In one embodiment of the present invention, the step of inputting the beat probability value, the downbeat probability value, and the BPM range into a dynamic Bayesian network for post-processing to obtain the precise measure timing and the final BPM value of the input music includes:

[0077] The state space of the dynamic Bayesian network is constructed, which contains three hidden state variables: beat period, beat phase, and re-beat mark. The range of the beat period is obtained by mapping the range of BPM. The beat phase represents the relative position of the current frame within one beat period. The re-beat mark is used to mark whether the current frame is a re-beat position.

[0078] Based on the beat probability value and beat phase of the current frame, a matching function for the beat observation probability is defined; based on the repeat probability value and repeat position of the current frame, a matching function for the repeat observation probability is defined.

[0079] The Viterbi algorithm is used to perform global decoding within the range of the beat period. The optimal state sequence is selected that maximizes the joint probability of the observation sequence that maximizes the beat observation probability and the re-beat observation probability, together with the hidden state sequence of the beat period, beat phase and re-beat marker.

[0080] Determine the decoded beat period from the optimal hidden state sequence, and calculate the final BPM value of the input music based on the beat period;

[0081] The frame position where the beat phase returns to zero is extracted from the optimal state sequence as the precise beat time point. The frame position marked as true by the re-beat is further extracted from the precise beat time point as the precise re-beat time point.

[0082] The frame interval between adjacent precise retake time points is defined as a segment based on the preset time number. Each precise retake time point is used as the start time of the corresponding segment to generate a sequence of precise segment time points.

[0083] It should be noted that in this embodiment, the state space of the dynamic Bayesian network contains three hidden state variables, and each time frame corresponds to a set of hidden states. The symbols, meanings, and value ranges of these three hidden state variables are explained in Table 1:

[0084] Table 1: Explanation of Hidden State Variables

[0085] Hidden state variables symbol meaning Range of values Beat cycle τ The time interval between two adjacent beats (unit: frames) <![CDATA[The value range of τ is mapped from the BPM range; let the BPM range be [B min , B max , the sampling rate be f s , and the frame shift be H. Then the beat period range is: [(60 · f s ) / (B max · H), (60 · f s ) / (B min · H)]. After rounding, it is output as a discrete state set]]> Beat phase The relative position of the current frame within the current beat cycle The range of values ​​for is {0, 1, 2, ..., τ-1}, where =0 indicates that the current frame is exactly at the beat point. Retake Mark m Is the current frame a retake position? The value of m ranges from {0, 1}, where 1 represents a repeat and 0 represents a non-repeat. m only takes values ​​within the range of {0, 1}. A frame with a value of 0 can be 1.

[0086] The state transitions of dynamic Bayesian networks follow the rhythmic pattern of the input music, i.e., the rhythmic phase. Incrementing at each time step (modulo τ), when from =τ-1 jumps to =0 indicates the start of a new beat. The beat period τ can be increased or decreased within a certain range to adapt to the tempo fluctuations in the music. The accent mark m determines its transition rule according to the preset time number (e.g., in 4 / 4 time, an accent occurs once every 4 beats).

[0087] In this embodiment, the beat probability value p of the t-th frame output by the recurrent neural network is used. beat (t) and the probability of retake p downbeat (t), defining the matching function for the observation probability, including:

[0088] Beat Observation Probability When the beat phase in the hidden state =0, meaning when the current frame should be a beat point, the beat observation probability is equal to p. beat (t) positively correlated; when When ≠0, the probability of beat observation is equal to 1-p. beat (t) is positively correlated, and it is preferable to use a Gaussian distribution for modeling, for example:

[0089]

[0090] Retake observation probability: When the retake flag m=1 in the hidden state, the retake observation probability is related to p. downbeat (t) positively correlated; when m=0, the probability of a retake observation is related to 1-p downbeat (t) is positively correlated.

[0091] Within a given range of candidate values ​​for the beat period τ, the Viterbi algorithm is used to solve for the optimal hidden state sequence. The Viterbi algorithm is a dynamic programming method that recursively calculates the maximum joint probability achievable in each frame and each possible hidden state, and records the path backtracking pointer. The algorithm flow is as follows:

[0092] (1) Initialize the hidden state probability of the first frame to a uniform distribution multiplied by its observation probability;

[0093] (2) For each subsequent frame, calculate the product of the transition probability and the observation probability of all hidden state combinations, and retain the probability value with the largest product result and the corresponding hidden state of the previous frame.

[0094] (3) Select the hidden state with the highest probability in the last frame, and extract the entire sequence in reverse along the backtracking pointer to obtain the optimal hidden state sequence.

[0095] Final BPM value: From the beat period τ of the optimal hidden state sequence, extract the stable mode or median as the decoded beat period τ0, and calculate BPM using the formula BPM = (60·f s The final BPM value of the input music is calculated by ) / (τ0·H).

[0096] Traverse the optimal hidden state sequence and extract all beat phases. The frame position with a value of 0 is converted to absolute time on the timeline: Absolute time = (frame index × H) / f s This yields a precise beat time point sequence; further, the time points of the re-beat marker m=1 for the corresponding frames are extracted from the precise beat time point sequence to obtain a precise re-beat time point sequence.

[0097] The frame interval between adjacent precise repeat time points is defined as a measure based on the preset beat number (for example, for 4 / 4 time, usually every 4 beats is a measure, where the first beat is the repeat). Each precise repeat time point is used as the start time of the corresponding measure to generate a precise measure time point sequence. The preset beat number can be specified by the user. Common beat numbers include 4 / 4 time (one repeat every 4 beats), 3 / 4 time (one repeat every 3 beats), etc.

[0098] In one embodiment of the present invention, the step of inputting the Mel-spectrum feature map into a pre-trained self-attention neural network and outputting the segmentation type of each frame of audio signal includes:

[0099] The Mel spectral feature map is input into a pre-trained self-attention neural network based on a structure that fuses convolutional layers and self-attention layers. The convolutional layers extract local time-frequency features from the Mel spectral feature map.

[0100] The local time-frequency features are compressed along the frequency dimension while retaining the time dimension to obtain a frame-level time feature sequence. The inter-frame dependencies at different time positions in the frame-level time feature sequence are calculated through a self-attention layer.

[0101] A fully connected layer is connected after the self-attention layer. Based on the inter-frame dependency, the probability of each frame belonging to each preset segment type is output, and the category corresponding to the highest probability is taken as the segment type of the frame.

[0102] It should be noted that, in this embodiment, the main structure of the self-attention neural network is as follows: convolutional feature extraction → frequency dimension compression → self-attention layer → fully connected classification layer, wherein:

[0103] Convolutional layers are used to extract local time-frequency patterns from Mel spectrum feature maps. Two or three two-dimensional convolutional layers can be stacked. The kernel size can be set to 3×3 or 5×5, the stride is 1, and same padding is used to keep the time dimension and frequency dimension unchanged.

[0104] Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The number of input channels is 1 (Mel spectrogram is processed as a single-channel image), and the number of output channels can be increased layer by layer. For example, the first layer outputs 32 channels, the second layer outputs 64 channels, and the third layer outputs 128 channels. After passing through the convolutional module, the dimensions of the output local time-frequency features are (number of time frames, number of Mel frequency bands, number of output channels).

[0105] The local time-frequency features are compressed along the Mel frequency band dimension while retaining the time dimension, resulting in a frame-level time feature sequence. The compression method can be global average pooling or global max pooling. After compression, the original three-dimensional features (number of time frames, number of Mel frequency bands, number of channels) are converted into two-dimensional features (number of time frames, compressed feature dimension). The dimension of the compressed feature is equal to the number of output channels (e.g., 128). Each frame in this feature sequence corresponds to a feature vector, representing the local time-frequency information of that frame.

[0106] The above frame-level temporal feature sequence is input into the self-attention layer to capture the long-range dependencies between different time frames. The self-attention layer allows each frame to directly focus on any other frame in the frame-level temporal feature sequence, thereby effectively modeling the long-range structural features across time in the music segment. This layer can use multi-head self-attention, preferably 4 or 8 heads. The outputs of each head are concatenated and then linearly transformed to obtain the final output.

[0107] A fully connected classification layer is connected after the self-attention layer. Its output dimension is equal to the total number of preset segment types, preferably the following 5: intro, verse, chorus, interlude, and outro. The Softmax function is applied to the output of this layer along the category dimension to convert the activation value of each time frame into a probability distribution, and the category corresponding to the highest probability is taken as the segment type of that frame.

[0108] It should be noted that the self-attention neural network mentioned above needs to be trained in advance on a large-scale dataset with labeled music structure, such as a music database with paragraph boundaries and type labels. The training loss function is preferably frame-level classification cross-entropy loss, which calculates the error between the predicted segment type and the true label frame by frame and sums and averages them.

[0109] In one embodiment of the present invention, after obtaining the segmentation type of each frame signal, the method further includes a step of combining precise measure time points to obtain complete segmentation information of the input music:

[0110] Obtain precise measure time points. Based on the audio signal's sampling rate and frame shift parameters, convert each precise measure time point into a corresponding frame index, which serves as the starting frame for the corresponding measure.

[0111] For each subsection, count the frequency of segment types in all frames within that subsection, and take the segment type with the highest frequency as the segment type of that subsection.

[0112] Traverse all sections along the timeline, merge adjacent sections with the same paragraph type into one paragraph, and record the start and end times of each paragraph as well as the paragraph type.

[0113] Map the start and end timestamps of each segment back to the audio timeline to generate complete segmentation information including start time, end time, and segment type.

[0114] It should be noted that in this embodiment, the precise segment time points are sequences in absolute time (seconds). To facilitate alignment with the frame-level segmentation type, the precise segment time points need to be converted into the corresponding frame index (frame_index). The conversion formula is as follows:

[0115] frame_index = round((time×f s ) / H);

[0116] Where: time is the precise time point of the segment (unit: seconds), f s H is the audio sampling rate, and H is the frame shift. After the above conversion, the start time of each section corresponds to a unique frame index, which serves as the starting frame of that section.

[0117] Each measure, that is, the interval between two adjacent precise measure time points, contains several consecutive time frames. The segmentation type of all frames in the measure is statistically analyzed, and the frequency of each segmentation type (such as intro, verse, chorus, interlude, and outro) in the measure is calculated. The segmentation type with the highest frequency is taken as the segment type of the measure. If multiple segmentation types have the same highest frequency, it is preferable to decide according to a preset priority (such as the chorus having a higher priority than the verse).

[0118] Traverse all sections along the timeline, merging adjacent sections with the same paragraph type into a single paragraph. Paragraphs of the same type but not adjacent are not merged. The merged paragraph contains the following information:

[0119] (1) Start time of the first paragraph: the start time of the first paragraph in this paragraph (unit: seconds);

[0120] (2) End time of the last measure: the end time of the last measure in the paragraph (i.e., the start time of the first measure after the paragraph, or the end time of the entire piece of music, in seconds).

[0121] (3) Paragraph type: The paragraph type of this paragraph (i.e. the paragraph type of each section before merging).

[0122] The start and end timestamps of each merged paragraph are preserved as time boundaries to generate the final complete segmentation information. An example of complete segmentation information is provided in JSON format:

[0123] [{"Paragraph Number": 1, "Start Time": 0.00, "End Time": 10.50, "Paragraph Type": "Intro"},

[0124] {"Segment Number": 2, "Start Time": 10:50, "End Time": 30:20, "Segment Type": "Verse"}

[0125] {"Section Number": 3, "Start Time": 30:20, "End Time": 50:80, "Section Type": "Chorus"}

[0126] {"Segment Number": 4, "Start Time": 50.80, "End Time": 70.00, "Segment Type": "Verse"}

[0127] {"Section Number": 5, "Start Time": 70.00, "End Time": 90.00, "Section Type": "Chorus"}

[0128] {"Section Number": 6, "Start Time": 90.00, "End Time": 98.50, "Section Type": "Outro"}]

[0129] In one embodiment of the present invention, the step of inputting the Mel spectrum feature map into a pre-trained deep auditory model to identify the chord type probability value of each frame of audio includes:

[0130] The Mel spectral feature map is input into a fully convolutional deep auditory model based on a fully convolutional network structure. The deep auditory model is composed of multiple stacked convolutional blocks, each of which contains a convolutional layer, a batch normalization layer, and an activation function.

[0131] The last convolutional block of the fully convolutional network structure outputs a two-dimensional feature map with time and channel dimensions. In the time dimension, each time position corresponds to a frame in the audio signal of the input music, and in the channel dimension, each channel corresponds to a preset chord type.

[0132] The Softmax function is applied to the channel dimension of the two-dimensional feature map to convert the channel value corresponding to each time position into a probability distribution, thereby obtaining the probability of each frame in the audio signal belonging to each preset chord type, which is used as the chord type probability value.

[0133] It should be noted that in this embodiment, a fully convolutional deep auditory model is used. Its main structure consists of multiple stacked convolutional blocks, without fully connected layers, to maintain the integrity of the temporal dimension. The overall structure is as follows: input layer → several convolutional blocks → output layer. Each convolutional block contains the following components:

[0134] Convolutional layer: Two-dimensional convolution along the time and frequency dimensions is used. The kernel size can be set to 3×3 or 5×3 (time axis × frequency axis). The stride can be set to (1,1) or (2,1), where the stride along the frequency dimension is usually kept at 1. The padding method is same padding to ensure that the time dimension remains unchanged after convolution.

[0135] Batch normalization layer: placed after the convolutional layer and before the activation function, used to accelerate training convergence and stabilize the distribution;

[0136] Activation function: ReLU function is used.

[0137] For an input Mel-band feature map with dimensions (time frame number T, Mel-band number F, 1), preferably, its Mel-band number dimension is compressed to 1 using any one or a combination of the following methods:

[0138] a. Increase the height of the convolution kernel: Use a convolution kernel that covers the entire or most of the Mel frequency band in subsequent convolution blocks (preferably such as 3×F or 5×F). A single convolution can compress the Mel frequency band to 1.

[0139] b. Stepwise pooling: Max pooling or average pooling is performed along the frequency dimension after each convolutional block. Preferably, the pooling window can be set to 2×1 or 3×1, and the Mel band number is gradually reduced until it is 1.

[0140] c. Global frequency pooling: After the output of the penultimate convolutional block, global average pooling is performed along the Mel frequency band, directly compressing it to 1.

[0141] The output dimension of the last convolutional block of the fully convolutional deep auditory model is (time frame number T, 1, C). K The feature map of ), where C K The number of channels after convolution, after removing the frequency dimension (the shape becomes T×C) K Then, the Softmax function is applied to its channel dimension to output the probability of each frame belonging to the preset chord type. The sum of the probabilities of all chord types is 1. The preset chord types can be defined according to application requirements. Preferably, it can be set as: 12 root notes × 4 types (major triad, minor triad, dominant seventh, half diminished seventh) = a total of 48 chord types, plus a "no chord" category, for a total of 49 types.

[0142] It is important to note that the above fully convolutional deep auditory model needs to be trained in advance on a large-scale labeled chord music dataset. The training loss function adopts frame-level classification cross-entropy loss, and the loss is calculated by summing and averaging the predicted chord probability distribution and the real chord label for each frame.

[0143] In one embodiment of the present invention, after obtaining the chord type probability value, the method further includes a step of decoding the chord type probability value using a linear chain conditional random field to obtain the chord sequence of the input music:

[0144] A linear chain conditional random field model is constructed, which includes a state transition feature function and a state feature function. The state feature function is defined based on the probability value of the chord type in each frame, and the state transition feature function is used to characterize the transition constraints of the chord type between adjacent frames.

[0145] By using a linear chain conditional random field model, the complete chord type probability value sequence of the audio signal is jointly decoded, and the chord type sequence that maximizes the global conditional probability is selected as the decoding result.

[0146] In the decoded chord type sequence, consecutive frames with the same chord type are merged into a chord segment, and the start frame position, end frame position, and chord type of each chord segment are recorded;

[0147] The start and end frame positions of each chord segment are mapped to start and end times on the audio timeline, generating a chord sequence that includes start time, end time, and chord type.

[0148] It should be noted that the linear-chain conditional random field (CRF) model is a probabilistic graphical model used for sequence labeling. For a given observation sequence x=(x1,x2,…,x…),… T (i.e., the complete chord probability value sequence of the audio signal), give the hidden state feature sequence y=(y1,y2,…,y) under this observation sequence. T ), which is the conditional probability of the chord type label sequence.

[0149] Its state characteristic function f s (y t (x,t) is the probability distribution vector p of the chords in the t-th frame output by the deep auditory model. t Specifically, for each candidate chord category c, its state feature is defined as follows:

[0150] f s,c (y t ,x,t) = ‖( y t =c)·log p t (c);

[0151] Among them, y t Given the hidden state features of the t-th frame in a given observation sequence, p t (c) represents the probability that the t-th frame output by the deep auditory model belongs to chord category c, and ||(·) is the indicator function, which takes 1 when the condition is met and 0 otherwise.

[0152] Its transfer characteristic function f t,c (y t-1 ,y t (x,t) is used to characterize the chord category transition constraints between adjacent frames. Common transition features include:

[0153] Basic transfer characteristics: for each pair of chord categories (c t-1 ,c t Define the transition characteristic function:

[0154] f t (y t-1 ,y t ,x,t) = ‖( y t-1 = c prev ∩ y t = c curr );

[0155] Among them, c t-1 c t Let represent the chord types of the t-th frame and the frame preceding it, respectively. This transition feature function allows the CRF to learn the transition probabilities between different chords.

[0156] Music theory constraints: for example, prohibiting a direct jump from a dominant seventh chord to an unrelated chord, or encouraging transitions that conform to common chord progressions (such as the circle of fifths, I-IV-VI, etc.) by assigning higher weights.

[0157] It is important to note that the weight parameters of CRF need to be trained on a dataset of labeled chord sequences. The training objective is to maximize the conditional log-likelihood, and optimization methods can include L-BFGS or stochastic gradient descent.

[0158] In this embodiment, the Viterbi algorithm is used for decoding. The Viterbi algorithm is a dynamic programming method that maintains the maximum probability value of each time step and each state and the corresponding predecessor state, and finally decodes the frame-by-frame chord category label sequence.

[0159] Since chords typically remain unchanged across multiple consecutive frames, it's necessary to merge consecutive frames of the same type into chord segments. A preferred merging rule is: iterate through the frame-by-frame sequence; when adjacent frames share the same chord type, group them into the same chord segment; when the type changes, end the current chord segment and begin a new one. Record each chord segment's starting frame index, ending frame index (excluding the starting frame of the next segment), and chord type, and map the frame indices back to the audio timeline. Using JSON format as an example, here's a sample of the final generated chord sequence:

[0160] [{"Start Time": 0.00, "End Time": 5.23, "Chord Type": "C"},

[0161] {"Start Time": 5.23, "End Time": 10.46, "Chord Type": "G"},

[0162] {"Start Time": 10:46, "End Time": 15:69, "Chord Type": "Am"},

[0163] {"Start Time": 15.69, "End Time": 20.92, "Chord Type": "F"}]

[0164] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for intelligent analysis and arrangement of musical accompaniment, characterized in that, The method includes: S1: Acquire the audio signal of the input music and convert it into a Mel-spectrum feature map through preprocessing; S2, input the Mel spectrum feature map into a pre-trained recurrent neural network to identify the beat probability value and downbeat probability value of each frame of audio signal; S3, input the Mel spectrum feature map into a pre-trained convolutional neural network, output an initial BPM value, and offset the BPM range up and down with this value as the center. Input the beat probability value, the downbeat probability value and the BPM range into a dynamic Bayesian network for post-processing to obtain the precise measure time point and the final BPM value of the input music. S4, input the Mel spectrum feature map into the pre-trained self-attention neural network, output the segment type of each frame of audio signal, and further combine it with the precise measure time point to obtain the complete segment information of the input music; S5, input the Mel spectrum feature map into the pre-trained deep auditory model, identify the chord type probability value of each frame of audio, decode the chord type probability value through a linear chain conditional random field, and obtain the chord sequence of the input music, including start time, end time and chord type; S6. The precise measure timing points, the final BPM value of the input music, the complete segmentation information of the input music, and the chord sequence of the input music are integrated into a unified music accompaniment analysis result data package and output.

2. The intelligent analysis and arrangement method for music accompaniment according to claim 1, characterized in that: The steps involved in acquiring the audio signal of the input music and converting it into a Mel-spectrum feature map through preprocessing include: The audio signal of the input music is acquired and resampled. The resampled audio signal is then processed into frames. A Hanning window is applied to each frame of the audio signal. The windowed signal is then subjected to a short-time Fourier transform to obtain a linear spectrum. The linear spectrogram is mapped through a set of Mel filters, wherein the center frequencies of each Mel filter are equally spaced on the Mel scale, to obtain the Mel spectral features of each frame of audio signal; Logarithmic energy compression is performed on the Mel-spectral features of all frames of audio signals to generate the final Mel-spectral feature map.

3. The intelligent analysis and arrangement method for music accompaniment according to claim 1, characterized in that: The step of inputting the Mel-spectrum feature map into a pre-trained recurrent neural network to identify the beat probability value and downbeat probability value of each frame of audio signal includes: The local features of the Mel spectral feature map are aggregated by a temporal convolutional block to obtain aggregated local features. The temporal convolutional block contains two convolutional layers, each followed by a batch normalization layer and a ReLU activation function. The local features are expanded along the time axis and sequentially input into a recurrent neural network based on a two-layer bidirectional long short-term memory network structure. A fully connected layer with an output dimension of 2 is connected after the recurrent network to output the activation values ​​of the beat point and the downbeat point of each frame of audio signal. The activation values ​​are mapped to the [0,1] interval using the Sigmoid activation function, serving as the beat probability value and downbeat probability value for each frame of audio signal.

4. The intelligent analysis and arrangement method for music accompaniment according to claim 3, characterized in that: After identifying the beat probability value and downbeat probability value of each frame of audio signal, the following steps are also included: Gaussian filtering is applied to the beat point probability value and the repeat point probability value respectively. Frames with smoothed beat point probability values ​​higher than a preset beat threshold are marked as candidate beat frames, and frames with smoothed repeat point probability values ​​higher than a preset repeat threshold are marked as candidate repeat frames. Time interval constraints are applied to the candidate beat frames and the candidate overbeat frames, and frames with time intervals less than the minimum beat period are removed. The minimum beat period is calculated based on the initial BPM value of the input music. The beat point probability value and replay point probability value of the remaining frames after the elimination process are output as the input parameters of the dynamic Bayesian network described in S3.

5. The intelligent analysis and arrangement method for music accompaniment according to claim 1, characterized in that: The step of inputting the Mel-frequency feature map into a pre-trained convolutional neural network, outputting an initial BPM value, and then obtaining the BPM range by shifting the value vertically around it includes: The Mel spectrum feature map is input into a pre-trained convolutional neural network based on a residual convolutional network structure, and its output is mapped through a global pooling layer and a fully connected layer to obtain the initial BPM value of the input music. Centered on the initial BPM value, offset upwards and downwards by 20% respectively to generate the BPM range including the lower limit and upper limit of BPM, and output the BPM range as the prior velocity constraint of the dynamic Bayesian network in S3.

6. The intelligent analysis and arrangement method for music accompaniment according to claim 1, characterized in that: The steps of inputting the beat probability value, the downbeat probability value, and the BPM range into a dynamic Bayesian network for post-processing to obtain the precise measure timing and the final BPM value of the input music include: A state space for a dynamic Bayesian network is constructed, which includes three hidden state variables: beat period, beat phase, and re-beat marker. The range of values ​​for the beat period is obtained by mapping the range of BPM. The beat phase represents the relative position of the current frame within one beat period. The re-beat marker is used to mark whether the current frame is a re-beat position. Based on the beat probability value and beat phase of the current frame, a matching function for the beat observation probability is defined; based on the repeat probability value and repeat position of the current frame, a matching function for the repeat observation probability is defined. The Viterbi algorithm is used to perform global decoding within the range of the beat period, and the optimal state sequence is selected that maximizes the joint probability of the observation sequence that maximizes the beat observation probability and the re-beat observation probability, together with the hidden state sequence of the beat period, beat phase and re-beat marker. The decoded beat period is determined from the optimal hidden state sequence, and the final BPM value of the input music is calculated based on the beat period. The frame position where the beat phase returns to zero is extracted from the optimal state sequence as the precise beat time point, and the frame position marked as true by the re-beat is further extracted from the precise beat time point as the precise re-beat time point. The frame interval between adjacent precise retake time points is defined as a segment based on the preset time number. Each precise retake time point is used as the start time of the corresponding segment to generate a sequence of precise segment time points.

7. The intelligent analysis and arrangement method for music accompaniment according to claim 1, characterized in that: The step of inputting the Mel-spectrum feature map into a pre-trained self-attention neural network and outputting the segmentation type of each frame of audio signal includes: The Mel spectrum feature map is input into a pre-trained self-attention neural network based on a structure that fuses convolutional layers and self-attention layers, and the convolutional layers extract local time-frequency features from the Mel spectrum feature map; The local time-frequency features are compressed along the frequency dimension while retaining the time dimension to obtain a frame-level time feature sequence. The inter-frame dependencies at different time positions in the frame-level time feature sequence are calculated through the self-attention layer. A fully connected layer is connected after the self-attention layer. Based on the inter-frame dependency relationship, the probability of each frame belonging to each preset segment type is output, and the category corresponding to the highest probability is taken as the segment type of the frame.

8. The intelligent analysis and arrangement method for music accompaniment according to claim 7, characterized in that: After obtaining the segment type of each frame signal, the process also includes a step of further combining the precise measure time points to obtain the complete segment information of the input music: The precise time points of each section are obtained, and based on the sampling rate and frame shift parameters of the audio signal, each precise time point is converted into a corresponding frame index, which serves as the starting frame of the corresponding section. For each subsection, count the frequency of segment types in all frames within that subsection, and take the segment type with the highest frequency as the segment type of that subsection. Traverse all sections along the timeline, merge adjacent sections with the same paragraph type into one paragraph, and record the start and end times of each paragraph as well as the paragraph type. Map the start and end timestamps of each segment back to the audio timeline to generate complete segmentation information including start time, end time, and segment type.

9. The intelligent analysis and arrangement method for music accompaniment according to claim 1, characterized in that: The step of inputting the Mel spectral feature map into a pre-trained deep auditory model to identify the chord type probability value of each frame of audio includes: The Mel spectral feature map is input into a fully convolutional deep auditory model based on a fully convolutional network structure. The deep auditory model is composed of multiple stacked convolutional blocks, each of which contains a convolutional layer, a batch normalization layer, and an activation function. The last convolutional block of the fully convolutional network structure outputs a two-dimensional feature map with time and channel dimensions. In the time dimension, each time position corresponds to a frame in the audio signal of the input music, and in the channel dimension, each channel corresponds to a preset chord type. The Softmax function is applied to the channel dimension of the dual-dimensional feature map to convert the channel value corresponding to each time position into a probability distribution, thereby obtaining the probability that each frame in the audio signal belongs to each preset chord type, which is used as the probability value of the chord type.

10. The intelligent analysis and arrangement method for music accompaniment according to claim 9, characterized in that: After obtaining the chord type probability value, the method further includes the step of decoding the chord type probability value using a linear chain conditional random field to obtain the chord sequence of the input music. A linear chain conditional random field model is constructed, which includes a state transition feature function and a state feature function. The state feature function is defined based on the probability value of chord type in each frame, and the state transition feature function is used to characterize the transition constraint of chord type between adjacent frames. The complete chord type probability value sequence of the audio signal is jointly decoded using the linear chain conditional random field model, and the chord type sequence that maximizes the global conditional probability is selected as the decoding result. In the decoded chord type sequence, consecutive frames with the same chord type are merged into a chord segment, and the start frame position, end frame position, and chord type of each chord segment are recorded; The start and end frame positions of each chord segment are mapped to start and end times on the audio timeline to generate a chord sequence containing start time, end time, and chord type.

Images