Chorus detection method, device, equipment and storage medium

Abstract

The present application discloses a chorus detection method, apparatus, device, and storage medium, relating to the field of audio processing. The method comprises: invoking a melody feature extraction model to extract features from a song to be detected, thereby obtaining song features; invoking a melody feature extraction model to extract features from a reference chorus segment, thereby obtaining chorus features; the reference chorus segment is a calibrated chorus segment in the song to be detected; multiplying the song features by the chorus features to calculate a correlation matrix between the song features and the chorus features; the correlation matrix is ​​a two-dimensional matrix, wherein the number of rows in the correlation matrix corresponds to the number of audio frames in the chorus segment, and the number of columns in the correlation matrix corresponds to the number of audio frames in the song to be detected; determining a target column number interval in the correlation matrix whose values ​​meet a threshold condition; and determining the target audio frame interval of the song to be detected corresponding to the target column number interval as the predicted chorus segment of the song to be detected. This method can improve chorus annotation efficiency.

Description

Technical Field

[0001] The present application relates to the field of audio processing, and in particular to a chorus detection method, apparatus, device and storage medium. Background Art

[0002] Modern pop music usually consists of a prelude, verse, chorus, interlude, etc. The chorus, also known as the climax of the song, usually contains multiple chorus segments in a song.

[0003] An excellent chorus detection model can accurately detect the start and end times of each chorus segment in a song. Training an excellent chorus detection model is a hot research topic in the field of music information retrieval, and organizing training data for the chorus detection model is a crucial step. The quality and quantity of training data directly determine the performance of the chorus detection model. Only by utilizing large, high-quality datasets can an excellent chorus detection model be trained.

[0004] However, the labeling process of chorus detection training datasets is usually done manually, which is very time-consuming and labor-intensive. Summary of the Invention

[0005] The embodiments of the present application provide a chorus detection method, apparatus, device, and storage medium, which can improve the efficiency of chorus annotation. The technical solution is as follows.

[0006] According to one aspect of the present application, a chorus detection method is provided, the method comprising:

[0007] Calling a melody feature extraction model to perform feature extraction on a song to be detected to obtain song features; calling the melody feature extraction model to perform feature extraction on a reference chorus segment to obtain chorus features; the reference chorus segment is a calibrated chorus segment in the song to be detected; the song features and the chorus features are both two-dimensional matrices, the width of the two-dimensional matrix represents the frequency range, and the length of the two-dimensional matrix represents the number of audio frames;

[0008] Multiplying the song feature and the chorus feature to calculate a correlation matrix between the song feature and the chorus feature; the correlation matrix is ​​a two-dimensional matrix, the number of rows of the correlation matrix corresponds to the number of audio frames of the chorus segment, and the number of columns of the correlation matrix corresponds to the number of audio frames of the song to be detected;

[0009] Determine a target column number interval in the correlation matrix whose values ​​meet a threshold condition;

[0010] The target audio frame interval of the song to be detected corresponding to the target column number interval is determined as the predicted chorus segment of the song to be detected.

[0011] According to another aspect of the present application, a chorus detection device is provided, the device comprising:

[0012] A feature extraction module is configured to call a melody feature extraction model to perform feature extraction on a song to be detected to obtain song features; call the melody feature extraction model to perform feature extraction on a reference chorus segment to obtain chorus features; the reference chorus segment is a calibrated chorus segment in the song to be detected; the song features and the chorus features are both two-dimensional matrices, the width of the two-dimensional matrix represents the frequency range, and the length of the two-dimensional matrix represents the number of audio frames;

[0013] a correlation module, configured to multiply the song feature by the chorus feature to calculate a correlation matrix between the song feature and the chorus feature; the correlation matrix being a two-dimensional matrix, the number of rows of the correlation matrix corresponding to the number of audio frames of the chorus segment, and the number of columns of the correlation matrix corresponding to the number of audio frames of the song to be detected;

[0014] A determination module, configured to determine a target column number interval in which values ​​in the correlation matrix meet a threshold condition;

[0015] The determination module is used to determine the target audio frame interval of the song to be detected corresponding to the target column number interval as the predicted chorus segment of the song to be detected.

[0016] According to another aspect of the present application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the chorus detection method described above.

[0017] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the chorus detection method described above.

[0018] According to another aspect of the embodiments of the present disclosure, a computer program product or computer program is provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the chorus detection method provided in the above-mentioned optional implementation.

[0019] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0020] A reference chorus segment from an already annotated song is used to automatically detect other chorus segments in the song. Features of the reference chorus segment and the song are extracted separately, and the correlation between the two features is calculated. Based on the correlation calculation results, segments with a high correlation with the reference chorus segment are matched from the song, and the chorus segments in the song are then annotated. This improves the efficiency of chorus annotation and saves costs. Obtaining a chorus detection model training set using the method proposed in the embodiments of this application can significantly improve the accuracy of the chorus detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 is a block diagram of a computer device provided by an exemplary embodiment of the present application;

[0023] Figure 2 is a method flow chart of a chorus detection method provided by another exemplary embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a chorus detection method provided by another exemplary embodiment of the present application;

[0025] Figure 4 is a schematic diagram of a chorus detection method provided by another exemplary embodiment of the present application;

[0026] Figure 5 is a schematic diagram of a chorus detection method provided by another exemplary embodiment of the present application;

[0027] Figure 6 is a method flow chart of a chorus detection method provided by another exemplary embodiment of the present application;

[0028] Figure 7 is a method flow chart of a chorus detection method provided by another exemplary embodiment of the present application;

[0029] Figure 8 is a schematic diagram of a chorus detection method provided by another exemplary embodiment of the present application;

[0030] Figure 9 is a method flow chart of a chorus detection method provided by another exemplary embodiment of the present application;

[0031] Figure 10is a method flow chart of a chorus detection method provided by another exemplary embodiment of the present application;

[0032] Figure 11 is a block diagram of a chorus detection device provided by another exemplary embodiment of the present application;

[0033] Figure 12 is a structural diagram of a server provided by another exemplary embodiment of the present application;

[0034] Figure 13 is a block diagram of a terminal provided by another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0036] Figure 1 A schematic diagram of a computer device 101 provided in an exemplary embodiment of the present application is shown. The computer device 101 may be a terminal or a server.

[0037] Terminals may include smartphones, laptops, desktop computers, tablets, all-in-one computers, Internet of Things devices, intelligent robot workstations, televisions, set-top boxes, smart glasses, smart watches, digital cameras, MP4 playback terminal devices, MP5 playback terminal devices, learning machines, reading machines, e-books, electronic dictionaries, in-vehicle terminal devices, virtual reality (VR) playback terminal devices or augmented reality (AR) playback terminal devices, etc.

[0038] In an optional implementation, the chorus detection method provided in this application can be applied to an application with a chorus detection function, which can be an audio playback application, an audio processing application, an audio recognition application, a video processing application, an audio publishing application, a video publishing application, a social application, a shopping application, a live broadcast application, a forum application, an information application, a lifestyle application, an office application, etc. Optionally, a client of the application is installed on the terminal.

[0039] The terminal and the server are connected to each other through a wired or wireless network.

[0040] The terminal includes a first memory and a first processor. The first memory stores a chorus detection algorithm 102; the chorus detection algorithm is called and executed by the first processor to implement the chorus detection method provided in this application. The first memory may include, but is not limited to, the following: random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM).

[0041] The first processor may be composed of one or more integrated circuit chips. Alternatively, the first processor may be a general-purpose processor, such as a central processing unit (CPU) or a network processor (NP). Alternatively, the first processor may implement the chorus detection method provided in the present application by running a program or code.

[0042] The server includes a second memory and a second processor. The second memory stores a chorus detection algorithm 102, which is invoked by the second processor to implement the chorus detection method provided herein. Optionally, the second memory may include, but is not limited to, RAM, ROM, PROM, EPROM, and EEPROM. Optionally, the second processor may be a general-purpose processor, such as a CPU or NP.

[0043] Figure 2 FIG1 shows a flow chart of a chorus detection method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, for example, Figure 1 The method includes the following steps.

[0044] Step 210: Call the melody feature extraction model to extract features of the song to be detected to obtain song features; call the melody feature extraction model to extract features of the reference chorus segment to obtain chorus features; the reference chorus segment is a calibrated chorus segment in the song to be detected; the song features and chorus features are both two-dimensional matrices, the width of the two-dimensional matrix represents the frequency range, and the length of the two-dimensional matrix represents the number of audio frames.

[0045] The melody feature extraction model is a neural network model. Optionally, the melody feature extraction model includes a preprocessing layer and a feature extraction layer.

[0046] The preprocessing layer is used to preprocess the input data (the song to be detected or the reference chorus segment) to obtain the mel-spectrogram features of the input data. The preprocessing can include at least one of the following: time-frequency conversion, filtering, and mel-spectrogram transformation.

[0047] Taking the song to be detected as an example, after the song to be detected is input into the preprocessing layer, the preprocessing layer performs STFT (Short-Time Fourier Transform) on the song to be detected to obtain at least two spectra corresponding to at least two frames of audio; Mel transform is performed on the at least two spectra to obtain Mel spectrum features, which include Mel spectrum features corresponding to at least two frames of audio.

[0048] STFT is a method for signal analysis in time and frequency, which decomposes the signal into spectral components in different time periods.

[0049] The STFT process can include the following two steps:

[0050] 1) Split the audio signal (the song to be detected) into at least two audio frames. The duration of each audio frame is fixed, such as 10ms or 20ms. There can also be a certain overlap between adjacent audio frames, with a common overlap ratio of 50% or 75%.

[0051] 2) Performing Discrete Fourier Transform (DFT) calculation on each audio frame to convert the time domain signal into a frequency domain signal to obtain the spectrum of each audio frame.

[0052] Exemplarily, the spectrum of each frame of audio includes amplitude values ​​(or energy values) at at least two frequencies. For example, a spectrum includes 256 amplitude values ​​at 256 frequencies.

[0053] Mel transform can convert a spectrum into a mel spectrum.

[0054] For example, a method for converting a spectrum into a Mel spectrum is given. For the spectrum of each frame of audio, the frequency is mapped to the Mel scale: the Mel scale has a logarithmic relationship with the frequency (i.e., the number of Hertz), and the frequency f (in Hertz) can be converted to the Mel scale m (in Mel) using the following formula: m = 2595*log10(1+f / 700). Among them, 700 is the reference frequency, which can also be adjusted to other values. The amplitude values ​​in the spectrum are logarithmically transformed to enhance the contrast of the low-amplitude part, and finally the Mel spectrum of each frame of audio is obtained.

[0055] Exemplarily, after the preprocessing layer preprocesses the song to be detected, the mel spectrum feature of the song to be detected is obtained. The preprocessing process can be: first, the song to be detected is divided into frames according to a preset time window, and then each frame is subjected to time-frequency conversion and mel transform to obtain the mel spectrum of each frame of audio. The mel spectrum of multiple frames of audio constitutes the mel spectrum feature of the song to be detected. For example, the time window can be set to 30ms (milliseconds), then a one-minute song to be detected can be divided into 2000 frames, and a mel spectrum can be obtained for each frame of audio, for a total of 2000 mel spectrums. The mel spectrum feature is the mel spectrum of 2000 frames.

[0056] Exemplarily, the mel spectrum in the mel spectrum feature is arranged in the time sequence of each frame of audio, that is, in the order of the mel spectrum of the first frame of audio, the mel spectrum of the second frame of audio, the mel spectrum of the third frame of audio, and so on.

[0057] Mel-spectrogram is a spectral identification method, also known as Mel-spectrogram. Mel-spectrogram is based on a spectrogram, which is obtained by analyzing an audio signal in time and frequency. The spectrogram is obtained by performing a short-time Fourier transform (STFT) on the audio signal in the time domain. Mel-spectrogram performs a series of transformations on the spectrogram to better adapt to the characteristics of human auditory perception. Mel-spectrogram uses the Mel scale instead of the linear frequency scale in the spectrogram. The Mel scale is more consistent with the characteristics of human auditory perception and can better reflect people's subjective perception of pitch.

[0058] Exemplarily, a Mel-spectrogram feature can be m*n dimensional data, where m is the number of audio frames and n is the amount of data in a Mel-spectrogram. For example, after time-frequency conversion and Mel-transformation of a song to be detected, a 2000-frame*256-dimensional Mel-spectrogram feature can be obtained. The song to be detected includes 2000 audio frames, and each audio frame is transformed to obtain 256 spectral data. The 256 spectral data can be 256 amplitude data at 256 Mel-scale (frequencies).

[0059] The feature extraction layer is used to extract the melody features from the data output by the preprocessing layer (Mel spectrum features).

[0060] Optionally, the feature extraction layer may employ a model architecture comprising a first RNN (Recurrent Neural Network), a CNN (Convolutional Neural Network), and a second RNN connected sequentially. For example, the first RNN and the second RNN each employ a bidirectional LSTM (Long Short Term Memory Network) model, and the CNN employs a 5-layer CNN grid. Thus, the feature extraction layer employs a network structure comprising a first LSTM, a 5-layer CNN, and a second LSTM connected sequentially.

[0061] Alternatively, the RNN can adopt the GRU (Gate Recurrent Unit) model. The CNN can adopt the ResNet (Residual Network).

[0062] Through the above feature extraction layer, we can first extract the most original time series features in the audio through LSTM, then use the CNN network to model the local features of the time series features, extract the corresponding melody features in each time frame, and finally use the LSTM model to link the local features with the context features to obtain the final melody features.

[0063] The reference chorus segment is an audio segment of the song to be detected. Exemplarily, the reference chorus segment is a manually annotated chorus segment of the song to be detected.

[0064] Exemplarily, the audio formats of the song to be detected and the reference chorus segment input to the melody feature extraction model are not limited. Optionally, the audio formats of the song to be detected and the reference chorus segment are the same.

[0065] Optionally, the features output by the melody feature extraction model are a two-dimensional matrix, the length of the two-dimensional matrix depends on the audio duration of the input data, and the width of the two-dimensional matrix is ​​a fixed width, wherein the width of the two-dimensional matrix is ​​used to represent the frequency range.

[0066] For example, the length of a song feature is equal to the number of audio frames obtained after the preprocessing layer segments the song to be detected. If the song to be detected is divided into 2000 frames and the frequency range includes amplitude data on a 256-Mel scale, the song feature is a two-dimensional matrix of 2000 × 256 dimensions. 2000 is the length (number of columns) of the two-dimensional matrix, and 256 is the width (number of rows) of the two-dimensional matrix.

[0067] For example, the length of the chorus feature is equal to the number of audio frames obtained after the preprocessing layer frames the reference chorus segment. If the reference chorus segment is divided into 100 frames and the frequency range includes amplitude data on the 256 Mel scale, the chorus feature is a two-dimensional matrix of 100 × 256 dimensions. Here, 100 is the length (number of columns) of the two-dimensional matrix, and 256 is the width (number of rows) of the two-dimensional matrix.

[0068] Exemplarily, the width (number of rows) of the song feature and the chorus feature are the same, and the length (number of columns) of the song feature is greater than the length (number of columns) of the chorus feature.

[0069] For example, Figure 3 As shown, it is a song feature 401, whose length is equal to the number of audio frames of the song to be detected, width is equal to the frequency range, color depth represents the numerical value, and the highlighted part in the figure can be used to characterize the melody trend of the song to be detected.

[0070] In step 220, the song features are multiplied by the chorus features to calculate a correlation matrix between the song features and the chorus features. The correlation matrix is ​​a two-dimensional matrix, the number of rows of the correlation matrix corresponds to the number of audio frames of the chorus segment, and the number of columns of the correlation matrix corresponds to the number of audio frames of the song to be detected.

[0071] For example, the two-dimensional matrix of the chorus is transposed, and matrix multiplication is performed on the transposed chorus features and the song features to obtain a correlation matrix.

[0072] For example, the dimension of a song feature is [A, B], where A is the frequency range and B is the number of audio frames in the song to be tested. The dimension of a chorus feature is [A, C], where A is the frequency range and C is the number of audio frames in the reference chorus segment. The chorus feature is then transformed into [C, A], and matrix multiplication of the transformed chorus feature with the song feature yields the correlation matrix: [C, A] * [A, B] = [C, B]. In other words, the length (number of columns) of the correlation matrix equals the number of audio frames (B) in the song to be tested, and the width (number of rows) of the correlation matrix equals the number of audio frames (C) in the reference chorus segment.

[0073] For example, if the dimension of the song feature is 256*2000 and the dimension of the chorus feature is 256*100, then the dimension of the correlation matrix is ​​100*2000.

[0074] For example, Figure 4 As shown, a correlation matrix 402 is shown, whose length is equal to the number of audio frames of the song to be detected, and whose width is equal to the number of audio frames of the reference chorus segment. The color depth represents the degree of correlation, and the highlighted part in the figure represents high correlation.

[0075] Step 230: Determine a target column number interval in the correlation matrix whose values ​​meet a threshold condition.

[0076] For example, the value of the value in the correlation matrix can represent the correlation between the song to be detected and the reference chorus segment at that position. For example, the value of the point (x, y) in the correlation matrix can represent the correlation between the melody of audio frame y in the song to be detected and the melody of audio frame x in the reference chorus segment. If the value is high, it means that the melody of audio frame y in the song to be detected is similar to that of audio frame x in the reference chorus segment; if the value is low, it means that the melody of audio frame y in the song to be detected is not similar to that of audio frame x in the reference chorus segment.

[0077] Therefore, the segments of the song to be detected that are highly correlated with the reference chorus segments appear as highlighted (high value) diagonals in the correlation matrix. By identifying the highlighted diagonals in the correlation matrix, the chorus segments in the song to be detected can be determined.

[0078] Suppose that frames 11 through 20 of the song to be tested are the chorus segment. Frame 11 of the song to be tested has a high correlation with frame 1 of the reference chorus segment, meaning that the value at position (1, 11) in the correlation matrix is ​​high. Frame 12 of the song to be tested has a high correlation with frame 2 of the reference chorus segment, meaning that the value at position (2, 12) in the correlation matrix is ​​high. Frame 13 of the song to be tested has a high correlation with frame 3 of the reference chorus segment, meaning that the value at position (3, 13) in the correlation matrix is ​​high. Frame 20 of the song to be tested has a high correlation with frame 10 of the reference chorus segment, meaning that the value at position (10, 20) in the correlation matrix is ​​high. As can be seen, the highlighted diagonal lines in the correlation matrix (from top left to bottom right) correspond to the chorus segments of the song to be tested.

[0079] Since the number of columns of the correlation matrix is ​​equal to the number of audio frames of the song to be detected, by determining the column interval where the diagonal line with a value higher than the threshold is located, the audio frame where the chorus segment of the song to be detected is located can be determined.

[0080] The threshold condition may include: the value on the diagonal within the column number interval in the correlation matrix is ​​higher than the threshold.

[0081] Optionally, the width of the column interval depends on the number of audio frames in the reference chorus segment (i.e., the width of the correlation matrix). That is, the width of the target column interval is equal to the width of the correlation matrix. For example, if the dimension of the correlation matrix is ​​100*2000, the width of the target column interval is 100. For example, the target column interval may include columns 11 to 110 of the correlation matrix.

[0082] In step 240 , the target audio frame interval of the song to be detected corresponding to the target column number interval is determined as the predicted chorus segment of the song to be detected.

[0083] For example, if the diagonal values ​​from the 11th column to the 20th column in the similarity matrix are higher than the threshold, then correspondingly, the 11th frame to the 20th frame in the song to be detected are the predicted chorus segments.

[0084] Optionally, the similarity matrix may include at least one target column number interval that meets a threshold condition, each target column number interval may determine a predicted chorus segment, and multiple target column number intervals may determine multiple predicted chorus segments.

[0085] Exemplarily, the predicted chorus segment determined in step 240 includes the reference chorus segment. For example, if the song to be tested includes three chorus segments and the reference chorus segment is the second chorus segment, then step 240 will determine the three chorus segments (predicted chorus segments) of the song to be tested.

[0086] Illustratively, the method provided in the embodiments of the present application can be used to collect training sample data for a chorus detection model, and use the training sample data to train the chorus detection model.

[0087] Exemplarily, the song to be detected is determined as sample input data, and the predicted chorus segment is determined as the sample chorus label; the chorus detection model is called to output the predicted chorus interval based on the sample input data; and the chorus detection model is trained until convergence based on the loss of the predicted chorus interval and the sample chorus label.

[0088] Exemplarily, the audio frame number where the predicted chorus segment is located is determined as the sample chorus label, and the chorus detection model is trained to output the audio frame number where the chorus segment is located in the song to be detected (predicted chorus interval).

[0089] For example, Figure 5 As shown, the song to be detected 501 is input into the melody feature extraction model to obtain song features 503; the reference chorus segment 502 is input into the melody feature extraction model to obtain chorus features 504.

[0090] Among them, the song to be detected 501 is an audio signal, and its signal length is equal to the duration of the song to be detected. The song feature 503 is a two-dimensional matrix, and its matrix length is equal to the number of audio frames of the song to be detected (the audio signal is framed to obtain at least one audio frame), and its width is equal to the frequency range preset by the model. The reference chorus segment 502 is an audio signal, and its signal length is equal to the duration of the reference chorus segment. The chorus feature 504 is a two-dimensional matrix, and its matrix length is equal to the number of audio true frames of the reference chorus segment (the audio signal of the reference chorus segment is framed to obtain at least one audio frame), and its width is equal to the frequency range preset by the model. Optionally, assuming that the song to be detected 501 contains two chorus segments, the reference chorus segment can be any one of the chorus segments.

[0091] Then, the two-dimensional matrix of song features is transposed, and matrix multiplication is performed on the chorus features and the transposed song features to obtain a correlation matrix 505. The matrix length of the correlation matrix 505 is equal to the number of audio frames of the song to be detected, and the matrix width of the correlation matrix 505 is equal to the number of audio frames of the reference chorus segment.

[0092] The column number interval corresponding to the target window matrix where the values ​​on the diagonal of the correlation matrix 505 are all higher than the threshold is the audio frame number interval where the chorus segment of the song to be detected is located. Based on the audio frame number where the target window matrix is ​​located, the predicted chorus segment 506 can be directly determined from the song to be detected.

[0093] In summary, the method provided in this embodiment utilizes a reference chorus segment from an already marked song to automatically detect other chorus segments in the song. The features of the reference chorus segment and the song are extracted separately, and the correlation between the two features is calculated. Based on the correlation calculation results, segments with a high correlation with the reference chorus segment are matched from the song, and the chorus segments in the song are then annotated. This improves the efficiency of chorus annotation and saves costs. Obtaining a chorus detection model training set using the method proposed in this embodiment of the application can significantly improve the accuracy of the chorus detection model.

[0094] The embodiments of the present application provide two exemplary methods for determining a target column number interval from a correlation matrix.

[0095] The first one: See Figure 6 , shows a flow chart of a chorus detection method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, for example, Figure 1 Based on the terminal or server shown. Figure 2 In the illustrated embodiment, step 230 includes step 231 .

[0096] Step 231 : Determine the values ​​in the correlation matrix that are higher than the first threshold as candidate points; and determine the column number interval in which the candidate points in the correlation matrix are diagonally distributed as the target column number interval.

[0097] For example, Figure 4 As shown, candidate points (highlighted points) with values ​​higher than the first threshold in the correlation matrix 402 are selected, and then a target column number interval in which the candidate points are diagonally distributed in the correlation matrix is ​​selected, wherein the width of the target column number interval is equal to the width of the correlation matrix, that is, the slope of the diagonal line is 45°.

[0098] Second: See Figure 7 , shows a flow chart of a chorus detection method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, for example, Figure 1Based on the terminal or server shown. Figure 2 In the illustrated embodiment, step 230 includes steps 232 to 235 .

[0099] Step 232: Use a sliding window to slide in the correlation matrix according to a preset step size, and obtain at least one window matrix from the correlation matrix. The number of rows and columns of the sliding window is the same as the number of rows of the correlation matrix.

[0100] Exemplarily, the width of the sliding window is equal to the width (number of rows) of the correlation matrix, and the length of the sliding window is equal to the width (number of rows) of the correlation matrix, that is, the sliding window is a square with equal length and width.

[0101] Taking the dimension of the correlation matrix as 100*2000 and the preset stride as 1 as an example, the dimension of the sliding window is 100*100. Then, the sliding window is used to slide from the initial position of the correlation matrix, and a total of 2000 window matrices can be intercepted, and the dimension of each window matrix is ​​100*100.

[0102] Step 233 : For each window matrix in the at least one window matrix, calculate the sum of the values ​​on the first diagonal to obtain a reference value corresponding to each window matrix; the first diagonal includes a diagonal from the upper left corner to the lower right corner of the window matrix.

[0103] Add the correlation values ​​on the diagonal from the upper left to the lower right of each window matrix to get the sum of the values ​​on the diagonal. If the current window matrix is ​​a chorus segment, the sum of the correlation values ​​on the diagonal is the largest.

[0104] After calculating the sum of the values ​​of the diagonals in each window matrix, a one-dimensional vector can be obtained. The value of the one-dimensional vector is the sum of the values ​​of the diagonals in each window matrix. The one-dimensional vectors are arranged in the order of the window matrices. Then, the following can be drawn based on the one-dimensional vector: Figure 8 In the line graph 403 shown, the window matrix corresponding to the peak point in the line graph 403 is the interval corresponding to the chorus segment.

[0105] Step 234: Determine the window matrix corresponding to the target peak point in at least one reference value as the target window matrix.

[0106] For example, Figure 8 As shown, after the reference value of each window matrix is ​​plotted as a line graph 403, three peak points can be obtained. The window matrix corresponding to each peak point is the position of the chorus segment.

[0107] Optionally, since the chorus segment of a song is typically repeated 2-4 times, the chorus segment can be screened based on the number of peak points. Specifically, the target peak point is determined by obtaining a candidate peak point from the at least one reference value whose reference value is greater than a second threshold; and if the number of candidate peak points falls within a preset range, determining the candidate peak point as the target peak point.

[0108] The second threshold is used to filter out peak points with small reference values ​​(small sum of similarities on the diagonal). The preset number interval can be set to 2 to 4 to filter out abnormal songs with less than or equal to 1 chorus segment and more than 4 chorus segments.

[0109] Optionally, when the number of the candidate peak points is not within a preset number range, the song to be detected is output as an abnormal song.

[0110] Step 235: Determine the column number interval corresponding to the target window matrix in the correlation matrix as the target column number interval.

[0111] For example, if the number of columns of the target window matrix in the correlation matrix ranges from 10 to 100, then the audio segments from the 10th frame to the 100th frame in the song to be detected are chorus segments.

[0112] In summary, the method provided in this embodiment uses a melody feature extraction model to extract the melody features of the reference chorus segment and the song to be detected, respectively. The melody features of the reference chorus segment are then used to match highly correlated segments from the melody features of the song to be detected, thereby selecting the predicted chorus segment from the song to be detected. This method can directly determine the audio frame interval where the predicted chorus segment is located from the correlation matrix of the two features, thereby improving the efficiency of chorus annotation and saving costs. Obtaining the chorus detection model training set using the method proposed in this embodiment of the application can significantly improve the accuracy of the chorus detection model.

[0113] The method provided in this embodiment uses a melody feature extraction model to extract the melody features of a reference chorus segment. Based on the principle of melody similarity, other chorus segments in the song to be detected are found based on the given reference chorus segment, thereby reducing the annotation cost of the chorus detection model training dataset and increasing the accuracy of the chorus annotation dataset.

[0114] The method provided in this embodiment extracts melodic features from a known reference chorus segment and the entire song to be tested. These melodic features are then combined through matrix multiplication to calculate the melodic feature correlation, resulting in a melodic correlation matrix. The melody correlation matrix is ​​then mapped to a one-dimensional vector to create a melody correlation vector, facilitating the search for similar segments. A peak detection algorithm is then used to detect peaks in the melody correlation vector. After filtering these peaks, other chorus segments within the song can be retrieved, improving the efficiency of chorus segment detection.

[0115] Exemplarily, the melody feature extraction model includes a preprocessing layer and a feature extraction layer. A training method for the melody feature extraction model is given below.

[0116] Figure 9 FIG1 shows a flow chart of a chorus detection method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, for example, Figure 1 The method is executed by the terminal or server shown in the figure. The method includes the following steps.

[0117] Step 310: Obtain sample data, where the sample data includes sample audio and sample labels. The sample labels are used to label the vocal melody of the sample audio.

[0118] For example, sample data can be obtained using the following method:

[0119] (1) Extract the fundamental frequency of each frame of audio in a cappella songs to obtain the a cappella melody of the a cappella songs, where the a cappella songs are unaccompanied human voice audio.

[0120] (2) According to the mapping relationship between fundamental frequency and pitch category, the pitch category corresponding to the fundamental frequency of each frame of audio in the a cappella melody is determined to obtain a sample label corresponding to the a cappella song; the pitch category includes at least one pitch category obtained by dividing the frequency range of the fundamental frequency into at least one sub-range.

[0121] (3) Acquire an accompaniment audio library, wherein the accompaniment audio library includes at least one segment of accompaniment audio.

[0122] (4) The a cappella song is weightedly mixed with a section of accompaniment audio in the accompaniment audio library to obtain a sample audio, and the sample label corresponding to the sample audio is the sample label corresponding to the a cappella song.

[0123] For example, a large number of a cappella songs (songs with only human voice and no accompaniment) are collected and the signal processing algorithm pYIN is used to extract the melody from these songs. The unit of melody is Hertz (Hz) (the melody in the song refers to the fundamental frequency). The window length used for melody extraction can be 25ms, and the window movement step is 10ms. Therefore, the final melody (fundamental frequency) is a sequence, and each value in the sequence represents the melody value at that position.

[0124] To facilitate model training, the fundamental frequency is divided into multiple cent categories, and the fundamental frequency values ​​of a cappella songs are mapped to specific cent categories. For example, the model can recognize fundamental frequencies ranging from 49.0Hz (G1) to 1174.66Hz (D6), spanning 55 semitones, each of which is 100 cents. Each semitone is divided into five parts (classes), each corresponding to 20 cents, for a total of 275 cent categories. In addition, a separate cent category is set to represent no melody (no singing), bringing the total number of cent categories to 276.

[0125] Then, a large number of accompaniments (only accompaniment without vocals) and common noises in life were collected and randomly mixed with a cappella songs during training to increase the generalization performance of the model.

[0126] To enable the model to accurately extract the melodic characteristics of the human voice from the original song (including accompaniment), accompaniment was randomly added to the training data (a cappella songs) during the training process. Specifically, the a cappella songs and their corresponding sample labels were cut into segments of equal length, each 10 seconds long. During training, for each input segment, an accompaniment audio was randomly selected from the accompaniment audio library. Then, 10 seconds of the selected accompaniment audio was randomly intercepted and mixed with the a cappella segment. The mixing weight of the a cappella segment was 1, and the mixing weight of the accompaniment audio was a random value in the range of 0.3 to 0.9.

[0127] To improve the model's noise tolerance, during training, we replaced the a cappella segments with pure instrumental audio with a probability of 1%, and assigned the corresponding labels to the cent categories corresponding to the absence of melody. This approach effectively prevented the model from mistakenly extracting melodies in places where there was no singing.

[0128] Step 320: Call the pre-processing layer to perform time-frequency conversion and filtering on the sample audio to obtain the Mel spectrum features of the sample audio.

[0129] Optionally, for the processing process of the pre-processing layer, please refer to the relevant description in step 210.

[0130] Step 330: Call the feature extraction layer to extract the Mel spectrum features to obtain audio features.

[0131] Optionally, for the processing of the feature extraction layer, please refer to the relevant description in step 210.

[0132] Step 340: Call the classification layer to classify the audio features into melody to obtain a predicted melody.

[0133] The classification layer is used to output the cent category (fundamental frequency) of each frame of audio in the sample audio based on the audio features, obtain multiple cent categories of multiple frames of audio, and then obtain the predicted melody of the sample audio (the cent category of multiple frames of audio).

[0134] In step 350 , the feature extraction layer and classification are trained until convergence based on the loss of the predicted melody and the sample label.

[0135] For example, a cross entropy loss function may be used to calculate the loss between the predicted melody and the sample label. For example, a Focal-Loss loss function may be used to calculate the loss between the predicted melody and the sample label.

[0136] In summary, the method provided in this embodiment provides a training method for a melody feature extraction model. This method uses the traditional signal processing algorithm pYIN to extract melodies from a cappella songs, which serve as training labels for the melody feature extraction model. The melodies are discretized into 276 categories to facilitate model training. Furthermore, during the melody feature extraction model training, accompaniment audio is randomly added to a cappella songs. This allows the model to accurately extract melodic features from songs with accompaniment, improving the model's robustness and generalization capabilities.

[0137] This embodiment of the present application proposes a method for organizing a training dataset for a chorus detection model. This method accurately identifies other chorus segments within each song based on a manually labeled chorus segment, thereby improving the training effectiveness of the chorus detection model. The following describes in detail how to implement "how to identify other chorus segments within a song based on a given chorus segment."

[0138] Different chorus segments in a song have the same melody trend. Based on this characteristic, a melody feature extraction model can be trained to extract the melody features of the chorus segments. The melody features of the known chorus segments are then correlated with the melody features of the entire song. The segment with the highest correlation is the chorus segment. Figure 10 , the specific process is as follows.

[0139] Step 510: Melody feature extraction model.

[0140] To find other chorus segments in a song based on a given chorus segment, we first need to train a melody feature extraction model.

[0141] The melody feature extraction model is used to extract melody features from songs. The input is the audio signal and the output is the melody feature. After the melody feature extraction model is trained, the features output by the melody feature extraction model are the melody features. The melody feature is a two-dimensional matrix. The length of the matrix represents the number of input audio frames, and the width of the matrix represents the frequency range of the melody. An example of melody feature visualization is as follows: Figure 3 As shown, Figure 3 It is the melody feature of an audio segment (song feature 401), wherein the highlighted part represents the melody trend of the song.

[0142] Step 520: Calculation of melody correlation.

[0143] The calculation of melody correlation is actually the multiplication of two melody feature matrices. Assume that there are two melody feature segments A and B, the dimension of A is [Ta, D], and the dimension of B is [D, Tb]. After calculating the correlation between A and B, the dimension of the correlation matrix obtained is [Ta, Tb]. After visualizing this correlation matrix 402, it is as follows Figure 4 shown.

[0144] The horizontal axis of the correlation matrix represents the length of segment B, and the vertical axis represents the length of segment A. The value of the coordinate [ta, tb] in the correlation matrix represents the correlation between the melody at ta of segment A and the melody at tb of segment B. The correlation is represented by a value of 1 to 0 from the highest to the lowest. Figure 4 In the correlation matrix, we can see three highlighted diagonal lines, located at positions 700-1000, 1390-1690, and 1850-2150, indicating that the melody of Section B (700-1000, 1390-1690, and 1850-2150) all coincides with the melody of Section A. (If Section A is the chorus and B is the entire song, the correlation matrix above shows that Section B (710-1010, 1380-1680, and 1840-2140) is the chorus.)

[0145] In order to improve the accuracy of chorus segment search, the similarity matrix is ​​processed through post-processing methods.

[0146] Step 530: post-processing of the melody correlation matrix.

[0147] In order to accurately find the chorus segment in the melody correlation matrix, the melody correlation matrix is ​​mapped to a one-dimensional vector. The mapping method is to set a square matrix with the same length and width as the vertical axis of the correlation matrix, and use the diagonal of this square matrix to sweep the correlation matrix from left to right. The sum of the diagonals of this square matrix at each position is calculated as the output, and finally a one-dimensional vector with the same length as the correlation matrix is ​​obtained. After normalizing this one-dimensional vector (maximum value is 1 and minimum value is 0), the similarity vector can be obtained. After visualizing this similarity vector, it is as follows Figure 8 As shown, Figure 8 The three peaks are where the chorus is located.

[0148] A peak detection algorithm is used to detect peaks in the similarity vector. The peaks found are the chorus segments to be searched. To improve the accuracy of chorus segment search, the peak detection algorithm only detects peaks greater than 0.8. In addition, the number of detected peaks must be between 2 and 4. Songs with less than or equal to 1 peak or greater than or equal to 5 peaks are considered abnormal and are not searched for chorus segments. (Based on experience, chorus segments in popular songs are usually repeated only 2 to 4 times.)

[0149] In summary, the method provided in this embodiment can find other chorus segments in a song based on a labeled chorus segment. Using this method to obtain a chorus detection model training set can significantly improve the accuracy of the chorus detection model.

[0150] The following is an embodiment of the device of the present application. For details not described in detail in the embodiment of the device, reference can be made to the corresponding records in the above method embodiment, and no further details will be given herein.

[0151] Figure 11 A schematic diagram of the structure of a chorus detection device provided by an exemplary embodiment of the present application is shown. The device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device includes:

[0152] The feature extraction module 701 is configured to call a melody feature extraction model to perform feature extraction on the song to be detected to obtain song features; call the melody feature extraction model to perform feature extraction on a reference chorus segment to obtain chorus features; the reference chorus segment is a calibrated chorus segment in the song to be detected; the song features and the chorus features are both two-dimensional matrices, where the width of the two-dimensional matrix represents the frequency range and the length of the two-dimensional matrix represents the number of audio frames;

[0153] A correlation module 702 is configured to multiply the song features by the chorus features to calculate a correlation matrix between the song features and the chorus features; the correlation matrix is ​​a two-dimensional matrix, the number of rows of the correlation matrix corresponds to the number of audio frames of the chorus segment, and the number of columns of the correlation matrix corresponds to the number of audio frames of the song to be detected;

[0154] A determination module 703 is configured to determine a target column number interval in the correlation matrix whose values ​​meet a threshold condition;

[0155] The determining module 703 is configured to determine the target audio frame interval of the song to be detected corresponding to the target column number interval as the predicted chorus segment of the song to be detected.

[0156] In an optional embodiment, the determination module 703 is used to determine the values ​​in the correlation matrix that are higher than a first threshold as candidate points; and determine the column number interval in which the candidate points in the correlation matrix are diagonally distributed as the target column number interval.

[0157] In an optional embodiment, the determining module 703 is configured to:

[0158] Sliding a sliding window in the correlation matrix according to a preset step size to obtain at least one window matrix from the correlation matrix, wherein the number of rows and columns of the sliding window is the same as the number of rows of the correlation matrix;

[0159] For each window matrix of the at least one window matrix, calculating the sum of the values ​​on a first diagonal to obtain a reference value corresponding to each window matrix; the first diagonal includes a diagonal from the upper left corner to the lower right corner of the window matrix;

[0160] Determine a window matrix corresponding to a target peak point in at least one reference value as a target window matrix;

[0161] The column number interval corresponding to the target window matrix in the correlation matrix is ​​determined as the target column number interval.

[0162] In an optional embodiment, the determining module 703 is configured to:

[0163] Obtaining a candidate peak point whose reference value is higher than a second threshold value from the at least one reference value;

[0164] When the number of the candidate peak points is within a preset number range, determining the candidate peak points as the target peak points;

[0165] The device further comprises:

[0166] When the number of the candidate peak points is not within a preset number range, the song to be detected is output as an abnormal song.

[0167] In an optional embodiment, the melody feature extraction model includes a preprocessing layer and a feature extraction layer; the device further includes:

[0168] A training module 704 is configured to obtain sample data, wherein the sample data includes sample audio and sample labels, wherein the sample labels are used to label the vocal melody of the sample audio;

[0169] The training module 704 is configured to call the preprocessing layer to perform time-frequency conversion and filtering on the sample audio to obtain Mel-spectrogram features of the sample audio;

[0170] The training module 704 is configured to call the feature extraction layer to extract the mel spectrum features to obtain audio features;

[0171] The training module 704 is configured to call a classification layer to perform melody classification on the audio features to obtain a predicted melody;

[0172] The training module 704 is configured to train the feature extraction layer and the classification layer according to the loss between the predicted melody and the sample label until convergence.

[0173] In an optional embodiment, the training module 704 is configured to extract the fundamental frequency of each frame of audio in a cappella song to obtain a cappella melody of the cappella song, wherein the cappella song is an a cappella audio of a human voice;

[0174] The training module 704 is configured to determine, based on a mapping relationship between fundamental frequency and pitch category, a pitch category corresponding to the fundamental frequency of each frame of audio in the a cappella melody, and obtain a sample label corresponding to the a cappella song; the pitch category includes at least one pitch category obtained by dividing the frequency range of the fundamental frequency into at least one subrange;

[0175] The training module 704 is configured to obtain an accompaniment audio library, wherein the accompaniment audio library includes at least one segment of accompaniment audio;

[0176] The training module 704 is used to weightedly mix the a cappella song with a section of accompaniment audio in the accompaniment audio library to obtain a sample audio, and the sample label corresponding to the sample audio is the sample label corresponding to the a cappella song.

[0177] In an optional embodiment, the device further includes:

[0178] A training module 704 is configured to determine the song to be detected as sample input data and the predicted chorus segment as a sample chorus label;

[0179] The training module 704 is configured to call the chorus detection model to output a predicted chorus interval based on the sample input data;

[0180] The training module 704 is configured to train the chorus detection model until convergence based on the loss of the predicted chorus interval and the sample chorus label.

[0181] Figure 12Schematic diagram of the structure of a server provided by one embodiment of the present application. Specifically, server 800 includes a central processing unit (CPU) 801, a system memory 804 including random access memory (RAM) 802 and read-only memory (ROM) 803, and a system bus 805 connecting system memory 804 and CPU 801. Server 800 also includes a basic input / output system (I / O system) 806 that facilitates information transmission between various components within the computer, and a mass storage device 807 for storing an operating system 813, application programs 814, and other program modules 815.

[0182] The basic input / output system 806 includes a display 808 for displaying information and an input device 809, such as a mouse and keyboard, for user account input. Both the display 808 and the input device 809 are connected to the central processing unit 801 via an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include an input / output controller 810 for receiving and processing input from various other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, printer, or other types of output devices.

[0183] The mass storage device 807 is connected to the central processing unit 801 via a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer-readable media provide non-volatile storage for the server 800. In other words, the mass storage device 807 may include computer-readable media (not shown) such as a hard disk or a CD-ROM drive.

[0184] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above-mentioned types. The above-mentioned system memory 804 and mass storage device 807 can be collectively referred to as memory.

[0185] According to various embodiments of the present application, the server 800 may also be connected to a remote computer on a network such as the Internet for operation. That is, the server 800 may be connected to the network 812 via the network interface unit 811 connected to the system bus 805, or the network interface unit 811 may be used to connect to other types of networks or remote computer systems (not shown).

[0186] The present application also provides a terminal, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the chorus detection method provided by each of the above method embodiments. It should be noted that the terminal can be as follows Figure 13 The terminal provided.

[0187] Figure 13 The following is a block diagram of a terminal 900 according to an exemplary embodiment of the present application. Terminal 900 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 900 may also be referred to as a user account device, portable terminal, laptop terminal, desktop terminal, or other similar names.

[0188] Typically, the terminal 900 includes a processor 901 and a memory 902 .

[0189] The processor 901 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 901 may be implemented in at least one of the following hardware forms: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computing operations related to machine learning.

[0190] Memory 902 may include one or more computer-readable storage media, which may be non-transitory. Memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 902 is used to store at least one instruction, which is executed by processor 901 to implement the chorus detection method or chorus detection method provided in the method embodiments of this application.

[0191] In some embodiments, terminal 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908, and a power supply 909.

[0192] The peripheral device interface 903 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0193] The RF circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Exemplarily, the RF circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user account identity module card, and the like. The RF circuit 904 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0194] Display screen 905 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When display screen 905 is a touchscreen display, it is also capable of collecting touch signals on or above the surface of display screen 905. These touch signals can be input as control signals to processor 901 for processing. Display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 905, located on the front panel of terminal 900. In other embodiments, there can be at least two display screens 905, located on different surfaces of terminal 900 or in a foldable design. In still other embodiments, display screen 905 can be a flexible display, located on a curved or foldable surface of terminal 900. Display screen 905 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0195] The camera assembly 906 is used to capture images or videos. Exemplarily, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0196] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user account and the environment, and convert the sound waves into electrical signals that are input into the processor 901 for processing, or input into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there can be multiple microphones, each located in different parts of the terminal 900. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0197] Positioning component 908 is used to locate the current geographic location of terminal 900 to implement navigation or LBS (Location Based Service). Positioning component 908 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, or Russia's Galileo system.

[0198] Power supply 909 is used to power various components in terminal 900. Power supply 909 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 909 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0199] In some embodiments, the terminal 900 further includes one or more sensors 910 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , a fingerprint sensor 914 , an optical sensor 915 , and a proximity sensor 916 .

[0200] The accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 900. For example, the accelerometer 911 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 901 can control the display screen 905 to display the user account interface in either a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 911. The accelerometer 911 can also be used to collect motion data for games or user accounts.

[0201] The gyroscope sensor 912 can detect the orientation and rotation angle of the terminal 900. It can also work with the accelerometer 911 to collect 3D motions of the user account on the terminal 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can implement the following functions: motion sensing (for example, changing the UI based on the user account's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0202] The pressure sensor 913 can be set on the side frame of the terminal 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is set on the side frame of the terminal 900, it can detect the user account's grip signal on the terminal 900, and the processor 901 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the display screen 905, the processor 901 controls the operable controls on the UI interface based on the pressure operation of the user account on the display screen 905. Operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0203] The fingerprint sensor 914 is used to collect the fingerprint of the user account. The processor 901 identifies the identity of the user account based on the fingerprint collected by the fingerprint sensor 914, or the fingerprint sensor 914 identifies the identity of the user account based on the collected fingerprint. When the identity of the user account is identified as a trusted identity, the processor 901 authorizes the user account to perform relevant sensitive operations, such as unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 914 can be set on the front, back, or side of the terminal 900. When a physical button or manufacturer logo is set on the terminal 900, the fingerprint sensor 914 can be integrated with the physical button or manufacturer logo.

[0204] The optical sensor 915 is used to detect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity detected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity detected by the optical sensor 915.

[0205] Proximity sensor 916, also known as a distance sensor, is typically located on the front panel of terminal 900. Proximity sensor 916 is used to detect the distance between the user account and the front of terminal 900. In one embodiment, when proximity sensor 916 detects that the distance between the user account and the front of terminal 900 is gradually decreasing, processor 901 controls display screen 905 to switch from the screen-on state to the screen-off state. When proximity sensor 916 detects that the distance between the user account and the front of terminal 900 is gradually increasing, processor 901 controls display screen 905 to switch from the screen-off state to the screen-on state.

[0206] Those skilled in the art will understand that Figure 13 The structure shown in the figure does not constitute a limitation on the terminal 900, and the terminal 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0207] The memory further includes one or more programs, which are stored in the memory and include a program for performing the chorus detection method provided in the embodiment of the present application.

[0208] The present application also provides a computer device, comprising: a processor and a memory, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, a code set, or an instruction set is loaded and executed by the processor to implement the chorus detection method provided by the above-mentioned method embodiments.

[0209] The present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the chorus detection method provided by the above-mentioned method embodiments.

[0210] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the chorus detection method provided in the above-mentioned optional implementation.

[0211] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0212] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0213] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A chorus detection method, characterized in that: The method comprises: Calling a melody feature extraction model to perform feature extraction on a song to be detected to obtain song features; calling the melody feature extraction model to perform feature extraction on a reference chorus segment to obtain chorus features; the reference chorus segment is a calibrated chorus segment in the song to be detected; the song features and the chorus features are both two-dimensional matrices, the width of the two-dimensional matrix represents the frequency range, and the length of the two-dimensional matrix represents the number of audio frames; Multiplying the song feature and the chorus feature to calculate a correlation matrix between the song feature and the chorus feature; the correlation matrix is ​​a two-dimensional matrix, the number of rows of the correlation matrix corresponds to the number of audio frames of the chorus segment, and the number of columns of the correlation matrix corresponds to the number of audio frames of the song to be detected; Determine a target column number interval in the correlation matrix whose values ​​meet a threshold condition; The target audio frame interval of the song to be detected corresponding to the target column number interval is determined as the predicted chorus segment of the song to be detected.

2. The method according to claim 1, characterized in that Determining a target column number interval in the correlation matrix where values ​​satisfy a threshold condition includes: Determine the values ​​in the correlation matrix that are higher than a first threshold as candidate points; The column number interval in which the candidate points in the correlation matrix are diagonally distributed is determined as the target column number interval.

3. The method according to claim 1, characterized in that Determining a target column number interval in the correlation matrix where values ​​satisfy a threshold condition includes: Sliding a sliding window in the correlation matrix according to a preset step size to obtain at least one window matrix from the correlation matrix, wherein the number of rows and columns of the sliding window is the same as the number of rows of the correlation matrix; For each window matrix of the at least one window matrix, calculating the sum of the values ​​on a first diagonal to obtain a reference value corresponding to each window matrix; the first diagonal includes a diagonal from the upper left corner to the lower right corner of the window matrix; Determine a window matrix corresponding to a target peak point in at least one reference value as a target window matrix; The column number interval corresponding to the target window matrix in the correlation matrix is ​​determined as the target column number interval.

4. The method according to claim 3, characterized in that The target peak point is determined by the following method: Obtaining a candidate peak point whose reference value is higher than a second threshold value from the at least one reference value; When the number of the candidate peak points is within a preset number range, determining the candidate peak points as the target peak points; The method further comprises: When the number of the candidate peak points is not within a preset number range, the song to be detected is output as an abnormal song.

5. The method according to any one of claims 1 to 4, characterized in that: The melody feature extraction model includes a preprocessing layer and a feature extraction layer; the melody feature extraction model is trained using the following method: Acquire sample data, the sample data including sample audio and sample labels, the sample labels being used to label the vocal melody of the sample audio; Calling the preprocessing layer to perform time-frequency conversion and filtering on the sample audio to obtain Mel spectrum features of the sample audio; Calling the feature extraction layer to extract the Mel spectrum features to obtain audio features; Calling the classification layer to perform melody classification on the audio features to obtain a predicted melody; The feature extraction layer and the classification layer are trained until convergence according to the loss between the predicted melody and the sample label.

6. The method according to claim 5, characterized in that The obtaining of sample data includes: Extracting the fundamental frequency of each frame of audio in a cappella song to obtain a cappella melody of the cappella song, wherein the cappella song is an a cappella audio of a human voice without accompaniment; Determining, based on a mapping relationship between fundamental frequency and pitch categories, a pitch category corresponding to the fundamental frequency of each frame of audio in the a cappella melody, and obtaining a sample label corresponding to the a cappella song; the pitch category includes at least one pitch category obtained by dividing the frequency range of the fundamental frequency into at least one subrange; Acquire an accompaniment audio library, wherein the accompaniment audio library includes at least one section of accompaniment audio; The a cappella song is weightedly mixed with a section of accompaniment audio in the accompaniment audio library to obtain a sample audio, and the sample label corresponding to the sample audio is the sample label corresponding to the a cappella song.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Determine the song to be detected as sample input data, and determine the predicted chorus segment as a sample chorus label; Calling the chorus detection model to output a predicted chorus interval based on the sample input data; The chorus detection model is trained until convergence according to the loss of the predicted chorus interval and the sample chorus label.

8. A chorus detection device, characterized in that: The device comprises: A feature extraction module is configured to call a melody feature extraction model to perform feature extraction on a song to be detected to obtain song features; call the melody feature extraction model to perform feature extraction on a reference chorus segment to obtain chorus features; the reference chorus segment is a calibrated chorus segment in the song to be detected; the song features and the chorus features are both two-dimensional matrices, the width of the two-dimensional matrix represents the frequency range, and the length of the two-dimensional matrix represents the number of audio frames; a correlation module, configured to multiply the song feature by the chorus feature to calculate a correlation matrix between the song feature and the chorus feature; the correlation matrix being a two-dimensional matrix, the number of rows of the correlation matrix corresponding to the number of audio frames of the chorus segment, and the number of columns of the correlation matrix corresponding to the number of audio frames of the song to be detected; A determination module, configured to determine a target column number interval in which values ​​in the correlation matrix meet a threshold condition; The determination module is used to determine the target audio frame interval of the song to be detected corresponding to the target column number interval as the predicted chorus segment of the song to be detected.

9. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the chorus detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the chorus detection method according to any one of claims 1 to 7.

Images