Beat point detection method and device, storage medium and electronic device

Abstract

The application discloses a beat point detection method and device, a storage medium and an electronic device. The method comprises the following steps: obtaining a candidate beat point extracted from a target audio signal; taking the candidate beat point as the center, and cutting out a target audio segment with a target time length from the target audio signal; inputting the target audio segment into a target prediction neural network model, obtaining a first feature vector of the target audio segment through a target feature extraction network, and transforming the first feature vector through a target transformation network in the target prediction neural network model to obtain a second feature vector; determining a target prediction result according to the second feature vector through a target prediction network in the target prediction neural network model; and when the target prediction result indicates that the target audio segment includes a beat point, determining the beat point in the target audio segment according to the candidate beat point. The application solves the technical problem of low accuracy in the beat point detection process in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a beat point detection method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the development of media information, various audio / video information has attracted public attention, for example, through mixing and cutting multiple original videos, a short video with diversified information content is obtained; by measuring the similarity between music, similar music can be recommended to users according to their historical playback records. Through the above methods, users can obtain more network information in a short time, and quickly obtain media information of their interest.

[0003] In related technologies, the beat point detection method is usually used to realize video editing or music recommendation, for example, by using the significant local pulse method, the input audio signal is subjected to short-time Fourier transform, and the result is logarithmized to obtain a logarithmic spectrum graph with time as the horizontal axis, frequency as the vertical axis, and intensity as the value. Further, by regularization, a curve representing the possibility of note start is obtained, and according to the curve, a preliminary result of beat detection is obtained.

[0004] However, in the process of obtaining the curve representing the possibility of note start by the above local pulse method, only the local maximum value on the curve is regarded as the preliminary result of beat detection, and the preliminary detection result usually contains a large amount of noise, causing the technical problem of low accuracy of beat point detection result.

[0005] At present, there is no effective solution to the above problems. SUMMARY

[0006] The embodiments of the present application provide a beat point detection method and device, a storage medium and an electronic device to at least solve the technical problem of low accuracy in beat point detection.

[0007] According to an aspect of some embodiments of the present application, a beat point detection method is provided, including: obtaining a candidate beat point extracted from a target audio signal; centering on the candidate beat point, a target audio segment of a target time length is cut from the target audio signal; inputting the target audio segment into a target prediction neural network model, obtaining a first feature vector of the target audio segment through a target feature extraction network in the target prediction neural network model, and transforming the first feature vector through a target transformation network in the target prediction neural network model to obtain a second feature vector; determining a target prediction result according to the second feature vector through a target prediction network in the target prediction neural network model; and when the target prediction result indicates that the target audio segment includes a beat point, determining the beat point in the target audio segment according to the candidate beat point.

[0008] Optionally, the target prediction result is determined according to the second feature vector through the target prediction network in the target prediction neural network model, including: inputting the second feature vector into a first prediction network in the target prediction network to obtain a first prediction result, and inputting the second feature vector into a second prediction network in the target prediction network to obtain a second prediction result, wherein the first prediction network is configured to determine the first prediction result according to the second feature vector, the first prediction result indicating whether the target audio segment includes a beat point, and the second prediction network is configured to determine the second prediction result according to the second feature vector, the second prediction result indicating a correction value for correcting the candidate beat point.

[0009] Optionally, when the target prediction result indicates that the target audio segment includes a beat point, the beat point in the target audio segment is determined according to the candidate beat point, including: when the first prediction result indicates that the target audio segment includes a beat point and the second prediction result indicates a first correction value, the beat point in the target audio segment is determined according to the candidate beat point and the first correction value.

[0010] Optionally, the beat point in the target audio segment is determined according to the candidate beat point and the first correction value, including: when the first correction value is less than 0, a first time point before the candidate beat point in the target audio segment is determined as the beat point, wherein a time length between the first time point and the candidate beat point is an absolute value of the first correction value; when the first correction value is greater than 0, a second time point after the candidate beat point in the target audio segment is determined as the beat point, wherein a time length between the candidate beat point and the second time point is the first correction value; and when the first correction value is 0, the candidate beat point in the target audio segment is determined as the beat point.

[0011] Optionally, the second feature vector is input into the second prediction network in the target prediction network to obtain the second prediction result, including: determining the correction value indicated by the second prediction result through the following formula: wherein, denotes a correction value, a is a preset upper limit value of the correction value, and is a weight parameter in the second prediction network, denotes a second feature vector.

[0012] Optionally, the method further includes: when the target prediction result indicates that the target audio segment includes a next beat point, determining the next beat point in the target audio segment according to the candidate beat point, wherein the next beat point is the first beat point in a measure in which the music segment is located.

[0013] Optionally, the determining, by the target prediction network in the target prediction neural network model, the target prediction result according to the second feature vector includes: inputting the second feature vector into a third prediction network in the target prediction network to obtain a third prediction result, and inputting the second feature vector into a fourth prediction network in the target prediction network to obtain a fourth prediction result, wherein the third prediction network is configured to determine the third prediction result according to the second feature vector, the third prediction result indicating whether the target audio segment includes a next beat point, and the fourth prediction network is configured to determine the fourth prediction result according to the second feature vector, the fourth prediction result indicating a correction value for correcting the candidate beat point.

[0014] Optionally, the determining, when the target prediction result indicates that the target audio segment includes a next beat point, the next beat point in the target audio segment according to the candidate beat point includes: when the third prediction result indicates that the target audio segment includes a next beat point and the fourth prediction result indicates a second correction value, determining the next beat point in the target audio segment according to the candidate beat point and the second correction value.

[0015] Optionally, the determining, according to the candidate beat point and the second correction value, the next beat point in the target audio segment includes: when the second correction value is less than 0, determining a third time point before the candidate beat point in the target audio segment as the next beat point, wherein a time length between the third time point and the candidate beat point is an absolute value of the second correction value; when the second correction value is greater than 0, determining a fourth time point after the candidate beat point in the target audio segment as the next beat point, wherein a time length between the candidate beat point and the fourth time point is the second correction value; and when the second correction value is 0, determining the candidate beat point in the target audio segment as the next beat point.

[0016] Optionally, the extracting, from the target audio signal, the target audio segment of the target time length centered on the candidate beat point includes: determining an interval time length between two adjacent candidate beat points in the target audio signal; and extracting, from the target audio signal, the target audio segment of the target time length centered on the candidate beat point, wherein the target time length is a multiple of the interval time length.

[0017] Optionally, the transforming, by the target transformation network in the target prediction neural network model, the first representation vector to obtain the second representation vector comprises: inputting the first representation vector and a set of representation vectors into the target transformation network, and obtaining the second representation vector through N layers of networks connected in sequence in the target transformation network, where N is a positive integer greater than or equal to 2, the first representation vector and the second representation vector have the same vector dimension, each of the N layers of networks comprises a self-attention module and a full connection module, and the set of representation vectors comprises a representation vector of each audio segment in a set of audio segments obtained by the target feature extraction network, the set of audio segments comprises an audio segment centered on each candidate beat point in a set of candidate beat points and cut from the target audio signal, and the set of candidate beat points comprises a plurality of continuous candidate beat points extracted from the target audio signal.

[0018] According to another aspect of the embodiments of the present application, a beat point detection apparatus is further provided, which comprises: a first acquisition unit configured to acquire a candidate beat point extracted from a target audio signal; a first processing unit configured to cut a target audio segment of a target duration from the target audio signal with the candidate beat point as a center; a second processing unit configured to input the target audio segment into a target prediction neural network model, obtain a first representation vector of the target audio segment through a target feature extraction network in the target prediction neural network model, and transform the first representation vector to obtain a second representation vector through a target transformation network in the target prediction neural network model; a first determination unit configured to determine a target prediction result according to the second representation vector through a target prediction network in the target prediction neural network model; and a second determination unit configured to determine a beat point in the target audio segment according to the candidate beat point when the target prediction result indicates that the target audio segment comprises the beat point.

[0019] According to still another aspect of the embodiments of the present application, a computer readable storage medium is further provided, which stores a computer program, where the computer program is configured to execute the beat point detection method when running.

[0020] According to still another aspect of the embodiments of the present application, a computer program product is further provided, which comprises computer programs / instructions, and the computer programs / instructions are configured to implement the steps of the method when executed by a processor.

[0021] According to still another aspect of the embodiments of the present application, an electronic device is further provided, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to execute the beat point detection method through the computer program.

[0022] In the embodiment of the present application, the target audio signal is divided into a target video segment of a target time length by obtaining the candidate beat points extracted from the target audio signal; then the target audio segment is input into a target feature extraction network in the target prediction neural network model to obtain a first feature vector of the target audio segment, the first feature vector is transformed by a target transformation network in the target prediction neural network model to obtain a second feature vector; finally, the target prediction result is determined according to the second guarantee vector and a target prediction network in the target prediction neural network model; and the beat points in the target video segment are determined according to the target prediction result and the candidate beat points. In other words, by combining the significant local pulse method with the target neural network model, the extracted candidate beat points are screened and corrected using the neural network, avoiding the correction disorder of the beat points between the target music segments, solving the technical problem of low accuracy in the beat point detection process, and achieving the technical effect of improving the accuracy of beat point detection. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0024] Figure 1 is a schematic diagram of an application scene of an optional beat point detection method according to an embodiment of the present application;

[0025] Figure 2 is a flowchart of an optional beat point detection method according to an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of an application scene of an optional beat point detection method according to an embodiment of the present application;

[0027] Figure 4 is a schematic diagram of an optional target prediction neural network model according to an embodiment of the present application;

[0028] Figure 5 is a schematic diagram of an optional target feature extraction network according to an embodiment of the present application;

[0029] Figure 6 is a schematic diagram of an optional beat point detection method according to an embodiment of the present application;

[0030] Figure 7 is a schematic diagram of an optional target transformation network according to an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of another optional target transformation network according to an embodiment of the present application;

[0032] Figure 9 is a schematic diagram of an optional target prediction network according to an embodiment of the present application;

[0033] Figure 10 is a schematic diagram of a first prediction network in an optional target prediction network according to an embodiment of the present application;

[0034] Figure 11 is a schematic diagram of an optional determination of a beat point according to a candidate beat point and a first correction value according to an embodiment of the present application;

[0035] Figure 12 is a schematic diagram of an optional determination of a next beat point according to a candidate beat point and a second correction value according to an embodiment of the present application;

[0036] Figure 13 is a schematic diagram of an optional training process of a prediction neural network model according to an embodiment of the present application;

[0037] Figure 14 is a schematic diagram of an optional structure of a beat point detection apparatus according to an embodiment of the present application;

[0038] Figure 15 is a schematic diagram of an optional structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the present application embodiment will be described clearly and completely below in combination with the drawings in the present application embodiment. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.

[0040] Firstly, the terms used in the present application embodiment or related are described as follows, and it can be understood that the following description is an explanation of the terms but not the only explanation:

[0041] Significant local impulse: a beat detection method that uses frequency domain information of music and prior knowledge that beats tend to be equally spaced to detect beats from an audio signal;

[0042] Next beat: the first beat of each measure of a music beat;

[0043] Chord recognition: identifying chords from an audio signal of music, wherein a chord refers to any combination of three or more notes;

[0044] Music transcription: the task of converting the audio signal of music into a musical score, specifically the task of converting the acoustic signal of music into some form of musical notation, such as converting an audio file in MP3, WAV, etc. format into a symbolic representation in MIDI, MusicXML, ABC, etc. format, so as to modify the music. Music transcription usually needs to detect the beat and downbeat of the audio signal first, divide the audio signal according to the detection results, and then identify the notes in each segment;

[0045] Music recommendation system: by measuring the similarity between music, similar music is recommended to the user according to the user's historical playback record.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0047] According to an aspect of an embodiment of the present application, a beat point detection method is provided. As an optional implementation, the beat point detection method can be applied to the application scenario as shown in Figure 1 , but is not limited thereto. In the application scenario as shown in Figure 1 , the terminal device 102 can communicate with the server 106 through the network 104, and the server 106 can perform operations on the database 108, such as writing data or reading data. The terminal device 102 can include a human-computer interaction screen, a processor and a memory. The human-computer interaction screen can be used to display the target audio segment, the beat points in the target audio segment, etc. on the terminal device 102. The processor can be used to respond to the human-computer interaction operation, perform corresponding operations, or generate corresponding instructions and send the generated instructions to the server 106. The memory is used to store related processing data, such as target video signal, candidate beat point, first feature vector, second feature vector, etc.

[0048] As an optional mode, the following steps in the beat point detection method can be executed on the terminal device 102: step S102, obtaining a candidate beat point extracted from a target audio signal; step S104, centering on the candidate beat point, a target audio segment of a target time length is intercepted from the target audio signal; step S106, inputting the target audio segment into a target prediction neural network model, obtaining a first feature vector of the target audio segment through a target feature extraction network in the target prediction neural network model, and transforming the first feature vector through a target transformation network in the target prediction neural network model to obtain a second feature vector; step S108, determining a target prediction result according to the second feature vector through a target prediction network in the target prediction neural network model; and step S110, when the target prediction result indicates that the target audio segment includes a beat point, determining the beat point in the target audio segment according to the candidate beat point.

[0049] As an optional example, the embodiment does not limit the execution subject of the above steps S102 to S110, for example, the above steps S102 to S110 can be executed on the terminal device 102 or the server 106, or can be partially executed on the terminal device 102 and partially executed on a computing server in communication with the server 106.

[0050] In the above manner, the candidate beat point extracted from the target audio signal is obtained, and the target audio signal is divided into a target video segment of a target time length; then the target audio segment is input into the target feature extraction network in the target prediction neural network model to obtain the first feature vector of the target audio segment, the first feature vector is transformed through the target transformation network in the target prediction neural network model to obtain the second feature vector; finally, the target prediction result is determined according to the second guarantee vector and the target prediction network in the target prediction neural network model; and the beat point in the target video segment is determined according to the target prediction result and the candidate beat point. In other words, by combining the significant local impulse method with the target neural network model, the extracted candidate beat point is screened and corrected using the neural network, avoiding the confusion of beat points between target music segments, solving the technical problem of low accuracy in the beat point detection process, and achieving the technical effect of improving the accuracy of beat point detection.

[0051] To solve the problem of low accuracy in the above beat point detection process, the embodiment of the present application provides a beat point detection method, Figure 2 is a flowchart of the beat point detection method according to the embodiment of the present application, which includes the following steps:

[0052] Step S202, obtaining a candidate beat point extracted from a target audio signal;

[0053] In this embodiment of the application, the method of extracting candidate beat points from the target audio signal may be limited to the use of the salient local impulse method. However, it should be noted that the salient local impulse method is only an example and does not constitute a limitation.

[0054] like Figure 3 As shown, audio signals have a high sampling rate; for example, 441,000 samples can be taken per second. Therefore, one minute of music would need to be represented by a time series of tens of millions of preset durations. Each time series of a musical segment typically contains a specified number of beat points, or next beat points, where the next beat point is the first beat of each measure of the musical beat.

[0055] The significant local impulse method is used to perform a Fourier transform on the input audio signal, and the logarithm of the Fourier transform result is taken to obtain a logarithmic spectrum with time on the horizontal axis, frequency on the vertical axis, and intensity on the vertical axis. The logarithmic spectrum is a commonly used audio representation in the field of signal analysis.

[0056] It's easy to understand that musical beats typically begin with a note, leading to an increase in the intensity of local frequencies in the logarithmic spectrum. Based on this, by differentiating the logarithmic spectrum along time (horizontal axis) and summing it along frequency (vertical axis), and then regularizing, a curve representing the probability of a note starting (onset envelope or novelty curve) is obtained. The local maxima on this curve can be considered as preliminary results of beat detection, i.e., candidate beats or candidate next beats.

[0057] Step S204: Using the candidate beat point as the center, extract the target audio segment of the target duration from the target audio signal;

[0058] As an alternative example, methods for determining the target audio segment centered on the beat point include:

[0059] Determine the interval between two adjacent candidate beat points in the target audio signal;

[0060] Centered on the candidate beat point, extract the target audio segment of the target duration from the target audio signal, where the target duration is a multiple of the interval duration.

[0061] like Figure 3 As shown, assume that the time points of two adjacent candidate beat points are t. i t i+1 Using this time point, the time interval between two adjacent candidate points is determined to be t. i+1 -t i According to the interval t i+1 -t i multiples of a(t)i+1 -t i ) to determine a target duration, and a plurality of different target audio segments are cut from the target audio signal according to the target duration, where a is a positive integer greater than or equal to 1. For example, when a = 2, the target duration is 2 (t i+1 -t i ), and when a = 3, the target duration is 3 (t i+1 -t i ).

[0062] It should be noted that the rhythm of the music usually does not change in a local range, and the beats of each audio segment are usually arranged at equal intervals. In order to avoid the problem of audio segment edge insensitivity, the candidate beat points are usually centered, and the audio signal is divided into mutually overlapping audio segments according to the determined target duration, for example, there is an overlapping candidate beat point between the audio segment 1 and the audio segment 2 shown in Figure 3 .

[0063] Through the above-mentioned division of the audio signal, a plurality of different audio segments are obtained as input of the target prediction neural network model. Since the problem of audio segment edge insensitivity is solved in the process of dividing the audio segment, the reliability of the sample data input into the target prediction neural network model is improved.

[0064] In step S206, the target audio segment is input into the target prediction neural network model, the first feature vector of the target audio segment is obtained through the target feature extraction network in the target prediction neural network model, and the first feature vector is transformed through the target transformation network in the target prediction neural network model to obtain the second feature vector;

[0065] In step S208, the target prediction result is determined according to the second feature vector through the target prediction network in the target prediction neural network model;

[0066] As shown in Figure 4 , the target prediction neural network model includes but is not limited to the target feature extraction network, the target transformation network, and the target prediction network. Among them, it is assumed that the target feature extraction network is a convolutional neural network, which is used to extract the feature information of the target audio segment ms1, and compresses the feature of the music segment to 1024 dimensions to obtain the first feature vector z i of the target audio segment.

[0067] The first feature vector z i is input into the target transformation network, and the second feature vector z i is obtained through the transformation processing of the first feature vector z i , and the second feature vector z i ; the second feature vector z iThe input target prediction network determines a target prediction result, where the target prediction result includes, but is not limited to, indicating that a beat point is included in the target audio segment.

[0068] It should be noted that in the present embodiment, the number of audio segments input into the target feature extraction network is not limited. For example, it can be 1, or any number of more than 2. In the case where the audio segments input into the target feature extraction network are multiple, for each input audio segment, an output first feature vector z i .

[0069] The target feature extraction network, the target transformation network and the target prediction network are described in detail below in combination with specific examples.

[0070] (I) Target feature extraction network

[0071] As an optional example, as shown in Figure 5 , it is assumed that the target feature extraction network includes 4 one-dimensional convolution layers and corresponding Relu activation functions, where the size of the convolution kernel of each convolution layer is 1x1. After pre-processing each audio segment and obtaining the corresponding 1x1024-dimensional audio segment feature, the 1x1024-dimensional audio segment feature is input into the convolutional neural network.

[0072] After 4 convolutional processing and normalization of the extracted features using the Relu activation function, i.e., the features of the audio segment are compressed to 1024 dimensions, the first feature vector z i of the target audio segment is obtained, where z i is 1x1024-dimensional.

[0073] It is easy to understand that the Relu activation function and the 4 one-dimensional convolution layers in the present embodiment are only an example and are not limited thereto. For example, the size of the convolution kernel can also be 3x3 or 5x5, and the 4 one-dimensional convolution layers in the convolutional neural network can also be 3 or 5 layers, etc.

[0074] In addition, for the 4 one-dimensional convolution layers in the convolutional neural network, the size of the convolution kernel is 1x1 in the 4 one-dimensional convolution layers, but the value of the convolution kernel of each layer can be different or partially the same and partially different.

[0075] (II) Target transformation network

[0076] As can be seen from the above analysis, the detection of the beat point needs to be combined with the music context information for 1 minute, and the music segment extracted in the above embodiment is only a few seconds. Therefore, in the present embodiment, the audio segment feature is input into the transformer network (for example, as shown in Figure 6The first feature vector z output by the convolutional neural network is shown as z i The attention mechanism is used to capture the timing information of the entire piece of music.

[0077] It should be noted that the 1-minute music is only an example and is not limited thereto. In actual application, the music duration can also be set to 2 minutes according to the music type. As the music duration changes, the number of feature vectors of the plurality of music segments will also change, avoiding the loss of context information of the entire piece of music and ensuring the integrity of the obtained music information.

[0078] As an optional example, the transformation of the first feature vector by the target transformation network in the target prediction neural network model to obtain the second feature vector includes:

[0079] The first feature vector and a set of feature vectors are input into the target transformation network, and the second feature vector is obtained through the N-layer network connected in sequence in the target transformation network, where N is a positive integer greater than or equal to 2, the vector dimensions of the first feature vector and the second feature vector are the same, each network in the N-layer network includes a self-attention module and a fully connected module, the set of feature vectors includes a feature vector of each audio segment in a set of audio segments obtained by the target feature extraction network, the set of audio segments includes an audio segment centered on each candidate beat point in a set of candidate beat points, and the set of candidate beat points includes a plurality of continuous candidate beat points extracted from the target audio signal.

[0080] When the first feature vector and the set of feature vectors are M feature vectors and M is a positive integer greater than or equal to 2, the self-attention module in the first network in the N-layer network performs the following operations on each feature vector in the M feature vectors to obtain M intermediate feature vectors in the first network, where each feature vector is the current feature vector when the following operations are performed:

[0081] The similarity between the current feature vector and M-1 feature vectors other than the current feature vector in the M feature vectors is determined to obtain M-1 similarities, and the M-1 similarities are used as weights to perform weighted summation on the M-1 feature vectors to obtain a current weighted feature vector. Then, the current weighted feature vector is added to the current feature vector to obtain a current intermediate feature vector in the first network;

[0082] The fully connected module in the first network is used to determine M output feature vectors output by the first network according to the M intermediate feature vectors in the first network;

[0083] wherein the self-attention module in the i-th network in the N-layer network performs the following operation on each of the M output representation vectors output by the (i-1)-th network to obtain M intermediate representation vectors in the i-th network, wherein each output representation vector is a current output representation vector when performing the following operation, 1 < i ≤ N:

[0084] determines similarities between the current output representation vector and M-1 output representation vectors in the M representation vectors other than the current output representation vector, respectively, to obtain M-1 similarities, and performs weighted summation on the M-1 output representation vectors by taking the M-1 similarities as weights to obtain a current weighted representation vector, and then adds the current weighted representation vector to the current output representation vector to obtain a current intermediate representation vector in the i-th network;

[0085] the fully connected module in the i-th network is configured to determine M output representation vectors output by the i-th network according to the M intermediate representation vectors in the i-th network,

[0086] wherein the M output representation vectors output by the N-th network include the second representation vector.

[0087] As an optional example, as shown in Figure 7 and Figure 8 The target transformation network includes, but is not limited to, a 4-layer network connected in sequence, and each layer network is composed of a self-attention module and a fully connected module. That is, the processing procedure in each layer network is the same, and the processing procedures of the self-attention module and the fully connected module will be introduced below by taking the first layer network structure as an example.

[0088] (I) Self-attention module

[0089] Suppose the target audio signal includes 3 audio segments in total, and the first representation vectors corresponding to the 3 audio segments are z1, z2 and z3 respectively, and the 3 first representation vectors are all 1x1024-dimensional. The first representation vectors z1, z2 and z3 are input into the self-attention module in the first layer network, and then in the self-attention module, the following operations are performed:

[0090] (a) the similarities between z1 and z2 and between z1 and z3 are calculated in sequence to obtain s2 and s3;

[0091] (b) the similarities s2 and s3 are taken as weights to perform weighted summation on the first representation vectors z2 and z3 to obtain a current weighted representation vector z2*s2+z3*s3;

[0092] (c) the current representation vector z1 is added to the current weighted representation vector z2*s2+z3*s3 to obtain a current intermediate representation vector z1' of the first layer network, z1'=z1+[z2*s2+z3*s3].

[0093] According to the above process, other current intermediate representation vectors z2' and z3' of the first layer network are calculated, wherein z2'=z2+[z1*s1+z3*s3], z3'=z3+[z1*s1+z2*s2].

[0094] (ii) Fully connected module

[0095] The three representation vectors z1', z2', and z3' output by the self-attention module are input into the fully connected module, and then for each representation vector input into the fully connected module, the processing of linear layer 1-GELU activation function-linear layer 2 is sequentially performed, so as to realize the fusion of the feature of each audio segment in different channels (1024 dimensions).

[0096] For example, assuming that each audio segment is a 1024-dimensional vector (for example, the second feature vector z1' of audio segment 1), after z1' is input into the fully connected layer, linear mapping of linear layer 1 is performed (for example, z1' is multiplied by a 1024×D-dimensional matrix), then processing of the GELU activation function is performed, and finally linear mapping of linear layer 2 is performed (for example, multiplication by a D×1024-dimensional matrix) to realize the fusion of audio segment 1 in different channels and output the corresponding vector z1''. Similarly, the fusion of audio segment 2 and audio segment 3 in different channels can also be performed, and the corresponding vectors z2'' and z3'' are output.

[0097] It should be noted that the self-attention module is used to determine the attention score of each audio segment feature and other audio segment features, that is, the self-attention module processes all audio segments to obtain the context information of the entire music; and the fully connected module processes each audio segment to realize the feature fusion of the audio segment in different channels.

[0098] Through the above analysis, it can be known that the input of the transformer network is the feature of all audio segments, and position encoding is added to the features of these segments to provide the position information of the segments in the entire music. Then the segment features are sent into the transformer network, and for each input audio segment, an output vector is obtained. The self-attention module calculates the attention score (similarity) of each music segment feature and all segment features, and then weights and sums all segment features according to the score, and adds the feature itself to realize the feature fusion of the entire music. The fully connected module is composed of linear layer-GELU activation function-linear layer, and respectively acts on different segment features to realize the fusion of the segment features in different channels (1024 dimensions), that is, the time sequence information of the entire music is learned.

[0099] Obviously, it is easy to understand that from the structure of the transformer network, it can be intuitively seen that for the second layer network in the transformer network (the target transformation network), the first layer network outputs z1'', z2'', and z3'' are executed by the self-attention module in the second layer network, and the same operations as steps (a)-(c) above are performed to obtain the current intermediate representation vector of the second layer network, and then the full connection layer in the second layer network is used to process each audio segment feature and output three representation vectors. The specific implementation process can refer to the description in the first layer network above, and will not be described here.

[0100] Based on the same principle, Figure 8 The processing processes of the self-attention modules and the full connection modules in the third layer network and the fourth layer network shown are also described in the first layer network and the second layer network part above, and will not be described here in this embodiment.

[0101] It should be noted that for each input audio segment feature z i , after processing by the target transformation network, an output vector z i ' is obtained, that is, the second representation vector z i ' of the target audio segment, and the vector dimensions of the first representation vector z i and the second representation vector z i ' are the same. For example, in the case of a first representation vector z i 1x1024, after transformation processing by the transformer network, the second representation vector z i ' is still 1x1024.

[0102] (Three) target prediction network

[0103] As shown in Figure 9 , the target prediction network includes but is not limited to a first prediction network, a second prediction network, a third prediction network, and a fourth prediction network. In the case of obtaining the second representation vector z i ' of the target audio segment after the target transformation network, the second representation vector z i ' is input into the target prediction network to obtain the target prediction result.

[0104] Step S210, when the target prediction result indicates that the target audio segment includes beat points, determining the beat points in the target audio segment according to the candidate beat points;

[0105] It is easy to understand that by inputting the above-mentioned second representation vector z i ' into the target prediction network, a prediction result corresponding to each audio segment can be obtained, wherein the prediction result can indicate that the target audio segment includes beat points, or the target audio segment does not include beat points.

[0106] For example, the second feature vector z1' of the audio segment 1 and the second feature vector z2' of the audio segment 2 are respectively input into the target prediction network to obtain a first prediction result and a second prediction result, where the first prediction result indicates that the audio segment 1 includes a beat point, and the second prediction result indicates that the audio segment 2 does not include a beat point. Then, only the beat point in the audio segment 1 needs to be determined according to the candidate beat point.

[0107] Through the above embodiments provided by the present application, the candidate beat points extracted from the target audio signal are obtained by using the significant local pulse method, and the target audio signal is divided into target video segments of a target time length; then the target video segments are input into the target neural network model, and sequentially pass through the target feature extraction network, the target transformation network and the target prediction network to determine the target prediction result. That is, by combining the significant local pulse method with the target neural network model, the determined candidate beat points are screened and corrected, avoiding the confusion of beat points between target music segments, solving the technical problem of low accuracy in the beat point detection process, and achieving the technical effect of improving the accuracy of beat point detection.

[0108] As an optional example, the target prediction result is determined according to the second feature vector through the target prediction network in the above target prediction neural network model, comprising:

[0109] The second feature vector is input into a first prediction network in the target prediction network to obtain a first prediction result, and the second feature vector is input into a second prediction network in the target prediction network to obtain a second prediction result, where the first prediction network is used to determine the first prediction result according to the second feature vector, the first prediction result indicating whether the target audio segment includes a beat point, and the second prediction network is used to determine the second prediction result according to the second feature vector, the second prediction result indicating a correction value for correcting the candidate beat point.

[0110] As shown in Figure 9 The target prediction network includes but is not limited to the first prediction network, the second prediction network, the third prediction network and the fourth prediction network, where the four prediction networks are each composed of a full connection layer and a respective activation function. For example, as shown in Figure 10 The first prediction network includes a full connection layer and a corresponding activation function, and the structures of the second prediction network to the fourth prediction network can refer to the structure of the first prediction network, which will not be described here.

[0111] In the process of determining the second feature vector z iIn the case of the first prediction network, by performing a binary classification task, it can be predicted whether the current target audio segment contains a beat point by the following formula (1), and then the output value of the full connection layer in the first prediction network is mapped to 0-1 by using the Sigmoid activation function in the following formula (2) to represent the probability of the binary classification task, that is, the first prediction result (which can be understood as a prediction probability) of the target audio segment containing a beat point :

[0112] (1)

[0113] (2)

[0114] wherein, is the output vector of the target transformation network, and is the weight of the full connection layer in the first prediction network, is the first prediction result, is also understood as Figure 9 the second feature vector z shown in i .

[0115] Then the first loss value L between the first prediction result and the actual value (the actual probability value pre-labeled) is calculated by the binary cross-entropy loss function in the following formula (3) 1-1 , wherein L 1-1 can also be calculated by the following formula (3):

[0116] (3)

[0117] Through the above first prediction network, it can be determined that the target audio segment includes a beat point when the first prediction result reaches a first preset threshold Th1 (for example, 0.8), avoiding the detection of candidate beat points in all audio segments, realizing the screening of candidate beat points, and improving the efficiency of beat point detection.

[0118] As an optional example, since the target audio segments extracted from the target audio signal by the above-mentioned candidate beat point as the center are mutually overlapped, it means that there may be the same candidate beat point between adjacent audio segments, which causes the problem of low accuracy of the candidate beat point in the target audio segment.

[0119] In order to solve the above problem of low accuracy of the candidate beat point, a second prediction network is also provided in this embodiment for determining whether the candidate beat point in the target video segment needs to be corrected, specifically comprising:

[0120] inputting the second feature vector into a second prediction network in the target prediction network to obtain a second prediction result, including:

[0121] The correction value represented by the second prediction result is determined by the following formula (4):

[0122] (4)

[0123] wherein, represents the correction value, a is a preset correction upper limit value, and is a weight parameter in the second prediction network, represents the second feature vector, is also understood as Figure 9 the second feature vector z i shown in the second prediction network.

[0124] It should be noted that the above a can be but is not limited to a parameter determined according to the target duration determined when the target audio segment is intercepted, for example, when the target duration is 1 second, a is set to 0.2 seconds. Then, in the process of correcting the candidate beat point by using the second prediction network, the SoftMax activation function and the L2 loss function are usually used, and the output result of the SoftMax activation function is mapped to [-0.2, 0.2], that is, the maximum value allowed for the correction of the candidate beat point is 0.2 seconds.

[0125] Obviously, it is easy to understand that only in the case where it is determined by the first prediction network that the target audio segment includes a beat point (that is, only when ), the training process of the second prediction network is activated, and in the case where the correction value represented by the second prediction result is determined , it is also necessary to determine whether to correct the candidate beat point in the target audio segment according to the correction value 2-1 by using the second loss value L , wherein the second loss value L 2-1 can be calculated by the following formula (5):

[0126] (5)

[0127] wherein, is the mapped correction value, is the time of the candidate beat point, is the labeled time.

[0128] As an optional implementation manner, the above determining the beat point in the target audio segment according to the candidate beat point when the target prediction result indicates that the target audio segment includes a beat point includes:

[0129] When the first prediction result indicates that the target audio segment includes beat points and the second prediction result indicates that the first correction value is used, the beat points in the target audio segment are determined based on the candidate beat points and the first correction value.

[0130] The specific implementation process includes:

[0131] When the first correction value is less than 0, the first moment before the candidate node beat point in the target audio segment is determined as the beat point, wherein the duration between the first moment and the candidate node beat point is the absolute value of the first correction value;

[0132] When the first correction value is greater than 0, the second moment after the candidate node beat point in the target audio segment is determined as the beat point, wherein the duration between the candidate node beat point and the second moment is the first correction value;

[0133] When the first correction value is 0, the candidate node beat points in the target audio segment are determined as beat points.

[0134] For example, such as Figure 11 As shown in (a), assuming that the target audio segment includes candidate beat points and the time of the candidate beat point is t1, when the first correction value is -0.2, the time t1 of the candidate beat point is corrected to the first time t1' earlier than t1, and the duration between t1 and the first time t1' is 0.2 seconds, that is, the absolute value of the first correction value -0.2, and the first time t1' is determined as the final beat point.

[0135] When the first correction value is 0.2, such as Figure 11 As shown in (b), the time t1 of the candidate beat point is corrected to the second time t2' which is later than t1, and the duration between t1 and the second time t2' is 0.2 seconds, and the second time t2' is determined as the final beat point.

[0136] Obviously, when the correction value is 0, the candidate beat points in the target audio segment are directly determined as beat points. It should be noted that the beat points or candidate beat points in the target audio segment correspond to each moment.

[0137] Using the second prediction network described above, it is possible to determine whether the target audio segment includes beat points, and the second loss value L 2-1 When the preset threshold (e.g., 0.6) is reached, the candidate beat points in the target audio segment are corrected according to the value of the first correction value, and the corrected candidate beat points are determined as the final beat points, thereby improving the accuracy of beat point detection.

[0138] In addition, in the related art, the beat detection method based on deep learning lacks prior knowledge of music theory and excessively relies on statistical information of the data set, thus being prone to overfitting and having poor generalization ability for out-of-dataset samples. In addition, the beat points of music are sparse, i.e., most of the time of a piece of music is not a beat point, which makes the number of positive and negative samples in the labeled data extremely unbalanced, limiting the performance of such methods. In the embodiments of the present application, the significant local impulse method is used to extract candidate beat points, and then a deep learning method (target prediction neural network model) is used for screening and correction, successfully introducing the prior of music theory and solving the problem of unbalanced positive and negative samples in the training of the deep learning model.

[0139] As an optional example, the target audio segment of the target duration is cut from the target audio signal centered on the candidate beat point, wherein each divided target audio segment corresponds to a measure, each measure includes different numbers of candidate beat points, and the first beat point in each measure is the next beat point. Through the detection of the next beat point, music symbol transcription, chord recognition, music similarity calculation, mixed video editing, etc. can be realized.

[0140] However, since the significant local impulse method is an algorithm designed based on limited music theory, the music theory used is not complete, especially for the beat detection of unpopular types of music and difficult samples, it cannot predict the next beat point in the target audio segment, which limits the application scenarios of the beat detection method.

[0141] To solve this problem, the network structure for predicting the next beat point is also designed in the target prediction network model in the embodiments of the present application, specifically including:

[0142] When the target prediction result indicates that the target audio segment includes the next beat point, the next beat point in the target audio segment is determined according to the candidate beat point, wherein the next beat point is the first beat point in the measure where the music segment is located.

[0143] As an optional implementation, the target prediction result is determined according to the second feature vector by the target prediction network in the target prediction neural network model, including:

[0144] The second feature vector is input into a third prediction network in the target prediction network to obtain a third prediction result, and the second feature vector is input into a fourth prediction network in the target prediction network to obtain a fourth prediction result, wherein the third prediction network is used to determine the third prediction result according to the second feature vector, the third prediction result indicating whether the target audio segment includes the next beat point, and the fourth prediction network is used to determine the fourth prediction result according to the second feature vector, the fourth prediction result indicating a correction value for correcting the candidate beat point.

[0145] and Figure 10 The first prediction network shown has the same structure, and the third and fourth prediction networks also consist of one fully connected layer and their respective activation functions.

[0146] In the second representation vector z i When the third prediction network is input, a binary classification task as shown in formula (1) above is performed to predict whether the current target audio segment contains the next beat point. Then, the Sigmoid activation function as shown in formula (2) above is used to map the output value of the fully connected layer in the third prediction network to between 0 and 1, which is used to represent the probability of the binary classification task, that is, the third prediction result (which can be understood as the prediction probability) that the target audio segment contains the next beat point. '.

[0147] When using the third prediction network to predict whether the target audio segment contains the next beat point, the activation function and loss function are the same as those used when using the first prediction network to predict whether the target audio segment contains the beat point, but the supervision signal is different; here, the supervision signal is the next beat label. The reason for separating the beat point and the next beat point into two separate binary classification tasks is that some datasets do not have the next beat label. In this case, it is only necessary to freeze the classification (third prediction network) and regression head (fourth prediction network) of the next beat point, and only train the classification (first prediction network) and regression head (second prediction network) of the beat point.

[0148] In addition, since the candidate beat points are extracted first using the significant local impulse method, the ratio of positive to negative samples in the beat is approximately 1:1, and the ratio of positive to negative samples in the next beat is approximately 1:4, which solves the problem of imbalance in the number of positive and negative samples.

[0149] Then, using the binary cross-entropy loss function in formula (3) above, the third loss value L between the third prediction result and the actual value (the pre-labeled actual probability value) is calculated. 1-2 , where L 1-2 It can be calculated using the formula (3) above.

[0150] By using the third prediction network described above, if the third prediction result reaches the second preset threshold Th2 (e.g., 0.7), it can be determined that the target audio segment includes the next beat point, thus avoiding the detection of candidate beat points in all audio segments and improving the efficiency of next beat point detection.

[0151] Meanwhile, referring to the description of the second prediction network section above, it is known that after determining that the target audio segment contains the next beat point, it is also necessary to... Figure 9 The fourth prediction network shown determines the correction value for the next beat point.

[0152] Specifically, by the above formula (4), the positive correction value for modifying the next beat point can be calculated .

[0153] It should be noted that in the process of calculating the second prediction result and the fourth prediction result by using the above formula (4), the values of the weight parameters and in the formula are different. That is, the values of the weight parameters and in the second prediction network and the fourth prediction network are different.

[0154] In the process of modifying the next beat point by using the fourth prediction network, the SoftMax activation function and the L2 loss function are usually used, and the output result of the SoftMax activation function is mapped to [-a, a], where a represents the upper limit value of the modification of the next beat point, for example, a = 0.1 seconds.

[0155] Obviously, it is easy to understand that the training process of the fourth prediction network will only be activated in the case that the target audio segment includes the next beat point determined by the third prediction network, and in the case that the correction value ' determined by the fourth prediction result is determined, it is also necessary to determine whether to modify the candidate beat point in the target audio segment according to the correction value 2-2 ' by using the fourth loss value L , for example, in the case that the fourth loss value L 2-2 reaches the preset threshold Th2, the candidate beat point is modified to obtain the next beat point. The fourth loss value L 2-2 can be calculated by the above formula (5). For detailed description process, please refer to the description of the above second prediction network, which will not be repeated here.

[0156] As an optional implementation manner, when the target prediction result indicates that the target audio segment includes the next beat point, the next beat point in the target audio segment is determined according to the candidate beat point, including:

[0157] When the third prediction result indicates that the target audio segment includes the next beat point, and the fourth prediction result indicates the second correction value, the next beat point in the target audio segment is determined according to the candidate beat point and the second correction value.

[0158] The specific implementation process includes:

[0159] When the second correction value is less than 0, the third time point before the candidate node beat point in the target audio segment is determined as the next beat point, wherein the time length between the third time point and the candidate node beat point is the absolute value of the second correction value;

[0160] When the second correction value is greater than 0, the fourth moment after the candidate node beat point in the target audio segment is determined as the next beat point, where the duration between the candidate node beat point and the fourth moment is the second correction value.

[0161] When the second correction value is 0, the candidate node beat points in the target audio segment are determined as the next beat points.

[0162] For example, such as Figure 12 As shown in (a), assuming that the measure in which the target audio segment is located includes two candidate beat points, and the times of the two candidate beat points are t1 and t2 respectively, then the first candidate beat point (time t2) in the measure is determined as the next beat point. When the second correction value is -0.1, the time t2 of the next beat point is corrected to the third time t3' earlier than t2, and the duration between t2 and the first time t3' is 0.1 seconds, that is, the absolute value of the second correction value -0.1, and the first time t3' is determined as the final next beat point.

[0163] When the second correction value is 0.2, such as Figure 12 As shown in (b), the time t2 of the next beat point is corrected to the fourth time t4', which is later than t2, and the duration between t2 and the fourth time t4' is 0.1 seconds. The fourth time t4' is then determined as the final beat point.

[0164] Obviously, when the correction value is 0, the time t2 corresponding to the next beat point in the target audio segment is directly determined as the next beat point. It should be noted that the beat points or candidate beat points in the target audio segment correspond to each time point.

[0165] The fourth prediction network described above can be used to determine whether the target audio segment includes the next beat point, and the fourth loss value L. 2-2 When the preset threshold (e.g., 0.9) is reached, the candidate beat points in the target audio segment are corrected according to the value of the second correction value, and the corrected candidate beat points are determined as the final next beat points, thereby improving the efficiency and accuracy of next beat point detection.

[0166] Based on the analysis of the above embodiments, it can be seen that the process of determining the beat classification result and the next beat classification result according to the first prediction result and the third prediction result obtained from the target prediction network includes, but is not limited to, at least one of the following:

[0167] Suppose the first prediction result is a and the third prediction result is b, where the values ​​of a and b are both arbitrary values ​​between [0,1].

[0168] i) If a ≥ first preset threshold Th1 (e.g., 0.5) and b < second preset threshold Th2 (e.g., 0.8), then the candidate beat points in the target audio segment are determined as beat points;

[0169] ii) If a ≥ first preset threshold Th1 (e.g., 0.5) and b ≥ second preset threshold Th2 (e.g., 0.8), then the candidate beat point in the target audio segment is determined as the next beat point;

[0170] iii) If a < first preset threshold Th1 (e.g., 0.5) and b < second preset threshold Th2 (e.g., 0.8), then the candidate beat point in the target audio segment is determined to be neither a beat point nor the next beat point, that is, the target audio segment does not actually include a beat point.

[0171] The following is combined Figure 13 The training process of the target prediction network model is described in detail.

[0172] S11, Obtain the target audio segment;

[0173] As an alternative example, candidate beat points are extracted from the target audio signal using the salient local impulse method described above, and then a target audio segment of the target duration is extracted from the target audio signal with the candidate beat points as the center.

[0174] S12, Obtain the first representation vector z of the target audio segment. i ;

[0175] The target audio segment is input into the initial feature extraction network of the prediction neural network model. For example, the target audio segment is input into the convolutional neural network in the above embodiment. By utilizing the four layers of one-dimensional convolution and the corresponding ReLU activation function in the convolutional neural network, the first representation vector z of the target audio segment is obtained. i .

[0176] S13, the first representation vector z i Input the initial transform network to obtain an output vector z for each input audio segment. i ';

[0177] For example, using such Figure 7 and Figure 8 The four-layer network in the transformer network shown above, for the first representation vector z i The transformation process is performed to obtain the second representation vector z corresponding to each input audio segment. i Each layer of the network consists of a self-attention module and a fully connected module. By utilizing the self-attention module, the temporal information of the entire piece of music can be captured.

[0178] S14, inputting the second feature vector z i into the initial prediction network to obtain the corresponding prediction results Y1', Y2', Y3', ….

[0179] Y2', Y3'….

[0180] As described in the above embodiments, the initial prediction network model includes but is not limited to the first prediction network, the second prediction network, the third prediction network, and the fourth prediction network. After the second feature vector z i is input into the initial prediction network, the first prediction result to the fourth prediction result are output simultaneously, wherein the first prediction result represents the first prediction result of the target audio segment including the beat point , the second prediction result represents the first correction value for correcting the candidate beat point , the third prediction result represents the third prediction result of the target audio segment including the next beat point , and the fourth prediction result represents the second correction value for correcting the next beat point .

[0181] S15, calculating the loss function L 总 .

[0182] According to the actual results Y1, Y2, Y3… and the prediction results Y1', Y2', Y3'… pre-labeled, and through the binary cross-entropy loss function in the above formula (3), the first loss value L 1-1 between the first prediction result and the actual value (the actual probability value pre-labeled) can be calculated. Through the above formula (5), the second loss value L for determining whether to correct the candidate beat point in the target audio segment according to the first correction value 2-1 can be calculated. Through the binary cross-entropy loss function in the above formula (3), the third loss value L 1-2 between the third prediction result and the actual value (the actual probability value pre-labeled) can be calculated. Through the above formula (5), the second loss value L for determining whether to correct the candidate beat point in the target audio segment according to the second correction value 2-2 .

[0183] In addition, it should be noted that for each input sample data X, it includes but is not limited to the time t i corresponding to the candidate node included in the target audio segment and the actual value c i pre-labeled.

[0184] As an optional example, by calculating the above L 1-1 , L 1-2 , L2-1 , L 2-2 The weighted sum is obtained as the total loss function L 总 = w1 L 1-1 + w2 L 1-2 + w3 L 2-1 + w4 L 2-2 .

[0185] S16, adjustment of model parameters;

[0186] In the case where L 总 does not reach the preset threshold, the parameters in the preset neural network model are adjusted, wherein the parameters that need to be adjusted include but are not limited to the value size of the convolution kernel in the initial feature extraction network, the weight , , , , etc. in the initial prediction network.

[0187] In the case where L 总 reaches the preset threshold, the training process is stopped, and the prediction neural network model after training is determined as the target prediction neural network model.

[0188] From the above training process, it can be seen that in the process of training the prediction neural network model by using the input samples X1, X2, X3…, the initial feature extraction network, the initial transformation network and the initial prediction network in the prediction neural network model are trained in a joint training manner, and in the case where the total loss function L 总 satisfies the preset condition, the training is stopped to obtain the target prediction neural network model, thereby improving the efficiency of model training and improving the reliability of the target prediction neural network model.

[0189] It should be noted that the technical solutions in the embodiments of the present application can be applied to the detection of beat points and next beat points, music transcription, chord recognition and video mixing in a music game scene, but are not limited thereto.

[0190] It should be noted that for each of the above method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0191] According to another aspect of the embodiments of the present application, a beat point detection device is also provided, as shown in Figure 14 .

[0192] The first obtaining unit 1402 is configured to obtain candidate beat points extracted from the target audio signal;

[0193] In this embodiment of the present application, the candidate beat points can be extracted from the target audio signal by using the salient local pulse method, but it should be noted that the salient local pulse method is only an example and does not constitute a limitation.

[0194] As shown in Figure 3 , the audio signal has a high sampling rate, for example, 441000 times per second, so that 1 minute of music needs to be represented by a time sequence of more than 10 million preset time lengths. In each time sequence representing a music segment, a specified number of beat points or downbeat points are usually included, wherein the downbeat point is the first beat of each measure of the music beat.

[0195] The input audio signal is subjected to Fourier transform by using the salient local pulse method, and the logarithm of the Fourier transform result is calculated to obtain a log spectrum graph with time as the horizontal axis, frequency as the vertical axis, and intensity as the value, wherein the log spectrum graph is a common audio representation form in the field of signal analysis.

[0196] Obviously, it is easy to understand that the beat of music is usually started by a note, which will cause the intensity of the local frequency in the log spectrum graph to increase, based on which, by differentiating the log spectrum graph along the time (horizontal axis) and summing along the frequency (vertical axis), a curve (onset envelope or novelty curve) representing the possibility of note onset is obtained after regularization. The local maximum on the curve can be regarded as the preliminary result of beat point detection, that is, the candidate beat point or the candidate downbeat point.

[0197] The first processing unit 1404 is configured to cut out a target audio segment of a target time length from the target audio signal with the candidate beat point as the center;

[0198] As an optional example, the implementation manner of determining the target audio segment with the candidate beat point as the center includes:

[0199] determining the interval time length between two adjacent candidate beat points in the target audio signal;

[0200] cutting out a target audio segment of a target time length from the target audio signal with the candidate beat point as the center, wherein the target time length is a multiple of the interval time length.

[0201] As shown in Figure 3 , it is assumed that the time points of two adjacent candidate beat points are t i , t i+1, by which the interval duration between two adjacent candidate points is determined as t i+1 -t i , the target duration is determined according to a multiple a (t i+1 -t i ) of the interval duration t i+1 -t i , and a plurality of different target audio segments are cut from the target audio signal according to the target duration, where a is a positive integer greater than or equal to 1. For example, when a = 2, the target duration is 2 (t i+1 -t i ), and when a = 3, the target duration is 3 (t i+1 -t i ).

[0202] It should be noted that the rhythm of the music usually does not change in a local range, and the beats of each audio segment are usually arranged at equal intervals. In order to avoid the problem of audio segment edge insensitivity, the audio signal is usually divided into mutually overlapping audio segments with the candidate beat point as the center and according to the determined target duration, for example, there is an overlapping candidate beat point between the audio segment 1 and the audio segment 2 shown in Figure 3

[0203] Through the above-mentioned division of the audio signal, a plurality of different audio segments are obtained as input of the target prediction neural network model. Since the problem of audio segment edge insensitivity is solved in the process of dividing the audio segment, the reliability of the sample data input into the target prediction neural network model is improved.

[0204] The second processing unit 1406 is configured to input the target audio segment into the target prediction neural network model, obtain a first feature vector of the target audio segment through a target feature extraction network in the target prediction neural network model, and obtain a second feature vector by transforming the first feature vector through a target transformation network in the target prediction neural network model.

[0205] The first determining unit 1408 is configured to determine a target prediction result according to the second feature vector through a target prediction network in the target prediction neural network model.

[0206] As shown in Figure 4 , the target prediction neural network model includes but is not limited to a target feature extraction network, a target transformation network, and a target prediction network. Assuming that the target feature extraction network is a convolutional neural network, it is used to extract feature information of the target audio segment ms1 and compress the feature of the music segment to 1024 dimensions to obtain a first feature vector z i of the target audio segment.

[0207] The first feature vector z​i Input target transformation network, through the first representation vector z i The transformation process yields the second representation vector z. i ', the second representation vector z i ';The second representation vector z i Input the target prediction network and determine the target prediction result, wherein the target prediction result includes, but is not limited to, indicating that the target audio segment includes beat points.

[0208] It should be noted that in this embodiment, the number of audio segments input to the target feature extraction network is not limited. For example, it can be one, or any number of two or more. When there are multiple audio segments input to the target feature extraction network, a first representation vector z will be obtained for each input audio segment. i .

[0209] For a detailed description of the target feature extraction network, target transformation network, and target prediction network, please refer to the detailed descriptions of the various embodiments in the above-mentioned beat point detection method; they will not be repeated here.

[0210] The second determining unit 1410 is used to determine the beat points in the target audio segment based on the candidate beat points when the target prediction result indicates that the target audio segment includes beat points.

[0211] It is easy to understand that by using the second representation vector z mentioned above... i The input is fed into the target prediction network, which can obtain the prediction result corresponding to each audio segment. The prediction result can indicate that the target audio segment includes beat points or does not include beat points.

[0212] For example, the second representation vector z1' of audio segment 1 and the second representation vector z2' of audio segment 2 are input into the target prediction network to obtain a first prediction result and a second prediction result. The first prediction result indicates that audio segment 1 includes a beat point, and the second prediction result indicates that audio segment 2 does not include a beat point. Therefore, it is only necessary to determine the beat point in audio segment 1 based on the candidate beat points.

[0213] The device is applied to obtain a candidate beat point extracted from a target audio signal, and divide the target audio signal into a target video segment with a target time length; then the target video segment is input into a target neural network model, sequentially passes through a target feature extraction network, a target transformation network and a target prediction network, and a target prediction result is determined. That is, by combining the local pulse method with the target neural network model, the determined candidate beat point is screened and corrected, the beat points between target music segments are avoided from being disordered, the technical problem of low accuracy in the beat point detection process is solved, and the technical effect of improving the accuracy of beat point detection is achieved.

[0214] Optionally, the first determination unit 1408 includes:

[0215] The first processing module is configured to input the second feature vector into a first prediction network in the target prediction network to obtain a first prediction result, and input the second feature vector into a second prediction network in the target prediction network to obtain a second prediction result, where the first prediction network is configured to determine the first prediction result according to the second feature vector, and the first prediction result indicates whether the target audio segment includes a beat point, and the second prediction network is configured to determine the second prediction result according to the second feature vector, and the second prediction result indicates a correction value for correcting the candidate beat point.

[0216] Optionally, the first determination unit 1410 includes:

[0217] The second processing module is configured to, when the first prediction result indicates that the target audio segment includes a beat point and the second prediction result indicates the first correction value, determine the beat point in the target audio segment according to the candidate beat point and the first correction value.

[0218] Optionally, the second processing module includes:

[0219] The first processing submodule is configured to, when the first correction value is less than 0, determine a first time point before the candidate beat point in the target audio segment as the beat point, where a time length between the first time point and the candidate beat point is an absolute value of the first correction value.

[0220] The second processing submodule is configured to, when the first correction value is greater than 0, determine a second time point after the candidate beat point in the target audio segment as the beat point, where a time length between the candidate beat point and the second time point is the first correction value.

[0221] The third processing submodule is configured to, when the first correction value is 0, determine the candidate beat point in the target audio segment as the beat point.

[0222] Optionally, the first processing module includes:

[0223] a fourth processing submodule, configured to determine a correction value represented by the second prediction result according to the following formula: wherein, the correction value, a is a preset correction upper limit value, and is a weight parameter in the second prediction network, represents the second feature vector.

[0224] Optionally, the apparatus further includes:

[0225] a third processing unit, configured to determine a next beat point in the target audio segment according to the candidate beat point when the target prediction result indicates that the target audio segment includes the next beat point, wherein the next beat point is a first beat point in a measure in which the music segment is located.

[0226] Optionally, the third processing unit includes:

[0227] a third processing module, configured to input the second feature vector into a third prediction network in the target prediction network to obtain a third prediction result, and input the second feature vector into a fourth prediction network in the target prediction network to obtain a fourth prediction result, wherein the third prediction network is configured to determine the third prediction result according to the second feature vector, and the third prediction result indicates whether the target audio segment includes the next beat point, and the fourth prediction network is configured to determine the fourth prediction result according to the second feature vector, and the fourth prediction result indicates a correction value for correcting the candidate beat point.

[0228] Optionally, the third processing unit further includes:

[0229] a fourth processing module, configured to determine the next beat point in the target audio segment according to the candidate beat point and the second correction value when the third prediction result indicates that the target audio segment includes the next beat point and the fourth prediction result indicates the second correction value.

[0230] Optionally, the fourth processing module includes:

[0231] a fifth processing submodule, configured to determine a third time point before the candidate beat point in the target audio segment as the next beat point when the second correction value is less than 0, wherein a time length between the third time point and the candidate beat point is an absolute value of the second correction value;

[0232] a sixth processing submodule, configured to determine a fourth time point after the candidate beat point in the target audio segment as the next beat point when the second correction value is greater than 0, wherein a time length between the candidate beat point and the fourth time point is the second correction value.

[0233] The seventh processing sub-module is configured to determine the candidate beat point in the target audio segment as a next beat point when the second correction value is 0.

[0234] Optionally, the device further includes:

[0235] The fourth processing unit is configured to determine an interval duration between two adjacent candidate beat points in the target audio signal.

[0236] The fifth processing unit is configured to extract a target audio segment of a target duration from the target audio signal with the candidate beat point as a center, where the target duration is a multiple of the interval duration.

[0237] Optionally, the device further includes:

[0238] The sixth processing unit is configured to input the first feature vector and a set of feature vectors into a target transformation network to obtain a second feature vector through N layers of networks connected in sequence in the target transformation network, where N is a positive integer greater than or equal to 2, the first feature vector and the second feature vector have the same vector dimension, each layer of network in the N layers of networks includes a self-attention module and a fully connected module, the set of feature vectors includes a feature vector of each audio segment in a set of audio segments obtained through the target feature extraction network, the set of audio segments includes an audio segment extracted from the target audio signal with each candidate beat point in a set of candidate beat points as a center, and the set of candidate beat points includes a plurality of continuous candidate beat points extracted from the target audio signal.

[0239] By applying the device to obtain the candidate beat points extracted from the target audio signal, the target audio signal is divided into target video segments of a target duration. Then, the target audio segment is input into the target feature extraction network in the target prediction neural network model to obtain the first feature vector of the target audio segment. The first feature vector is transformed through the target transformation network in the target prediction neural network model to obtain the second feature vector. Finally, the target prediction result is determined according to the second feature vector and the target prediction network in the target prediction neural network model. The beat points in the target video segment are determined according to the target prediction result and the candidate beat points. In other words, by combining the salient local impulse method with the target neural network model, the extracted candidate beat points are screened and corrected using the neural network, avoiding the disorder of the beat points between the target music segments, solving the technical problem of low accuracy in the beat point detection process, and achieving the technical effect of improving the accuracy of beat point detection.

[0240] It should be noted that the embodiments of the beat point detection device herein can refer to the embodiments of the beat point detection method described above, which will not be described herein again.

[0241] According to another aspect of the embodiments of the present application, an electronic device for implementing the beat point detection method is also provided, which can be Figure 15 The electronic device is taken as a background device in this embodiment for illustration. As shown in the figure, the electronic device includes a memory 1502 and a processor 1504, the memory 1502 stores a computer program, and the processor 1504 is configured to execute the steps in any of the above method embodiments through the computer program. Figure 15 The electronic device includes a memory 1502 and a processor 1504, the memory 1502 stores a computer program, and the processor 1504 is configured to execute the steps in any of the above method embodiments through the computer program.

[0242] Optionally, in this embodiment, the electronic device can be located in at least one of the network devices in the computer network.

[0243] Optionally, in this embodiment, the processor can be configured to execute the following steps through the computer program:

[0244] S1, obtaining a candidate beat point extracted from a target audio signal;

[0245] S2, centering on the candidate beat point, a target audio segment of a target duration is cut from the target audio signal;

[0246] S3, inputting the target audio segment into a target prediction neural network model, obtaining a first feature vector of the target audio segment through a target feature extraction network in the target prediction neural network model, and transforming the first feature vector through a target transformation network in the target prediction neural network model to obtain a second feature vector;

[0247] S4, determining a target prediction result according to the second feature vector through a target prediction network in the target prediction neural network model;

[0248] S5, when the target prediction result indicates that the target audio segment includes a beat point, determining the beat point in the target audio segment according to the candidate beat point.

[0249] Optionally, those skilled in the art can understand that Figure 15 The structure shown in the figure is only schematic, and the electronic device can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a mobile Internet device (MID), a PAD, etc. Figure 15 It does not limit the structure of the electronic device. For example, the electronic device can include more or less components (such as network interfaces, etc.) than those shown in the figure, or have a different configuration from that shown in the figure. Figure 15 Figure 15

[0250] ​​The memory 1502 can be used to store software programs and modules, such as program instructions / modules corresponding to the beat point detection method and device in the embodiments of the present application. The processor 1504 executes various functions and data processing by running the software programs and modules stored in the memory 1502, that is, implements the beat point detection method described above. The memory 1502 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 1502 can further include a memory remotely arranged with respect to the processor 1504, which can be connected to the terminal through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. Specifically, the memory 1502 can be used to store, but is not limited to, target audio segments, first feature vectors, second feature vectors, and the like. As an example, as shown in FIG. 2, the memory 1502 can include, but is not limited to, the first acquisition unit 1402, the first processing unit 1404, the second processing unit 1406, the first determination unit 1408, and the second determination unit 1410 in the beat point detection device described above. In addition, other module units in the beat point detection device described above can also be included, but are not limited to, which will not be described in detail in this example. Figure 15

[0251] Optionally, the transmission device 1506 described above is used to receive or send data via a network. Specific examples of the above network can include wired networks and wireless networks. In one example, the transmission device 1506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers through a network cable to communicate with the Internet or a local area network. In one example, the transmission device 1506 is a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0252] In addition, the electronic device described above further includes a display 1508 for displaying the allowed push virtual resource, and a connection bus 1510 for connecting various module components in the electronic device.

[0253] In other embodiments, the target terminal or server can be a node in a distributed system, where the distributed system can be a blockchain system, which can be a distributed system formed by the plurality of nodes communicating through a network. The nodes can form a peer-to-peer (P2P, Peer To Peer) network, and any form of computing device, such as a server, a terminal, and other electronic devices, can become a node in the blockchain system by joining the peer-to-peer network.​

[0254] According to an aspect of the present application, a computer program product or computer program is provided, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the beat detection method provided in various optional implementation manners of the server verification processing aspect, wherein the computer program is configured to perform the steps in any of the method embodiments when executed.

[0255] Optionally, in the embodiment, the computer readable storage medium described above can be configured to store a computer program for performing the following steps:

[0256] S1, obtaining a candidate beat point extracted from a target audio signal;

[0257] S2, centering on the candidate beat point, a target audio segment of a target duration is cut from the target audio signal;

[0258] S3, inputting the target audio segment into a target prediction neural network model, obtaining a first feature vector of the target audio segment through a target feature extraction network in the target prediction neural network model, and transforming the first feature vector through a target transformation network in the target prediction neural network model to obtain a second feature vector;

[0259] S4, determining a target prediction result according to the second feature vector through a target prediction network in the target prediction neural network model;

[0260] S5, when the target prediction result indicates that the target audio segment includes a beat point, determining the beat point in the target audio segment according to the candidate beat point.

[0261] Optionally, in the embodiment, those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the target terminal through a program, and the program can be stored in a computer readable storage medium, and the storage medium can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0262] The serial numbers of the embodiments of the present application described above are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0263] The integrated units in the above embodiments, if implemented in the form of software function units and sold or used as independent products, can be stored in the above computer-readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions to make one or more computer devices (which can be personal computers, servers or network devices, etc.) execute all or part of the steps of the methods of the various embodiments of the present application.

[0264] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0265] In the several embodiments provided by the present application, it should be understood that the disclosed client can be implemented in other ways. Of course, the above device embodiment is only illustrative, and the division of units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0266] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment according to actual needs.

[0267] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0268] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principle of the present application, some improvements and refinements can be made, and these improvements and refinements should be regarded as the protection scope of the present application.

Claims

1. A method for detecting beat points, characterized in that, include: Obtain candidate beat points extracted from the target audio signal; Centered on the candidate beat point, extract the target audio segment of the target duration from the target audio signal; The target audio segment is input into the target prediction neural network model. The target feature extraction network in the target prediction neural network model is used to obtain the first representation vector of the target audio segment. The first representation vector is then transformed by the target transformation network in the target prediction neural network model to obtain the second representation vector. The second representation vector is input into the first prediction network of the target prediction neural network model to obtain a first prediction result, and the second representation vector is input into the second prediction network of the target prediction neural network model to obtain a second prediction result; When the first prediction result indicates that the target audio segment includes a beat point, the beat point in the target audio segment is determined based on the correction value for correcting the candidate beat point indicated by the second prediction result and the candidate beat point.

2. The method according to claim 1, characterized in that, When the first prediction result indicates that the target audio segment includes a beat point, determining the beat point in the target audio segment based on the correction value indicated by the second prediction result and the candidate beat point includes: When the first prediction result indicates that the target audio segment includes the beat point, and the second prediction result indicates the first correction value, the beat point in the target audio segment is determined based on the candidate beat point and the first correction value.

3. The method according to claim 2, characterized in that, The step of determining the beat point in the target audio segment based on the candidate beat point and the first correction value includes: When the first correction value is less than 0, the first moment before the candidate beat point in the target audio segment is determined as the beat point, wherein the duration between the first moment and the candidate beat point is the absolute value of the first correction value; When the first correction value is greater than 0, the second moment after the candidate beat point in the target audio segment is determined as the beat point, wherein the duration between the candidate beat point and the second moment is the first correction value; When the first correction value is 0, the candidate beat point in the target audio segment is determined as the beat point.

4. The method according to claim 1, characterized in that, The step of inputting the second representation vector into the second prediction network of the target prediction neural network model to obtain the second prediction result includes: The correction value represented by the second prediction result is determined by the following formula: , in, This represents the correction value, where 'a' is a preset upper limit for correction. and These are the weight parameters in the second prediction network. This represents the second representation vector.

5. The method according to claim 1, characterized in that, The method further includes: When the first prediction result indicates that the target audio segment includes a next beat point, the next beat point in the target audio segment is determined based on the candidate beat points, wherein the next beat point is the first beat point in the measure in which the audio segment is located.

6. The method according to claim 5, characterized in that, The method further includes: The second representation vector is input into the third prediction network in the target prediction neural network model to obtain a third prediction result, and the second representation vector is input into the fourth prediction network in the target prediction neural network model to obtain a fourth prediction result. The third prediction network is used to determine the third prediction result based on the second representation vector. The third prediction result indicates whether the target audio segment includes the next beat point. The fourth prediction network is used to determine the fourth prediction result based on the second representation vector. The fourth prediction result indicates a correction value used to correct the candidate beat point.

7. The method according to claim 6, characterized in that, When the first prediction result indicates that the target audio segment includes the next beat point, determining the next beat point in the target audio segment based on the candidate beat points includes: When the third prediction result indicates that the target audio segment includes the next beat point and the fourth prediction result indicates the second correction value, the next beat point in the target audio segment is determined based on the candidate beat point and the second correction value.

8. The method according to claim 7, characterized in that, The step of determining the next beat point in the target audio segment based on the candidate beat point and the second correction value includes: When the second correction value is less than 0, the third moment before the candidate beat point in the target audio segment is determined as the next beat point, wherein the duration between the third moment and the candidate beat point is the absolute value of the second correction value; When the second correction value is greater than 0, the fourth moment after the candidate beat point in the target audio segment is determined as the next beat point, wherein the duration between the candidate beat point and the fourth moment is the second correction value; When the second correction value is 0, the candidate beat point in the target audio segment is determined as the next beat point.

9. The method according to any one of claims 1 to 8, characterized in that, The step of extracting a target audio segment of a target duration from the target audio signal, centered on the candidate beat point, includes: Determine the interval duration between two adjacent candidate beat points in the target audio signal; Centered on the candidate beat point, a target audio segment of the target duration is extracted from the target audio signal, wherein the target duration is a multiple of the interval duration.

10. The method according to any one of claims 1 to 8, characterized in that, The step of transforming the first representation vector through the target transformation network in the target prediction neural network model to obtain the second representation vector includes: The first representation vector and a set of representation vectors are input into the target transformation network. The second representation vector is obtained by passing through N layers of networks connected sequentially in the target transformation network, where N is a positive integer greater than or equal to 2. The first representation vector and the second representation vector have the same vector dimension. Each layer of the N-layer network includes a self-attention module and a fully connected module. The set of representation vectors includes the representation vector of each audio segment in a set of audio segments obtained by the target feature extraction network. The set of audio segments includes audio segments extracted from the target audio signal with each candidate beat point in a set of candidate beat points as the center. The set of candidate beat points includes a plurality of consecutive candidate beat points extracted from the target audio signal.

11. A device for detecting beat points, characterized in that, include: The first acquisition unit is used to acquire candidate beat points extracted from the target audio signal; The first processing unit is used to extract a target audio segment of target duration from the target audio signal, centered on the candidate beat point. The second processing unit is used to input the target audio segment into the target prediction neural network model, obtain the first representation vector of the target audio segment through the target feature extraction network in the target prediction neural network model, and transform the first representation vector through the target transformation network in the target prediction neural network model to obtain the second representation vector. The device is further configured to input the second representation vector into the first prediction network of the target prediction neural network model to obtain a first prediction result, and input the second representation vector into the second prediction network of the target prediction neural network model to obtain a second prediction result; The second determining unit is configured to determine the beat point in the target audio segment based on the correction value for correcting the candidate beat point indicated by the second prediction result and the candidate beat point when the first prediction result indicates that the target audio segment includes a beat point.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program is executed by a processor to perform the method of any one of claims 1 to 10.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of any one of claims 1 to 10.

14. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to perform the method of any one of claims 1 to 10 via the computer program.

Images