Method and system for analyzing music rhythm in real time

Abstract

The invention provides a method and a system for analyzing music rhythm in real time, in which an audio signal is received through an input interface of a device, and the audio signal is decoded through an audio signal processing circuit to obtain audio frame data, including sampling rate, sampling rate, sampling rate, sampling rate, sampling rate, sampling rate and sampling rate. An audio frame jump step length is obtained according to the audio frame size and the overlapped audio frame size set by the system, an audio frame rate is calculated according to the sampling rate and the audio frame jump step length, and an initial value of a beat period is calculated according to the sampling rate and an initial value of a beat number per minute; and the beat point position can be obtained according to the number of sampling points covered by one beat. And then, the system speculates the position of the next beat point regularly by using a recursive algorithm, recalculates a new beat period according to audio frame data of a past period of time, reckons the position of the next beat point according to the new beat period, and further adds a beat prompt tone at the position of the beat point.

Description

Technical Field

[0001] The present invention relates to a technology for adding a beat prompt sound to music, and in particular to a method and system for real-time analysis of music rhythm by analyzing an audio signal to determine whether a beat prompt sound can be added under a stable rhythm. Background Art

[0002] When someone is listening to music or watching a music video (MV), they may want to sing along with the music. However, it is difficult for inexperienced or young people to catch the beat. Therefore, the prior art proposes a method of adding a beat prompt sound to the music to make it easier for users to sing along with the music.

[0003] In view of this, according to the existing technology, there are players that can provide beat prompt sounds when playing music, but the relevant algorithms for estimating rhythm are all offline models, and the entire song must be obtained at one time and then analyzed to get the result. In this way, preparations can only be made before playing, and the beat data cannot be obtained in real time during on-demand playback. Summary of the invention

[0004] In order to provide rhythm data when ordering music, the present invention proposes a method and system for real-time analysis of music rhythm. The system can be run in a device by circuit or software method. In addition to the circuit for processing audio signals in this device, an audio signal analysis module is provided, which is connected to the audio signal processing circuit in the device, and the method for real-time analysis of music rhythm is executed.

[0005] In the method, an audio signal is received through an input interface, the audio signal is decoded through an audio signal processing circuit, and audio frame data including a sampling rate is obtained, then an audio frame jump step is obtained according to an audio frame size and an overlapping audio frame size, and an audio frame rate is calculated according to the sampling rate and the audio frame jump step. Then, an initial value of a beat cycle is calculated according to the sampling rate and the initial value of the number of beats per minute, so as to obtain the beat point position according to the number of sampling points covered by one beat.

[0006] Afterwards, a recursive algorithm is used to estimate the next beat point position, wherein the number of audio frames covered in the past period of time is calculated based on the audio frame rate, and then a new beat cycle is calculated based on the number of beats per minute. The next beat point position can be estimated based on this new beat cycle.

[0007] In this way, the system can be set to recalculate a new beat cycle based on the audio frame data of the past period of time at regular intervals to recalculate the next beat point position.

[0008] Preferably, when the audio signal is a multi-channel audio signal modulated by a pulse code, it is necessary to perform an averaging operation on the digital signals of multiple audio frames at the same time to form a monophonic audio signal for real-time analysis of the music rhythm.

[0009] Preferably, the audio frame rate represents the number of proxy audio frames in one second, wherein the proxy audio frame is obtained based on an audio frame size of an original audio frame and an overlapping audio frame size set by the system.

[0010] According to one embodiment, in the method of calculating a new beat period, an autocorrelation function can be used to find a repetitive pattern using the correlation of audio frame data at different time points in the audio signal, and then the new beat period is recalculated based on the interval of the maximum value of the autocorrelation function.

[0011] Furthermore, a starting point feature value is introduced into the autocorrelation function, that is, the starting point feature value of each proxy audio frame in a plurality of proxy audio frames in a time segment is calculated, and the maximum value of a calculated value of the starting point feature value or a related starting point feature value in this time segment is determined to deduce a new beat cycle.

[0012] Furthermore, a beat point detection value can be introduced, wherein a maximum beat point feature value among the sampling points near the feature value of the starting point of each proxy audio frame in the time segment is first calculated, which can be used as the beat point detection value of the proxy audio frame; then the beat point detection value of the next beat cycle is inferred based on all the beat point detection values ​​of the beat cycle in the past period of time, that is, the beat point detection value with the largest value in the next beat cycle is used as the beat point position of the next beat cycle.

[0013] Furthermore, when the next beat point position of the new beat cycle is obtained, the beat point positions can be traced back multiple times and multiple time differences between adjacent beat point positions can be calculated to determine whether the audio signal has a stable rhythm. If it is determined that the audio signal has a stable rhythm, a beat prompt sound is added to each beat point position. Afterwards, when the audio signal processing circuit outputs the processed audio signal through the output interface, the beat prompt sound added to each beat point position can be synthesized in the audio signal.

[0014] To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a system architecture embodiment for executing a method for analyzing music rhythm in real time is shown;

[0016] Figure 2A flow chart of an embodiment of a method for analyzing music rhythm in real time is shown;

[0017] Figure 3 A flow chart of a first pre-processing process embodiment in a method for real-time analysis of music rhythm is shown;

[0018] Figure 4 A flow chart of an embodiment of a second pre-processing process in a method for real-time analysis of music rhythm is shown;

[0019] Figure 5 A schematic diagram of an embodiment of obtaining an audio frame skipping step and an audio frame rate in a second pre-processing flow is shown;

[0020] Figure 6 A flow chart of an embodiment of subsequent beat point detection processing in a method for real-time analysis of music rhythm is shown;

[0021] Figure 7 A flow chart of an embodiment of detecting the position of the next beat point according to the estimated beat cycle is shown;

[0022] Figure 8 An embodiment diagram of a method for determining a characteristic value of an onset point by using an autocorrelation function is shown;

[0023] Fig. 9 An embodiment diagram showing a beat point position described by a beat point detection value is shown;

[0024] Fig.10 An embodiment diagram showing a method of describing the estimated next beat point position using a beat point detection value; and

[0025] Fig.11 A diagram of an audio frame data embodiment for describing a method for determining whether the rhythm in an audio signal is stable is shown.

[0026] Description of reference numerals:

[0027] 111: Audio and video data 100: Audio and video device 105: Input interface

[0028] 107: output interface 101: audio signal processing circuit 103: video signal processing circuit

[0029] 113: Display 115: Speaker 120: Audio signal analysis module

[0030] 121: Music rhythm analysis unit 123: Beat prompt sound adding unit 501: First audio frame

[0031] 502: second audio frame 503: third audio frame 511: first proxy audio frame

[0032] 512: First proxy audio frame 513: First proxy audio frame 521: Audio frame jump step

[0033] 523: audio frame size 81: first group of data 82: second group of data

[0034] 901: Current beat point 903, 905: Reference beat point 1001: Current beat point

[0035] 1003: next beat cycle 1005, 1007: reference beat cycle 1101, 1102: audio frame

[0036] 1103, 1104: Proxy audio frames 1105, 1106: Audio frame jump step

[0037] Step S201: receiving an audio signal

[0038] Step S203: Decode to obtain audio frame data

[0039] Step S205: Convert to a mono audio signal

[0040] Step S207: First pre-processing flow (obtaining audio frame size, overlapping audio frames and beats per minute)

[0041] Step S209: Second pre-processing flow (obtaining audio frame skipping step length and audio frame rate)

[0042] Step S211: Calculate the beat period

[0043] Step S213: Estimate the next beat point position

[0044] Step S215: Beat stability determination

[0045] Step S217: Adding beat prompt sound

[0046] Step S219: Outputting an audio frame with a beat prompt tone

[0047] Step S221: Directly output the audio frame (without adding the beat prompt sound)

[0048] Steps S301 to S307: Flow of the first pre-processing process

[0049] Steps S401 to S409: Flow of the second pre-processing process

[0050] Steps S601 to S609: Subsequent beat point detection process

[0051] Steps S701 to S707: Process of detecting the next beat point position DETAILED DESCRIPTION

[0052] The following is an explanation of the embodiments of the present invention through specific specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention through the contents disclosed in this application. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the concept of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. It is stated in advance. The following embodiments will further describe the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention.

[0053] It should be understood that, although the terms "first", "second", "third", etc. may be used herein to describe various components or signals, these components or signals should not be limited by these terms. These terms are mainly used to distinguish one component from another component, or one signal from another signal. In addition, the term "or" used herein may include any one or more combinations of the associated listed items depending on the actual situation.

[0054] The present invention provides a method and system for real-time analysis of music rhythm, which are used in a device. When the device receives a video and audio data, the video signal and the audio signal are decoded, decompressed and processed by the software and hardware for processing the video signal and the audio signal, respectively, and then the video signal and the audio signal can be properly encoded into a broadcastable image and sound. In particular, when the content of the input device is a music-related video and audio content, such as a music video (MV), or an audio signal of music that only has sound but no image, the method and system for real-time analysis of music rhythm proposed by the present invention can add a beat prompt sound to the music, such as adding a beat prompt sound (such as a beep or other rhythm sound) to the accent part of a measure, or adding a beat prompt sound to both the accent and the secondary accent of a measure, or regularly adding a beat prompt sound to each beat, so that the user can easily sing along.

[0055] According to one embodiment, reference may be made to Figure 1 The schematic diagram of the system architecture embodiment of the method for running real-time analysis of music rhythm is shown. The system for real-time analysis of music rhythm is set in a specific device, which can be a circuit system implemented in hardware running in the device, or a software system implemented in a computer program running in the device.

[0056] The method for real-time analysis of music rhythm is run in a device, and the device can be implemented as a set-top box (STB), a smart TV, or a computer device. Figure 1The example shown provides a device 100 that can be a built-in or external device of a television system, including an audio and video processing circuit that cooperates with hardware in a software manner.

[0057] According to the illustrated embodiment, the main circuits in the device 100 include input / output interfaces (input interface 105 and output interface 107) and data processing circuits therein, and the audio signal and video signal parts can be processed by the audio signal processing circuit 101 and the video signal processing circuit 103 respectively, and can be output to the display 113 and the speaker 115 connected to the device 100 respectively. The technology is known to technicians in the relevant fields. According to the system embodiment of real-time analysis of music rhythm proposed by the present invention, the input and output ends of the audio signal processing circuit 101 that processes the audio signal part include an audio signal analysis module 120 connected to the audio signal processing circuit 101. The audio signal analysis module 120 can implement the music rhythm analysis unit 121 and the beat prompt sound adding unit 123 in the circuit, firmware or software, wherein the music rhythm analysis unit 121 is mainly used to analyze the beat cycle in the audio signal to obtain the beat point position, and the beat prompt sound adding unit 123 adds the beat prompt sound to the audio signal according to the beat point position obtained by real-time analysis.

[0058] The audio signal analysis module 120 executes the method for real-time analysis of music rhythm. In the method, the audio and video data 111 including audio signal and video signal is received through the input interface 105 of the device 100. The data of the video signal part is processed by the video signal processing circuit 103. After the video signal processing (the details are not repeated here), the image can be output through the output interface 107, and the screen is displayed through the display 113. The data of the audio signal part is processed by the audio signal processing circuit 101, and the audio signal is decoded to obtain the output sound, which is output to the speaker 115 through the output interface 107 for playback. In addition, in particular, the beat point position in the audio signal is obtained by the music rhythm analysis unit 121 of the audio signal analysis module 120, and the beat prompt sound added at each beat point position is synthesized in the audio signal by the beat prompt sound adding unit 123. The audio signal is then encoded in a specific format by the audio signal processing circuit 101 and output.

[0059] The system provided in the device 100 may be an embedded system provided in the device, and is used to perform a method for real-time analysis of music rhythm, which may be referred to in Figure 2 The flowchart of the method embodiment is shown.

[0060] According to the flowchart of the embodiment, at the beginning, the system running in the device receives an audio signal through the input interface of the device (step S201), and decodes the audio signal through the audio signal processing circuit to obtain the audio frame data therein, such as the sampling rate (sample rate) of the number of samples (samples) covered in one second in the audio signal (step S203). The sampling rate depends on the data of the audio signal, and the sampling rate will be different according to the audio file.

[0061] Next, according to an embodiment, when the audio signal is a multi-channel audio signal, a software method can be used to perform an averaging operation on the digital signals of multiple audio frames at the same time to form a monophonic audio signal for real-time analysis of music rhythm (step S205).

[0062] Next, the pre-processing flow of the audio signal is entered, one of which is recorded in step S207 (the first pre-processing flow), and the system setting values ​​include the audio frame size (frame size), the overlapped audio frame size (overlapped framesize) and the beats per minute (BPM). Thus, in step S209 (the second pre-processing flow), an audio frame hop size (hop size) can be obtained based on the audio frame size and the overlapped audio frame size, and the audio frame rate (frame per second) can be calculated based on the sampling rate and the audio frame hop size. It is worth mentioning that, reference can be made to Figure 5 The proxy audio frame is obtained based on the audio frame size of an original audio frame and the overlapping audio frame size set by the system. The audio frame rate represents the number of proxy audio frames in one second. The calculation process of the audio frame rate can be referred to in the embodiment. Figure 4 .

[0063] It is worth mentioning that in the field of digital audio signals, adjacent audio frames in an audio signal under a specific encoding format will have overlapping audio frames when sampling, because in a relatively unstable audio signal, adjacent audio frames will have slight changes. In order to reduce this change, adjacent audio frames can be allowed to have overlapping parts (the size of the overlapping audio frame is approximately 1 / 2 to 2 / 3 of the original audio frame). Because overlapping audio frames can effectively hide errors in the audio signal (effective error concealment), when decoding the audio signal, it is necessary to first calculate the audio frame jump step according to the audio frame size set by the system and the overlapping audio frame size, so as to be able to calculate the subsequent beat cycle. The audio frame jump step represents the distance between the starting point of an audio frame and the starting point of the next audio frame (sampling points), which is equal to the audio frame size (the number of sampling points covered by an audio frame) minus the overlapping audio frame size (that is, the number of sampling points covered by the overlapping audio frame). Among them, the audio frame rate represents the number of audio frames covered per second. The beat cycle represents the number of audio frames covered between two beat points, and the beat cycle can be used to infer the next beat point.

[0064] In particular, according to an embodiment, after step S211, the method for real-time analysis of music rhythm uses a recursive algorithm to infer the next beat point position. According to an embodiment, the number of audio frames covered in the past period of time can be calculated based on the audio frame rate, and a new beat cycle can be calculated based on the number of beats per minute, so the next beat point position can be inferred based on the new beat cycle (step S213). According to the recursive algorithm, the system can be set to recalculate the new beat cycle based on the audio frame data of the past period of time at regular intervals to recalculate the next beat point position.

[0065] When the system can recalculate the new beat cycle and estimate the next node position at regular intervals, the system will further determine whether the beat in the audio signal is stable, such as Figure 2 Step S215 determines the beat stability, wherein the mechanism is to trace back multiple beat point positions and calculate multiple time differences between adjacent beat point positions to determine whether the audio signal has a stable rhythm.

[0066] according to Figure 2 In the process shown, if it is judged that the rhythm is stable in step S215, a beat cue sound can be added to the beat point position judged in the above process by software method (step S217), and when the audio signal processing circuit outputs the processed audio signal through the output interface, the beat cue sound added at each beat point position can be synthesized in the audio signal to output an audio signal of an audio frame with a beat cue sound (step S219).

[0067] According to one of the embodiments of adding a beat reminder sound, when the next beat point position is determined and it is determined to be a stable rhythm, a beep sound can be automatically added as a beat reminder sound. Therefore, when the user randomly selects streaming audio and video, the device can be controlled to turn on this beat reminder sound function as a body rhythm aid or a performance practice aid. When turning on this beat reminder sound function, you can choose general (adding a beat reminder sound directly according to the beats per minute (BPM)), or select fast tempo (for example, adding a beat reminder sound at twice the speed BPM), or slow tempo (for example, adding a beat reminder sound at 0.5 times the speed BPM) as the beat point of the auxiliary sound effect.

[0068] On the contrary, in Figure 2 In step S215 of the process, if it is determined that the rhythm is not stable, the audio signal composed of the audio frames (without adding the beat prompt sound) can be directly output without adding the beat prompt sound (step S221).

[0069] Above Figure 2 The first pre-processing flow described in step S207 can be referred to Figure 3 The flowchart of the embodiment shown in the figure describes that when the system receives an audio signal through a device (step S301), and decodes the audio signal (step S303), if the audio signal is a multi-channel audio signal using pulse code modulation (PCM), the multi-channel audio frame f(t) shown in the figure will be generated after decoding (step S305).

[0070] The output of the audio signal decoder shown in the figure on the time axis is in units of audio frames. At each time point, there will be one or more audio frames of the channel. Each audio frame is a digital audio signal in the form of pulse code modulation (PCM). It is schematically shown that the audio frames of channel 0 (ch(0)) are f(0)ch(0), f(1)ch(0), ... f(t)ch(0); the audio frames of channel 1 (ch(1)) are f(0)ch(1), f(1)ch(1), ... f(t)ch(1); the audio frames of channel m (ch(m)) are f(0)ch(m), f(1)ch(m), ... f(t)ch(m).

[0071] Then, the multi-channel is converted to mono. One way is to perform an averaging operation on the digital signals of multiple audio frames at the same time to form a mono audio signal for real-time analysis of music rhythm: f(0), f(1), ... f(t) (step S307).

[0072] Figure 2 The second pre-processing process described in step S209 can be referred to Figure 4In the flowchart of the embodiment shown, the basic formula of the embodiment of audio signal processing is as described in Formula 1 for steps (a) to (g), wherein the numerical values ​​are for illustrative purposes only and are not intended to limit the scope of application of the invention.

[0073] The illustrated process can refer to Formula 1. The system first obtains a mono audio frame (step S401). If the sampling rate of the audio signal is 44100 (sample / second), it means that the audio signal consists of 44100 samples per second (step S403).

[0074] The system setting value may include an audio frame size (frame size, which is the number of sampling points included in an audio frame), for example, 1024, indicating that an original audio frame covers 1024 sampling points (step S405). The system also sets the size of the overlapped audio frame, indicating that the overlapped audio frame between two adjacent audio frames covers 512 sampling points. This example shows that multiple sampling points are covered, such as 512.

[0075] In this way, the audio frame hop step can be obtained based on the audio frame size (such as 1024) and the overlapping audio frame size (such as 512). The audio frame hop step is equal to the audio frame size minus the overlapping audio frame size. In this example, the audio frame hop step is calculated to be 512 sampling points (hop size = frame size-overlapped frame size, 1024-512 = 512) (step S407).

[0076] Next, the audio frame rate is obtained by dividing the sampling rate by the audio frame hop size. Formula 1 shows that the audio frame rate is equal to 44100 divided by 512, which is equal to 86.13, indicating that an audio frame has 86.13 sampling points (frame rate = sample rate / hop size) (step S409).

[0077] Figure 5 To describe the second pre-processing process ( Figure 2 Schematic diagram of an embodiment of obtaining the audio frame skipping step and the audio frame rate in step S209). As described above, adjacent audio frames in an audio signal under a specific coding format may have overlapping audio frames during sampling, and overlapping audio frames can effectively hide errors in the audio signal. Therefore, when decoding the audio signal, it is necessary to first calculate the audio frame skipping step according to the audio frame size set by the system and the overlapping audio frame size, so as to be able to calculate the subsequent beat cycle.

[0078] According to the legend, it is schematically shown that the first audio frame (audio frame f(0)) 501, the second audio frame (audio frame f(1)) 502 and the third audio frame (audio frame f(2)) 503 are received successively, each having an audio frame size 523. There are overlapping parts between adjacent audio frames. In order to calculate the beat cycle, it is necessary to first obtain the audio frame jump step. According to the legend, after eliminating the overlapping parts between adjacent audio frames, it is shown that there are a first proxy audio frame (audio frame f'(0)) 511, a first proxy audio frame (audio frame f'(1)) 512 and a first proxy audio frame (audio frame f'(2)) 513. Therefore, the audio frame jump step 521 can be obtained according to the audio frame size 523 and the overlapping audio frame size. In this way, the proxy audio frame (f'(t)) is obtained after overlapping the adjacent audio signal frames (f(t)) and then used for beat point detection processing.

[0079] This example shows that the audio frame jump step 521 is located between the start point of the first audio frame 501 and the start point of the second audio frame 502, and the distance points (sampling points) are equal to the audio frame size (the number of sampling points covered by an audio frame) minus the overlapping audio frame size (that is, the number of sampling points covered by the overlapping audio frame). Therefore, the proxy audio frame is obtained based on the audio frame size of the original audio frame and the overlapping audio frame size set by the system. When the audio frame rate can be calculated according to the sampling rate and the audio frame jump step, the audio frame rate represents the number of proxy audio frames (f'(t)) in one second.

[0080] After the above pre-processing process is completed, the initial value of a beat period (i.e., the initial value of the number of sampling points τ covered by one beat) is calculated according to the sampling rate obtained by the above process and the initial value of the number of beats per minute, so as to obtain the beat point position according to the number of sampling points covered by one beat (step S211). As in step (f) of formula 1, when the number of beats per minute is given to be 120, in step (g), it can be calculated that the beat period is 43.06 audio frames, that is, every 43.06 audio frames has a beat point position.

[0081] Formula 1:

[0082] (a) Given a sampling rate (sample rate) = 44100 (sample / second);

[0083] (b) The system sets the audio frame size (frame size) = 1024 (samples);

[0084] (c) The system sets the overlapped audio frame size (overlapped frame size) = 512 (samples);

[0085] (d) The audio frame hop size is obtained as follows: frame size - overlapped frame size = 512

[0086] (sample);

[0087] (e) Audio frame rate (frame rate) = sample rate / hop size = 44100 / 512 = 86.13 (frames per second).

[0088] (f) Given beats per minute (BPM) = 120;

[0089] (g) Calculate beat period = (60*(sample rate / BPM)) / hot size = 60*(44100)

[0090] / 120) / 512=43.06(frame).

[0091] According to the above Figure 2 The process of the method for analyzing music rhythm is as follows: Figure 3 and Figure 4 After the pre-processing process described, the initial value of the beat period (τ) is calculated according to the sampling rate obtained from the audio signal and the initial value of the beats per minute (BPM) set by the system or obtained from the audio signal. The number of sampling points (b) covered by a beat can be calculated, that is, a beat point will appear every b sampling points, and the position of the next beat point can be further estimated later.

[0092] Generally speaking, the next beat point position can be inferred based on the initial beat cycle obtained. Under a stable music rhythm, the beat prompt sound should be added. However, in the face of various forms of music, it is impossible to guarantee that the rhythm of the music is always the same. Therefore, in the method of real-time analysis of music rhythm, the beat point position will be continuously detected during the music playback process to determine whether the beat prompt sound can be stably output. For examples, please refer to Figure 6 Flowchart of the subsequent beat point detection process shown.

[0093] The illustrated process adopts a dynamic programming method. According to the method for real-time analysis of music rhythm proposed by the present invention, a recursive algorithm is used to infer the beat point position. According to the process embodiment, the initial value of the beats per minute (BPM) is first determined (step S601). It can be the initial value of the beats per minute obtained from the original audio signal when the music starts to play, or a system setting value is used. Then, the initial value of the beat period (τ) is calculated based on the sampling rate obtained from the audio signal and the audio frame jump step set by the system (step S603). This part can refer to Formula 1.

[0094] For example, the initial value of the number of beats per minute generally falls between 60 and 160, and the sampling points of the corresponding beat period (τ) fall between 44100 and 16537.5. According to the audio frame rate, the number of audio frames is between 86.13 and 32.

[0095] Afterwards, the system can be pre-set to re-detect the beat point position at a certain time interval (such as u seconds), so the beat cycle (τ') can be recalculated based on the data of the past period of time (u seconds). According to the embodiment, in the step of recalculating the beat cycle, the audio frame rate (the number of audio frames per second) is also calculated from the sampling rate and the audio frame jump step, and then the number of proxy audio frames (f'(t)) covered by the past period of time (u seconds) is calculated (step S605). The calculation of the number of proxy audio frames can refer to Formula 2.

[0096] Formula 2:

[0097] The data of u seconds is u*frame rate=u*(sample rate / hop size)=u*(44100 / 512)=u*86.13 f'(t).

[0098] After the number of proxy audio frames covered in a period of time is obtained, a new beat cycle is recalculated according to the beats per minute (BPM) (step S607), and then the next beat point position is estimated (step S609).

[0099] For example, if u is 6 seconds, the number of proxy audio frames covered in u seconds can be obtained as: 6*(44100 / 512)=516.79 proxy audio frames (f'(t)). The beats per minute (BPM) of general music is 60 to 160. According to the calculation results of the above formula, the beat period (τ) is between 86.13 and 32 proxy audio frames, which means that there can be 6 (516.79 / 86.13 is approximately equal to 6) to 16 (516.79 / 32 is approximately equal to 16) beat points in 6 seconds. In this way, it will be possible to determine whether there is a stable rhythm and infer the next beat point.

[0100] According to the embodiment of calculating the new beat cycle (step S607), in the face of continuously received audio signals, continuously calculating the new beat cycle will consume a large amount of computing power, and the amount of computing power can be further reduced. According to one embodiment, in the method of real-time analysis of music rhythm, the beat cycle can be recalculated only when a certain proxy audio frame is calculated as a beat location, rather than calculating it for each proxy audio frame.

[0101] In another embodiment, in order to reduce the amount of calculation, when calculating the number of proxy audio frames covered in the past period of time (such as u seconds), an auto-correlation function (ACF) is used to find the repetitive pattern by using the correlation of the audio frame data at different time points in the audio signal, and the beat period is recalculated based on the interval of the maximum value of the auto-correlation function.

[0102] One of the purposes of the method for real-time analysis of music rhythm proposed in the present invention is to infer the position of the next beat point. The relevant embodiment process is as follows Figure 7 The flowchart of the embodiment of detecting the position of the next beat point according to the estimated beat cycle is shown.

[0103] Steps start as above Figure 6 In step S605, a proxy audio frame (f'(t)) is first obtained (step S701), and as described in the above embodiment, in addition to calculating a new beat period), an onset feature value (onset value, O(t)) is calculated according to an autocorrelation function (step S703).

[0104] In the method of calculating the new beat period, you can refer to Figure 8 The method for calculating the beat period using the autocorrelation function is shown in the figure, that is, recalculating the new beat period based on the data (audio frames) of a period of time (such as u seconds) in the past. According to the embodiment, as shown in Formula 3, the new beat period (τ') is recalculated as the interval z with the maximum value of the autocorrelation function (ACF), which can be expressed as: τ'=arg Max ACF(z), where z=τ+20,…,τ-20. In order to reduce the amount of calculation, the system is set to search only a certain number (such as 20) of proxy audio frames f'(t) before and after, and the larger this number is, the larger the amount of calculation of the system is.

[0105] Taking the past u seconds as an example, assuming that the data of u seconds includes u' proxy audio frames f'(t), it can be expressed as u'=u*(sample rate / hop size). The autocorrelation function (ACF) is expressed as the following formula 3, where an onset value (onset value O(t)) is introduced into the autocorrelation function, which can be referred to formula 4.

[0106] Formula 3:

[0107] ACF(z)=∑ k O(k)O(kx);k=t-0,…,t-(u'-1);

[0108] if kz<0,O(kz)=0.

[0109] Formula 4:

[0110] O(t)=∑ k f′ k (t)f′ k (t); k = 1,…, frame size.

[0111] where f′ k (t) represents the kth sampling point in f'(t) (for example, each f'(t) includes 1024 sampling points), and O(t) is the sum of the energy values ​​of all sampling points in f'(t).

[0112] according to Figure 8 Multiple proxy audio frames in a time segment are shown, where the proxy audio frame f'(t) represents the tth proxy audio frame. The onset feature value (onset value O(t)) of each proxy audio frame f'(t) can be calculated using Formula 4, see Formula 4. Figure 8 It shows that multiple proxy audio frames are covered in the past u seconds, and this time interval includes a first group of data 81: f'(t-0), f'(t-1),…, f'(t-(u'-1)), and a second group of data 82: f'(tz), f'((t-1)-z),…, f'((t-(u'-1))-z), separated by interval z.

[0113] For example, if the beat period τ=30, it means that a beat point will appear every 30 proxy audio frames f'(t), and the value z is between τ-20 and τ+20. Taking τ=30 as an example, τ-20,…,τ+20=10,11,…,50. Where u' is the amount of data in u seconds.

[0114] Thus, when estimating a new beat cycle, the onset feature value (O(t)) of the proxy audio frame f'(t) is calculated. This is a concept of energy envelope. It is expected that each beat point should have a stronger or strongest energy. Therefore, when estimating the beat cycle, the maximum value of the onset feature value (or the value calculated between the related onset feature values) will be determined, that is, the beat cycle is estimated using the onset feature value of the proxy audio frame within a period of time. If the interval z of the maximum value of the autocorrelation function (ACF) is the correct beat cycle, Figure 8 The multiplication of the onset point eigenvalues ​​on the beats separated by z between the first group of data 81 and the second group of data 82 (as shown in the figure O(t)O(tz), O(t-1)O((t-1)-z), O(t-(u'-1))O((t-(u'-1))-z)) will be a very large value. If this z value can be added to these multiplied onset point eigenvalues ​​and then added together (O(t)O(tz)+O(t-1)O((t-1)-z)+…+O(t-(u'-1))O((t-(u'-1))-z)), it will be the maximum. This z value is the latest estimated value of the beat period (τ). Therefore, the new beat period can be calculated through the value of specific operations (such as multiplication and addition) between the onset point eigenvalues ​​in the autocorrelation function.

[0115] It should be mentioned here that in addition to the above method using the autocorrelation function, the following method can also be used, and the actual implementation is not limited to a specific method. For example, there is an energy difference method that examines the energy envelope difference value between two adjacent audio frames or proxy audio frames (f'(t)). The larger the difference value, the greater the energy change between the two audio frames, which can be used to determine the onset value.

[0116] There is also a spectral difference method, which first performs a Fourier transform on the audio frame or the proxy audio frame to obtain the energy difference of a specific frequency band. The larger the energy difference, the greater the energy change in this frequency band, which can be used to determine the onset feature value. For example, the energy change can be determined based on the spectrum distribution of the drum sound responsible for the rhythm in the music, and the beat point position can be found based on the onset feature value.

[0117] There is also a phase deviation method, which is a method for detecting the onset feature value in the audio signal based on the phase. Another onset feature value detection technology called complex spectral difference can also be used to track the beat point position in the music.

[0118] When the sum of the energy values ​​of all sampling points in a specific proxy audio frame (f'(t)) is calculated, the onset feature value (O(t)) in the audio signal is obtained. The larger the onset feature value, the more onset features there are, and the beat position can be determined accordingly. However, in order to determine the beat position more accurately, the beat detection value (beat detection value, C(t)) can be further calculated (step S705).

[0119] According to the above embodiment, the autocorrelation function (ACF) uses the onset feature value of each audio frame as the feature of autocorrelation, so the beat period (τ) can be derived from it. However, further, according to the sensitivity of ordinary people to rhythm, it is necessary to confirm the beat point position more accurately. Therefore, in order to ensure that the obtained beat period is correct, it is hoped that the beat period is not only calculated based on the onset feature value, that is, whether the period in the past period of time also has the same characteristics. According to the embodiment, a beat point detection value (C(t)) is introduced, that is, the sampling points near the onset feature value (O(t)) are obtained by using the defined beat point detection value.

[0120] The beat point detection value (C(t)) is defined as in Formula 5.

[0121] Formula 5:

[0122] C(t)=(1-α)O(t)+α*Max(C(t+v)*W(v));

[0123] in,

[0124]

[0125] Wherein, τ is the beat period, and the value α is the weight value. The system proposed by the present invention will give α as a fixed initial value, and use this value α to determine the weight of the operation between the feature value of the onset point and the beat point detection value (C(t+v)) in the past period of time. The earlier the beat point detection value, the lower the weight. W (log-Gaussian transition weighting function) is used as the coefficient. The reference is "'Real-Time Beat-Synchronous Analysis of Musical Audio'AM Stark, MEP Davies and MD Plumbley. In Proceedings of the 12th International Conference on Digital Audio Effects (DAFx-09) Como, Italy, September 1-4, 2009". In addition, v is defined as how long ago the beat point detection value should be taken into consideration. This example shows v = -(2τ),…,-(τ / 2).

[0126] According to Formula 5, when 0<=t<=2τ, C(t) will have a non-existent C(t+v) value, and will be replaced by 0. Taking C(0) as an example, since C(-2τ) to C(-τ / 2) do not exist, Max(C(t+v)) is actually 0, that is, C(0)=(1-α)O(t). The meaning at this time is to only refer to the onset feature value of the proxy audio frame (f'(0)) at t=0.

[0127] Fig. 9 An embodiment diagram of describing the beat point position with the beat point detection value is shown, thereby describing the beat point detection value (C(t)) of the current beat point 901 to be calculated in Formula 5, and the beat point detection values ​​of the two reference beat points 903 and 905 represented by "α*Max(C(t+v)*W(v))" in Formula 5: C(t-(τ / 2)) and C(t-2τ).

[0128] As shown in Formula 5, the maximum value (Max) of the beat point detection value is found from several past beat cycles (τ). According to the example shown in the figure, the maximum value of the beat point detection values ​​of the two reference beat points 903 and 905 in the figure is multiplied by the coefficient W(v), which is used as the beat point detection value (C(t)) of the current beat point 901 (step S705).

[0129] After the above steps, the onset feature value (O(t)) of each proxy audio frame is calculated according to Formula 5, and the maximum beat point detection value (Max(C(t+v)*W(v))) of the sampling points near the onset feature value is calculated. The two are added after the weight value α to serve as the beat point detection value of this proxy audio frame. The larger the value, the more likely this audio frame is to be a beat point.

[0130] Next Figure 7 Step S707 of estimating the beat location in the process. According to one embodiment, in order to reduce the amount of calculation, it is determined that the next beat location is estimated only when the next proxy audio frame (f'(t)) is estimated to be at the position of half (τ / 2) of the beat period, rather than calculating for each proxy audio frame (f'(t)).

[0131] When estimating the next beat point position, the beat point detection value of the next beat cycle is first estimated based on all beat point detection values ​​of the beat cycle in the past period of time, and reference may be made to Formula 6.

[0132] Formula 6:

[0133] C(t+x)=Max(C(t+x+v)*W(v));

[0134] Among them, x=1,…,τ; v=-(2τ),…,-(τ / 2);

[0135]

[0136] The beat point detection value C(t+x) refers to the beat point detection value of the proxy audio frame (f'(t+x)) at each time point t+x within the next beat cycle (x=1,…,τ) of time point t, and then the beat cycle with the maximum beat point detection value (C(t+x)) is taken out (represented by x', time is t'), which is used as the position of the beat point in the next beat cycle, that is, it is estimated that the next beat point appears in a certain proxy audio frame (f'(t')).

[0137] For reference Fig.10 The following is an example of using the beat point detection value to describe the next beat point position estimation. According to Formula 6, x = 1 and τ and v = -(2τ) and -(τ / 2) are substituted into the formula to form Fig.10The multiple beat point detection values ​​indicated in FIG. include the beat point detection value C(t) of the current beat point 1001, and the beat point detection values ​​of the next beat cycle 1003: C(t+τ) to C(t+1), and include the beat point detection values ​​of the reference beat cycle 1005 for obtaining the maximum value in formula 6: C(t+τ-(τ / 2)) to C(t+τ-2τ), and the beat point detection values ​​of the reference beat cycle 1007: C(t+1-(τ / 2)) to C(t+1-2τ).

[0138] During the calculation, the maximum value (Max) in the reference beat cycles 1005 and 1007 is multiplied by the coefficient W(v) and used as the beat point detection value in the next beat cycle 1003. According to the example shown in the figure, the audio frame where the maximum value of the beat point detection value in the next beat cycle 1003 appears is inferred as the position where the next beat point appears. For example, the maximum value of the beat point detection values ​​(C(t+τ-3), C(t+1),…, C,(t+τ)) in the next beat cycle 1003 shown in the figure, if C(t') is = C(t+τ-3), then the proxy audio frame (f'(t+τ-3)) is inferred to be the audio frame where the next beat point appears. At this point, the step of inferring the position of the next beat point is completed ( Figure 2 Step S213: Figure 7 Step S707), in this way, a beat prompt sound can be added to the audio signal according to the inferred beat point position and then output.

[0139] However, in the system for real-time analysis of music rhythm, it is still necessary to avoid the problem that the music rhythm may change at any time and the beat prompt sound interferes with the music. Therefore, the beat stability judgment is also performed in the method for real-time analysis of music rhythm ( Figure 2 Step S215).

[0140] According to the embodiment, the system first defines what a stable rhythm is, please refer to Fig.11 The diagram shown is a diagram of an audio frame data embodiment for describing a method for determining whether the rhythm in an audio signal is stable.

[0141] According to the audio frame data schematically shown in the figure, if the current audio frame (f(t)) is judged as a beat point, the new beat cycle is re-estimated using the autocorrelation function based on the energy record of the onset point in the past n seconds. The system defines a mechanism to determine whether the rhythm is stable. If it is stable, the current audio frame (f(t)) is the beat point position, called the rhythm audio frame (beatframe), that is, the beat prompt sound is added. The method for determining whether the rhythm is stable can refer to Formula 7.

[0142] The figure schematically shows an audio frame (f(t)) 1101 and an audio frame (f(t+1)) 1102 at two consecutive time points. After calculating the audio frame jump step (h(n)) 1105 and the audio frame jump step (h(n+1)) 1106, a proxy audio frame (f'(n)) 1103 and a proxy audio frame (f'(n+1)) 1104 are obtained. Assume that f'(t0'), f'(t1'), f'(t2'), ..., f'(tn') are rhythmic audio frames obtained through speculation. When the next beat point position of the new beat cycle is obtained, the beat point positions are traced back multiple times and multiple time differences (d values) between adjacent beat point positions are calculated to determine whether the audio signal has a stable rhythm.

[0143] If we push back the d value (time difference) of i times (d n-0 ,d n-1 ,…,d n-i ) are the same, indicating that the predicted beat point intervals are the same each time, and the rhythm is considered stable, and a beat prompt tone can be added at each beat point position. As shown in Formula 7.

[0144] Formula 7:

[0145] d n-0 =t n-0’ -t n-1’ ;

[0146] d n-1 =t n-1’ -t n-2’ ;

[0147] …

[0148] d n-i =t n-i-1’ -t n-i’ .

[0149] When the method is applied in a specific device, after determining that the received data is an audio signal with a stable rhythm, the beat prompt sound function can be enabled, wherein the beat prompt sound can be added in a normal, fast or slow mode. For example, taking Formula 7 as an example, the beat prompt sound can be added to the rhythm audio frame determined to be the beat point position, such as the rhythm audio frame of one beat per minute (BPM) (normal mode): f'(t n’ ),f'(t n’ +d); rhythmic audio frame with twice the BPM (Allegro mode): f'(t n’ ),f'(t n’ +d / 2); or a rhythmic audio frame with half the beats per minute (slow mode): f'(t n’ ),f'(t n’ +2d).

[0150] Furthermore, an intelligent model for learning various rhythms can be introduced so that the system equipped with the intelligent model can accurately infer the beat cycle, thereby achieving the purpose of real-time analysis of music rhythm and accurate addition of beat prompt sounds.

[0151] To summarize, according to the method and system for real-time analysis of music rhythm described in the above embodiments, the beat period is determined by analyzing the audio frame data in the audio signal, and then the beat point position is determined. A beat prompt sound can be added under a stable rhythm, thereby realizing real-time analysis of the rhythm in the music and the purpose of adding a beat prompt sound in real time to the on-demand audio signal.

[0152] The contents disclosed above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent technical changes made using the contents of the present invention's specification and drawings are within the protection scope of the present invention.

Claims

1. A method for real-time analysis of music rhythm, applied in a system, comprising: receiving an audio signal, and obtaining audio frame data including a sampling rate after decoding the audio signal; deriving an audio frame skipping step according to an audio frame size and an overlapping audio frame size, and calculating an audio frame rate according to the sampling rate and the audio frame skipping step; Calculating an initial value of a beat cycle according to the sampling rate and an initial value of the number of beats per minute, so as to obtain a beat point position according to the number of sampling points covered by one beat; and A recursive algorithm is used to estimate a beat point position, wherein the number of audio frames covered in the past period of time is calculated according to the audio frame rate, a new beat cycle is calculated according to the number of beats per minute, and then the next beat point position is estimated according to the new beat cycle; The system is configured to recalculate the new beat cycle based on the audio frame data of the past period of time at regular intervals to recalculate the next beat point position.

2. The method for real-time analysis of music rhythm according to claim 1, characterized in that: When the audio signal is a multi-channel audio signal, an average operation is performed on the digital signals of multiple audio frames at the same time to form a monophonic audio signal for real-time analysis of the music rhythm.

3. The method for real-time analysis of music rhythm according to claim 1, characterized in that: The audio frame rate indicates the number of proxy audio frames in one second, wherein the proxy audio frame is derived based on the audio frame size of an original audio frame and the overlapping audio frame size set by the system.

4. The method for real-time analysis of music rhythm according to claim 3, characterized in that: In the method for calculating the new beat period, an autocorrelation function is used to find a repetitive pattern using the correlation of audio frame data at different time points in the audio signal, and then the new beat period is recalculated based on the interval of the maximum value of the autocorrelation function.

5. The method for real-time analysis of music rhythm according to claim 4, characterized in that: A starting point feature value is introduced into the autocorrelation function, that is, the starting point feature value of each proxy audio frame in a plurality of proxy audio frames in a time segment is calculated, and the maximum value of a calculated value of the starting point feature value or a related starting point feature value in the time segment is determined to deduce the new beat cycle.

6. The method for real-time analysis of music rhythm according to claim 5, characterized in that: A beat point detection value is introduced, and a maximum beat point feature value among the sampling points near the onset point feature value of each proxy audio frame in the time segment is calculated as the beat point detection value of the proxy audio frame; then the beat point detection value of the next beat cycle is inferred based on all the beat point detection values ​​of the beat cycle in the past period of time, and the beat point detection value with the largest value in the next beat cycle is taken as the beat point position of the next beat cycle.

7. The method for real-time analysis of music rhythm according to any one of claims 1 to 6, characterized in that: When the next beat point position of the new beat cycle is obtained, the beat point positions are traced back multiple times and multiple time differences of adjacent beat point positions are calculated to determine whether the audio signal has a stable rhythm; When it is determined that the audio signal has a stable rhythm, a beat prompt sound is added at each beat point.

8. A system for real-time analysis of music rhythm, provided in a device, comprising: An audio signal analysis module is connected to an audio signal processing circuit in the device, wherein a method for real-time analysis of music rhythm is performed, including: Receiving an audio signal through an input interface, wherein the audio signal is decoded by the audio signal processing circuit to obtain audio frame data including a sampling rate; deriving an audio frame skipping step according to an audio frame size and an overlapping audio frame size, and calculating an audio frame rate according to the sampling rate and the audio frame skipping step; Calculating an initial value of a beat cycle according to the sampling rate and an initial value of the number of beats per minute, so as to obtain a beat point position according to the number of sampling points covered by one beat; and A recursive algorithm is used to estimate a beat point position, wherein the number of audio frames covered in the past period of time is calculated according to the audio frame rate, a new beat cycle is calculated according to the number of beats per minute, and then the next beat point position is estimated according to the new beat cycle; The system is configured to recalculate the new beat cycle based on the audio frame data of the past period of time at regular intervals to recalculate the next beat point position.

9. The system for real-time analysis of music rhythm according to claim 8, characterized in that: In the method for calculating the new beat period, an autocorrelation function is used to find a repetitive pattern by utilizing the correlation of audio frame data at different time points in the audio signal, and then a starting point feature value is introduced into the autocorrelation function, that is, the starting point feature value of each proxy audio frame in a plurality of proxy audio frames in a time segment is calculated, and the maximum value of the starting point feature value or a calculated value related to the starting point feature value in the time segment is determined to deduce the new beat period.

10. The system for real-time analysis of music rhythm according to claim 8 or 9, characterized in that: When the next beat point position of the new beat cycle is obtained, the beat point positions are traced back multiple times and multiple time differences between adjacent beat point positions are calculated to determine whether the audio signal has a stable rhythm.

Images