Audio visualization control method and device, equipment and medium

Abstract

The invention relates to the field of network live broadcast, and discloses an audio visualization control method and device, equipment and a medium, and the method comprises the steps: tracking and determining the spectrum information of each audio frame in an audio stream of a live broadcast room in real time, and determining the energy change rate of each audio frame; corresponding to each current audio frame, determining whether the current audio frame has candidate beat points or not based on the energy change rate of the current audio frame; for each current audio frame, determining an ideal beat point based on an energy change sequence formed by the energy change rates of all the current and previous audio frames; and judging whether the candidate beat point belongs to an actual beat point or not by using the ideal beat point, so that the actual beat point is used for driving the live broadcast room to play the audio visual special effect. By analyzing the energy change rate of the audio stream in real time, accurate beat point detection is realized, an audio visualization effect can be presented more accurately, and the interactive experience and immersion of a user in a live broadcast environment are effectively improved.

Description

Technical Field

[0001] The present application relates to network live broadcast technology, and in particular to an audio visualization control method and its device, equipment, and medium. Background Art

[0002] With the rapid development of Internet technology, live streaming has become an indispensable part of people's daily lives. In the live streaming process, in order to enhance the user's interactive experience and immersion, the performance of audio effects has become particularly important. Especially for music live streaming, audio visualization effects can transform invisible sounds into intuitive visual elements, greatly enriching the user's viewing experience.

[0003] Although existing audio visualization techniques have achieved the visualization of audio signals to a certain extent, they still have some defects in practical applications.

[0004] First, existing audio visualization technologies often lack sufficient flexibility and adaptability when dealing with complex music styles and audio signals. For example, when the music style changes frequently or the audio signal strength fluctuates greatly, existing technologies are difficult to accurately capture and reflect the subtle changes in the audio signal, resulting in a disconnect between the visualization effect and the music rhythm.

[0005] Secondly, most existing audio visualization technologies rely on simple spectrum analysis, which can only provide single-dimensional information of audio signals, such as frequency distribution. This single dimension limits the richness and expressiveness of the visualization effect, making it difficult to meet users' needs for more dynamic and diverse visual effects.

[0006] Furthermore, existing audio visualization technology may be implemented using time domain analysis technology, which also has certain limitations when dealing with background noise. In a noisy live broadcast environment, background noise may interfere with the analysis and processing of audio signals, resulting in inaccurate audio rhythm and negatively affecting the visualization effect.

[0007] In addition, when implementing audio visualization, existing technologies may apply more complex deep learning algorithms to determine the rhythm of music, which requires higher computing resources and processing time, which may affect the system's response speed and user experience in real-time live broadcast scenarios.

[0008] It can be seen that although the existing audio visualization technologies have achieved the visualization of audio signals to a certain extent, they still need to be improved in adapting to complex audio environments and improving the richness and accuracy of visualization effects. Summary of the invention

[0009] The purpose of the present application is to solve the above-mentioned problems and to provide an audio visualization control method and its corresponding device, equipment, and non-volatile readable storage medium.

[0010] According to one aspect of the present application, there is provided an audio visualization control method, comprising the following steps: real-time tracking and determining the spectrum information of each audio frame in the audio stream of a live broadcast room, and determining the energy change rate of each audio frame; corresponding to each current audio frame, determining whether there is a candidate beat point in the current audio frame based on the energy change rate of the current audio frame; corresponding to each current audio frame, determining an ideal beat point based on an energy change sequence composed of the energy change rates of all current and previous audio frames; using the ideal beat point to determine whether the candidate beat point belongs to an actual beat point, so as to use the actual beat point to drive the live broadcast room to play audio visualization special effects.

[0011] According to another aspect of the present application, an audio visualization control device is provided, including: a spectrum analysis module, configured to track and determine in real time the spectrum information of each audio frame in the audio stream of a live broadcast room, and determine the energy change rate of each audio frame; a single frame analysis module, configured to correspond to each current audio frame, and determine whether there is a candidate beat point in the current audio frame based on the energy change rate of the current audio frame; a sequence analysis module, configured to correspond to each current audio frame, and determine an ideal beat point based on an energy change sequence composed of the energy change rates of all current and previous audio frames; a beat determination module, configured to use the ideal beat point to determine whether the candidate beat point belongs to an actual beat point, so as to use the actual beat point to drive the live broadcast room to play audio visualization special effects.

[0012] According to another aspect of the present application, an audio visualization control device is provided, comprising a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the audio visualization control method described in the present application.

[0013] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the audio visualization control method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

[0014] The technical solution of the present application provides significant advantages for audio visualization in live broadcast scenarios. By analyzing the spectrum information and energy change rate of the audio stream in real time, it can accurately capture the detailed changes in the music rhythm and ensure that the visualization effect is synchronized with the music rhythm. The solution improves the accuracy of beat detection and the dynamics of audio visualization effects by dynamically adjusting the determination of beat points to adapt to the real-time changes of audio signals. At the same time, the intelligent decision-making mechanism enhances the accuracy of beat point determination, reduces the interference of background noise, and ensures high-quality audio visualization effects even in noisy environments. In addition, the present application avoids relying on resource-intensive deep learning algorithms, reduces computing resource consumption and processing time, and enables live viewers to instantly enjoy accurate audio visualization effects, improving the interactivity and viewing experience of live broadcasts. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 An exemplary network architecture of the present application that is suitable for applying the audio visualization control method of the present application;

[0016] Figure 2 A flowchart of an embodiment of the audio visualization control method of the present application;

[0017] Figure 3 This is a principle block diagram of the audio visualization control device of the present application;

[0018] Figure 4 This is a structural schematic diagram of an audio visualization control device used in this application. DETAILED DESCRIPTION

[0019] To facilitate understanding of the various embodiments of the audio visualization control method of the present application, an exemplary network architecture is first introduced. Figure 1 As shown, the exemplary network architecture includes multiple computer devices, including multiple servers and multiple terminal devices. Through the interaction between these devices, the network live broadcast service can be ensured, and information transmission can be achieved by transmitting live streams, specifically including image streams and audio streams. For example, a live server 83 and a media server 85 are deployed in the network architecture, and at the same time, the terminal device 90 of the anchor user and the terminal device 92 of the audience user are allowed to access the network architecture to communicate with each server.

[0020] The media server 85 is responsible for processing the media data of the live broadcast room of the network live broadcast, mainly processing the live stream. In one embodiment, the media server 85 receives the live stream generated by the anchor user, decodes the live stream, and implements various technical processing in the decoding space, including mixing, editing, and analyzing and processing the audio stream in the live stream according to the audio visualization control method of the present application to realize the control of audio visualization special effects, etc. Then, the media server 85 corresponds to each terminal device 90, 92 in the live broadcast room, and encodes and transmits the processed live stream to each audience user for playback.

[0021] In another embodiment, the host user's terminal device 90 can analyze and process the audio stream in the live stream generated in the terminal device according to the audio visualization control method of the present application to realize the control of the audio visualization special effects. The various data obtained by the host user's terminal device 90 through analysis and processing can be constructed as control parameters, which are broadcast to the audience user's terminal device 92 through the live broadcast room, and then the terminal device 92 controls the presentation of the audio visualization special effects.

[0022] In another embodiment, the terminal device 92 of the audience user can also execute the audio visualization control method of the present application, analyze and process the audio stream in the live stream, obtain corresponding control parameters, and locally control the presentation of corresponding audio visualization effects.

[0023] The live broadcast server 83 is a bridge that maintains the operation of the live broadcast room to connect the anchor user and the audience user. To this end, the live broadcast server 83 is not only responsible for providing the users in the live broadcast room with the address for pulling the live broadcast stream from the media server 85, but also handles interactive functions related to the live broadcast, such as barrage, gifts and comments, thereby enhancing the interactivity of the live broadcast and the audience's sense of participation.

[0024] In such a network architecture, the computer program product that implements the audio visualization control method of the present application can be flexibly deployed in different network nodes. For anchor users and audience users, the program product can be integrated into the network live broadcast program and deployed in their terminal devices 90, 92, such as smart phones, tablet computers, personal computers or game consoles. On the terminal devices 90, 92, the program can analyze and process the audio stream in real time to obtain each actual beat point for controlling the audio visualization special effects, which can reflect the significant advantages in computing efficiency.

[0025] In addition, the computer program product can also be directly deployed in the media server 85 for centralized processing of audio streams in live streams. On the media server 85, the program can simultaneously process a large number of live streams from different anchors, determine the rhythm information and energy information in the audio stream in the live stream of each anchor user, and send it to the terminal device 92 of each audience user in the corresponding live room for controlling the corresponding audio visualization special effects.

[0026] From the above description of the network architecture, it can be seen that the method of the present application can be implemented on the terminal devices 90, 92 or the media server 85, providing a high degree of flexibility and a wide range of application potentials.

[0027] See also Figure 2 According to an audio visualization control method provided by the present application, it can be implemented as a computer program product, installed and run in a terminal device or a server, and the audio stream in the live stream is analyzed in real time and the display of the audio visualization special effects is controlled according to the analysis results, so as to express the rhythm information and / or energy information of the audio through the audio visualization special effects. In some embodiments thereof, the following steps are included:

[0028] Step S3100: Track and determine the spectrum information of each audio frame in the audio stream of the live broadcast room in real time, and determine the energy change rate of each audio frame;

[0029] A live stream is a data stream consisting of continuous audio and video data. Usually, in a live broadcast environment, it is generated in real time by the host end where the host user is located and transmitted to the server, and then distributed by the server to the audience end where the audience user is located. As a component of the live stream, the audio stream contains the audio data during the live broadcast process. In a voice room where communication is based only on voice, the live stream can only contain the audio stream.

[0030] After obtaining the audio stream, it is necessary to analyze the audio stream in real time to extract the spectrum information. The spectrum information reflects the energy distribution of the audio signal at different frequencies. This process can be achieved through time-frequency transformation, which is an analysis method that can convert the time domain signal into a spectrum representation that represents the change of the signal frequency over time. For example, after framing the audio stream, each audio frame can be processed using fast Fourier transform (FFT) or other time-frequency analysis techniques to achieve this step, thereby obtaining the spectrum information of each audio frame.

[0031] In this embodiment, the spectrum information and energy change rate of each audio frame in the audio stream of the live broadcast room are tracked in real time by continuously receiving the audio stream during the live broadcast and processing it at predetermined time intervals. This process can be implemented with the help of a buffer, which continuously updates the data of the audio stream at fixed time intervals. Whenever new audio data flows in and meets the frame length requirement of an audio frame, a time-frequency transform, such as a fast Fourier transform (FFT), is performed on it to obtain the spectrum information of continuous audio frames.

[0032] As the live broadcast progresses, the present application continuously uses the newly generated audio frame as the current audio frame, and repeats this analysis process for the current audio frame. For each audio frame, its spectrum information is extracted, mainly the spectrum information corresponding to the high-frequency content (High Frequency Content, HFC) therein, and the ratio between the energy change rate of the current frame and the historical high-frequency energy level value reflecting the historical overall energy change rate, i.e., hfcRatio, is calculated. The historical high-frequency energy level value can be calculated by a continuously updated sliding average or exponential moving average to reflect the recent audio energy change trend in real time. In this way, the energy changes of each audio frame can be tracked in real time to determine the energy change rate, and quickly respond to any significant changes in the audio stream, providing a basis for accurate beat point detection and real-time generation of audio visualization effects.

[0033] The energy change rate is used to characterize the degree of energy change of the sum of the energy of the high-frequency content part in the audio spectrum of an audio frame relative to the historical high-frequency energy level of the corresponding part of the previous audio frame. It is particularly sensitive to sudden changes or percussion points in the audio signal and is very suitable for beat detection.

[0034] In order to calculate the energy change rate hfcRatio corresponding to the current audio frame, it is first necessary to extract the energy value of the high-frequency content part, that is, the high-frequency energy value, from the spectrum information of the current audio frame. In specific implementation, the high-frequency energy value of the current audio frame can be obtained by selecting the frequency points within a specific high-frequency range in the spectrum information and summing up the energy values ​​of these frequency points. Then, the historical high-frequency energy level value calculated in advance corresponding to the previous audio frame is obtained, and the weighted value is fused with the high-frequency energy value of the current audio frame to obtain the historical high-frequency energy level value corresponding to the current audio frame. Among them, starting from the first audio frame, the determination of the historical high-frequency energy level value corresponding to each audio frame is iteratively calculated in the same way, so the historical high-frequency energy level value is the weighted summation result of the high-frequency energy values ​​of the current audio frame and all the previous historical audio frames. It can respond to rapid changes in the audio signal more sensitively by tracking the high-frequency energy value of each audio frame for iterative update, while reducing the misjudgment caused by short-term fluctuations of the audio signal, and can provide accurate tracking of audio signal changes without sacrificing real-time performance.

[0035] Finally, the high-frequency energy value of the current audio frame is compared with the historical high-frequency energy level value corresponding to the current audio frame, which is determined by integrating the high-frequency energy values ​​of the current audio frame and all its previous audio frames, and the energy change rate hfcRatio corresponding to the current audio frame is calculated. This ratio reflects the degree of energy change of the current audio frame relative to the historical high-frequency energy level. If hfcRatio exceeds a preset threshold, this usually indicates that there is a significant energy change in the audio signal, which may correspond to a candidate beat point.

[0036] In this embodiment, a variety of strategies can be used to update the historical high-frequency energy level value in real time. In one embodiment, an exponential moving average (EMA) method is used, which gives a greater weight to the historical high-frequency energy level value corresponding to the previous audio frame and a relatively lower weight to the high-frequency energy value of the current audio frame, thereby fusing the historical high-frequency energy level value corresponding to the current audio frame, so that the historical high-frequency energy level value is gradually adjusted as new data is added. This method is very suitable for tracking dynamic changes in data while reducing the impact of short-term fluctuations on the results, which helps to improve the accuracy and stability of hfcRatio, thereby improving the performance of the entire audio visualization control method, including beat detection, audio visualization, and the overall user experience.

[0037] In addition to the exponential moving average method, in another embodiment, the historical high-frequency energy level value can also be updated by a sliding average method. The sliding average method is used to calculate the average value of a set of continuous observations, which usually includes several recent audio frames. For example, the high-frequency energy values ​​of the most recent 100 frames corresponding to the current audio frame can be selected, and their arithmetic mean can be calculated as the historical high-frequency energy level value corresponding to the current audio frame. This method achieves smoothing of the historical high-frequency energy level value by reducing the influence of extreme values ​​and random noise.

[0038] Step S3200: corresponding to each current audio frame, determining whether there is a candidate beat point in the current audio frame based on the energy change rate of the current audio frame;

[0039] Significant energy changes in audio signals are usually associated with beat points in music and are the basis for music beat analysis. The energy change rate of an audio frame reflects the degree of change in the overall high-frequency energy level of the audio frame relative to all other previous audio frames. Therefore, the energy change rate of the current audio frame can be used to determine whether there are candidate beat points in the current audio frame.

[0040] Specifically, the energy change rate of the current audio frame, i.e., hfcRatio, is used to determine whether there is a candidate beat point. hfcRatio is the ratio of the high frequency content (HFC) of the current audio frame to the historical high frequency energy level (HFC). avg). HFC refers to the sum of the energy of the high frequency part in the current audio frame, and HFC avg It is obtained by fusing the past HFC values.

[0041] In the judgment process, the energy change rate hfcRatio of the current audio frame is compared with the preset hit point threshold to determine whether there is a candidate beat point in the current audio frame. If hfcRatio exceeds this threshold, it indicates that the energy change of the current audio frame is significant, so it is considered that there is a candidate beat point, otherwise it is determined that there is no candidate beat point in the current audio frame. At this time, the starting time sequence position of the current audio frame can be recorded as the time sequence position of the candidate beat point.

[0042] The strike point threshold is used to distinguish whether the energy change in the audio frame is significant enough to constitute a beat point. In one embodiment, this threshold can be an empirical threshold, which can be preset. In another embodiment, the strike point threshold may not be fixed, but dynamically adjusted to adapt to different audio content and environments. Specifically, the adjustment of the strike point threshold can be based on the historical data of hfcRatio. If the current hfcRatio exceeds the current strike point threshold, and the amplitude of this excess is significant, the strike point threshold can be increased accordingly to avoid too many misjudgments when the audio content changes. On the contrary, if hfcRatio does not exceed the threshold, the threshold may be reduced to capture more subtle changes.

[0043] For example, if a song transitions from a quiet intro to a loud chorus, the energy of the audio signal will suddenly increase. By dynamically adjusting the strike point threshold, it is possible to keep the sensitivity low during the intro and increase the sensitivity during the chorus, accurately capturing the beginning of the music.

[0044] In addition, in order to adapt to different music styles and the complexity of audio signals, a variety of specific embodiments may also be included. For example, the adjustment strategy of the strike point threshold may also be customized according to the actual application scenario, such as adopting a more conservative threshold adjustment strategy in a noisy live broadcast environment. Specifically, first, the hfcRatio distribution of the audio frame over a period of time is monitored to determine an initial threshold. Then, the change of hfcRatio is continuously tracked. If it is detected that the hfcRatio of multiple consecutive audio frames exceeds the initial threshold, the strike point threshold is moderately increased, so as to avoid erroneously increasing the threshold due to occasional noise. At the same time, a maximum threshold upper limit may be preset to prevent the strike point threshold from being too high and causing important strike points to be missed. In addition, the threshold adjustment speed may be dynamically adjusted according to the noise level of the live broadcast environment. The higher the noise level, the slower the threshold adjustment, so as to better adapt to environmental changes and reduce misjudgment. Through this customized threshold adjustment strategy, even in a noisy live broadcast environment, the real strike point in the audio stream can be effectively identified, thereby determining the candidate beat point.

[0045] It is not difficult to understand that after the corresponding energy change rate of each audio frame is determined, it will be processed as the current audio frame in this step, thereby iteratively corresponding to each audio frame to determine whether there is a candidate beat point.

[0046] Step S3300: corresponding to each current audio frame, determining an ideal beat point based on an energy change sequence formed by energy change rates of all current and previous audio frames;

[0047] The ideal beat point refers to the beat point that is theoretically most consistent with the rhythm of music, determined based on the energy change rate of the audio signal during the audio analysis process. It is determined by calculating and analyzing the energy change sequence of multiple consecutive audio frames in the audio stream, finding the time granularity with the largest total energy change, and thus determining the beat period and starting phase. The determination of the ideal beat point reflects the regularity of the music rhythm and is used to determine whether the candidate beat in the corresponding audio frame belongs to the actual beat point. Accordingly, in order to more accurately reflect the rhythm changes of the audio signal, it is necessary to determine the ideal beat point based on the energy change sequence composed of the energy change rates of all current and previous audio frames.

[0048] First, the energy change rate of the current audio frame is added to a cached energy change sequence. This sequence contains the energy change rates of all audio frames from the beginning of the audio stream to the present. Then, this energy change sequence is analyzed, and the possible beat period and initial phase are found in the analysis process. In this analysis process, the sum of energy changes at different time granularities can be counted, where the time granularity is characterized by different combinations of beat periods and deviation phases.

[0049] Specifically, in order to find the ideal beat point, based on the energy change sequence corresponding to the current audio frame and the audio frame before it, the sum of energy changes under multiple preset possible beat cycles and phases is calculated. To this end, in one embodiment, a range of a beat cycle is pre-set, for example, 30 to 200 beats per minute (BPM 30-200). Within this range, multiple beat cycle values ​​and multiple deviation phase values ​​are tested, and a corresponding energy change sum is calculated for each time granularity composed of a beat cycle value and a deviation phase value. This sum is obtained by multiplying the energy change rate of each audio frame by a cosine function (the period and phase of the function match the tested beat cycle value and the deviation phase value) and summing them.

[0050] According to the above processing method, a sum of energy changes can be obtained for each possible beat cycle and deviation phase, so the beat cycle and deviation phase with the largest sum of energy changes can be selected as the ideal beat cycle and ideal deviation phase. This selection is based on the assumption that the most significant beat point in the audio signal will cause the largest sum of energy changes in the energy change sequence.

[0051] For example, if you need to analyze a piece of electronic music and find that the total energy change is the largest at 120 beats per minute (ideal beat cycle) and at the 32nd sample point from the beginning of the audio (ideal deviation phase), then you can determine that the timing position corresponding to this beat cycle and deviation phase is the ideal beat point.

[0052] It is not difficult to understand that after the corresponding energy change rate of each audio frame is determined, it will be processed as the current audio frame in this step, thereby iteratively determining the corresponding ideal beat point for each audio frame.

[0053] Step S3400: Use the ideal beat point to determine whether the candidate beat point belongs to the actual beat point, so as to use the actual beat point to drive the live broadcast room to play audio visualization effects.

[0054] After determining the ideal beat point, the ideal beat point can be used to determine whether the candidate beat point belongs to the actual beat point, thereby achieving accurate beat detection and ensuring that only candidate beat points that conform to the rhythm of the music are confirmed as actual beat points, thereby driving the accurate performance of the audio visualization effect.

[0055] In the specific implementation process, the temporal distance between the ideal beat point and the candidate beat point is first calculated. The ideal beat point is calculated based on the energy change rate of all previous audio frames in the audio stream, which represents the beat position that theoretically best matches the music rhythm. The candidate beat point is determined based on the energy change rate of a single audio frame compared with the strike point threshold, indicating a possible beat point.

[0056] Next, a preset time tolerance window is obtained to determine whether the candidate beat point falls near the ideal beat point. If the temporal distance between the candidate beat point and the ideal beat point is less than or equal to the width of the time tolerance window, and no other actual beat points have been determined in the current beat cycle, the candidate beat point is confirmed as the actual beat point.

[0057] For example, assuming that the time tolerance window is set to 200 milliseconds, if at a certain moment, the timing distance between the calculated ideal beat point and the candidate beat point is 150 milliseconds, and no other candidate beat point has been confirmed as the actual beat point since the actual beat point was last determined, then the candidate beat point will be determined as the actual beat point.

[0058] In addition, in order to avoid identifying multiple actual beat points within an ideal beat cycle, in one embodiment, a search status flag can be maintained to record whether the actual beat point has been determined within the current beat cycle. If it has been determined, subsequent candidate beat points will not be considered until the next ideal beat point arrives. In this way, the actual beat point can be accurately identified from the audio stream, and the actual beat point can be used to drive the audio visualization effects of the live broadcast room. This not only improves the accuracy of beat detection, but also makes the audio visualization effect more vivid and synchronized, greatly improving the user's viewing experience.

[0059] When implementing this step, multiple specific embodiments can also be adopted. For example, the size of the time tolerance window can be dynamically adjusted according to the style of music and the speed of rhythm to adapt to different music types. In addition, the maintenance of the state identification can also adopt different data structures and algorithms to optimize performance and response speed. These embodiments make this step more flexible and efficient.

[0060] In one embodiment, by applying the same algorithm logic as in the embodiments of the present application, but shortening the beat period used to determine the time granularity to 1 / 2 of the original, the detection of the actual beat point corresponding to the half beat can be achieved. Therefore, the detection of the actual beat point of the full beat and / or half beat can be achieved in combination with the adjustment of the beat period, making the beat detection more comprehensive and accurate.

[0061] It should be pointed out that the frequency band division of the audio signal in this application corresponds to the audible range of the human ear and can be implemented flexibly, but the spectrum from 20Hz to 20kHz is usually divided into three main areas: low frequency, medium frequency and high frequency. The low frequency part starts at 20Hz and extends to about 250Hz, including extremely low frequency and low frequency. These frequency bands usually contain the basic rhythm and depth of music. The medium frequency part is roughly from 250Hz to 2kHz, covering medium-low frequency to medium-high frequency. It is the main frequency band of human voice and most musical instruments, responsible for the main information and clarity of the sound. The high frequency part starts from 2kHz to 20kHz, including high frequency and very high frequency, which is responsible for the detailed texture and clarity of the sound, such as the sound of cymbals and bells. Such frequency band division helps to accurately reflect the rhythm, harmony and details of music in audio visualization, thereby providing a rich and dynamic audio-visual experience.

[0062] According to the above embodiments, the technical solution of the present application shows significant technical advantages in live broadcast scenarios, especially in applications that require real-time presentation of audio visualization effects:

[0063] First of all, this application can accurately capture the detailed changes of audio signals by analyzing the spectrum information and energy change rate of each audio frame in the audio stream in real time, which is crucial for quickly responding to changes in music rhythm in live broadcast scenarios. This application not only focuses on the frequency distribution of audio signals, but also on energy changes, providing rich and multi-dimensional audio information, thereby showing higher flexibility and adaptability when processing various music styles and complex audio signals. This effectively avoids the problem of the visualization effect being out of sync with the music rhythm due to rapid changes in music style or fluctuations in audio signal strength, ensuring that even in a rapidly changing live broadcast environment, the visualization effect can be updated in real time to keep consistent with the music rhythm.

[0064] Secondly, the present application determines the ideal beat point by analyzing the energy change sequence composed of the energy change rate of the current and previous audio frames, and judges the actual beat point accordingly. This method can more accurately reflect the rhythm changes of the audio signal, especially in live broadcast scenarios, and can dynamically adjust the determination of the beat point to adapt to the real-time changes of the audio signal, thereby providing highly accurate and dynamic beat point detection. This greatly improves the real-time and dynamic nature of the audio visualization effect, allowing the visualization effects to closely follow the changes in the music rhythm, providing users with a smooth and immersive viewing experience.

[0065] Furthermore, the application makes intelligent decisions on candidate beat points through ideal beat points, achieving accurate beat point determination. This intelligent decision-making mechanism improves the accuracy of beat detection, while reducing the interference of background noise on audio signal analysis, and provides high-quality audio visualization even in a noisy live broadcast environment, ensuring that the audience can feel accurate beats and excellent visualization even in a complex sound background.

[0066] In addition, when implementing audio visualization, this application avoids relying on deep learning algorithms with high computational overhead, thereby reducing the demand for computing resources and processing time. This is particularly important in real-time live broadcast scenarios, because it enables the technical solution of this application to respond quickly, provide a smoother user experience, and solve the problem of system performance affected by computing resource and processing time limitations in the prior art. This efficiency improvement allows live viewers to enjoy accurate audio visualization effects without waiting, greatly improving the interactivity and viewing experience of live broadcasts.

[0067] Based on any embodiment of the method of the present application, the spectrum information of each audio frame in the audio stream of the live broadcast room is tracked and determined in real time, and the energy change rate of each audio frame is determined, including:

[0068] Step S3110: performing time-frequency transformation on each audio frame in the audio stream of the live broadcast room in real time to obtain spectrum information corresponding to each audio frame;

[0069] The audio stream of the live broadcast room is first divided into a series of continuous audio frames. Each frame is a short segment in the audio stream, and its length is usually in the millisecond level, for example, the frame length can be 10 to 20 milliseconds. This length should be short enough to capture the beats and rhythm changes in the music, but long enough to ensure the accuracy of the spectrum analysis.

[0070] Each audio frame after segmentation will be transformed in time and frequency. The most commonly used method is Fast Fourier Transform (FFT). FFT is an efficient frequency domain analysis algorithm that can convert time domain signals into frequency domain signals. Through FFT, the spectrogram of each audio frame can be obtained. The spectrogram provides spectrum information and shows the intensity of each frequency component in the frame, that is, the energy distribution.

[0071] In practical applications, in order to improve the efficiency and real-time performance of spectrum analysis, some optimized FFT algorithms can be used, such as the Overlap-Save method or the Overlap-Add method. These methods allow reducing the amount of calculation when processing consecutive audio frames and maintaining the continuity and stability of the spectrum.

[0072] In addition, in order to adapt to different audio content and environments, the parameters of the time-frequency transform, such as the window size and type of FFT, can be dynamically adjusted. For example, when the music rhythm is fast or the audio signal is more complex, a smaller window and a shorter frame length may be required to capture details; when the music rhythm is slow or the audio signal is relatively stable, a larger window and a longer frame length can be used to obtain a smoother spectrum.

[0073] Step S3120: Counting high-frequency energy values ​​corresponding to all frequency points of high-frequency content in the current audio frame based on the frequency spectrum information;

[0074] Since the energy value of the high-frequency content in the audio frame is closely related to the rhythm and hitting points of the music, the high-frequency energy values ​​corresponding to all frequency points of the high-frequency content in the current audio frame can be first counted based on the spectrum information.

[0075] The spectrum information obtained above shows the energy distribution of the audio signal at each frequency. This step focuses on the high-frequency content because the frequency of the high-frequency content contains a lot of rhythm information and the clarity of the music. High-frequency content refers to the higher frequency range in the spectrum, which is usually associated with elements such as human voice and percussion instruments.

[0076] In order to calculate the high-frequency energy value corresponding to the current audio frame, select the frequency points pre-defined as the high-frequency range in the spectrum information of the current audio frame. Add up the energy values ​​of these frequency points to obtain the high-frequency energy values ​​corresponding to all frequency points in the high-frequency content of the current audio frame. In specific implementation, the high-frequency energy value HFC of the current audio frame can be calculated by the following formula:

[0077]

[0078] Where S[i] represents the amplitude of the i-th frequency point, and n is the spectrum width of the high-frequency range. This formula calculates HFC by weighted summing the energy values ​​of each frequency point in the selected high-frequency range, where the index i of the frequency point is used as a weight to reflect the contribution of different frequency components to the rhythm information.

[0079] In practical applications, the selection of the high frequency range is determined based on the audio content and the level of detail required for analysis. For example, for pop music and electronic music, the high frequency range may start from a few thousand hertz, while for classical music, the high frequency range may be lower because the spectrum distribution of classical music may be wider. In one embodiment, a music classification model may be used to classify the audio frame sequence in the audio stream to determine its corresponding music type, and then the corresponding high frequency range is determined according to the music type.

[0080] The calculated high-frequency energy value of the current audio frame will be used for energy change rate analysis in the subsequent steps to determine the degree of energy change of the audio frame. By focusing on the change of high-frequency energy, the starting point and hitting point of the music rhythm can be captured more accurately, providing an important basis for achieving accurate beat detection and audio visualization effects.

[0081] Step S3130: weighted sum the high frequency energy value of the current audio frame and the historical high frequency energy level value corresponding to the previous audio frame, and iteratively determine the historical high frequency energy level value corresponding to the current audio frame;

[0082] In this step, the high frequency energy value HFC of the current audio frame is weighted and summed with the historical high frequency energy level value corresponding to the previous audio frame to iteratively determine the historical high frequency energy level value corresponding to the current audio frame. This step is to more smoothly track the high frequency energy changes of the audio signal and reduce the impact caused by transient noise or short-term signal fluctuations.

[0083] In this step, a method called exponential moving average (EMA) is used to calculate the historical high-frequency energy level value. This calculation method can quickly adapt to changes in the signal while reducing the impact of extreme values.

[0084] Specifically, the following formula is used to calculate the historical high frequency energy level value (HFC) of the current audio frame: avg ):

[0085] HFC avg =α*HFC avg +(1―α)*HFC

[0086] in:

[0087] HFC avg It is the historical high-frequency energy level value determined by statistics of the high-frequency energy values ​​of the historical audio frames that have been generated. The right side of the formula represents the historical high-frequency energy level value corresponding to the previous audio frame, and the left side of the formula represents the historical high-frequency energy level value corresponding to the current audio frame.

[0088] HFC is the high frequency energy value of the current audio frame.

[0089] α is a smoothing factor between 0 and 1, which is used to determine the weight of the historical high-frequency energy level value and the high-frequency energy value of the current audio frame.

[0090] In practical applications, the choice of α depends on how much influence you want to give to new data points. A larger α value means that new data points have a greater impact on the weighted result, making the EMA more sensitive to changes in the signal. On the contrary, a smaller α value means that historical data has a greater impact on the weighted result, making the EMA smoother.

[0091] In this embodiment, the α value is set to a higher value relative to (1-α), for example, the α value is set to 0.8, thereby reducing the weight of the high-frequency energy value of the current audio frame, strengthening the role of historical data, and making the historical high-frequency energy level value corresponding to the current audio frame have a smoother effect.

[0092] It can be seen that this step can generate a smooth historical high-frequency energy level value for each audio frame, which helps to more accurately calculate the energy change rate in the subsequent steps, and then identify significant changes and beat points in the audio signal.

[0093] Step S3140: Determine the ratio between the high-frequency energy value of the current audio frame and the historical high-frequency energy level value corresponding to the current audio frame as the energy change rate corresponding to the current audio frame.

[0094] The historical high-frequency energy level value corresponding to the current audio frame has been calculated in the previous step, which can be used to calculate the energy change rate (hfcRatio) to measure the degree of energy change between the high-frequency energy value of the current audio frame and its historical high-frequency energy level.

[0095] Specifically, the following formula can be used to calculate hfcRatio:

[0096] HFC Ratio=HFC / HFC avg

[0097] in:

[0098] hfcRatio is the energy change ratio of the current audio frame and is used as the energy change rate.

[0099] HFC is the high frequency energy value of the current audio frame.

[0100] HFC avg It is the historical high-frequency energy level value corresponding to the current audio frame.

[0101] The calculation result of the energy change rate hfcRatio is a ratio that reflects the relationship between the high-frequency energy of the current audio frame and its historical high-frequency energy level value. If hfcRatio is significantly higher than the corresponding set hit point threshold, such as 1, this indicates that the high-frequency energy of the current audio frame has increased significantly, which may correspond to a hit point or a significant change in the audio signal. On the contrary, if hfcRatio is lower than the hit point threshold, this indicates that the energy of the current audio frame is relatively low.

[0102] It can be seen that this embodiment can generate an energy change rate for each audio frame, which is used to determine whether there is a candidate beat point in the current audio frame in the subsequent steps.

[0103] In this embodiment, by dividing the audio stream into continuous audio frames and using fast Fourier transform (FFT) to obtain the spectrum information of each frame, it is ensured that the subtle beats and rhythm changes in the music can be captured. Next, by focusing on the energy value calculation of the high-frequency content, the signal characteristics closely related to the rhythm of the music can be identified. The exponential moving average (EMA) method is used to smooth the historical high-frequency energy level values, further improving the anti-interference ability for transient noise and short-term fluctuations, and ensuring the stability of the analysis results. Finally, by calculating the ratio of the current high-frequency energy value to the historical high-frequency energy level value to determine the energy change rate of the current audio frame, it is possible to quantify and highlight the significant energy changes in the audio signal, which is the key indicator of the music beat point.

[0104] It can be seen that the technical effect achieved by the series of steps in this embodiment is that they jointly realize a detection mechanism that is extremely sensitive to changes in music rhythm, which not only improves the accuracy of beat detection, but also enhances the adaptability to different music styles and complex audio environments. In addition, this real-time analysis method provides a solid foundation for the presentation of audio visualization special effects, allowing the special effects to change synchronously with the music rhythm, thereby significantly improving the viewing experience and immersion of live viewers.

[0105] On the basis of any embodiment of the method of the present application, corresponding to each current audio frame, determining whether the current audio frame has a candidate beat point based on the energy change rate of the current audio frame includes:

[0106] Step S3210, determining whether the energy change rate of the current audio frame exceeds a preset strike point threshold, and when it exceeds the strike point threshold, determining that the current audio frame has a candidate beat point, and taking the start timing of the current audio frame as the timing position of the candidate beat point; otherwise, determining that the current audio frame does not have a candidate beat point;

[0107] In order to detect whether there is an actual beat point in the current audio frame, it is necessary to first determine whether the energy change rate (hfcRatio) of the current audio frame indicates the existence of a candidate beat point, which can be determined by comparing the energy change rate hfcRatio of the current audio frame with a preset beat point threshold (thres).

[0108] Specifically, first, the energy change rate hfcRatio of the current audio frame is obtained. As disclosed above, it is obtained by dividing the high-frequency energy value (HFC) of the current audio frame by the historical high-frequency energy level value (HFC) corresponding to the current audio frame. avg ). Then, the calculated hfcRatio is compared with the preset strike point threshold, and the formula example is as follows:

[0109]

[0110] in:

[0111] onset is used to indicate whether there is a candidate strike point in the current audio frame.

[0112] hfcRatio represents the energy change rate of the current audio frame.

[0113] thres indicates the strike point threshold.

[0114] It is not difficult to understand based on the above exemplary formula that if hfcRatio exceeds the strike point threshold, this indicates that the energy change of the current audio frame is significant, which is likely to mark a strike point or rhythm point in the music. In this case, it will be determined that there are candidate beat points in the current audio frame, and the starting timing of the current audio frame will be recorded as the timing position of the candidate beat point. This timing position can be used to synchronize audio visualization effects in subsequent steps. Of course, if hfcRatio does not exceed the strike point threshold, there are no candidate beat points in the current audio frame, and there can be no actual beat points.

[0115] The hit point threshold (thres) is a key parameter that determines the sensitivity to energy changes. In this embodiment, this threshold is not static, but is dynamically adjusted based on historical data to adapt to different audio content and environments. For example, in a quiet music passage, the threshold may be set higher to avoid mistakenly treating weak noise as a beat point. On the contrary, in music with a strong rhythm and large energy changes, the threshold may be set relatively low to ensure that important beat points are not missed.

[0116] Step S3220: adapting to the energy change rate of the current audio frame relative to the energy jitter trend represented by the strike point threshold, readjusting the strike point threshold within a preset threshold range.

[0117] This step focuses on adjusting the hit point threshold (thres) to adapt to the energy jitter trend shown by the energy change rate (hfcRatio) of the current audio frame. This dynamic adjustment mechanism is to more accurately identify candidate beat points while avoiding misjudgment in different audio content and environments.

[0118] In this step, the hit point threshold is adjusted according to the energy change rate hfcRatio of the current audio frame. Specifically, if the hfcRatio of the current audio frame exceeds the existing hit point threshold (thres), it indicates that the energy change is significant and the hit point threshold needs to be increased to avoid misidentifying too many points as candidate beat points. On the contrary, if hfcRatio does not exceed the hit point threshold, the threshold needs to be lowered to capture more possible beat points.

[0119] The adjustment of the strike point threshold can be achieved by the following formula:

[0120]

[0121] in:

[0122] thres is the current strike point threshold.

[0123] threshold max and threshold min They are the maximum and minimum values ​​of the strike point threshold, respectively. They are parameters that can be adjusted by the external interface and constitute a threshold interval to ensure that the threshold is within a reasonable range.

[0124] When hfcRatio>thres, it means that the energy change of the current audio frame is large, so the hitting point threshold needs to be increased to reduce misjudgment.

[0125] When hfcRatio≤thres, it means that the energy change of the current audio frame is small, so the threshold can be appropriately lowered to better capture the beat point.

[0126] Through this dynamic adjustment strategy, it is possible to more flexibly adapt to different audio environments and contents, thereby improving the accuracy and robustness of beat detection. This adaptive mechanism is particularly suitable for real-time audio analysis, such as live broadcast scenarios, where the audio content may change at any time.

[0127] In addition, by setting the threshold appropriately max and threshold min , you can avoid overreaction while maintaining high sensitivity. For example, by default, threshold max =3.0 and threshold min =1.0, these values ​​can be adjusted according to the actual application scenarios to achieve the best beat detection effect.

[0128] In this embodiment, step S3210 effectively identifies audio frames with significant energy changes as candidate beat points by comparing hfcRatio with the dynamically adjusted strike point threshold thres. This design reduces misjudgment in complex audio environments while ensuring sensitivity to changes in music rhythm. Step S3220 further optimizes the dynamic adjustment strategy of the strike point threshold. By gradually adjusting thres according to the actual value of hfcRatio, it can adapt to changes in audio signals and work stably in both quiet and noisy environments. This dynamic threshold adjustment strategy is particularly suitable for real-time audio processing scenarios, such as live broadcasts, where the dynamic nature of the audio content places high demands on high adaptability and flexibility.

[0129] Combining the various steps in this embodiment, this embodiment demonstrates the following significant technical advantages: first, it improves the accuracy of beat detection, ensuring that the beat points can be accurately identified even when the audio content changes frequently; second, the adaptive adjustment of the strike point threshold reduces the dependence on preset parameters, so that stable performance can be maintained under different music styles and audio signal strengths; finally, the real-time nature of this method ensures that it can be applied to real-time audio analysis, such as online live broadcast, greatly enhancing the user's interactive experience and immersion. Therefore, this embodiment provides a more efficient and reliable solution in terms of achieving the accuracy, adaptability and real-time nature of audio beat detection.

[0130] Based on any embodiment of the method of the present application, for each current audio frame, an ideal beat point is determined based on an energy change sequence composed of energy change rates of all current and previous audio frames, including:

[0131] Step S3310: adding the energy change rate of the current audio frame to the pre-cached energy change sequence, so that the energy change sequence includes the energy change rates of all current and previous audio frames;

[0132] In the process of performing audio analysis on the audio stream to determine the ideal beat point corresponding to each audio frame, a large number of time granularities are required to count the total energy changes for each audio frame, which inevitably leads to a large number of computing tasks. In order to prevent such computing tasks from reducing the real-time recognition, this embodiment further introduces a cache mechanism.

[0133] First, in this step, an initialized and iteratively updated buffer area is used to store the energy change sequence. Whenever a new audio frame is processed, its corresponding energy change rate, that is, the ratio of the high-frequency energy of the audio frame to its historical high-frequency energy level value, is calculated. Subsequently, this new energy change rate is added to the end of the energy change sequence, thereby updating the entire sequence.

[0134] The energy change sequence is maintained and updated continuously. As the audio stream continues to play, the sequence continues to grow and contains the energy change rate of each audio frame since the beginning of the audio stream. This continuous recording method can capture the dynamic characteristics of the audio signal over time and provide rich historical data support for beat analysis.

[0135] The cached energy change sequence can not only identify the pattern of audio rhythm, but also enhance the ability to respond to sudden or periodic changes in audio signals. By analyzing this sequence, the rhythm and beat points of the music can be judged more accurately, thus achieving accurate audio visualization effects.

[0136] Step S3320: Count the energy change sums of different time granularities corresponding to the energy change rates in the energy change sequence, and cache the energy change sums of different time granularities in association with the current audio frame for reuse when performing the same statistics on the next audio frame, wherein the time granularity is jointly characterized by different beat periods and deviation phase combinations;

[0137] This step will count the sum of energy changes in the energy change sequence, which are calculated based on different time granularities and are associated with different beat cycles and deviation phases. In order to improve the efficiency of audio stream analysis, this step continues to reuse the calculation results through the cache mechanism, reduce repeated calculations, and ensure the real-time nature of audio rhythm analysis.

[0138] In order to calculate the sum of energy changes in the current audio frame, a series of time granularities are first determined. These time granularities can be pre-set and represent possible beat cycles. Each time granularity is associated with a specific beat cycle and deviation phase, which together characterize different possibilities of musical rhythm. For example, different beat cycles consisting of 30 beats per minute to 200 beats per minute can be set, thereby determining 170 different beat cycles; at the same time, different deviation phases from 0 degrees to 360 degrees can be set, thereby determining 360 different deviation phases. The combination of any beat cycle and any deviation phase defines a corresponding time granularity. For this example, 170*360=61200 time granularities can be obtained.

[0139] For each time granularity, it is necessary to calculate the sum of the energy change rates corresponding to this time granularity in all audio frames of the energy change sequence. This can be achieved by multiplying the energy change rate of each audio frame by the cosine function corresponding to the time granularity and then summing the results. The cosine function is used to weight the energy change rate to reflect the energy change pattern at different time granularities. The corresponding formula can be expressed as:

[0140]

[0141] in:

[0142] Su m It is the sum of energy changes at a specific time granularity.

[0143] hfcRatio[i] is the energy change rate of the i-th audio frame.

[0144] frameLenMs is the time length of the audio frame.

[0145] frameLenMs is the tick period in that particular time granularity.

[0146] t0 is the deviation phase in that particular time granularity.

[0147] According to the above formula, it can be understood that for each time granularity, a corresponding energy change sum can be obtained by statistically analyzing the energy change rates of all audio frames in the energy change sequence. Subsequently, the time granularity that best represents the rhythm pattern of the audio stream can be determined by comparing the energy change sums of these different time granularities, thereby determining the ideal beat point corresponding to the current audio frame.

[0148] In order to improve efficiency, the sum of energy changes of each audio frame corresponding to each time granularity needs to be cached. Specifically, it can be associated with the corresponding audio frame in the energy change sequence and stored in the cache area. When a new audio frame arrives, it is only necessary to update the partial calculation related to the frame, without recalculating the sum of energy changes of the audio frames corresponding to different time granularities that have been calculated in the entire energy change sequence. This means that for each new frame, it is only necessary to add the energy change rate of the new frame, and then calculate the value of the energy change rate weighted by the cosine correlation function at different time granularities, and then according to the corresponding relationship of the time granularity, add the value of each time granularity and the sum of energy changes of the previous audio frame already stored in the cache area to obtain the sum of energy changes of the new frame. According to this principle, if the current audio frame is not the first frame in the audio stream, this method can be applied for statistics in the same way.

[0149] It can be seen that the cache mechanism greatly reduces the amount of repeated calculations, ensuring high performance and real-time response even when processing large amounts of audio data. By reusing the previously calculated sum of energy changes, the amount of calculation is reduced, so that it can quickly adapt to changes in audio signals and update the detection of ideal beat points in real time, which is of obvious benefit to live broadcast scenarios that require high timeliness.

[0150] In one embodiment, the cached energy variation sums can also be reused across multiple audio frames to further reduce computational cost. For example, if an audio stream is identified to have a stable rhythm over a period of time, the energy variation sums associated with the rhythm can be used in multiple consecutive audio frames without having to be recalculated for each frame.

[0151] Step S3330: corresponding to the current audio frame, the beat period and the deviation phase of the time granularity with the largest total energy change are respectively used as the ideal beat period and the ideal deviation phase, and the ideal beat point and its temporal position in the current audio frame are determined.

[0152] The sum of energy changes at different time granularities has been calculated and cached in the previous text. Each time granularity is composed of a specific beat cycle and deviation phase combination. For each current audio frame, the sum of energy changes of multiple such time granularities has been calculated.

[0153] In this step, these energy change sums are continuously evaluated to determine which time granularity has the largest energy change sum. This largest sum indicates that the audio stream exhibits the most significant rhythmic pattern at this time granularity. Therefore, the beat period and deviation phase associated with this largest sum are considered to be the ideal beat period and ideal deviation phase.

[0154] The determination of the ideal beat period (T) and the ideal deviation phase (φ) is based on analyzing the sum of energy changes at different time granularities in the energy change sequence. By comparing the sum of energy changes corresponding to each time granularity and selecting the time granularity with the largest sum, the period and phase corresponding to the most significant beat point in the audio signal can be found. This selection reflects the regularity of audio rhythm, where the time granularity is defined by a specific beat period and deviation phase combination. Therefore, the beat period and deviation phase corresponding to the time granularity with the largest sum of energy changes are considered to be the ideal parameters that most accurately reflect the rhythm of the music.

[0155] Once the ideal beat period and ideal deviation phase are determined, these parameters can be used to define the ideal beat point. The ideal beat point refers to the timing position of the beat point that theoretically best matches the rhythm of the music within the audio frame. beat (n)) can be expressed by the following formula:

[0156] t beat (n) = n*T + φ(n∈Z)

[0157] in:

[0158] t beat (n) is the timing position of the nth ideal beat point.

[0159] T is the ideal beat period.

[0160] φ is the ideal deviation phase.

[0161] n is the beat number, an integer, indicating the beat number in the current audio frame.

[0162] In this way, the corresponding ideal beat point can be determined for the current audio frame, which represents the most likely beat position of the music rhythm in the current audio frame.

[0163] This embodiment searches for the ideal beat point by accurately matching each audio frame to a finer time granularity, and combines a cache mechanism to avoid repeated calculations in the process of searching for the ideal beat point, thereby showing significant technical effects in live broadcast scenarios with high real-time requirements:

[0164] First, by subdividing the time granularity, the rhythmic pattern of the audio signal can be analyzed more carefully, so that the ideal beat point can be determined with higher accuracy. This detailed analysis ensures that the beat point can be accurately captured even when the audio rhythm is complex or changes rapidly.

[0165] Secondly, the introduced cache mechanism greatly improves the efficiency of finding the ideal beat point. By caching the sum of energy changes corresponding to different time granularities for each audio frame, repeated calculations when processing subsequent audio frames are avoided. This mechanism significantly reduces the amount of calculations and speeds up processing, maintaining high performance and real-time response even in a live broadcast environment.

[0166] Furthermore, since the caching mechanism reduces the consumption of computing resources, more resources can be allocated to other tasks, such as the generation and rendering of audio visualization effects. This not only improves the overall system efficiency, but also reduces resource overhead, making it more stable and efficient.

[0167] In addition, in terms of user experience, this efficient beat detection mechanism ensures that the audio visualization effects can be updated synchronously with the music rhythm, providing users with a smooth and immersive viewing experience. The audience can see the visual effects that perfectly match the music rhythm in real time during the live broadcast, thus enhancing the interactivity and viewing experience of the live broadcast.

[0168] Based on any embodiment of the method of the present application, using the ideal beat point to determine whether the candidate beat point belongs to the actual beat point includes:

[0169] Step S3410, determining a timing distance between the timing position of the ideal beat point and the timing position of the candidate beat point, wherein the timing position of the candidate beat point is the starting timing position of the current audio frame corresponding to the candidate beat point;

[0170] As previously revealed, the temporal position of the ideal beat point is determined based on the ideal beat period and ideal deviation phase calculated during the audio stream analysis process. Specifically, the ideal beat points are marked according to the pre-calculated ideal beat period (T) and ideal deviation phase (φ), which represent the theoretical position of the most significant beat points in the music rhythm. These ideal beat points are obtained by a detailed analysis of the energy change sequence of historical audio frames, reflecting the regularity of the music rhythm.

[0171] The timing position of the candidate beat point refers to the starting timing of the beat point determined based on the energy change rate of the frame exceeding the preset hit point threshold when analyzing the current audio frame in real time. It is also the starting timing of the corresponding audio frame. The candidate beat points represent possible beat positions, which are obtained by comparing the energy change rate with the historical average when processing audio frames in real time.

[0172] After determining the timing positions of the ideal beat point and the candidate beat point corresponding to the current audio frame, the timing distance between the ideal beat point and the candidate beat point can be calculated. This can be achieved by simply subtracting the timing position of the ideal beat point from the timing position of the candidate beat point and finding the absolute value. This distance reflects the gap between the two on the timeline and is an important basis for judging whether the two are aligned.

[0173] Step S3420, determining whether the timing distance enters a preset time tolerance window. When entering the time tolerance window and the search state flag indicates that no actual beat point has been determined within the current beat cycle, the candidate beat point is determined as the actual beat point, and the current beat cycle is the beat cycle corresponding to the ideal beat point, and is counted from the last actual beat point;

[0174] The time tolerance window is a predefined time range used to account for small changes in audio tempo and potential delays in real-time processing. If the timing distance between the ideal beat point and the candidate beat point falls within this tolerance window, it indicates that the candidate beat point is aligned with the ideal beat point within an acceptable time and can be considered as the actual beat point.

[0175] Specifically, first check the timing position t of each candidate beat point onset The timing position t with the ideal beat point beat The time distance between |t onset -t beat If the timing distance is less than or equal to the threshold corresponding to the time tolerance window, the search status flag is checked. This flag is used to track whether an actual beat point has been determined in the current beat cycle.

[0176] The current beat cycle refers to the time period from the last determined actual beat point to the audio frame currently being analyzed. This cycle is based on the regularity of the audio rhythm and is determined by the ideal beat cycle, which is determined by analyzing the sequence of energy changes in the audio stream. During the real-time analysis process, once a new actual beat point is determined, a new beat cycle will be started, which is the beat cycle (T) corresponding to the ideal beat point.

[0177] The introduction of the search status indicator is to ensure that only one actual beat point is determined in each beat cycle. If the search status indicator shows that the actual beat point has not been determined in the current beat cycle, and the timing distance between the timing position of the ideal beat point and the timing position of the candidate beat point falls within the time tolerance window, then the current candidate beat point will be determined as the actual beat point.

[0178] To express this relationship, refer to the following formula:

[0179]

[0180] in:

[0181] isFinalBeat(t onset ) is an indicator function used to determine the candidate beat point t onset Is it an actual beat point?

[0182] ·t onset is the temporal position of the candidate beat point.

[0183] ·t beat It is the timing position of the ideal beat point.

[0184] ·t window It is used to measure the threshold corresponding to the time tolerance window.

[0185] ·flag is a search status indicator, which is used to indicate whether an actual beat point has been determined in the current beat cycle.

[0186] If the candidate beat point passes the test of the time tolerance window and the search status flag is 0 (indicating that the actual beat point has not been determined in the current beat cycle), the candidate beat point is determined as the actual beat point, and the search status flag will be updated to 1, indicating that the actual beat point has been determined in the current beat cycle. It is not difficult to understand that during the period when the search status flag is 1 and the current beat cycle has not ended, even if the timing distance between the candidate beat point of the audio frame and the corresponding ideal beat point falls within the time tolerance window, the candidate beat point will not be confirmed as the actual beat point.

[0187] It can be seen that this step can accurately identify the actual beat points from the audio stream and use these beat points to drive the audio visualization effects in the live broadcast room. This not only improves the accuracy of beat detection, but also makes the audio visualization effect more vivid and synchronized, greatly improving the user's viewing experience.

[0188] Step S3430, mark the search status mark to indicate that the actual beat point has been determined in the current beat cycle, and switch the search status mark to indicate that the actual beat point has not been determined until the end of the current beat cycle.

[0189] In the process of determining whether a candidate beat point becomes an actual beat point, the maintenance of the mark search status flag is the key to ensure accurate identification of the actual beat point. This flag is used to track whether the actual beat point has been identified in the current beat cycle, ensuring that only one actual beat point is determined in each beat cycle to avoid repeated counting of beat points.

[0190] When a candidate beat point passes the test of the time tolerance window and the corresponding search status is 0 (indicating that the actual beat point has not been determined in the current beat cycle), the candidate beat point is identified as the actual beat point. At this time, the search status is updated to 1, indicating that an actual beat point has been successfully determined in the current beat cycle.

[0191] After the search state flag is updated to 1, the state is maintained until the end of the current beat cycle. The end of the beat cycle means that the time from the last actual beat point has reached the length of the ideal beat cycle (T). During the beat cycle, even if there are new candidate beat points and the timing distance of the ideal beat point is within the time tolerance window, the search state flag will not be updated again, so that no additional actual beat points will be determined in the same beat cycle.

[0192] This mechanism ensures the consistency and accuracy of the audio rhythm, and also provides an accurate synchronization point for audio visualization effects. For example, in a live broadcast environment, once the actual beat point is determined and the corresponding audio visualization effect is triggered, it will wait until the next beat cycle starts before looking for a new actual beat point again, and then adjust the audio visualization effect according to the new actual beat point.

[0193] This embodiment achieves efficient and accurate recognition of actual beat points in an audio stream through a series of organically coordinated steps, thereby providing significant technical advantages in application scenarios such as real-time live broadcasting.

[0194] First, by calculating the timing distance between the ideal beat point and the candidate beat point and comparing it with the time tolerance window, this embodiment can accurately synchronize the audio visualization effect with the music rhythm, ensuring accuracy and real-time performance in various music styles and complex audio environments.

[0195] Secondly, the introduced search status identifier further enhances the ability to judge the actual beat point, ensuring that only one actual beat point is determined in each beat cycle, avoiding repeated confirmation of multiple candidate beat points in the same beat cycle. This mechanism not only improves the accuracy of beat detection, but also reduces the waste of computing resources and improves processing efficiency.

[0196] Furthermore, the present embodiment further ensures the continuity and stability of the audio rhythm by maintaining the search state flag as determined until the current beat cycle ends after the actual beat point is determined. In this way, subsequent audio frames within the same beat cycle are not processed unnecessarily, thereby saving computing resources and allowing centralized processing of new beat cycles.

[0197] In addition, the organic combination of these steps enables accurate capture and real-time feedback of audio rhythm in high-demand environments such as live broadcasting, significantly improving the user's interactive experience and immersion. By ensuring the synchronization of audio visualization effects and actual music rhythm, this embodiment provides live broadcast viewers with a richer and more attractive visual experience, enhancing the attractiveness and viewing value of live broadcasting.

[0198] On the basis of any embodiment of the method of the present application, after using the ideal beat point to determine whether the candidate beat point belongs to an actual beat point, so as to use the actual beat point to drive the live broadcast room to play audio visualization special effects, the method includes:

[0199] Step S3500: displaying a preset audio visualization special effect in the live broadcast room to show the special effect image therein;

[0200] To display the audio information of the live broadcast room in the live broadcast room interface, you can use the preset special effects template to present the corresponding audio visualization effects, and display the special effects images through audio visualization effects, so that users can visually perceive the audio changes.

[0201] The special effect template can be designed flexibly. An exemplary special effect template of this embodiment can be set as a spectrogram composed of multiple brightly colored vertical bars, each bar representing a specific frequency range in the spectrum. When the music is playing, the length and color of these bars can change according to the energy change of the corresponding frequency, thereby reflecting the rhythm and energy of the music in real time.

[0202] Specifically, when an actual beat point is identified, the special effect image will change significantly in synchronization with the actual beat point. For example, whenever an actual beat point is determined, the background color of the spectrogram can undergo a corresponding mutation, or the brightness of the bar will suddenly increase, so as to highlight the location of the actual beat point. In addition, if the actual beat point identified has a particularly high energy change rate, the bars in the spectrogram can also be controlled to grow rapidly, and even some special visual effects can be triggered, such as fireworks with explosion effects appearing on the screen to enhance the visual impact.

[0203] In terms of the expression of energy information, special effect templates can be further refined. For example, the energy change rate corresponding to high-frequency content can control the color and height of the upper bars of the spectrum. When the high-frequency energy change rate is high, the corresponding bar may become a brighter color and grow rapidly to the top of the screen. Conversely, the energy information of low-frequency content, such as the amplitude of low-frequency oscillations, can affect the width and color changes of the lower bars of the spectrum. When the amplitude of low-frequency oscillations is large, the corresponding bar may become wider and emit a pulsed light effect.

[0204] In addition, special effects templates can also include some detailed designs. For example, as the rhythm of the music speeds up, the dynamic change speed of the entire spectrum will increase accordingly, or at certain specific beat points in the music, such as half beats or triplets, the special effects image will show preset decorative patterns or animation effects.

[0205] The live broadcast room can automatically load the special effects template corresponding to the audio visualization special effect, and open the corresponding control interface for subsequent steps to pass parameters, so as to implement change control on the special effects image in the audio visualization special effect.

[0206] Step S3600, controlling the special effect image to be synchronized with the generation of the actual beat point to present a periodic change;

[0207] Take the spectrogram special effects template as an example. This template consists of multiple brightly colored vertical bars, each of which represents the energy of a specific frequency range in the audio signal. When the music plays, the length and color of these bars will dynamically adjust according to the energy changes of the corresponding frequency, realizing the visual expression of the audio signal.

[0208] In the process of controlling the special effect image to present periodic changes, the display effect of the spectrogram is adjusted according to the actual beat points identified. For example, every time a new actual beat point is determined, the background color of the spectrogram can change suddenly, or the brightness of a specific frequency bar can suddenly increase, so as to highlight the location of the actual beat point. This change is synchronized with the rhythm of the music, allowing the audience to intuitively feel the beat of the music.

[0209] In order to achieve periodic changes, the rhythm of the music can be continuously tracked, and the dynamic effects of the special effects image can be adjusted according to the actual beat points. For example, a beat trigger mechanism can be preset, and every time an actual beat point is generated, the special effects image will be updated according to the preset mode, such as the growth of the bar, the change of color, or the flashing of the background. This periodic change is perfectly synchronized with the beat of the music, providing users with a dynamic and rhythmic visual experience.

[0210] In addition, the speed of the special effect image change can be adjusted according to the rhythm of the music, that is, the length of time between two adjacent actual beat points. Fast-paced music will cause the dynamic change of the spectrogram to speed up, while slow-paced music will slow down the change, thus ensuring that the change of the special effect image is consistent with the rhythm of the music.

[0211] Step S3700: Control the special effect image to express changes in energy information of the frequency spectrum information.

[0212] The specific content of the energy information can be obtained by analyzing the spectrum information of the audio frame. Specifically, the spectrum information of the audio frame is obtained through fast Fourier transform (FFT) or other time-frequency analysis techniques, which reflects the energy distribution of the audio signal at different frequencies. On this basis, the energy value of a specific frequency band can be extracted, such as the high frequency range and the low frequency range.

[0213] For example, the energy change rate of high-frequency content, which is one of the energy information, can be obtained by calculating the ratio of the total energy of the high-frequency frequency points to the historical average value. This ratio reflects the degree of change of high-frequency energy relative to the past level. Low-frequency content can be processed similarly, and its energy change information can be obtained by analyzing the energy value in the low-frequency range.

[0214] After obtaining this energy information, the special effect image in the special effect template can be controlled to show the energy change of the audio. Taking the spectrogram special effect template as an example, the change of high-frequency energy can control the color and height of the upper bar of the spectrogram, while the change of low-frequency energy affects the width and color of the lower bar. When the high-frequency energy change rate is high, the corresponding bar may become brighter and grow rapidly; on the contrary, when the low-frequency oscillation amplitude is large, the corresponding bar may become wider and emit a pulsed light effect.

[0215] In addition, the energy value of a specific frequency band in the audio frame can be used as energy information to control other aspects of the special effect image. For example, if an audio frame has high energy in the mid-frequency band, the bar representing the frequency band in the spectrogram can be made to flash or change color to highlight the energy change of this frequency band.

[0216] The control of special effect images is not limited to the color and height changes of the bars, but can also include changes in other visual elements. For example, as the audio energy increases, the background brightness in the special effect image can gradually increase, or dynamic patterns synchronized with the rhythm of the music can appear. These changes can be pre-set or dynamically adjusted based on real-time analyzed data.

[0217] The audio visualization special effects of this embodiment can realize the visualization display of live audio based on multi-dimensional features such as time domain features and frequency domain features. By capturing the rhythm and energy distribution of the music, this information is presented through visual elements such as dynamically adjusted special effect images, providing users with a richer and more detailed audio visualization experience, and also providing richer control information for the design of audio visualization special effects for live broadcast platforms. This multi-dimensional analysis method that combines time domain and frequency domain features not only improves the accuracy and real-time performance of music visualization, but also enhances the user's immersion and interactivity in music perception. This technical advantage is particularly prominent in scenarios with high real-time requirements such as live broadcasts.

[0218] Based on any embodiment of the method of the present application, the method includes: after real-time tracking and determining the spectrum information of each audio frame in the audio stream of the live broadcast room, the method includes:

[0219] Step S4100: interpolating low-frequency content in the spectrum information of the current audio frame to improve the resolution of the low-frequency content;

[0220] Since the resolution of spectrum information obtained by spectrum analysis methods such as fast Fourier transform (FFT) is relatively low in the low-frequency area, manual interpolation technology can effectively improve the details of this area. For example, mathematical methods such as Lagrange interpolation or spline interpolation can be used to predict and fill in the unknown frequency point data in the low-frequency area through interpolation calculation of known frequency point data, so as to make the visualization of low-frequency content more accurate and detailed.

[0221] Step S4200: smoothing the frequency spectrum information of each audio frame in the audio stream using a preset sliding window;

[0222] In order to reduce random noise and short-term fluctuations in audio signals, it is an effective method to use a preset sliding window to smooth the spectrum information of each audio frame in the audio stream. By setting a fixed-length window (such as a moving average window or a Gaussian window) and performing weighted averaging on the spectrum information of consecutive audio frames, spectrum smoothing can be achieved. This smoothing process helps to highlight the long-term trends and periodic features in the audio signal, while suppressing instantaneous noise interference, providing more stable energy information for beat detection and audio visualization.

[0223] Step S4300: extract energy information of a specified frequency band from the spectrum information of the current audio frame for controlling the special effect image, wherein the energy information includes the energy change rate and / or energy values ​​of multiple frequency points in the current audio frame.

[0224] Extracting energy information of a specified frequency band from the spectrum information of the current audio frame is the basis for realizing audio visualization special effects control. Therefore, the corresponding energy information can be extracted from the spectrum information of the current audio frame according to the special effects image control parameters preset in the special effects template. According to the requirements of the control parameters, the energy information can be the energy change rate in the current audio frame, or the energy value of multiple frequency points, or a combination thereof. For example, corresponding to the special effects template in the previous example, the ratio of the energy value in the high-frequency range to the historical average value can be calculated as the energy change rate, which is used to control the color and height of the upper bars of the spectrogram; while the energy value in the low-frequency range can control the width and color of the lower bars. In this way, the special effects image can be dynamically adjusted according to the energy changes of the audio signal to achieve synchronous changes with the music rhythm and energy distribution.

[0225] This embodiment provides a more detailed and accurate audio visualization control method by optimizing the resolution of low-frequency content and accurately extracting energy data from spectrum information. The advantage of this technology is that it not only enhances the visual expression of music, but also improves the user's audio-visual experience, especially in scenes with high real-time requirements such as live broadcasts, it can ensure that the real-time update of special effect images is consistent with the rhythm of music, providing users with a rich and dynamic audio visualization experience.

[0226] See also Figure 3 According to one aspect of the present application, an audio visualization control device is provided, comprising a spectrum analysis module 3100, a single frame analysis module 3200, a sequence analysis module 3300, and a beat determination module 3400, wherein the spectrum analysis module 3100 is configured to track and determine the spectrum information of each audio frame in the audio stream of the live broadcast room in real time, and determine the energy change rate of each audio frame; the single frame analysis module 3200 is configured to correspond to each current audio frame, and determine whether there is a candidate beat point in the current audio frame based on the energy change rate of the current audio frame; the sequence analysis module 3300 is configured to correspond to each current audio frame, and determine the ideal beat point based on the energy change sequence composed of the energy change rates of all current and previous audio frames; the beat determination module 3400 is configured to use the ideal beat point to determine whether the candidate beat point belongs to the actual beat point, so as to use the actual beat point to drive the live broadcast room to play audio visualization special effects.

[0227] Based on any embodiment of the device of the present application, the spectrum analysis module 3100 includes: a time-frequency conversion module, which is configured to perform time-frequency conversion on each audio frame in the audio stream of the live broadcast room in real time to obtain spectrum information corresponding to each audio frame; an energy detection module, which is configured to count the high-frequency energy values ​​corresponding to all frequency points of the high-frequency content in the current audio frame based on the spectrum information; a weighted smoothing module, which is configured to weighted sum the high-frequency energy value of the current audio frame with the historical high-frequency energy level value corresponding to the previous audio frame, and iteratively determine the historical high-frequency energy level value corresponding to the current audio frame; a change determination module, which is configured to determine the ratio between the high-frequency energy value of the current audio frame and the historical high-frequency energy level value corresponding to the current audio frame as the energy change rate corresponding to the current audio frame.

[0228] Based on any embodiment of the device of the present application, the single frame analysis module 3200 includes: a threshold judgment module, configured to judge whether the energy change rate of the current audio frame exceeds a preset strike point threshold; when it exceeds the strike point threshold, determining that there is a candidate beat point in the current audio frame, and taking the start timing of the current audio frame as the timing position of the candidate beat point; otherwise, determining that there is no candidate beat point in the current audio frame; a threshold update module, configured to adapt to the energy jitter trend represented by the energy change rate of the current audio frame relative to the strike point threshold, and readjusting the strike point threshold within a preset threshold range.

[0229] Based on any embodiment of the device of the present application, the sequence analysis module 3300 includes: a data caching module, configured to add the energy change rate of the current audio frame to the pre-cached energy change sequence, so that the energy change sequence contains the energy change rates of all current and previous audio frames; a statistical analysis module, configured to count the sum of energy changes corresponding to different time granularities of the energy change rate in the energy change sequence, and cache the sum of energy changes of different time granularities associated with the current audio frame for reuse when the next audio frame performs the same statistics, and the time granularity is jointly represented by different beat period and deviation phase combinations; a position inference module, configured to correspond to the current audio frame, and take the beat period and deviation phase of the time granularity with the largest sum of energy changes as the ideal beat period and ideal deviation phase, respectively, to determine the ideal beat point and its timing position in the current audio frame.

[0230] Based on any embodiment of the device of the present application, the beat determination module 3400 includes: a time slot determination module, which is configured to determine the timing distance between the timing position of the ideal beat point and the timing position of the candidate beat point, and the timing position of the candidate beat point is the starting timing position of the current audio frame corresponding to it; a candidate judgment module, which is configured to determine whether the timing distance enters a preset time tolerance window. When entering the time tolerance window, and the search status mark indicates that the actual beat point has not been determined in the current beat cycle, the candidate beat point is determined as the actual beat point, and the current beat cycle is the beat cycle corresponding to the ideal beat point, and is counted from the last actual beat point; a status marking module, which is configured to mark the search status mark so that it indicates that the actual beat point has been determined in the current beat cycle, and the search status mark is switched to indicate that the actual beat point has not been determined until the end of the current beat cycle.

[0231] Based on any embodiment of the device of the present application, the device of the present application also includes: a special effect loading module, configured to display a preset audio visualization special effect in the live broadcast room to show the special effect image therein; a rhythm expression module, configured to control the special effect image to be synchronized with the generation of the actual beat point and present periodic changes; an energy expression module, configured to control the special effect image to express the change of energy information of the spectrum information.

[0232] On the basis of any embodiment of the device of the present application, it includes: the device of the present application also includes: a low-frequency interpolation module, configured to interpolate the low-frequency content in the spectrum information of the current audio frame to improve the resolution of the low-frequency content; a spectrum smoothing module, configured to smooth the spectrum information of each audio frame in the audio stream with a preset sliding window; an information extraction module, configured to extract energy information of a specified frequency band from the spectrum information of the current audio frame for controlling the special effect image, the energy information including the energy change rate and / or energy values ​​of multiple frequency points in the current audio frame.

[0233] Another embodiment of the present application also provides an audio visualization control device. Figure 4 As shown, a schematic diagram of the internal structure of an audio visualization control device. The audio visualization control device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable non-volatile storage medium of the audio visualization control device stores an operating system, a database, and computer-readable instructions. The database may store an information sequence. When the computer-readable instructions are executed by the processor, the processor may implement an audio visualization control method.

[0234] The processor of the audio visualization control device is used to provide computing and control capabilities to support the operation of the entire audio visualization control device. The memory of the audio visualization control device may store computer-readable instructions, which, when executed by the processor, may enable the processor to execute the audio visualization control method of the present application. The network interface of the audio visualization control device is used to connect and communicate with a terminal.

[0235] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the present application scheme, and does not constitute a limitation on the audio visualization control device to which the present application scheme is applied. The specific audio visualization control device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0236] In this embodiment, the processor is used to execute Figure 3The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the audio visualization control device of this application, and the server can call the program code and data of the server to execute the functions of all modules.

[0237] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the audio visualization control method of any embodiment of the present application.

[0238] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0239] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0240] In summary, this application has significant advantages in improving the interactivity and viewing experience of live broadcasts. It has broad application prospects in online live broadcasts, music playback applications and other fields that require audio visualization, and is expected to become the mainstream direction of future audio visualization control technology.

Claims

1. An audio visualization control method, characterized in that: include: Track and determine the spectrum information of each audio frame in the audio stream of the live broadcast room in real time, and determine the energy change rate of each audio frame; Corresponding to each current audio frame, determining whether there is a candidate beat point in the current audio frame based on the energy change rate of the current audio frame; Corresponding to each current audio frame, determining an ideal beat point based on an energy change sequence formed by energy change rates of all current and previous audio frames; The ideal beat point is used to determine whether the candidate beat point belongs to an actual beat point, so that the actual beat point is used to drive the live broadcast room to play audio visualization effects.

2. The audio visualization control method according to claim 1, characterized in that: Track and determine the spectrum information of each audio frame in the audio stream of the live broadcast room in real time, and determine the energy change rate of each audio frame, including: Perform time-frequency transformation on each audio frame in the audio stream of the live broadcast room in real time to obtain the spectrum information corresponding to each audio frame; Based on the spectrum information, high-frequency energy values ​​corresponding to all frequency points of the high-frequency content in the current audio frame are counted; Performing a weighted summation of the high-frequency energy value of the current audio frame and the historical high-frequency energy level value corresponding to the previous audio frame, and iteratively determining the historical high-frequency energy level value corresponding to the current audio frame; The ratio between the high frequency energy value of the current audio frame and the historical high frequency energy level value corresponding to the current audio frame is determined as the energy change rate corresponding to the current audio frame.

3. The audio visualization control method according to claim 1, characterized in that: Corresponding to each current audio frame, determining whether the current audio frame has a candidate beat point based on the energy change rate of the current audio frame includes: Determine whether the energy change rate of the current audio frame exceeds a preset strike point threshold. If it exceeds the strike point threshold, determine that the current audio frame has a candidate beat point, and use the start timing of the current audio frame as the timing position of the candidate beat point; otherwise, determine that the current audio frame does not have a candidate beat point; The strike point threshold is readjusted within a preset threshold range to adapt to the energy change rate of the current audio frame relative to the energy jitter trend represented by the strike point threshold.

4. The audio visualization control method according to claim 1, characterized in that: For each current audio frame, an ideal beat point is determined based on an energy change sequence consisting of energy change rates of all current and previous audio frames, including: Adding the energy change rate of the current audio frame to the pre-cached energy change sequence, so that the energy change sequence includes the energy change rates of all current and previous audio frames; Counting the energy change sums of different time granularities corresponding to the energy change rates in the energy change sequence, and buffering the energy change sums of different time granularities in association with the current audio frame for reuse when performing the same statistics on the next audio frame, wherein the time granularity is jointly characterized by different beat periods and deviation phase combinations; Corresponding to the current audio frame, the beat period and deviation phase of the time granularity with the largest total energy change are respectively taken as the ideal beat period and ideal deviation phase, and the ideal beat point and its temporal position in the current audio frame are determined.

5. The audio visualization control method according to claim 1, characterized in that: Using the ideal beat point to determine whether the candidate beat point belongs to the actual beat point includes: Determine a timing distance between the timing position of the ideal beat point and the timing position of the candidate beat point, wherein the timing position of the candidate beat point is the starting timing position of the current audio frame corresponding to the candidate beat point; Determine whether the timing distance enters a preset time tolerance window. When entering the time tolerance window and the search state flag indicates that no actual beat point has been determined in the current beat cycle, determine the candidate beat point as the actual beat point, and the current beat cycle is the beat cycle corresponding to the ideal beat point, and is counted from the last actual beat point; The search state mark is marked to indicate that the actual beat point has been determined within the current beat cycle, and the search state mark is switched to indicate that the actual beat point has not been determined until the end of the current beat cycle.

6. The audio visualization control method according to any one of claims 1 to 5, characterized in that: After using the ideal beat point to determine whether the candidate beat point belongs to an actual beat point, so as to use the actual beat point to drive the live broadcast room to play audio visualization special effects, the method includes: Displaying preset audio visualization effects in the live broadcast room to show the special effect images therein; Controlling the special effect image to be synchronized with the generation of the actual beat point to present periodic changes; The special effect image is controlled to express the change of energy information of the frequency spectrum information.

7. The audio visualization control method according to claim 6, characterized in that: include: After real-time tracking and determining the spectrum information of each audio frame in the audio stream of the live broadcast room, it includes: Interpolate the low-frequency content in the spectrum information of the current audio frame to improve the resolution of the low-frequency content; Smoothing the frequency spectrum information of each audio frame in the audio stream with a preset sliding window; Energy information of a specified frequency band is extracted from the spectrum information of the current audio frame to control the special effect image, wherein the energy information includes the energy change rate and / or energy values ​​of multiple frequency points in the current audio frame.

8. An audio visualization control device, characterized in that: include: A spectrum analysis module is configured to track and determine spectrum information of each audio frame in the audio stream of the live broadcast room in real time, and determine the energy change rate of each audio frame; A single frame analysis module is configured to correspond to each current audio frame and determine whether there is a candidate beat point in the current audio frame based on the energy change rate of the current audio frame; A sequence analysis module is configured to correspond to each current audio frame and determine an ideal beat point based on an energy change sequence composed of energy change rates of all current and previous audio frames; The beat determination module is configured to use the ideal beat point to determine whether the candidate beat point belongs to the actual beat point, so as to use the actual beat point to drive the live broadcast room to play audio visualization effects.

9. An audio visualization control device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images