Audio data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of multimedia, and discloses an audio data processing method and device, equipment and a storage medium, and the method comprises the steps: determining an audio data size according to a target playing rhythm and an audio sampling rate; wherein the audio data volume is a data volume needing to be played in a future time period; generating to-be-played audio data corresponding to the audio data volume according to the original digital audio data of the target tone; and writing the audio data to be played into an underlying data buffer area of the audio playing equipment in advance of the current playing progress of the audio playing equipment. By applying the technical scheme of the invention, the problem that cross-platform consistency, real-time controllability, sound effect quality and the like are difficult to consider in the prior art can be solved, the method is used for shielding difference and instability of rhythm-type sound effects in different systems and hardware expressions, and the audio playing performance and the user experience are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of multimedia, and in particular to an audio data processing method and device, an audio data processing apparatus, and a storage medium. BACKGROUND

[0002] In modern intelligent devices and audio applications, users have increasingly growing demands for the stability and customizability of sound effects. Users can expect rhythm-type sound effects (such as metronomes, short prompt tones, and looped background tones) to maintain a highly consistent playback effect on different hardware platforms and operating systems.

[0003] However, the underlying implementation of existing audio devices differs significantly, resulting in significant differences in the playback timing, volume curve, and mixing effect of sound effects, which not only affects the user experience but also limits the development efficiency of cross-platform audio applications. SUMMARY

[0004] In view of the above problems, embodiments of the present application provide an audio data processing method, device, apparatus, and storage medium, to solve the problems of difficulty in balancing cross-platform consistency, real-time controllability, and sound quality in the prior art.

[0005] According to an aspect of an embodiment of the present application, an audio data processing method is provided, the method comprising:

[0006] determining an audio data amount according to a target playback rhythm and an audio sampling rate, wherein the audio data amount is an amount of data to be played in a future time period;

[0007] generating to-be-played audio data corresponding to the audio data amount according to original digital audio data of a target timbre;

[0008] writing the to-be-played audio data into an underlying data buffer of an audio playback device ahead of a current playback progress of the audio playback device.

[0009] According to another aspect of an embodiment of the present application, an audio data processing apparatus is provided, the apparatus comprising:

[0010] a first processing unit configured to determine an audio data amount according to a target playback rhythm and an audio sampling rate, wherein the audio data amount is an amount of data to be played in a future time period;

[0011] a second processing unit configured to generate to-be-played audio data corresponding to the audio data amount according to original digital audio data of a target timbre;

[0012] The third processing unit is used to write the audio data to be played into the underlying data buffer of the audio playback device ahead of the current playback progress of the audio playback device.

[0013] According to another aspect of the embodiments of this application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0014] The memory is used to store at least one executable instruction that causes the processor to perform the operation of the audio data processing method provided in the embodiments of this application.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction that causes an electronic device / apparatus to perform the operation of the audio data processing method provided in the embodiments of this application.

[0016] This application's embodiments construct a novel, self-closing audio playback control model through the synergistic effect of pre-calculating the amount of audio data needed in the future, accurately generating audio data on demand, and writing it into the underlying buffer in advance. This successfully shifts the precision of rhythm control from the unreliable software timing domain to the domain of precise mathematical calculations and stable hardware data consumption, thereby solving the three core problems of existing technologies: unstable rhythm, tampering with sound quality, and inability to customize. It is especially suitable for application scenarios with stringent requirements for timing and sound quality, such as metronomes, drum machines, and game sound effects. It can significantly improve the software's audio playback performance and user experience without increasing hardware costs.

[0017] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0018] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0019] Figure 1 A flowchart illustrating an audio data processing method provided in an embodiment of this application;

[0020] Figure 2 A flowchart illustrating another audio data processing method provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0024] With the development of technology, there are more and more audio devices on the market (such as smartphones with different operating systems, standalone audio devices, etc.), and users' demands for the stability and customizability of sound effects are also growing. For example, in music creation, game sound effect design, smart hardware interaction (such as the vibration feedback sound effect of smartwatches), and virtual reality (VR) / augmented reality (AR) scenarios, rhythmic sound effects (such as metronomes, short cue sounds, looping background sounds, etc.) need to maintain a high degree of consistency in playback across different hardware platforms and operating systems (such as Android, iOS, Windows, embedded systems, etc.).

[0025] However, different audio devices employ different underlying algorithms for handling audio data. For example, some audio devices use timers to trigger rhythmic sound effects. However, timer accuracy is unstable, especially on low-performance devices or under high load, where system scheduling delays can cause playback cycle deviations. Additionally, some audio devices use fade-in algorithms to gradually increase the volume of sound, which is very unfriendly to short, sharp sound effects and does not meet the user's desired effect.

[0026] Furthermore, users' personalized demands for sound effects (such as custom timbres, adjusted rhythm intervals, and layering multiple sound effects) further complicate the technical implementation. For example, users may want to achieve rhythmic sound effects such as "short, powerful drum beats" or "smooth, crescendoing ambient sounds" on different devices using the same application. However, due to differences in the underlying audio processing logic, existing technologies result in significant variations in the timing of sound effect playback, volume curves, and mixing effects. This inconsistency not only affects the user experience but also limits the development efficiency of cross-platform audio applications.

[0027] To address the aforementioned issues, this application provides an audio data processing method that converts time control into data volume control. By combining dynamic audio synthesis with a custom mixing algorithm, it achieves consistent sound effects across platforms and hardware. Specifically, by pre-calculating the audio data volume and dynamically writing it into the audio buffer, the time precision issue is transformed into a data volume management issue, thereby avoiding the instability of system-level timers. Furthermore, through real-time mixing algorithms (such as audio data overlay and attenuation coefficient control) and audio data overflow handling mechanisms, it ensures volume balance and smooth playback when multiple sound effects are overlaid, ultimately achieving a unified sound effect performance across different devices and systems.

[0028] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0029] It should be noted that the entity executing the audio data processing method provided in this application embodiment can be an audio data processing device. This processing device can be deployed on electronic devices such as mobile phones, tablets, servers, computers, independent audio devices, independent digital audio workstations (DAWs), or embedded audio modules. This application embodiment does not impose any limitations, and the method of this application can be implemented by software, hardware, or a combination of software and hardware.

[0030] Figure 1 This is a flowchart illustrating an audio data processing method provided in an embodiment of this application. This method can be executed by an electronic device. Figure 1 As shown, the method may include the following steps:

[0031] Step 110: Determine the amount of audio data based on the target playback rhythm and audio sampling rate; where the amount of audio data is the amount of data to be played in the future time period.

[0032] For example, the target playback rhythm refers to the sound effect triggering timing rules set by the user or preset by the program. It can be expressed as beat speed (e.g., 120 beats per minute), a specific drum beat sequence timestamp (e.g., [0ms, 500ms, 1000ms]), or any custom pronunciation pattern. This application embodiment does not impose any limitations. The target playback rhythm defines when the sound effect needs to be played.

[0033] Audio sampling rate is a fixed parameter in the audio digitization process, representing the number of audio samples captured (or played) per second, measured in Hertz (Hz), such as 44100Hz, 48000Hz, etc. Audio sampling rate is a constant measure of the amount of data converted from a duration of audio input.

[0034] Audio data volume is usually expressed as the number of sampling points, but it can also be converted to the number of bytes based on the quantization bit depth (such as 16-bit or 24-bit).

[0035] In this embodiment, the electronic device first needs to determine the target playback tempo (usually expressed in beats per minute, BPM) and audio sampling rate (e.g., 44.1kHz, 48kHz, etc.). Based on these two parameters, the theoretical amount of audio data (usually expressed in the number of sample points or bytes) that needs to be played in a specific time period in the future (e.g., the next 100 milliseconds, 500 milliseconds, or a rhythm cycle) can be calculated.

[0036] For example, based on a target playback rhythm, an electronic device can analyze all planned vocal events to occur within an upcoming "future time period" (e.g., the next 100 milliseconds). Then, based on the "length of the future time period" and the "audio sampling rate," and using the formula audio bytes = duration × sampling rate × number of channels × bit depth bytes, it can precisely determine how many audio data samples are needed for that time period. For each vocal event within that time period, it can also calculate its specific data volume and its starting position within the total data volume based on the duration of its corresponding timbre sample.

[0037] For example, if the target playback tempo is 120 BPM (i.e., 0.5 seconds per beat) and the audio sampling rate is 48 kHz, then the number of sampling points that need to be played in the next beat is: 48000 × 0.5 = 24000 sampling points.

[0038] By precisely calculating the amount of audio data to be played in the future, this application provides a quantitative basis for the subsequent synthesis and buffering of audio data. By converting time control into data volume control, the timing precision of playback no longer depends on the triggering precision of the operating system's timers, but rather on the amount of audio data to be written to the buffer in this calculation. As long as the time length corresponding to this data volume is accurate, the timing of the data being played in the buffer is accurate. This fundamentally avoids the impact of timer jitter caused by system load and thread scheduling on the rhythm.

[0039] Step 120: Generate audio data to be played based on the original digital audio data of the target timbre and the corresponding audio data volume.

[0040] For example, the target timbre refers to the desired timbre characteristics of the final output sound, which can be defined by instrument samples, synthesizer parameters, sound models, user input, etc. The raw digital audio data of the target timbre refers to Pulse Code Modulation (PCM) data representing the target timbre that has not been processed by the system audio pipeline. For example, the complete waveform data of a single "clapping sound" or "piano key sound." As the "raw material," it is decoupled from the "rhythm," allowing the user to freely select and switch timbres. The audio data to be played refers to a complete digital audio signal that has already been processed (such as splicing, synthesis, effects rendering, etc.) and can be directly sent to a playback device for digital-to-analog conversion and playback.

[0041] In this embodiment, the electronic device can invoke an audio synthesis engine, a sample player, or a digital signal processing module to process the raw digital audio data according to the audio data volume calculated in the aforementioned steps. Processing methods may include, but are not limited to: extracting audio segments of corresponding length from a sample library, generating waveforms of a specified duration in real time using a synthesis algorithm, or performing speed, pitch, or mixing processing on the basic waveform to match playback rhythm and pitch requirements. This embodiment does not impose any limitations on these methods.

[0042] Then, based on the determined data volume framework and sound event layout, the corresponding raw digital audio data is filled into the appropriate positions to generate the audio data to be played, corresponding to the audio data volume. For example, if there is only one sound event in the future time period, the raw digital audio data of the corresponding timbre can be directly copied to the specified position. If there are multiple sound events that overlap in time, the sample values ​​corresponding to the raw digital audio data of each timbre need to be summed at each overlapping sample point for mixing. If there are no sound events in the future time period, a corresponding length of all-zero data (silence data) is generated.

[0043] Optionally, after performing the above summation, it is also possible to check whether the result value exceeds the representation range of the data format (e.g., -32768 to +32767 for a 16-bit signed integer). If it does, it is clamped to the boundary value to prevent harsh distortion caused by digital overflow. Furthermore, before mixing, a decay coefficient that increases over time can be introduced for each playing sound effect. The sampled values ​​are weighted before mixing, and when their impact is negligible, they are removed from the calculation to reduce computational load.

[0044] This step enables on-demand generation of audio data, ensuring that the length of the generated data precisely matches future playback requirements. Since all processing, including mixing, is completed at the application layer, the generated data is the final audio data, bypassing automated effects such as "fade-in / fade-out" and "equalization" that might be applied by the operating system or audio driver. This ensures the integrity and impact of short sound waveforms, achieving original sound quality reproduction. Simultaneously, users can freely combine different "raw digital audio data" (timbre) with "target playback tempo," which are then synthesized in real-time by this step, breaking the flexibility limitations imposed by pre-rendered fixed audio files.

[0045] Step 130: Write the audio data to be played into the underlying data buffer of the audio playback device ahead of the current playback progress of the audio playback device.

[0046] For example, an audio playback device refers to a hardware device or its driver program responsible for performing digital-to-analog audio signal conversion (DAC) and driving the output of speakers or headphones. The current playback progress refers to the playback time position corresponding to the data being read and converted in the buffer by the audio playback device. The underlying data buffer is a memory area directly managed by the audio playback device, used to temporarily store digitized audio data that will soon be read and played by the playback device.

[0047] In this embodiment, the electronic device can submit and write audio data to be played to the underlying data buffer in a time sequence that precedes the data point currently being played by the device, via the audio application programming interface (API) provided by the operating system. For example, steps 110 and 120 can be periodically triggered (e.g., every 100 milliseconds) by a separate "fill timer" or event, and the generated audio data to be played for a sufficient duration (e.g., 200 milliseconds) can be written to the buffer, forming a pipeline where "data supply" always leads "data consumption". By controlling the amount and timing of the written data, it is ensured that there is always data greater than zero in the buffer that is to be played. For example, the write strategy can be set so that the buffer always retains audio data for at least 100-200 milliseconds in the future.

[0048] The advance write mechanism in this application creates a data safety cushion. Even if there is a slight delay in the mechanism that triggers data filling, playback will not be interrupted as long as there is still data in the buffer. This transforms the risk of timing inaccuracies into a controllable problem of buffer data margin management. Since the final control is applied to the data stream written to the underlying buffer, and the behavior of audio playback devices consuming data at a fixed sampling rate is consistent across platforms, this method can fundamentally shield the differences in high-level scheduling and processing between different operating systems and different audio frameworks, enabling rhythmic sound effects to exhibit highly consistent stability and sound quality across various devices.

[0049] In this embodiment, a novel, self-closing audio playback control model is constructed through the synergistic effect of pre-calculating the amount of audio data required in the future, accurately generating audio data on demand, and writing to the underlying buffer in advance. This successfully shifts the precision of rhythm control from the unreliable software timing domain to the domain of precise mathematical calculation and stable hardware data consumption, thereby solving the three core problems of unstable rhythm, tampering with sound quality, and inability to customize existing technologies. It is particularly suitable for application scenarios with stringent requirements for timing and sound quality, such as metronomes, drum machines, and game sound effects. It can significantly improve the audio playback performance and user experience of software without increasing hardware costs.

[0050] Figure 2 This is a flowchart illustrating another audio data processing method provided in an embodiment of this application, which can be executed by an electronic device. Figure 2 As shown, the method may include the following steps:

[0051] Step 210: Start the timed write task; wherein, the timed write task includes a predetermined write cycle, the write cycle being less than the total playback duration of the underlying data buffer.

[0052] For example, a timed write task refers to a software task that executes periodically at fixed time intervals. Its function is not to directly trigger playback, but rather to periodically trigger subsequent data preparation and writing processes (steps 220-240). The timed write task can be driven by an operating system timer, a high-precision clock, or a dedicated audio thread. The write cycle refers to the time interval between two consecutive triggers of the timed write task, for example, 100 milliseconds. This cycle represents the rhythm at which the system actively replenishes data.

[0053] The total playback duration of the underlying data buffer refers to the continuous playback time that the underlying data buffer (usually a circular buffer) of the audio playback device can support when it is full of data. For example, a 44100Hz, stereo, 16-bit buffer with a size of 8192 samples has a total playback duration of approximately 8192 / (44100×2)≈93 milliseconds. In this embodiment, the write cycle is set to be less than the total playback duration to ensure that the amount of data consumed by the audio playback device within a single write cycle (at most equal to the total playback duration) will not exceed the buffer capacity, thereby preventing the data from being exhausted too quickly and causing playback interruption. At the same time, this also provides operational space for subsequent advance writing.

[0054] For example, an electronic device can create and start a high-precision timer (such as using a multimedia timer or high-resolution clock API provided by the operating system) during the initialization of the audio playback process, configuring its period as a write cycle. This write cycle must be set strictly less than the total playback duration of the underlying data buffer. For instance, if the total playback duration of the buffer is 1000 milliseconds, the write cycle can be set to 100 milliseconds or 200 milliseconds, etc.

[0055] By introducing periodically triggered timed write tasks, data writing that might have been event-driven or triggered at uncertain times is transformed into periodic tasks with a fixed rhythm. This makes the data flow of the entire system more controllable and predictable. This not only avoids excessively high CPU or I / O load peaks at a single point in time, making system resource usage smoother and more predictable, but also facilitates real-time personalized control by users. Furthermore, this application sets the write cycle to be less than the total playback time of the buffer, ensuring that a portion of the data written each time is played and consumed before the next write cycle is triggered. This maintains the buffer in a healthy state of being neither empty nor full, providing redundancy to cope with brief system stutters.

[0056] Step 220: When each timed write task is triggered, determine the amount of audio data based on the target playback rhythm, audio sampling rate, and write cycle.

[0057] For example, whenever the timer is triggered and the timed write task begins execution, the electronic device calculates the amount of audio data to be supplied next based on the current playback parameters and the write cycle. For instance, at the moment the timed write task is triggered, the electronic device reads the current target playback tempo and audio sampling rate. At this time, the "future time period" considered is equivalent to the write cycle. Based on these three parameters—target playback tempo, audio sampling rate, and write cycle—the amount of audio data expected to be played in the next write cycle can be calculated. Understandably, if the target playback tempo affects the playback rate (e.g., variable speed playback), a speed factor also needs to be included in the calculation; this embodiment does not impose such limitations.

[0058] Optionally, in one possible embodiment, determining the amount of audio data based on the target playback rhythm, audio sampling rate, and write cycle may include:

[0059] S1. Determine the start time, end time, and time length of the future time period to be written this time based on the execution progress of the timed write task and the write cycle;

[0060] S2. Determine the pronunciation events that will occur within the future time period according to the target playback rhythm and start and end times;

[0061] S3. Calculate the amount of audio data corresponding to the pronunciation events based on the audio sampling rate and time length.

[0062] Exemplarily, the execution progress of the timed write task refers to the number of times the timed task has been triggered for execution since it started, or the current logical time point aligned with the task cycle deduced based on the system absolute time. For example, if the write cycle is 100 milliseconds and it is the 5th trigger after the system starts, its execution progress can be characterized as the interval of the 400 - 500 milliseconds on the time axis.

[0063] The start and end times of the future time period define the start time (T_start) and end time (T_end) of the target time window on the playback time axis. The time length is ΔT = T_end - T_start. In the embodiments of the present application, this length may be equal to the write cycle.

[0064] The electronic device can maintain a logical clock or serial number synchronized with the audio playback time axis. Each time the timed task is triggered, according to the current execution progress (such as the Nth execution) and the fixed write cycle (P), calculate the future time window corresponding to this task. For example, T_start = N×P, T_end = (N + 1)×P. This window is the upcoming playback time period for which audio data needs to be prepared this time.

[0065] A pronunciation event refers to an instruction or mark that needs to emit a specific sound during audio playback. The electronic device pre - analyzes the target playback rhythm into a rhythm time series containing a series of pronunciation events. Each pronunciation event includes at least one absolute trigger timestamp and an associated target timbre identifier. Using the start and end times [T_start, T_end] of the future time period obtained in step S1 as the screening condition, traverse the rhythm time series, and optionally select all pronunciation events whose trigger timestamps fall within this interval (i.e., T_start ≤ trigger time < T_end). These screened pronunciation events are all the sounds that are scheduled to "sound" within the future time period corresponding to this write cycle.

[0066] Furthermore, based on the time length ΔT (seconds) obtained in step S1, the fixed audio sampling rate, number of channels, and sample bit depth, the total number of audio samples and bytes corresponding to the entire future time period can be calculated, thereby determining the overall size of the audio data block to be generated and played. Wherein, the total number of audio samples = time length × audio sampling rate × number of channels; the total number of bytes = total number of audio samples × sample bit depth.

[0067] Furthermore, the data volume and location corresponding to each pronunciation event can be calculated. For each pronunciation event determined in step S2, the corresponding raw digital audio data (a segment of PCM data) can be loaded according to the timbre identifier. If its sample duration is D_sample (seconds), the data volume of a complete pronunciation event is theoretically (sample duration D_sample × audio sampling rate × number of channels) samples. However, it is also necessary to consider whether it will be truncated by the boundary of future time periods (for example, if the event starts late, its samples may exceed T_end). The data volume corresponding to the pronunciation event can be calculated based on the effective segments / locations of the pronunciation event in future time periods, thereby improving the control accuracy. This application embodiment does not impose any limitations.

[0068] Understandably, if there are no vocal events in the future time period, the data generated this time will be silent data with a length equal to the total data volume (all sample values ​​are zero).

[0069] This optional embodiment constructs a complete technical chain, from periodic timed triggering to time window planning, then to rhythmic event filtering, and finally to accurate time-to-data-volume conversion, effectively transforming time control into data-volume control. It enables the audio playback system to pre-prepare precise audio data for any future playback time in a deterministic manner, thereby shifting the stability of playback timing from relying on unreliable software scheduling to relying on precise mathematical calculations and stable hardware consumption, fundamentally solving the cross-platform stability problem of rhythmic sound effect playback.

[0070] Step 230: Generate audio data to be played based on the original digital audio data of the target timbre and the corresponding audio data volume.

[0071] For example, in this embodiment of the application, when the audio data volume is determined to be periodically triggered by the timed write task, the generation of audio data to be played and the write operation will occur at the same rhythm as the write cycle (e.g., every 100ms). When generating the audio data to be played corresponding to the audio data volume based on the original digital audio data of the target timbre, different processing methods can be adopted according to the number of pronunciation events existing in the future time period.

[0072] Optionally, in one possible embodiment, generating playback audio data corresponding to the amount of audio data based on the original digital audio data of the target timbre may include:

[0073] S10. When there is only one sound event in the future time period, fill the original digital audio data of the target timbre corresponding to the sound event into the data position corresponding to the amount of audio data to generate the audio data to be played.

[0074] S20. When multiple sound events exist within a future time period, determine the temporally overlapping sample intervals of the original digital audio data of the target timbre corresponding to each sound event; within the overlapping sample intervals, add the sample values ​​of the original digital audio data corresponding to each sound event to generate mixed data as the audio data to be played in that sample interval; within the non-overlapping sample intervals, fill the corresponding data positions with the original digital audio data of the sound event as the audio data to be played in that sample interval.

[0075] S30. When there are no sound events in the future time period, the generated audio data to be played is silent data with a length equal to the data volume corresponding to the future time period.

[0076] For example, if only one sound effect needs to be triggered within the future time period of this processing, its starting sample index in the final output data block can be calculated based on the trigger time offset of the sound event. Then, starting from that index position, the original digital audio data (a segment of PCM data) of the associated target timbre is directly copied to the audio data buffer allocated for this time period. If the duration of the timbre sample exceeds the remaining part of the future time period, it is truncated; if the duration is insufficient, the subsequent part remains at the initial value (usually zero). By directly filling in the data, unnecessary mixing computation overhead is avoided, the processing latency is extremely low, and the purity of the original timbre is guaranteed not to be interfered with by other operations. This provides optimal performance for the system when processing single-note melodies and simple sound effects.

[0077] If multiple sound events are triggered within the future timeframe of this processing, resulting in a complex scenario with multiple sound effects playing simultaneously or interleaved, autonomous mixing processing is required. Specifically, the electronic device can calculate the coverage range of each sound event on the time axis based on the starting sample index and the length of its timbre data. By comparing these ranges, the sample intervals where two or more timbre data are simultaneously valid are precisely identified. At each overlapping sample point, the sample values ​​of each timbre data at that point are algebraically summed. For example, for a 16-bit signed PCM format, the two sample values ​​are directly added, and the resulting mixed data is used as the audio data to be played in that sample interval. In intervals where only one timbre data is valid, the processing method is the same as in S10: direct filling is performed, filling the corresponding data position with the original digital audio data of the corresponding sound event, which is used as the audio data to be played in that sample interval.

[0078] Optionally, in one possible embodiment, after summing the sampled values ​​of the original digital audio data corresponding to each pronunciation event, the method provided in this application embodiment further includes:

[0079] S21. Check whether the sampled value after addition exceeds the preset numerical representation range;

[0080] S22. If the range is exceeded, the summed sampled value will be set as the boundary value of the numerical representation range.

[0081] For example, within the overlapping sample intervals, the sampled values ​​of the original digital audio data corresponding to each sound event are summed. After obtaining the summed result, the electronic device can also check whether the summed sampled value exceeds the representation range of the PCM format (e.g., -32768 to +32767 of a 16-bit signed integer). If it does, amplitude limiting processing is required, that is, setting it to the boundary value (maximum or minimum value) of the range to prevent distortion.

[0082] Optionally, in one possible embodiment, before summing the sampled values ​​of the original digital audio data corresponding to each pronunciation event, the method provided in this application embodiment may further include:

[0083] S01. Determine the current attenuation coefficient of the sound event; wherein, the attenuation coefficient increases with time from the start of writing the sound event, and when the attenuation coefficient reaches a preset threshold, stop using the original digital audio data corresponding to the sound event;

[0084] S02. Based on the current attenuation coefficient of the sound event, perform attenuation weighting processing on the sampled values ​​of the original digital audio data corresponding to the sound event.

[0085] For example, the attenuation coefficient is a numerical parameter used to quantify the degree of loudness decay of a sound event. Its initial value is typically set to 1.0 (representing no attenuation, full loudness), or it can be set according to the characteristics of the sound effect. This coefficient is specifically used for the mixed weighting of audio data and is independent of the audio file format. In each audio processing cycle (e.g., each time a timed write task is triggered), the system updates the attenuation coefficients of all active sound events. The increasing pattern can be linear (e.g., increasing by a fixed step size each time), exponential, or according to a preset envelope curve; this embodiment of the application does not impose any limitations.

[0086] For example, for a continuous sound effect, its attenuation coefficient might start at 1.0, increasing by 0.1 every 100 milliseconds until it reaches a preset threshold. The preset threshold is a predefined critical value; when the attenuation coefficient reaches or exceeds this value, the sound event is considered "over" or its contribution to the final mix is ​​negligible. For example, the threshold could be set to 10.0, or the coefficient corresponding to the value that makes the absolute value of the weighted sample values ​​less than a certain minimum audible threshold. When the attenuation coefficient of a sound event reaches the threshold, the system will no longer read, process, or mix any original digital audio data corresponding to that event in all subsequent audio data generation cycles.

[0087] Once the current attenuation coefficient of the sound event is determined, for each sample value of the original digital audio data corresponding to the sound event, before identifying any temporal overlap requiring sample value addition (mixing), it is first multiplied by a weighting factor. This weighting factor is determined by the current attenuation coefficient. Typically, the weighting factor is the reciprocal of the attenuation coefficient or a related attenuation function value, ensuring that a larger coefficient results in a smaller output loudness. For example, weight = 1.0 / attenuation coefficient.

[0088] For long-duration sound effects, as the attenuation coefficient increases, the amplitude of the weighted sample value decreases rapidly. When the amplitude becomes small enough, its numerical contribution to the mixing result is negligible even without immediate cessation. This provides a basis for introducing approximate calculations or early termination processing, thereby significantly reducing continuous and meaningless full-precision calculations throughout the long sound effect's lifespan and optimizing the utilization of computing resources. Simultaneously, it reduces the risk of digital overflow when multiple large sample values ​​are directly added, serving as a pre-processing supplement to "anti-distortion processing" and further improving output sound quality. Furthermore, continuous attenuation weighting ensures that the end of the sound effect sounds natural and gradual, rather than abruptly interrupted, enhancing the user experience.

[0089] For example, if no sound events occur within the future time period of this processing, i.e., a silent segment occurs, the required total number of samples can be calculated directly based on parameters such as the duration ΔT of the future time period and the sampling rate. Then, a memory area of ​​the corresponding size can be allocated and initialized to zero to generate silent data of equal size. In this way, silent data is still written to the underlying buffer even when there is no sound effect playback, ensuring that the audio hardware always has data to read, completely avoiding playback stagnation, popping sounds, or abnormal release of the system audio channel caused by "data interruption".

[0090] This optional embodiment covers all possible playback states through three paths: single-event filling, multi-event mixing, and event-free mute, collectively forming a complete and robust application-layer audio synthesis engine. In particular, by moving the crucial mixing operation from the uncontrollable system layer to the application layer, it solves the problem of inconsistent sound effects across platforms.

[0091] Step 240: Write the audio data to be played into the underlying data buffer of the audio playback device ahead of the current playback progress of the audio playback device.

[0092] For example, within each write cycle, after data generation is complete, the electronic device can append these small chunks of audio data to be played into the underlying data buffer of the audio playback device via the audio API. The write position always precedes the device's current playback progress, ensuring a continuous data stream in the buffer.

[0093] By combining a periodic writing mechanism, the system achieves micro-level, high-frequency replenishment of data in the buffer, ensuring that the amount of data in the buffer always fluctuates within a controllable small range. This allows the system to be more sensitive to changes in playback status (such as a user suddenly adjusting the rhythm) and respond quickly, adjusting the data generation strategy in subsequent cycles, thus improving the system's interactivity and adaptability.

[0094] Optionally, in one possible embodiment, writing the audio data to be played into the underlying data buffer of the audio playback device ahead of the current playback progress of the audio playback device may include: after writing the audio data to be played into the underlying data buffer ahead of the current playback progress of the audio playback device, the playback end time point corresponding to the data already written but not played in the underlying data buffer is at least one writing cycle ahead of the current playback time point of the audio playback device.

[0095] For example, the current playback time of the audio playback device refers to the absolute playback moment corresponding to the audio data sample that the audio hardware is converting into an analog signal. Data written to the buffer but not yet played refers to data located between the current read and write pointers; this data has been submitted by the application but has not yet been consumed by the audio hardware. The playback end time refers to the future moment corresponding to the point in time when the audio hardware, assuming it starts from the current moment and continuously consumes all existing but unplayed data in the buffer, finishes consuming the last sample.

[0096] Each time a scheduled write task is triggered, the electronic device can not only calculate the amount of data to be generated, but also predict or calculate the future time point at which the buffer data will be reached after this batch of new data is written. By controlling the duration of the generated and written audio data, the lead time can be controlled. For example, if the write cycle of the scheduled write task is 100 milliseconds, then at 0 milliseconds, the amount of data that can be played for 200 milliseconds can be written in advance; at 100 milliseconds, the data that needs to be played for 200 to 300 milliseconds can be written, and so on. The written audio data always leads the actual playback time.

[0097] Understandably, since audio data cannot be undone once it is written to the audio buffer, the write cycle of the timed write task cannot be too long in order to respond promptly to user operations (such as modifying the target playback rhythm, reselecting the target timbre, etc.).

[0098] This optional embodiment limits the data written to the underlying data buffer to be at least one write cycle ahead. This ensures stability and consistency across different systems and hardware, provided the timer error for the scheduled write task does not exceed one write cycle, while also maintaining response speed. By transforming the "timer accuracy" problem, which is unsolvable in traditional solutions, into an easily manageable and guaranteed "buffer data volume maintenance" problem, unprecedented, near-hardware-limit-level playback stability and timing accuracy are provided for rhythmic sound effects in various complex and heterogeneous hardware and software environments.

[0099] In this embodiment, a timed write task is introduced, which drives a complete "time-to-data conversion" and "direct underlying write" process. On the one hand, it fundamentally solves the problem of inaccurate rhythmic sound effect playback caused by the instability of the system timer. By precisely converting playback timing control into audio data volume calculation based on a fixed sampling rate, and ensuring that the playback end time corresponding to the data written to the underlying buffer always leads the current playback progress, the final playback timing depends only on the arrangement of data in the buffer and the hardware sampling rate. This completely decouples it from upstream task scheduling jitter and achieves sub-millisecond rhythm accuracy. On the other hand, by directly writing the raw audio data to the underlying buffer, it bypasses the automatic processing that the operating system's audio service may impose, ensuring the originality and impact of short sound effect waveforms and solving the problem of sound quality tampering. In addition, by autonomously performing mixing, anti-distortion, and attenuation processing at the application layer, and using the target playback rhythm and target timbre as flexibly configurable input parameters, it perfectly supports arbitrary user customization of timbre and rhythm while ensuring cross-platform playback stability and sound quality consistency, overcoming the lack of flexibility of pre-rendered fixed audio files.

[0100] For example, in practical applications, the audio data processing method provided in the embodiments of this application can be divided into the following three stages.

[0101] During system initialization, after the user completes parameter settings (including but not limited to timbre, playback tempo, and write cycle), the electronic device can parse one or more WAV format audio files specified by the user and decode them into raw pulse-code modulation data upon application startup. To simplify subsequent processing, all audio files must have uniform format attributes, such as stereo, 16-bit sampling depth, and a 44100Hz sampling rate. Furthermore, a target write cycle T_cycle (e.g., 50 milliseconds) is defined, representing the ideal time interval at which the system plans to replenish the audio data to be played. A state variable T_last_fill is initialized to record the last successful write of data to the audio buffer. Simultaneously, the audio playback device is initialized, setting its output attributes (number of channels, sampling rate, bit depth) to be completely consistent with the aforementioned audio files to ensure data format matching and avoid unnecessary resampling or conversion.

[0102] Then, the initial calculation, generation, and writing of data begins. During the initial calculation, based on the initial T_last_fill and the target playback tempo, the amount and content of audio data to be played within the first T_cycle time period are calculated. To quickly establish a safe data buffer, during the initial write, audio data equivalent to at least two T_cycles is written to the underlying data buffer of the audio playback device. For example, if T_cycle is 50 milliseconds, 100 milliseconds of audio data are written initially, ensuring that the amount of data in the underlying data buffer is ahead of the audio hardware's playback consumption from the outset. After the initial write is complete, T_last_fill is updated to the current time.

[0103] Further, the main cycle of periodic data supply is entered, for example, through a timed write task. At each polling trigger, the following logical judgments and operations are executed: (1) Calculate the actual interval: Obtain the current system time T_now and calculate the actual time interval since the last fill: ΔT_actual = T_now - T_last_fill. (2) Interval judgment: Check if ΔT_actual is greater than or equal to the preset target write cycle T_cycle (i.e., 50 milliseconds). If not (ΔT_actual < T_cycle): It means the planned data replenishment point has not yet been reached. This poll will not perform any operation and will return directly, waiting for the next poll. If yes (ΔT_actual ≥ T_cycle): It means the planned replenishment point has been reached or exceeded, and a data filling process should be executed immediately. (3) Execute the data filling process: a. Update the time base: Update the recorded last filling time to T_last_fill = T_last_fill + T_cycle (or update to T_now); this ensures that the internal time base of the system is based on successful data supply events, rather than unreliable external timing signals. b. Calculate the amount of data: Based on the updated T_last_fill and T_cycle, determine the next future time period [T_last_fill, T_last_fill + T_cycle]. According to the target playback rhythm, parse out all the pronunciation events that need to be triggered within this time period. Combined with the audio sampling rate, accurately calculate the total amount of audio data required to play these events, as well as the specific position (sample offset) of each event data in the total data block. c. Synthesize audio data: Based on the calculation results of step b, extract timbre samples from the corresponding original digital audio data. If there are multiple pronunciation events within this time period and the times overlap, then in the overlapping sample interval, add the sample values ​​of each timbre sample (mixing) and perform anti-digital overflow processing (anti-distortion). For non-overlapping portions or single events, the corresponding data is directly filled in. If there are no vocal events within the time period, silence data (all zeros) of the corresponding length is generated. During this process, attenuation coefficients can be applied to vocal events with longer durations to optimize computational performance. d. Write to the buffer: The final synthesized audio data to be played is written to the underlying data buffer of the audio playback device. After one filling operation is completed, the process returns to the polling waiting state and continues to run in a loop. When the user performs an operation (such as changing the rhythm, switching the timbre, etc.), the new instruction is recorded and immediately adopted in the next or current data calculation step (b), thus affecting the subsequently generated audio content in real time.

[0104] During the loop phase, the actual data filling time, rather than the timer trigger time, is used as the internal timing benchmark. The filling timing is determined by frequently polling to compare the difference between the actual interval and the target period. This completely absorbs the jitter of the external timing mechanism. Even if polling is delayed due to system busy, as long as the data is filled after the delay, the continuity of audio playback (guaranteed by buffer margin) and the timing accuracy (determined by the sample arrangement of the filled data) remain unaffected. Combined with the advance writing strategy that always maintains at least one T_cycle of data lead, this embodiment can achieve highly stable, low-latency, and customizable rhythmic sound effect playback even on resource-constrained platforms or platforms with variable system loads.

[0105] Figure 3 This is a schematic diagram of an audio data processing device provided in an embodiment of this application. Figure 3 As shown, the device 30 includes: a first processing unit 301, a second processing unit 302, and a third processing unit 303.

[0106] The first processing unit 301 is used to determine the amount of audio data based on the target playback rhythm and audio sampling rate; wherein, the amount of audio data is the amount of data to be played in the future time period;

[0107] The second processing unit 302 is used to generate audio data to be played that corresponds to the amount of audio data based on the original digital audio data of the target timbre.

[0108] The third processing unit 303 is used to write the audio data to be played into the underlying data buffer of the audio playback device ahead of the current playback progress of the audio playback device.

[0109] In one alternative embodiment, the first processing unit 301 is specifically used for:

[0110] Start a timed write task; the timed write task includes a predetermined write cycle, which is less than the total playback duration of the underlying data buffer;

[0111] When each timed write task is triggered, the amount of audio data is determined based on the target playback rhythm, audio sampling rate, and write cycle.

[0112] In one alternative embodiment, the first processing unit 301 is specifically used for:

[0113] Based on the execution progress and writing cycle of the scheduled write task, determine the start and end times and duration of the future time period to be written.

[0114] Based on the target playback rhythm and start and end times, determine the pronunciation events that will occur in the future time period;

[0115] Based on the audio sampling rate and duration, the amount of audio data corresponding to the pronunciation event is calculated.

[0116] In one alternative embodiment, the second processing unit 302 is specifically used for:

[0117] When there is only one sound event in the future time period, the original digital audio data of the target timbre corresponding to the sound event is filled into the data position corresponding to the amount of audio data to generate the audio data to be played.

[0118] When multiple sound events occur within a future time period, determine the temporally overlapping sample intervals of the original digital audio data of the target timbre corresponding to each sound event; within the overlapping sample intervals, add the sample values ​​of the original digital audio data corresponding to each sound event to generate mixed data as the audio data to be played in that sample interval; within the non-overlapping sample intervals, fill the corresponding data positions with the original digital audio data of the sound event as the audio data to be played in that sample interval.

[0119] When there are no sound events in the future time period, the generated audio data to be played is silent data with a length equal to the data volume corresponding to the future time period.

[0120] In one alternative embodiment, after summing the sampled values ​​of the original digital audio data corresponding to each phonation event, the second processing unit 302 further performs the following:

[0121] Check whether the summed sampled values ​​exceed the preset numerical representation range;

[0122] If the range is exceeded, the summed sampled value will be set as the boundary value of the numerical representation range.

[0123] In one alternative embodiment, the second processing unit 302, before summing the sampled values ​​of the original digital audio data corresponding to each phonation event, is further configured to:

[0124] Determine the current attenuation coefficient of the sound event; wherein, the attenuation coefficient increases over time from the start of writing the sound event, and when the attenuation coefficient reaches a preset threshold, stop using the original digital audio data corresponding to the sound event;

[0125] Based on the current attenuation coefficient of the sound event, the sampled values ​​of the original digital audio data corresponding to the sound event are subjected to attenuation weighting processing.

[0126] In one alternative embodiment, the third processing unit 303 is specifically used for:

[0127] Before the current playback progress of the audio playback device, after writing the audio data to be played into the underlying data buffer, the playback end time corresponding to the data already written to but not yet played in the underlying data buffer is at least one write cycle ahead of the current playback time of the audio playback device.

[0128] As can be seen from the above, the audio data processing device provided in this application embodiment constructs a novel, self-closing audio playback control model through the synergistic effect of pre-calculating the amount of audio data required in the future, accurately generating audio data on demand, and writing it into the underlying buffer in advance. It successfully transfers the precision of rhythm control from the unreliable software timing domain to the domain of precise mathematical calculation and stable hardware data consumption, thereby solving the three core problems of unstable rhythm, tampering with sound quality, and inability to customize existing technologies. It is especially suitable for application scenarios with strict requirements for timing and sound quality, such as metronomes, drum machines, and game sound effects. It can significantly improve the audio playback performance and user experience of software without increasing hardware costs.

[0129] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0130] like Figure 4 As shown, the electronic device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0131] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other network elements such as clients or other servers. The processor 402 executes program 410, specifically performing the relevant steps described in the above embodiment of the audio data processing method.

[0132] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0133] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The electronic device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0134] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0135] Specifically, program 410 can be called by processor 402 to enable the electronic device to execute the relevant steps in the above-described audio data processing method embodiment.

[0136] This application provides a computer-readable storage medium storing at least one executable instruction that, when executed on an electronic device / app, causes the electronic device / app to perform the audio data processing method in any of the above method embodiments.

[0137] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Furthermore, the embodiments in this application are not directed to any particular programming language.

[0138] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. Similarly, for the purpose of simplification and aiding understanding of one or more aspects of the invention, in the above description of exemplary embodiments of this application, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0139] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

[0140] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for processing audio data, characterized in that, The method includes: The amount of audio data is determined based on the target playback rhythm and audio sampling rate; wherein, the amount of audio data is the amount of data to be played in the future time period; Based on the original digital audio data of the target timbre, generate audio data to be played that corresponds to the amount of audio data; The audio data to be played is written to the underlying data buffer of the audio playback device ahead of the current playback progress of the audio playback device.

2. The method according to claim 1, characterized in that, The process of determining the amount of audio data based on the target playback rhythm and audio sampling rate includes: Start a timed write task; wherein the timed write task includes a predetermined write cycle, the write cycle being less than the total playback duration of the underlying data buffer; Each time the timed write task is triggered, the amount of audio data is determined based on the target playback rhythm, the audio sampling rate, and the write cycle.

3. The method according to claim 2, characterized in that, The process of determining the amount of audio data based on the target playback rhythm, the audio sampling rate, and the writing cycle includes: Based on the execution progress of the timed write task and the write cycle, the start and end times and duration of the future time period to be written are determined. Based on the target playback rhythm and the start and end times, determine the pronunciation events that will occur within the future time period; Based on the audio sampling rate and the time length, the amount of audio data corresponding to the pronunciation event is calculated.

4. The method according to claim 1, characterized in that, The step of generating audio data to be played corresponding to the amount of audio data based on the original digital audio data of the target timbre includes: When there is only one sound event within the future time period, the original digital audio data of the target timbre corresponding to the sound event is filled into the data position corresponding to the amount of audio data to generate the audio data to be played. When multiple pronunciation events occur within the future time period, the sample intervals where the original digital audio data of the target timbre corresponding to each pronunciation event overlap in time are determined; within the overlapping sample intervals, the sample values ​​of the original digital audio data corresponding to each pronunciation event are added together to generate mixed data as the audio data to be played in that sample interval; within the non-overlapping sample intervals, the original digital audio data of the corresponding pronunciation event is filled into the corresponding data position as the audio data to be played in that sample interval. When there are no sound events within the future time period, the generated audio data to be played is silent data with a length equal to the data volume corresponding to the future time period.

5. The method according to claim 4, characterized in that, After summing the sampled values ​​of the original digital audio data corresponding to each pronunciation event, the method further includes: Check whether the summed sampled values ​​exceed the preset numerical representation range; If the value exceeds the limit, the summed sampled value is set as the boundary value of the numerical representation range.

6. The method according to claim 4, characterized in that, Before summing the sampled values ​​of the original digital audio data corresponding to each pronunciation event, the method further includes: Determine the current attenuation coefficient of the sound event; wherein the attenuation coefficient increases over time from the start of writing the sound event, and when the attenuation coefficient reaches a preset threshold, stop using the original digital audio data corresponding to the sound event; Based on the current attenuation coefficient of the sound event, the sampled values ​​of the original digital audio data corresponding to the sound event are subjected to attenuation weighting processing.

7. The method according to any one of claims 1-6, characterized in that, The step of writing the audio data to be played into the underlying data buffer of the audio playback device, which is ahead of the current playback progress of the audio playback device, includes: Before the current playback progress of the audio playback device, after the audio data to be played is written to the underlying data buffer, the playback end time corresponding to the data already written to but not played in the underlying data buffer is at least one writing cycle ahead of the current playback time of the audio playback device.

8. An audio data processing apparatus, characterized in that, The device includes: The first processing unit is used to determine the amount of audio data based on the target playback rhythm and audio sampling rate; wherein, the amount of audio data is the amount of data to be played in the future time period; The second processing unit is used to generate audio data to be played that corresponds to the amount of audio data, based on the original digital audio data of the target timbre. The third processing unit is used to write the audio data to be played into the underlying data buffer of the audio playback device ahead of the current playback progress of the audio playback device.

9. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the audio data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on an electronic device, causes the electronic device to perform the operation of the audio data processing method as described in any one of claims 1-7.

Images