A multi-track audio buffering method, apparatus, device and readable storage medium

Abstract

The application discloses a kind of multi-track audio buffering method, device, equipment and readable storage medium, applied to digital audio buffering technical field, comprising: obtaining initial multi-track audio accompaniment data;Utilize decoder to carry out decoding processing to initial multi-track audio accompaniment data, obtain multiple decoded audio accompaniment data, and utilize first decoding buffer pool to store multiple decoded audio accompaniment data;From first decoding buffer pool, obtain target decoded audio accompaniment data, utilize first mixer to process target decoded audio accompaniment data, obtain mixed multi-track audio accompaniment data;Utilize player to play mixed multi-track audio accompaniment data.Compared with current directly buffering directly playing to multi-track audio, the application is decoded to multi-track audio, and decoded data is stored in first buffer pool, so that decoded audio accompaniment data can be directly obtained from first buffer pool, so that subsequent waiting time is reduced by buffering, and the fluency of audio playing is improved.

Description

Technical Field

[0001] This invention relates to the field of digital audio buffering technology, and in particular to a multitrack audio buffering method, apparatus, device, and readable storage medium. Background Technology

[0002] Audio decoding and playback speeds vary across different devices. Due to limitations in device performance and operating environment, some devices may have significantly different decoding and playback speeds, easily causing stuttering and interruptions. Therefore, audio playback usually requires buffering to achieve a balance between decoding and playback, reducing the perceived stuttering in audio output. However, current solutions only provide single-level buffering for single-track audio, and there is no relevant technology for buffering multi-track audio.

[0003] Therefore, there is a need to provide a technical solution for buffering multi-track audio to improve the playback effect of multi-track audio. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a multitrack audio buffering method, apparatus, device and readable storage medium, which solves the technical problem of poor playback effect of multitrack audio in the prior art.

[0005] To solve the above-mentioned technical problems, the present invention provides a multi-track audio buffering method, comprising:

[0006] Obtain initial multitrack audio accompaniment data;

[0007] The initial multitrack audio accompaniment data is decoded using a decoder to obtain multiple decoded audio accompaniment data, and the multiple decoded audio accompaniment data are stored in a first-level decoding buffer pool; wherein, the number of decoders is the same as the number of tracks in the initial multitrack audio accompaniment data;

[0008] The target decoded audio accompaniment data is obtained from the first-level decoding buffer pool, and the target decoded audio accompaniment data is processed by the first mixer to obtain mixed multitrack audio accompaniment data.

[0009] The mixed multitrack audio accompaniment data is played using a player.

[0010] Optionally, the step of using a decoder to decode the initial multitrack audio accompaniment data to obtain multiple decoded audio accompaniment data, and storing the multiple decoded audio accompaniment data using a first-level decoding buffer pool, includes:

[0011] Obtain the remaining storage space of the first-level decoding buffer pool;

[0012] If the remaining storage space is sufficient to store a preset number of audio data frames, the decoder is used to decode the initial multitrack audio accompaniment data to obtain the multiple decoded audio accompaniment data, and the primary decoding buffer pool is used to store the multiple decoded audio accompaniment data.

[0013] If the remaining storage space is insufficient to store the preset number of audio frames, decoding will stop until the remaining storage space of the first-level decoding buffer pool is sufficient to store the preset number of audio frames, at which point a command to start decoding will be sent.

[0014] Optionally, the step of obtaining the target decoded audio accompaniment data from the first-level decoding buffer pool and processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data includes:

[0015] Obtain the lengths of all frames corresponding to all the target decoded audio accompaniment data in the first-level decoding buffer pool;

[0016] Determine whether the length of all frames reaches the preset mixing frame length;

[0017] When the length of all frames reaches the preset mixing processing frame length, the target decoded audio accompaniment data is uniformly obtained from the first-level decoding buffer pool;

[0018] The target decoded audio accompaniment data is processed using the first mixer to obtain the mixed multitrack audio accompaniment data.

[0019] Optionally, after obtaining the target decoded audio accompaniment data from the first-level decoding buffer pool and processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data, the method further includes:

[0020] Get or create a secondary mixing buffer pool;

[0021] The secondary mixing buffer pool is used to store the mixed multitrack audio accompaniment data and / or the target decoded audio accompaniment data that reaches the target frame length.

[0022] Optionally, after obtaining the target decoded audio accompaniment data from the first-level decoding buffer pool and processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data, the method further includes:

[0023] The mixed multitrack audio accompaniment data is processed with sound effects to obtain the target audio accompaniment data;

[0024] Accordingly, playing the mixed multitrack audio accompaniment data using a player includes:

[0025] The target audio accompaniment data is played using the player.

[0026] Optionally, after obtaining the target decoded audio accompaniment data from the first-level decoding buffer pool, processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data, the method further includes:

[0027] Obtain the target mixed multitrack audio accompaniment data;

[0028] The target mixed multitrack audio accompaniment data is stored in a playback buffer pool;

[0029] Accordingly, playing the mixed multitrack audio accompaniment data using a player includes:

[0030] The target mixed multitrack audio accompaniment data is obtained from the playback buffer pool, and the target mixed multitrack audio accompaniment data is played using the player.

[0031] Optionally, storing the target mixed multitrack audio accompaniment data in a playback buffer pool includes:

[0032] Acquire the voice buffer data in the voice buffer pool, as well as the target mixed multitrack audio accompaniment data;

[0033] The target mixed multitrack audio accompaniment data is stored as the accompaniment for the vocal buffer data in the playback buffer pool.

[0034] Optionally, before acquiring the voice buffer data in the voice buffer pool, the method further includes:

[0035] Acquire initial human voice data;

[0036] The initial human voice data is processed with different types of sound effects to obtain multiple processed human voice data.

[0037] The second mixer is used to process the multiple audio effect processing voice data to obtain mixed processing voice buffer data;

[0038] The human voice buffer pool is used to store the mixed-process human voice buffer data.

[0039] Optionally, storing the target mixed multitrack audio accompaniment data as the accompaniment for the vocal buffer data in the playback buffer pool includes:

[0040] The target mixed multitrack audio accompaniment data is used as accompaniment and, after being aligned with the vocal buffer data, is stored in the playback buffer pool.

[0041] The present invention also provides a multitrack audio buffer device, comprising:

[0042] The initial multitrack audio accompaniment data acquisition module is used to acquire the initial multitrack audio accompaniment data;

[0043] The decoding buffer module is used to decode the initial multitrack audio accompaniment data using a decoder to obtain multiple decoded audio accompaniment data, and to store the multiple decoded audio accompaniment data using a first-level decoding buffer pool; wherein, the number of decoders is the same as the number of tracks in the initial multitrack audio accompaniment data;

[0044] The mixing module is used to obtain target decoded audio accompaniment data from the first-level decoding buffer pool, and process the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data.

[0045] The playback module is used to play the mixed multitrack audio accompaniment data using a player.

[0046] The present invention also provides a multitrack audio buffer device, comprising:

[0047] Memory, used to store computer programs;

[0048] A processor for implementing the above-described multitrack audio buffering method when executing the computer program.

[0049] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the above-described multitrack audio buffering method.

[0050] As can be seen, this invention acquires initial multi-track audio accompaniment data; decodes the initial multi-track audio accompaniment data using a decoder to obtain multiple decoded audio accompaniment data, and stores these multiple decoded audio accompaniment data using a first-level decoding buffer pool; wherein the number of decoders is consistent with the number of tracks in the initial multi-track audio accompaniment data; obtains target decoded audio accompaniment data from the first-level decoding buffer pool, processes the target decoded audio accompaniment data using a first mixer to obtain mixed multi-track audio accompaniment data; and plays the mixed multi-track audio accompaniment data using a player. Compared with current methods that do not buffer multi-track audio, this invention can utilize a first-level decoding buffer pool to store multiple decoded audio accompaniment data, fully leveraging the performance of a multi-core CPU (Central Processing Unit / Processor) through buffering, reducing subsequent waiting time, and improving the smoothness of subsequent audio playback.

[0051] In addition, the present invention also provides a multitrack audio buffer device, apparatus and readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0053] Figure 1 A flowchart of a multitrack audio buffering method provided in an embodiment of the present invention;

[0054] Figure 2 A flowchart of another multitrack audio buffering method provided in an embodiment of the present invention;

[0055] Figure 3 A flowchart illustrating another multitrack audio buffering method provided in an embodiment of the present invention;

[0056] Figure 4 A flowchart illustrating a multitrack audio buffering method provided in an embodiment of the present invention;

[0057] Figure 5 This is a flowchart illustrating the overall process framework of a multitrack audio buffering method provided in an embodiment of the present invention.

[0058] Figure 6 A schematic diagram of a first-level cache decoding buffer provided in an embodiment of the present invention;

[0059] Figure 7 A schematic diagram of a two-stage mixing buffer provided in an embodiment of the present invention;

[0060] Figure 8 A schematic diagram of a two-stage voice buffer provided in an embodiment of the present invention;

[0061] Figure 9 A schematic diagram of a three-level playback buffer provided in an embodiment of the present invention;

[0062] Figure 10 This is a schematic diagram of a multi-buffered, multi-track audio decoding and playback flow field provided in an embodiment of the present invention;

[0063] Figure 11 This is a schematic diagram of a three-level buffer processing mode provided in an embodiment of the present invention;

[0064] Figure 12 This is a schematic diagram of the structure of a multitrack audio buffer device provided in an embodiment of the present invention;

[0065] Figure 13This is a schematic diagram of the structure of a multitrack audio buffer device provided in an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Please refer to Figure 1 , Figure 1 A flowchart illustrating a multitrack audio buffering method provided in an embodiment of the present invention. The method may include:

[0068] S100, acquire initial multitrack audio accompaniment data.

[0069] The execution subject of this embodiment is a smart device, which can be a computer, tablet, mobile phone, etc.

[0070] S101, the initial multitrack audio accompaniment data is decoded using a decoder to obtain multiple decoded audio accompaniment data, and the multiple decoded audio accompaniment data are stored in a first-level decoding buffer pool; wherein, the number of decoders is consistent with the number of tracks in the initial multitrack audio accompaniment data.

[0071] The decoder in this embodiment is an audio decoder. This embodiment is not limited to a specific decoder, as long as it can decode the initial multi-track audio accompaniment data. The decoder in this embodiment can be a digital audio signal decoder based on an FPGA (Field-Programmable Gate Array); or it can be a context-based entropy decoder. This embodiment can use the audio decoder to decode the initial multi-track audio accompaniment data to obtain multiple decoded audio accompaniment data. This embodiment does not limit the number of frames that the decoder can decode at any given time. For example, the decoder can decode one frame of data at a time; or it can decode two frames of data at a time. It is understood that in this embodiment, there are N buffer pools in the first-level decoding buffer pool, where N refers to the number of audio tracks in the audio file. Each audio track corresponds to one decoder and one buffer pool.

[0072] It should be noted that the above-mentioned decoding process of the initial multi-track audio accompaniment data using a decoder to obtain multiple decoded audio accompaniment data, and storing these multiple decoded audio accompaniment data in a primary decoding buffer pool, may include: obtaining the remaining storage space of the primary decoding buffer pool; if the remaining storage space is sufficient to store a preset number of audio data frames, then the decoder is used to decode the initial multi-track audio accompaniment data to obtain multiple decoded audio accompaniment data frames, and these multiple decoded audio accompaniment data frames are stored in the primary decoding buffer pool; if the remaining storage space is insufficient to store the preset number of audio data frames, then decoding is stopped until the remaining storage space of the primary decoding buffer pool is sufficient to store the preset number of audio data frames, at which point a command to start decoding is sent. In this embodiment, the decoder decodes one frame of audio data each time. When the remaining space in the buffer pool is sufficient to store one frame of audio data, it notifies the decoding thread to submit a decoding task to the thread pool, and the decoded audio data is stored in the buffer pool. When the remaining space in the buffer pool is insufficient to store one frame of audio data, no decoding task is submitted; instead, the buffer pool remains dormant, waiting for the data in the buffer pool to be consumed to free up space. This embodiment determines whether to continue decoding by obtaining the remaining space in the primary decoding buffer pool, in order to prevent the data processed by decoding from being unable to be stored in the primary decoding buffer pool and thus causing data loss.

[0073] S102: Obtain the target decoded audio accompaniment data from the first-level decoding buffer pool, and process the target decoded audio accompaniment data using the first mixer to obtain mixed multi-track audio accompaniment data.

[0074] In this embodiment, the target decoded audio accompaniment data can be all the decoded audio accompaniment data in the primary decoding buffer pool; or, in this embodiment, the target decoded audio accompaniment data can be a portion of the primary decoding buffer pool. This embodiment obtains the target decoded audio accompaniment data from the primary decoding buffer pool, and uses a first mixer to perform mixing processing on the target decoded audio accompaniment data to obtain mixed multitrack audio accompaniment data.

[0075] It should be noted that the above-mentioned process of obtaining target decoded audio accompaniment data from the primary decoding buffer pool and processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data may include: obtaining all frame lengths corresponding to all target decoded audio accompaniment data in the primary decoding buffer pool; determining whether all frame lengths have reached the preset mixing processing frame length; when all frame lengths have reached the preset mixing processing frame length, uniformly obtaining target decoded audio accompaniment data from the primary decoding buffer pool; and processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data. This embodiment does not limit the specific length of the preset mixing processing frame length. For example, the preset mixing processing frame length can be one frame length; or the preset mixing processing frame length in this embodiment can also be two frames long. This embodiment considers that the decoding of each audio track is performed on different threads, and the audio data in each primary buffer pool may be inconsistent in length due to different thread execution times. Here, the mixer will wait for the audio data in N buffer pools to reach one frame length before retrieving them simultaneously for subsequent mixing processing to achieve the purpose of aligning audio timing points. If alignment is not performed, the audio track data will be inconsistent in time and length, resulting in poor audio quality after mixing, such as noise and echo.

[0076] It should be noted that after obtaining the target decoded audio accompaniment data from the primary decoding buffer pool and processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data, the process may further include: performing sound effect processing on the mixed multitrack audio accompaniment data to obtain the target audio accompaniment data; correspondingly, the aforementioned playback of the mixed multitrack audio accompaniment data using a player may include: playing the target audio accompaniment data using a player. This embodiment performs sound effect processing on the mixed multitrack audio accompaniment data. In current products, sound effect processing involves controlling the audio, such as processing instrument tracks and positions, allowing for customized processing of instruments and improving the audio quality.

[0077] S103 uses a player to play mixed multitrack audio accompaniment data.

[0078] This embodiment does not limit the specific process of playing the mixed multitrack audio accompaniment data using a player. For example, the mixed multitrack audio accompaniment data can be played directly; or the mixed multitrack audio accompaniment data after sound effects processing can be played; or the mixed multitrack audio accompaniment data can be synthesized before playback.

[0079] The multi-track audio buffering method provided in this invention embodiment may include: acquiring initial multi-track audio accompaniment data; decoding the initial multi-track audio accompaniment data using a decoder to obtain multiple decoded audio accompaniment data, and storing the multiple decoded audio accompaniment data in a first-level decoding buffer pool; wherein the number of decoders is consistent with the number of tracks in the initial multi-track audio accompaniment data; acquiring target decoded audio accompaniment data from the first-level decoding buffer pool, processing the target decoded audio accompaniment data using a first mixer to obtain mixed multi-track audio accompaniment data; and playing the mixed multi-track audio accompaniment data using a player. As can be seen, compared to the current method of directly playing multi-track audio accompaniment data without buffering, this invention can store multiple decoded audio accompaniment data obtained by decoding the initial multi-track audio accompaniment data using a decoder in a first-level buffer pool, thereby achieving buffered playback of audio, fully utilizing the performance of multi-core CPUs, reducing subsequent waiting time, and improving the smoothness of subsequent audio playback. Furthermore, it determines whether to perform decoding based on the remaining space in the first-level buffer pool to prevent decoding from occurring when the first-level buffer pool storage space is insufficient, thus avoiding the loss of decoded data. Furthermore, time alignment is performed on all target decoded audio accompaniment data in the primary decoding buffer pool to prevent inconsistent audio data lengths within the pool, thereby improving audio playback quality. Additionally, sound effects processing is applied to the mixed multi-track audio accompaniment data to add different details and enhance the overall audio experience.

[0080] For a clearer understanding of this invention, please refer to the following details. Figure 2 , Figure 2 A flowchart of another multitrack audio buffering method provided in an embodiment of the present invention may specifically include:

[0081] S200: Acquire initial multitrack audio accompaniment data.

[0082] S201, the initial multitrack audio accompaniment data is decoded using a decoder to obtain multiple decoded audio accompaniment data, and the multiple decoded audio accompaniment data are stored in a first-level decoding buffer pool; wherein, the number of decoders is the same as the number of tracks in the initial multitrack audio accompaniment data.

[0083] S202: Obtain the target decoded audio accompaniment data from the first-level decoding buffer pool, and process the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data;

[0084] S203, Obtain or create a secondary mixing buffer pool.

[0085] This embodiment creates a secondary mixing buffer if no secondary mixing buffer has been created previously; if a secondary mixing buffer has already been created, it is directly retrieved. In this embodiment, the secondary mixing buffer is equivalent to the secondary level of the decoding buffer. This secondary mixing buffer is used to store multi-track integrated mixes.

[0086] S204 utilizes a secondary mixing buffer pool to store mixed multitrack audio accompaniment data and / or target decoded audio accompaniment data that reaches the target frame length.

[0087] This embodiment further utilizes a secondary mixing buffer pool to store mixed multitrack audio accompaniment data and / or target decoded audio accompaniment data that reaches the target frame length.

[0088] S205: Obtain the mixed multitrack audio accompaniment data from the secondary mixing buffer pool, and play the mixed multitrack audio accompaniment data using a player.

[0089] This invention utilizes a primary decoding buffer pool and a secondary mixing buffer pool to perform multi-level buffering of data during the playback of multi-track audio data, thereby optimizing the playback experience and achieving smooth playback.

[0090] For a clearer understanding of this invention, please refer to the following details. Figure 3 , Figure 3 A flowchart of another multitrack audio buffering method provided in this embodiment of the invention may specifically include:

[0091] S300, acquire initial multitrack audio accompaniment data.

[0092] S301 uses a decoder to decode the initial multitrack audio accompaniment data to obtain multiple decoded audio accompaniment data, and uses a first-level decoding buffer pool to store the multiple decoded audio accompaniment data; wherein, the number of decoders is consistent with the number of tracks in the initial multitrack audio accompaniment data.

[0093] S302: Obtain the target decoded audio accompaniment data from the first-level decoding buffer pool, and process the target decoded audio accompaniment data using the first mixer to obtain mixed multi-track audio accompaniment data.

[0094] S303, acquire the target mixed multitrack audio accompaniment data.

[0095] S304 uses the playback buffer pool to store the target mixed multitrack audio accompaniment data.

[0096] This embodiment does not limit the specific process of storing the target mixed multitrack audio accompaniment data in the playback buffer pool. For example, the target mixed multitrack audio accompaniment data can be directly obtained and stored. In this case, the playback buffer pool is a secondary playback buffer pool. When the target mixed multitrack audio accompaniment data is obtained from the secondary mixing buffer pool, the playback buffer pool becomes a tertiary playback buffer pool.

[0097] It should be noted that the above-mentioned use of the playback buffer pool to store the target mixed multitrack audio accompaniment data may include:

[0098] S21: Obtain the vocal buffer data in the vocal buffer pool, as well as the target mixed multitrack audio accompaniment data;

[0099] S22: Store the target mixed multitrack audio accompaniment data as the accompaniment data of the vocal buffer data into the playback buffer pool.

[0100] In this embodiment, when human voice data is present, human voice buffer data in the human voice buffer pool can be obtained, and the target mixed multitrack audio accompaniment data can be stored as the accompaniment of the human voice buffer data in the playback buffer pool, thereby forming a technical solution of multi-level buffer parallel processing, which improves the synchronous playback of human voice and accompaniment.

[0101] It should be further explained that, before obtaining the voice buffer data in the voice buffer pool as described above, the process may further include: obtaining initial voice data; performing different types of sound effect processing on the initial voice data to obtain multiple sound effect processed voice data; processing the multiple sound effect processed voice data using a second mixer to obtain mixed sound processed voice buffer data; and storing the mixed sound processed voice buffer data in the voice buffer pool. In this embodiment, the voice data can be copied multiple times as needed, processed with different sound effects to add different details, and finally mixed and placed into the buffer. Alternatively, in this embodiment, the voice data can be copied multiple times as needed, processed with different sound effects to add different details, and finally mixed and processed with sound effects before being placed into the buffer.

[0102] It should be further explained that the aforementioned storage of the target mixed multitrack audio accompaniment data as the accompaniment to the vocal buffer data in the playback buffer pool can include: storing the target mixed multitrack audio accompaniment data as the accompaniment, aligning it with the vocal buffer data, and then storing it in the playback buffer pool. After storing the target mixed multitrack audio accompaniment data as the accompaniment, aligning it with the vocal buffer data, and then storing it in the playback buffer pool, this embodiment can directly play the aligned data. This embodiment uses the target mixed multitrack audio accompaniment data as the accompaniment, aligns it with the vocal buffer data, and then stores it in the playback buffer pool, improving the playback effect.

[0103] S305: Obtain the target mixed multitrack audio accompaniment data from the playback buffer pool, and play the target mixed multitrack audio accompaniment data using the player.

[0104] This embodiment combines a primary decoding buffer pool with a playback buffer pool to achieve multi-level buffering, thereby improving the playback hierarchy and thus enhancing the playback effect.

[0105] For a clearer understanding of this invention, please refer to the following details. Figure 4 , Figure 4 A flowchart illustrating a multitrack audio buffering method provided in this embodiment of the invention may specifically include:

[0106] The S400 acquires multi-track audio, decodes the multi-track audio using a decoder to obtain audio data, and stores the audio data in a primary decoding buffer. When the primary decoding buffer is insufficient to store one frame of data, it enters a sleep state until the primary decoding buffer can store one frame of data, at which point it sends a decoding command to the decoder.

[0107] The overall flowchart corresponding to this embodiment is shown below. Figure 5 , Figure 5 This is a flowchart illustrating the overall process of a multi-track audio buffering method provided in this invention. The flowchart describes parallel buffering of multi-track audio and vocal data. Multi-track audio processing includes a first-level decoding buffer, a second-level mixing buffer, and a third-level playback buffer. Vocal data processing includes a second-level vocal buffer and a third-level playback buffer. Data is acquired from the playback buffer and played using a player, or synthesized using a synthesizer before playback. Specifically, the multi-track audio file undergoes two stages of processing: single-track decoding and mixing. Vocal data undergoes sound effects and mixing processing. Finally, the vocal data and multi-track mixed data are mixed and processed with sound effects. Buffering is performed at each stage to reduce latency and jitter in data transmission. The processed data is then converted into a format recognizable by the player and sent for playback; or it is encoded and finally synthesized into a single audio file.

[0108] The corresponding level-one cache decoding buffer in this embodiment is [see...]. Figure 6 , Figure 6This is a flowchart illustrating a first-level cache decoding buffer provided in an embodiment of the present invention. The first-level decoding buffer consists of N buffer pools, where N represents the number of audio tracks in the audio file. Each track corresponds to one decoder and one buffer pool. Each time a decoder decodes one frame of audio data, when the remaining space in the buffer pool is sufficient to store one frame, it notifies the decoding thread to submit a decoding task to the thread pool. The decoded audio data, after sound effects processing, is stored in the buffer pool. When the remaining space in the buffer pool is insufficient to store one frame of audio data, a decoding task is not submitted; instead, the buffer pool remains idle, waiting for data to be consumed to free up space.

[0109] S401: Obtain the target audio data that has reached the preset frame length threshold in the first-level decoding buffer pool, process the target audio data using a mixer to obtain mixed audio, and store it in the second-level mixing buffer pool.

[0110] For easier understanding, please refer to Figure 7 , Figure 7 This is a flowchart illustrating a two-stage mixing buffer provided in an embodiment of the present invention. Each audio track is decoded on a separate thread. Due to differences in thread execution time, the audio data in each primary buffer pool may have inconsistent lengths. The mixer waits until the audio data in N buffer pools has reached the length of one frame before retrieving it simultaneously for subsequent mixing processing, thus aligning the audio timings. Without alignment, inconsistencies in the timing and length of the audio track data may occur, resulting in poor audio quality after mixing.

[0111] S402 performs different sound effects processing on the human voice data to obtain multiple sound effect processed human voices.

[0112] S403 uses a mixer to process multiple audio effects and vocals to obtain a mixed vocal, which is then stored in a vocal buffer pool.

[0113] For easier understanding, please refer to Figure 8 , Figure 8 This is a flowchart illustrating a two-level human voice buffer provided in an embodiment of the present invention. The human voice data can be copied multiple times according to actual needs, processed with different sound effects, and have different details added. Finally, after mixing and sound effect processing, the data is placed into the buffer.

[0114] S404: Obtain the mixed vocals from the vocal buffer pool and the mixed audio from the secondary mixing buffer pool. Use a mixer to process the mixed vocals and mixed audio to obtain multi-track audio, and store the multi-track audio into the playback buffer pool.

[0115] For easier understanding, please refer to Figure 9 , Figure 9This is a flowchart illustrating a three-level playback buffer provided in an embodiment of the present invention. The three-level playback buffer includes a buffer pool for storing data after time-consuming audio processing, aligning the secondary vocal buffer data and the secondary mixing buffer data, using multi-track audio data as accompaniment, aligning the vocal data, performing mixing and audio processing, and storing it in the playback buffer for subsequent playback or saving actions.

[0116] The S405 uses a player or synthesizer to process multitrack audio in the playback buffer pool.

[0117] Please refer to the following for easier understanding. Figure 10 , Figure 10 The schematic diagram of a multi-buffered multi-track audio decoding and playback flow field provided in this embodiment of the invention clearly shows that the multi-buffered solution produces smoother audio playback with normal waveforms and a spectrum within a reasonable range.

[0118] For easier understanding, please refer to Figure 11 , Figure 11 This diagram illustrates a three-level buffering processing mode provided by an embodiment of the present invention. Multi-level parallel buffering can maximize the utilization of multi-core CPU capabilities in synthetic scenarios, forming a pipelined processing mode that significantly reduces processing time.

[0119] The following describes a multitrack audio buffer device provided by an embodiment of the present invention. The multitrack audio buffer device described below can be referred to in correspondence with the multitrack audio buffer method described above.

[0120] Please refer to the details. Figure 12 , Figure 12 A schematic diagram of a multitrack audio buffer device provided in an embodiment of the present invention may include:

[0121] The initial multitrack audio accompaniment data acquisition module 100 is used to acquire initial multitrack audio accompaniment data;

[0122] The decoding buffer module 200 is used to decode the initial multitrack audio accompaniment data using a decoder to obtain multiple decoded audio accompaniment data, and to store the multiple decoded audio accompaniment data using a first-level decoding buffer pool; wherein, the number of decoders is consistent with the number of tracks in the initial multitrack audio accompaniment data;

[0123] The mixing module 300 is used to obtain target decoded audio accompaniment data from the first-level decoding buffer pool, and process the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data.

[0124] The playback module 400 is used to play the mixed multitrack audio accompaniment data using a player.

[0125] Furthermore, based on the above embodiments, the decoding buffer module 200 may include:

[0126] The remaining space acquisition unit is used to acquire the remaining storage space of the first-level decoding buffer pool;

[0127] The decoding unit is used to decode the initial multitrack audio accompaniment data using the decoder if the remaining storage space can store a preset number of audio data, to obtain the multiple decoded audio accompaniment data, and to store the multiple decoded audio accompaniment data using the first-level decoding buffer pool.

[0128] The waiting decoding unit is used to stop decoding if the remaining storage space is insufficient to store the preset number of audio frames, until the remaining storage space of the first-level decoding buffer pool is sufficient to store the preset number of audio frames, and then send a command message to start decoding.

[0129] Furthermore, based on any of the above embodiments, the mixing module 300 may include:

[0130] The frame length acquisition unit is used to acquire the length of all frames corresponding to all the target decoded audio accompaniment data in the first-level decoding buffer pool.

[0131] A frame length determination unit is used to determine whether the length of all frames reaches the preset mixing processing frame length.

[0132] The target decoded audio accompaniment data unified acquisition unit is used to uniformly acquire the target decoded audio accompaniment data from the first-level decoding buffer pool when the length of all frames reaches the preset mixing processing frame length.

[0133] The mixing unit is used to process the target decoded audio accompaniment data using the first mixer to obtain the mixed multitrack audio accompaniment data.

[0134] Furthermore, based on any of the above embodiments, the multitrack audio buffer device may further include:

[0135] The secondary mixing buffer determination module is used to obtain or create a secondary mixing buffer.

[0136] A secondary mixing buffer processing module is used to store the mixed multitrack audio accompaniment data and / or the target decoded audio accompaniment data that reaches the target frame length.

[0137] Furthermore, based on any of the above embodiments, the multitrack audio buffer device may further include:

[0138] The audio effects processing module is used to process the mixed multitrack audio accompaniment data to obtain the target audio accompaniment data.

[0139] Accordingly, the aforementioned playback module 400 includes:

[0140] The playback unit is used to play the target audio accompaniment data using the player.

[0141] Furthermore, based on any of the above embodiments, the multitrack audio buffer device may further include:

[0142] The target mixed multitrack audio accompaniment data acquisition unit is used to acquire target mixed multitrack audio accompaniment data;

[0143] The playback buffer pool storage unit is used to store the target mixed multitrack audio accompaniment data in the playback buffer pool;

[0144] Accordingly, the aforementioned playback module 400 includes:

[0145] The playback unit is used to obtain the target mixed multitrack audio accompaniment data from the playback buffer pool and play the target mixed multitrack audio accompaniment data using the player.

[0146] Furthermore, based on the above embodiments, the playback unit according to the playback buffer pool may include:

[0147] The vocal buffer data acquisition subunit is used to acquire vocal buffer data in the vocal buffer pool, as well as the target mixed multitrack audio accompaniment data.

[0148] The vocal and accompaniment buffer subunit is used to store the target mixed multitrack audio accompaniment data as the accompaniment of the vocal buffer data in the playback buffer pool.

[0149] Furthermore, based on any of the above embodiments, the multitrack audio buffer device may further include:

[0150] The initial human voice data acquisition module is used to acquire initial human voice data;

[0151] The human voice effect processing module is used to perform different types of sound effect processing on the initial human voice data to obtain multiple sound effect processed human voice data.

[0152] The human voice mixing module is used to process the multiple audio effect processing human voice data using a second mixer to obtain mixed processing human voice buffer data.

[0153] A human voice buffer pool module is used to store the human voice buffer data of the mixing process.

[0154] Furthermore, based on the above embodiments, the aforementioned simultaneous vocal and accompaniment buffering subunit may include:

[0155] The data alignment subunit is used to align the target mixed multitrack audio accompaniment data with the vocal buffer data and store it in the playback buffer pool.

[0156] It should be noted that the order of the modules and units in the aforementioned multitrack audio buffer device can be changed without affecting the logic.

[0157] The multitrack audio buffer device provided in this embodiment of the invention may include: an initial multitrack audio accompaniment data acquisition module 100, used to acquire initial multitrack audio accompaniment data; a decoding buffer module 200, used to decode the initial multitrack audio accompaniment data using a decoder to obtain multiple decoded audio accompaniment data, and to store the multiple decoded audio accompaniment data using a first-level decoding buffer pool; wherein the number of decoders is consistent with the number of tracks in the initial multitrack audio accompaniment data; a mixing module 300, used to acquire target decoded audio accompaniment data from the first-level decoding buffer pool, and to process the target decoded audio accompaniment data using a first mixer to obtain mixed multitrack audio accompaniment data; and a playback module 400, used to play the mixed multitrack audio accompaniment data using a player. As can be seen, compared to the current method of directly playing multi-track audio accompaniment data without buffering, this invention can store multiple decoded audio accompaniment data obtained by decoding the initial multi-track audio accompaniment data using a decoder in a primary buffer pool. This enables buffered playback of the audio, fully utilizing the performance of multi-core CPUs, reducing subsequent waiting time, and improving the smoothness of subsequent audio playback. Furthermore, it determines whether to perform decoding based on the remaining space in the primary buffer pool, preventing decoding when the primary buffer pool is insufficient and thus avoiding data loss. Additionally, it performs time alignment processing on all target decoded audio accompaniment data in the primary decoding buffer pool to prevent inconsistent audio data lengths, thereby improving the audio playback effect. Moreover, it performs sound effect processing on the mixed multi-track audio accompaniment data, adding different details and improving the audio effect; and again, it performs time alignment processing on all target decoded audio accompaniment data in the primary decoding buffer pool to prevent inconsistent audio data lengths, thereby improving the audio playback effect. Furthermore, it performs sound effect processing on the mixed multi-track audio accompaniment data, adding different details and improving the audio quality; and, based on the first-level decoding buffer pool, it combines with the playback buffer pool to achieve multi-level buffering, improving the playback hierarchy and thus enhancing the playback effect.

[0158] The following describes a multitrack audio buffer device provided by an embodiment of the present invention. The multitrack audio buffer device described below can be referred to in correspondence with the multitrack audio buffer method described above.

[0159] Please refer to Figure 13 , Figure 13 A schematic diagram of a multitrack audio buffer device provided in an embodiment of the present invention may include:

[0160] Memory 10 is used to store computer programs;

[0161] Processor 20 is used to execute computer programs to implement the multitrack audio buffering method described above.

[0162] The memory 10, processor 20, and communication interface 30 all communicate with each other through the communication bus 40.

[0163] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions:

[0164] Obtain initial multitrack audio accompaniment data;

[0165] The initial multitrack audio accompaniment data is decoded using a decoder to obtain multiple decoded audio accompaniment data, and these multiple decoded audio accompaniment data are stored in a first-level decoding buffer pool; the number of decoders is the same as the number of tracks in the initial multitrack audio accompaniment data.

[0166] The target decoded audio accompaniment data is obtained from the first-level decoding buffer pool, and the target decoded audio accompaniment data is processed by the first mixer to obtain mixed multi-track audio accompaniment data.

[0167] Use a media player to play the mixed multitrack audio accompaniment data.

[0168] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0169] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0170] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0171] The communication interface 30 can be an interface for the communication module, used to connect with other devices or systems.

[0172] Of course, it should be noted that, Figure 13 The structure shown does not constitute a limitation on the multitrack audio buffer device in the embodiments of the present invention. In practical applications, the multitrack audio buffer device may include more than Figure 13 More or fewer components as shown, or combinations of certain components.

[0173] The following describes the computer-readable storage medium provided in the embodiments of the present invention. The computer-readable storage medium described below can be referred to in correspondence with the multitrack audio buffering method described above.

[0174] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multitrack audio buffering method described above.

[0175] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0177] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0178] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0179] The above provides a detailed description of a multitrack audio buffering method, apparatus, device, and readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-track audio buffering method, characterized in that, include: Obtain initial multitrack audio accompaniment data; The initial multitrack audio accompaniment data is decoded using a decoder to obtain multiple decoded audio accompaniment data, and the multiple decoded audio accompaniment data are stored in a first-level decoding buffer pool; wherein, the number of decoders is the same as the number of tracks in the initial multitrack audio accompaniment data; The target decoded audio accompaniment data is obtained from the first-level decoding buffer pool, and the target decoded audio accompaniment data is processed by the first mixer to obtain mixed multitrack audio accompaniment data. Play the mixed multitrack audio accompaniment data using a player; Specifically, the initial multitrack audio accompaniment data is decoded using a decoder to obtain multiple decoded audio accompaniment data sets, which are then stored in a primary decoding buffer pool, including: Obtain the remaining storage space of the first-level decoding buffer pool; If the remaining storage space is sufficient to store a preset number of audio data frames, the decoder is used to decode the initial multitrack audio accompaniment data to obtain the multiple decoded audio accompaniment data, and the primary decoding buffer pool is used to store the multiple decoded audio accompaniment data. If the remaining storage space is insufficient to store the preset number of audio frames, decoding will stop until the remaining storage space of the first-level decoding buffer pool is sufficient to store the preset number of audio frames, at which point a command to start decoding will be sent.

2. The multitrack audio buffering method according to claim 1, characterized in that, The step of obtaining target decoded audio accompaniment data from the first-level decoding buffer pool and processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data includes: Obtain the lengths of all frames corresponding to all the target decoded audio accompaniment data in the first-level decoding buffer pool; Determine whether the length of all frames reaches the preset mixing frame length; When the length of all frames reaches the preset mixing processing frame length, the target decoded audio accompaniment data is uniformly obtained from the first-level decoding buffer pool; The target decoded audio accompaniment data is processed using the first mixer to obtain the mixed multitrack audio accompaniment data.

3. The multitrack audio buffering method according to claim 1, characterized in that, After obtaining the target decoded audio accompaniment data from the first-level decoding buffer pool, and processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data, the method further includes: Get or create a secondary mixing buffer pool; The secondary mixing buffer pool is used to store the mixed multitrack audio accompaniment data and / or the target decoded audio accompaniment data that reaches the target frame length.

4. The multitrack audio buffering method according to claim 1, characterized in that, After obtaining the target decoded audio accompaniment data from the first-level decoding buffer pool, and processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data, the method further includes: The mixed multitrack audio accompaniment data is processed with sound effects to obtain the target audio accompaniment data; Accordingly, playing the mixed multitrack audio accompaniment data using a player includes: The target audio accompaniment data is played using the player.

5. The multitrack audio buffering method according to any one of claims 1 to 4, characterized in that, After obtaining the target decoded audio accompaniment data from the first-level decoding buffer pool, processing the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data, the process further includes: Obtain the target mixed multitrack audio accompaniment data; The target mixed multitrack audio accompaniment data is stored in a playback buffer pool; Accordingly, playing the mixed multitrack audio accompaniment data using a player includes: The target mixed multitrack audio accompaniment data is obtained from the playback buffer pool, and the target mixed multitrack audio accompaniment data is played using the player.

6. The multitrack audio buffering method according to claim 5, characterized in that, The method of storing the target mixed multitrack audio accompaniment data in a playback buffer pool includes: Acquire the voice buffer data in the voice buffer pool, as well as the target mixed multitrack audio accompaniment data; The target mixed multitrack audio accompaniment data is stored as the accompaniment for the vocal buffer data in the playback buffer pool.

7. The multitrack audio buffering method according to claim 6, characterized in that, Before acquiring the voice buffer data in the voice buffer pool, the method further includes: Acquire initial human voice data; The initial human voice data is processed with different types of sound effects to obtain multiple processed human voice data. The second mixer is used to process the multiple audio effect processing voice data to obtain mixed processing voice buffer data; The human voice buffer pool is used to store the mixed-process human voice buffer data.

8. The multitrack audio buffering method according to claim 6, characterized in that, The step of storing the target mixed multitrack audio accompaniment data as the accompaniment of the vocal buffer data into the playback buffer pool includes: The target mixed multitrack audio accompaniment data is used as accompaniment and, after being aligned with the vocal buffer data, is stored in the playback buffer pool.

9. A multi-track audio buffer device, characterized in that, The multitrack audio buffering method according to any one of claims 1 to 8 includes: The initial multitrack audio accompaniment data acquisition module is used to acquire the initial multitrack audio accompaniment data; The decoding buffer module is used to decode the initial multitrack audio accompaniment data using a decoder to obtain multiple decoded audio accompaniment data, and to store the multiple decoded audio accompaniment data using a first-level decoding buffer pool; wherein, the number of decoders is the same as the number of tracks in the initial multitrack audio accompaniment data; The mixing module is used to obtain target decoded audio accompaniment data from the first-level decoding buffer pool, and process the target decoded audio accompaniment data using the first mixer to obtain mixed multitrack audio accompaniment data. The playback module is used to play the mixed multitrack audio accompaniment data using a player.

10. A multitrack audio buffer device, characterized in that, include: Memory, used to store computer programs; A processor for implementing the multitrack audio buffering method as described in any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the multitrack audio buffering method as described in any one of claims 1 to 8.

Images