Audio processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an audio processing method and device, electronic equipment and storage medium, and relates to the technical field of computers. The method comprises obtaining current collected audio data, current audio collection time corresponding to the current collected audio data, and current accompaniment delay data; modifying the current audio collection time based on the current accompaniment delay data to obtain modified audio collection time corresponding to the current collected audio data; reading target accompaniment audio data from an accompaniment audio queue based on the modified audio collection time; and mixing the target accompaniment audio data and the current collected audio data to obtain mixed audio data. The embodiments of the present disclosure can realize accurate alignment of accompaniment audio data and collected audio data, are suitable for various different scenarios, can better cope with changes in system environment, improve system operation stability, avoid maintenance cost problems caused by system updates, and thus improve user experience.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an audio processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of the internet, online karaoke has become a common form of entertainment. Through online karaoke, users can sing along with accompaniment, gaining a more immersive singing experience. However, in actual use, users often encounter misalignment between vocals and accompaniment, which affects the listening experience to some extent. This is because karaoke users need to hear the accompaniment before they start singing, but when the recorded vocals are mixed with the accompaniment, it cannot be guaranteed that the current vocals and the accompaniment being mixed are precisely matched, resulting in misalignment. This problem becomes more pronounced when switching device peripherals. Currently, a common method is to add a fixed offset when mixing the accompaniment to reduce the deviation between the accompaniment and vocals. However, this solution is unstable because the time deviation between the accompaniment and vocals is not fixed. For example, system performance fluctuations and routing changes can cause variations in the deviation, easily leading to misalignment issues again. Therefore, the above method has limited application scenarios, and due to the continuous updates of mobile devices and business applications, periodic alignment tests are needed to obtain the ideal offset value, resulting in a certain level of maintenance cost. Summary of the Invention

[0003] In view of the aforementioned technical problems, this disclosure proposes an audio processing method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of the embodiments of this disclosure, an audio processing method is provided, the method comprising:

[0005] Obtain the currently acquired audio data, the current audio acquisition time corresponding to the currently acquired audio data, and the current accompaniment delay data; the current accompaniment delay data is determined based on the current playback delay data corresponding to the currently acquired audio data and the current acquisition delay data corresponding to the currently acquired audio data;

[0006] Based on the current accompaniment delay data, the current audio acquisition time is modified to obtain the modified audio acquisition time corresponding to the current acquired audio data.

[0007] Based on the modified audio acquisition time, the target accompaniment audio data is read from the accompaniment audio queue;

[0008] The target accompaniment audio data and the currently acquired audio data are mixed to obtain mixed audio data.

[0009] According to another aspect of the present disclosure, an audio processing apparatus is provided, the apparatus comprising:

[0010] The first data acquisition module is used to acquire the currently acquired audio data, the current audio acquisition time corresponding to the currently acquired audio data, and the current accompaniment delay data; the current accompaniment delay data is determined based on the current playback delay data corresponding to the currently acquired audio data and the current acquisition delay data corresponding to the currently acquired audio data.

[0011] The modification module is used to modify the current audio acquisition time based on the current accompaniment delay data to obtain the modified audio acquisition time corresponding to the current acquired audio data;

[0012] The target accompaniment reading module is used to read target accompaniment audio data from the accompaniment audio queue based on the modified audio acquisition time;

[0013] The mixing processing module is used to perform mixing processing on the target accompaniment audio data and the currently acquired audio data to obtain mixed audio data.

[0014] Optionally, the device further includes:

[0015] The first overlay processing module is used to overlay the current playback delay data and the current acquisition delay data to obtain the current cumulative delay data corresponding to the current acquisition audio data;

[0016] The accompaniment delay acquisition module is used to determine the current accompaniment delay data based on the current cumulative delay data.

[0017] Optionally, the device further includes:

[0018] The historical data acquisition module is used to acquire the historical cumulative delay data corresponding to at least one historical accompaniment audio data in the current period; the historical cumulative delay data corresponding to each historical accompaniment audio data is obtained by superimposing the historical playback delay data and the historical acquisition delay data corresponding to each historical accompaniment audio data.

[0019] Accordingly, the accompaniment delay acquisition module includes:

[0020] The data distribution analysis module is used to perform data distribution analysis on the historical cumulative delay data corresponding to the current cumulative delay data and the at least one historical accompaniment audio data respectively, to obtain the delay distribution index data corresponding to the current period; the delay distribution index data is used to indicate the central tendency of the historical cumulative delay data corresponding to the current cumulative delay data and the at least one historical accompaniment audio data respectively.

[0021] The first execution module is used to use the delay distribution index data as the current accompaniment delay data.

[0022] Optionally, the data distribution analysis module includes:

[0023] The sorting module is used to sort the current cumulative delay data and the historical cumulative delay data corresponding to each of the at least one historical accompaniment audio data in descending order to obtain a cumulative delay data sequence;

[0024] The delay data filtering module is used to determine target cumulative delay data from the cumulative delay data sequence based on a preset ratio; the ratio of the number of cumulative delay data less than or equal to the target cumulative delay data in the cumulative delay data sequence to the number of cumulative delay data in the cumulative delay data sequence satisfies the preset ratio.

[0025] The second execution module is used to use the target cumulative delay data as the delay distribution index data.

[0026] Optionally, the device further includes:

[0027] The second data acquisition module is used to acquire the current accompaniment audio data and the current data reading time when the accompaniment playback command is detected.

[0028] The first-time writing module is used to write the current data reading time into the current accompaniment audio data;

[0029] The data adding module is used to add the current accompaniment audio data to the accompaniment audio queue.

[0030] Optionally, the device further includes:

[0031] The third data acquisition module is used to acquire the audio transmission delay data and the hardware playback delay data corresponding to the current accompaniment audio data; the audio transmission delay data represents the delay from reading the current accompaniment audio data to transmitting it to the audio player; the hardware playback delay data represents the hardware playback delay of the audio player playing the current accompaniment audio data;

[0032] The second overlay processing module is used to overlay the audio transmission delay data and the hardware playback delay data to obtain the current playback delay data.

[0033] Optionally, the device further includes:

[0034] The sending time acquisition module is used to acquire the data sending time corresponding to the audio data sending instruction when an audio data sending instruction for the current accompaniment audio data is detected; the audio data sending instruction is used to instruct the current accompaniment audio data to be sent to the audio player;

[0035] The first difference analysis module is used to perform difference analysis on the data transmission time and the current data reading time in the current accompaniment audio data to obtain the audio transmission delay data.

[0036] Optionally, the device further includes:

[0037] The fourth data acquisition module is used to acquire the first acquisition delay data and the second acquisition delay data corresponding to the current accompaniment audio data; the first acquisition delay data represents the acquisition hardware delay of the audio acquisition device; the second acquisition delay data represents the delay from acquiring the current acquired audio data corresponding to the current accompaniment audio data to detecting the mixing command for the current acquired audio data;

[0038] The third overlay processing module is used to overlay the first acquisition delay data and the second acquisition delay data to obtain the current acquisition delay data.

[0039] Optionally, the device further includes:

[0040] The receiving time acquisition module is used to acquire the current data receiving time when the audio data currently being acquired is sent by the audio acquisition device.

[0041] The second time writing module is used to write the current data reception time into the currently acquired audio data;

[0042] The mixing time acquisition module is used to acquire the current mixing start time when a mixing command for the currently acquired audio data is detected.

[0043] The second difference analysis module is used to perform difference analysis on the current mixing start time and the current data reception time in the currently acquired audio data to obtain the second acquisition delay data.

[0044] Optionally, the target accompaniment reading module includes:

[0045] The head audio acquisition module is used to acquire the head accompaniment audio data in the accompaniment audio queue;

[0046] The matching processing module is used to perform matching processing on the data reading time and the modified audio acquisition time in the head accompaniment audio data to obtain the target matching result;

[0047] The third execution module is used to, when the target matching result indicates that the data reading time in the head accompaniment audio data matches the modified audio acquisition time, use the head accompaniment audio data as the target accompaniment audio data and remove the head accompaniment audio data from the accompaniment audio queue.

[0048] Optionally, the device further includes:

[0049] The fourth execution module is used to retain the head accompaniment audio data when the target matching result indicates that the data reading time in the head accompaniment audio data does not match the modified audio acquisition time.

[0050] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the above-described audio processing method.

[0051] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-described audio processing method.

[0052] According to another aspect of the present disclosure, a computer program product containing instructions is provided that, when run on a computer, causes the computer to perform the above-described audio processing method.

[0053] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0054] The system acquires the currently captured audio data, the corresponding current audio capture time, and the current accompaniment delay data. The current accompaniment delay data is determined based on the current playback delay data and the current capture delay data corresponding to the currently captured audio data. This allows for accurate prediction of the time difference between the audio capture and the accompaniment. Combined with the current accompaniment delay data, the current audio capture time is modified to obtain the modified audio capture time corresponding to the currently captured audio data. This avoids system-induced deviations that could affect the accuracy of the audio capture time. Next, based on the modified audio capture time, the target accompaniment audio data is read from the accompaniment audio queue. This ensures more accurate reading of accompaniment audio data that matches the currently captured audio data. Finally, the target accompaniment audio data and the currently captured audio data are mixed to obtain mixed audio data. This achieves accurate alignment between the accompaniment audio data and the captured audio data, making it suitable for various scenarios and better able to handle changes in the system environment, improving system stability and avoiding maintenance costs associated with system updates, thereby enhancing the user experience.

[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0057] Figure 1 This is a schematic diagram illustrating an application system according to an exemplary embodiment;

[0058] Figure 2 This is a flowchart illustrating an audio processing method according to an exemplary embodiment;

[0059] Figure 3 This is a schematic diagram illustrating a process of reading current accompaniment audio data according to an exemplary embodiment;

[0060] Figure 4 This is a schematic diagram of the architecture of an audio processing device according to an exemplary embodiment;

[0061] Figure 5 This is a flowchart illustrating an audio processing method according to an exemplary embodiment;

[0062] Figure 6 This is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment;

[0063] Figure 7 This is a block diagram illustrating an electronic device for mixing acquired audio data and its corresponding accompaniment audio data according to an exemplary embodiment;

[0064] Figure 8 This is a block diagram illustrating another electronic device for mixing acquired audio data and its corresponding accompaniment audio data, according to an exemplary embodiment. Detailed Implementation

[0065] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0066] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0067] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0068] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application system according to an exemplary embodiment. The application system can be used in the audio processing method of this application. The application system may include at least a server 01 and a terminal 02.

[0069] In this embodiment, server 01 can be used to mix the target accompaniment audio data and the currently acquired audio data. Specifically, server 01 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0070] In this embodiment, terminal 02 can be used to acquire currently collected audio data. Terminal 02 may include physical devices such as smartphones, desktop computers, tablets, laptops, smart speakers, in-vehicle terminals, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices, and may also include software running on the physical device, such as applications. The operating system running on terminal 02 in this embodiment may include, but is not limited to, Android, GNU / Linux, and Windows systems.

[0071] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by this disclosure. In practical applications, other application environments may also be included. For example, the process of mixing the target accompaniment audio data and the currently acquired audio data can also be implemented on terminal 02.

[0072] In the embodiments described in this specification, the terminal 02 and the server 01 can be directly or indirectly connected through wired or wireless communication, and this application does not limit this connection.

[0073] It should be noted that this specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one of many possible execution orders and does not represent the only possible execution order.

[0074] Specifically, Figure 2 This is a flowchart illustrating an audio processing method according to an exemplary embodiment. Figure 2 As shown, this audio processing method can be used in electronic devices such as terminals or servers, and specifically includes the following steps:

[0075] S201: Obtain the currently acquired audio data, the current audio acquisition time corresponding to the currently acquired audio data, and the current accompaniment delay data.

[0076] In one specific embodiment, the currently acquired audio data can refer to the currently acquired audio data. Specifically, the currently acquired audio data can be object speech audio data with a corresponding data frame duration of a first preset duration. The object speech audio data can include human voice audio data. Further, the currently acquired audio data can be a data frame of fixed length. Optionally, the first preset duration can be 10ms or 20ms, etc.

[0077] In one specific embodiment, audio can be acquired using an audio acquisition device to obtain currently acquired audio data. The audio acquisition device can be used to acquire sound and convert it into audio data. Specifically, upon receiving an audio acquisition command, the audio acquisition device begins audio acquisition, obtaining at least one target audio data set; the currently acquired audio data can be determined based on the at least one target audio data set. Each target audio data set can be audio data with a corresponding data frame duration of a second preset duration. Optionally, the second preset duration can be 2ms, 5ms, or 10ms, etc. Further, if the second preset duration is less than the first preset duration, multiple target audio data sets can be concatenated to obtain the currently acquired audio data; if the second preset duration is equal to the first preset duration, a single target audio data set can be used as the currently acquired audio data. For example, when the audio acquisition device receives an audio acquisition command, it starts audio acquisition. Assuming that the acquired at least one target audio data includes target audio data 1 (i.e., the audio data that is first in the order of time among the acquired at least one target audio data) and target audio data 2 (i.e., the audio data that is second in the order of time among the acquired at least one target audio data), and the data frame duration of the above target audio data is 5ms, and the first preset duration is 10ms, the above target audio data 1 and the above target audio data 2 can be concatenated to obtain the currently acquired audio data.

[0078] In one specific embodiment, the current audio acquisition time can refer to the time when the aforementioned currently acquired audio data is obtained. Specifically, the aforementioned current audio acquisition time can be the time when the aforementioned currently acquired audio data needs to begin mixing processing.

[0079] In one specific embodiment, the time when a mixing instruction for the currently acquired audio data is detected can be used as the current audio acquisition time. The mixing instruction for the currently acquired audio data can be used to instruct mixing processing to be performed on the currently acquired audio data.

[0080] In a specific embodiment, the current accompaniment delay data can be determined based on the current playback delay data corresponding to the currently acquired audio data and the current acquisition delay data corresponding to the currently acquired audio data. Specifically, the current playback delay data characterizes the playback delay of the current accompaniment audio data. The current acquisition delay data characterizes the acquisition delay of the current acquired audio data. It can be understood that the current playback delay data can indicate the delay from when the system reads the accompaniment from the music library until the accompaniment is played; the current acquisition delay data can indicate the delay from when the system acquires the vocals until the system begins mixing the accompaniment with the acquired vocals.

[0081] In one specific embodiment, the current accompaniment delay data may be obtained in the following ways:

[0082] The current playback delay data and the current acquisition delay data are superimposed to obtain the current cumulative delay data corresponding to the current acquired audio data;

[0083] Based on the current cumulative delay data, determine the current accompaniment delay data.

[0084] In a specific embodiment, the current cumulative delay data can refer to the cumulative delay obtained by combining the playback delay of any accompaniment audio data during playback with the acquisition delay of the acquisition audio data corresponding to that accompaniment audio data. The acquisition audio data corresponding to any accompaniment audio data can refer to the object speech audio data that matches that accompaniment audio data. It can be understood that, assuming any accompaniment audio data is the accompaniment audio data with lyrics "ABC" in a song, the acquisition audio data corresponding to that accompaniment audio data can refer to the object speech audio data of a human voice singing the lyrics "ABC". Accordingly, when the current acquisition audio data is the acquisition audio data with lyrics "ABC", the current cumulative delay data can be the delay from reading the accompaniment with lyrics "ABC" from the music library until the accompaniment is played, and the delay from the system acquiring the human voice until the system starts mixing the accompaniment with the acquired human voice.

[0085] In one specific embodiment, the current playback delay data can be added to the current acquisition delay data to obtain the current cumulative delay data.

[0086] In one specific embodiment, the current cumulative delay data can be used as the current accompaniment delay data.

[0087] In the above embodiments, by superimposing the current playback delay data and the current acquisition delay data, the current cumulative delay data corresponding to the current acquisition audio data is obtained. Based on the current cumulative delay data, the current accompaniment delay data is determined, which can achieve accurate prediction of the delay between acquisition and accompaniment. This enables accurate alignment of accompaniment audio data and acquisition audio data, is applicable to various scenarios, and can better cope with changes in the system environment, improve system stability, avoid maintenance costs caused by system updates, and thus improve user experience. Understandably, in scenarios like singing, users hear the accompaniment before starting to sing. During mixing, the capture will experience a certain delay compared to the accompaniment. This delay includes the time from reading the accompaniment from the music library to feeding it to the system, the time from feeding the accompaniment to the system to its playback, the time from capture to vocals to the system sending the captured data back to the application, and the time from the application receiving the captured data to starting the accompaniment. Since these delays can fluctuate under different conditions and have a significant impact on mixing accompaniment, by superimposing the current playback delay data and the current capture delay data, we can obtain the current cumulative delay data corresponding to the current captured audio data. Based on the current cumulative delay data, we can determine the current accompaniment delay data, enabling accurate prediction of the delay between capture and accompaniment. This makes it applicable to various scenarios and better able to cope with changes in the system environment, improving system stability.

[0088] In one specific embodiment, the above method may further include:

[0089] Obtain the historical cumulative delay data corresponding to at least one historical accompaniment audio data within the current period;

[0090] Accordingly, the determination of the current accompaniment delay data based on the current cumulative delay data may include:

[0091] Perform data distribution analysis on the current cumulative delay data and the historical cumulative delay data corresponding to at least one historical accompaniment audio data to obtain the delay distribution index data corresponding to the current period;

[0092] Use the delay distribution index data as the current accompaniment delay data.

[0093] In a specific embodiment, the current cycle can refer to the current processing cycle. Specifically, the duration of each processing cycle can be 1 second to 2 seconds. It is understood that, taking a song as an example, a song can correspond to multiple processing cycles, and each processing cycle can correspond to multiple audio data frames. Assuming the duration of a processing cycle is 1 second, and the duration of an audio data frame is 10 ms, then the first 100 audio data frames acquired in chronological order can belong to processing cycle 1, and the 101st to 200th audio data frames can belong to processing cycle 2.

[0094] In one specific embodiment, at least one historical accompaniment audio data can refer to accompaniment audio data prior to the current time and belonging to the current period. The historical cumulative delay data corresponding to each historical accompaniment audio data can be obtained by superimposing the historical playback delay data and the historical acquisition delay data corresponding to each historical accompaniment audio data.

[0095] In one specific embodiment, historical cumulative delay data belonging to the current period can be obtained from a preset memory, serving as the historical cumulative delay data corresponding to each of the at least one historical accompaniment audio data. Specifically, the preset memory can store the stored cumulative delay data corresponding to multiple stored accompaniment audio data and the stored audio acquisition time corresponding to each stored accompaniment audio data, wherein the stored audio acquisition time corresponding to any stored accompaniment audio data can be the modified audio acquisition time corresponding to any of the stored accompaniment audio data. Further, based on the multiple stored audio acquisition times stored in the preset memory, the stored audio acquisition time belonging to the current period can be determined, and correspondingly, the stored cumulative delay data corresponding to the stored audio acquisition time belonging to the current period can be used as the aforementioned historical cumulative delay data.

[0096] In one specific embodiment, the delay distribution index data can be used to indicate the central tendency of the historical cumulative delay data corresponding to the current cumulative delay data and at least one historical accompaniment audio data.

[0097] In one specific embodiment, the delay distribution index data corresponding to the current period can be obtained by averaging the historical cumulative delay data corresponding to the current cumulative delay data and at least one historical accompaniment audio data.

[0098] In a specific embodiment, the above-mentioned data distribution analysis of the historical cumulative delay data corresponding to the current cumulative delay data and at least one historical accompaniment audio data to obtain the delay distribution index data corresponding to the current period may include:

[0099] Sort the current cumulative delay data and the historical cumulative delay data corresponding to at least one historical accompaniment audio data in descending order to obtain a cumulative delay data sequence;

[0100] Based on the preset ratio data, the target cumulative delay data is determined from the cumulative delay data sequence;

[0101] The cumulative delay data of the target is used as the delay distribution index data.

[0102] In one specific embodiment, the cumulative delay data sequence can refer to the sequence of historical cumulative delay data corresponding to the current cumulative delay data and at least one historical accompaniment audio data, arranged in descending order.

[0103] In one specific embodiment, the preset ratio data can be used to indicate the location of the data range to be analyzed. Specifically, the preset ratio data can be set according to the actual application needs, and this disclosure does not limit it. For example, the preset ratio data can be 90%.

[0104] In one specific embodiment, the target cumulative delay data can be the cumulative delay data located at the position corresponding to a preset proportion of data in the cumulative delay data sequence. Specifically, the ratio of the number of cumulative delay data less than or equal to the target cumulative delay data in the cumulative delay data sequence to the total number of cumulative delay data in the cumulative delay data sequence satisfies a preset proportion.

[0105] In one specific embodiment, the target data location information can be determined based on the number of elements in the cumulative delay data sequence and a preset proportion of data. Correspondingly, the target cumulative delay data can be determined from the cumulative delay data sequence based on the target data location information. Specifically, the target data location information can be obtained by multiplying the number of elements in the cumulative delay data sequence by a preset proportion of data. If the target data location information is an integer, the cumulative delay data corresponding to the target data location information can be used as the target cumulative delay data. If the target data location information is not an integer, the cumulative delay data sequence can be interpolated based on the target data location information, and the interpolated cumulative delay data can be used as the target cumulative delay data. For example, assuming the target data location information is 5.5, and the cumulative delay data for location information 5 is "3", and the cumulative delay data for location information 6 is "1", we can use interpolation to insert the cumulative delay data "2" between the cumulative delay data corresponding to location information 5 and the cumulative delay data corresponding to location information 6. Accordingly, since the cumulative delay data corresponding to the above target data location information is "2", we can determine that the target cumulative delay data is the cumulative delay data "2".

[0106] In the above embodiments, by sorting the current cumulative delay data and the historical cumulative delay data corresponding to at least one historical accompaniment audio data in descending order, a cumulative delay data sequence is obtained. Based on a preset ratio, a target cumulative delay data is determined from the cumulative delay data sequence. The target cumulative delay data is used as a delay distribution index data, which can realize the central trend analysis of the cumulative delay data in the current period, thereby improving the accuracy of accompaniment delay prediction.

[0107] In one specific embodiment, if there is no historical accompaniment audio data in the current period, the current cumulative delay data can be used as the current accompaniment delay data.

[0108] In the above embodiments, by acquiring the historical cumulative delay data corresponding to at least one historical accompaniment audio data in the current period, performing data distribution analysis on the current cumulative delay data and the historical cumulative delay data corresponding to at least one historical accompaniment audio data, the delay distribution index data corresponding to the current period is obtained. Using the delay distribution index data as the current accompaniment delay data can improve the accuracy and stability of accompaniment delay prediction, thereby achieving the alignment of accompaniment audio data and acquired audio data.

[0109] In one specific embodiment, the above method may further include:

[0110] Upon detecting a accompaniment playback command, obtain the current accompaniment audio data and the current data reading time;

[0111] Write the current data reading time into the current accompaniment audio data;

[0112] Add the current accompaniment audio data to the accompaniment audio queue.

[0113] In one specific embodiment, the accompaniment playback instruction can be used to instruct the audio player to play corresponding audio data. Specifically, the accompaniment playback instruction may include audio playback identifier information. This audio playback identifier information can be used to indicate the audio data that currently needs to be played.

[0114] In one specific embodiment, the current accompaniment audio data can refer to the accompaniment audio data that needs to be played now. The current accompaniment audio data can be a portion of the complete accompaniment audio data within the target multimedia resource. The target multimedia resource can refer to the multimedia resource currently being interacted with by the first target object. Specifically, the first target object can be the user account of the resource content creator.

[0115] In one specific embodiment, the current data reading time may refer to the time it takes to read the current accompaniment audio data.

[0116] In a specific embodiment, upon receiving a resource playback instruction from a first target object for a target multimedia resource, the resource accompaniment audio data corresponding to the target multimedia resource can be obtained; the resource accompaniment audio data can be segmented to obtain multiple audio data to be played and the playback order corresponding to the multiple audio data to be played; and based on the playback order, an accompaniment playback instruction for each audio data to be played can be generated sequentially.

[0117] In a specific embodiment, when a accompaniment playback command is detected, the resource accompaniment audio data corresponding to the playback audio identifier information in the corresponding accompaniment playback command can be obtained from the multimedia resource storage; the resource accompaniment audio data can be used as the current accompaniment audio data, and correspondingly, the current time can be obtained and used as the current data reading time.

[0118] In one specific embodiment, the current data read time can be written as a timestamp into the current accompaniment audio data. It is understood that during subsequent processing of the current accompaniment audio data, the current data read time can be continuously carried within the current accompaniment audio data.

[0119] In one specific embodiment, the current accompaniment audio data can be added to the end of the accompaniment audio queue.

[0120] In one specific embodiment Figure 3 This is a schematic diagram illustrating a process of reading current accompaniment audio data according to an exemplary embodiment. Specifically, as shown below... Figure 3 As shown, when a accompaniment playback command is detected, the current accompaniment audio data and the current data reading time are obtained. The current data reading time is written into the current accompaniment audio data, and the current accompaniment audio data is added to the accompaniment audio queue. This achieves caching by writing the current accompaniment audio data into the accompaniment audio queue while the current accompaniment audio data is being played.

[0121] In the above embodiments, when a accompaniment playback command is detected, the current accompaniment audio data and the current data reading time are obtained, the current data reading time is written into the current accompaniment audio data, and the current accompaniment audio data is added to the accompaniment audio queue. This can realize playback-driven accompaniment caching, thereby achieving synchronization between the accompaniment and the playback status. It avoids the mixing of unplayed accompaniment audio data and acquired audio data due to playback abnormalities, and can further improve the alignment accuracy of accompaniment audio data and acquired audio data.

[0122] In one specific embodiment, the current playback delay data may be obtained in the following ways:

[0123] Get the audio transmission delay data and hardware playback delay data corresponding to the current accompaniment audio data;

[0124] The audio transmission delay data and the hardware playback delay data are superimposed to obtain the current playback delay data.

[0125] In one specific embodiment, audio transmission delay data can characterize the delay from reading the current accompaniment audio data to transmitting it to the audio player.

[0126] In one specific embodiment, audio transmission delay data may be obtained in the following ways:

[0127] If an audio data sending command for the current accompaniment audio data is detected, the data sending time corresponding to the audio data sending command is obtained;

[0128] By performing a difference analysis on the data transmission time and the current data reading time in the current accompaniment audio data, audio transmission delay data is obtained.

[0129] In one specific embodiment, the audio data sending instruction can be used to instruct the current accompaniment audio data to be sent to the audio player, so that the audio player can play the current accompaniment audio data. The audio player can be used to implement the playback of the audio data.

[0130] In one specific embodiment, the current accompaniment audio data can refer to the accompaniment audio data that needs to be played now. Specifically, the current accompaniment audio data can be the accompaniment audio data that matches the currently acquired audio data mentioned above.

[0131] In one specific embodiment, the data transmission time corresponding to the audio data transmission command can be used to characterize the time when the current accompaniment audio data needs to be sent to the audio player. Specifically, the data transmission time corresponding to the audio data transmission command can be the time when the audio data transmission command is received.

[0132] In one specific embodiment, upon receiving an audio data transmission instruction, the current time can be obtained and used as the data transmission time.

[0133] In one specific embodiment, the audio transmission delay data can be obtained by subtracting the current data reading time from the data transmission time corresponding to the current accompaniment audio data. It is understood that after the accompaniment data is read, it may undergo 3A processing, mixing with other streams, volume changes, and sound effects, and the impact of these operations on the delay cannot be ignored. Therefore, by continuously carrying the timestamp "current data reading time" and reading the current data transmission time just before feeding it to the audio player, the audio transmission delay data can be obtained.

[0134] In the above embodiments, by detecting an audio data sending command for the current accompaniment audio data, obtaining the data sending time corresponding to the audio data sending command, performing a difference analysis on the data sending time and the current data reading time in the current accompaniment audio data, and obtaining audio transmission delay data, it is possible to accurately predict the delay from reading the current accompaniment audio data to transmitting it to the audio player, thereby achieving accurate alignment between the accompaniment audio data and the acquired audio data.

[0135] In one specific embodiment, hardware playback latency data can characterize the hardware playback latency of the audio player when playing the current accompaniment audio data.

[0136] In a specific embodiment, hardware playback latency data can be acquired by calling the first latency acquisition interface. Specifically, upon detecting the call instruction of the first latency acquisition interface, the target system can send a first latency acquisition instruction to the audio player. Further, upon receiving the first latency acquisition instruction, the audio player can perform a first latency prediction operation to obtain first predicted latency data; the audio player can then return the first predicted latency data. Upon receiving the first predicted latency data, the target system can use it as hardware playback latency data. It is understood that after data is fed to the audio player, the audio player also has its own internal hardware playback latency, which varies depending on the device and routing, etc. Hardware playback latency data can be acquired by calling the first latency acquisition interface.

[0137] In one specific embodiment, the audio player can obtain the amount of data to be consumed and the current queue consumption rate of the internal queue of the player; based on the amount of data to be consumed and the current queue consumption rate, the player can perform queue consumption delay prediction on the internal queue to obtain queue consumption delay data; accordingly, based on the queue consumption delay data, the first predicted delay data can be determined.

[0138] In one specific embodiment, the current playback delay data can be obtained by adding the audio transmission delay data to the hardware playback delay data.

[0139] In the above embodiments, the audio transmission delay data and the hardware playback delay data corresponding to the current accompaniment audio data are obtained. The audio transmission delay data and the hardware playback delay data are superimposed to obtain the current playback delay data. This can achieve accurate prediction of the delay during the accompaniment playback process, and thus achieve accurate alignment between the accompaniment audio data and the acquired audio data. It is applicable to a variety of different scenarios and can better cope with changes in the system environment, thereby improving the stability of system operation.

[0140] In a specific embodiment, the currently acquired delay data may be obtained in the following ways:

[0141] Obtain the first acquisition delay data and the second acquisition delay data corresponding to the current accompaniment audio data;

[0142] The first and second acquisition delay data are superimposed to obtain the current acquisition delay data.

[0143] In one specific embodiment, the first acquisition delay data can characterize the acquisition hardware delay of the audio acquisition device.

[0144] In a specific embodiment, the first acquisition delay data can be acquired by calling the second delay acquisition interface. Specifically, upon detecting the call instruction of the second delay acquisition interface, the target system can send a second delay acquisition instruction to the audio acquisition device. Further, upon receiving the second delay acquisition instruction, the audio acquisition device can perform a second delay prediction operation to obtain second predicted delay data; the audio acquisition device can then return the second predicted delay data. Upon receiving the second predicted delay data, the target system can use it as the first acquisition delay data. It is understood that the delay from the acquisition of human voice to the audio acquisition device returning the human voice to the system can vary depending on the device and routing, etc., and the first acquisition delay data can be acquired by calling the second delay acquisition interface.

[0145] In one specific embodiment, the audio acquisition device can perform a hardware acquisition delay prediction operation to obtain hardware acquisition delay data; the audio acquisition device can perform a driver buffer delay prediction operation to obtain driver buffer delay data; the audio acquisition device can perform an application layer buffer delay prediction operation to obtain application layer buffer delay data; the audio acquisition device can superimpose the above-mentioned hardware acquisition delay data, the above-mentioned driver buffer delay data, and the above-mentioned application layer buffer delay data to obtain second predicted delay data.

[0146] In one specific embodiment, the second acquisition delay data can characterize the delay from acquiring the current acquired audio data corresponding to the current accompaniment audio data to detecting the mixing instruction for the current acquired audio data.

[0147] In one specific embodiment, the second acquisition delay data may be obtained in the following ways:

[0148] Upon receiving the currently acquired audio data from the audio acquisition device, obtain the current data reception time;

[0149] Write the current data reception time into the currently acquired audio data;

[0150] If a mixing command for the currently acquired audio data is detected, the current mixing start time is obtained;

[0151] A difference analysis is performed on the current mixing start time and the current data reception time in the currently acquired audio data to obtain the second acquisition delay data.

[0152] In one specific embodiment, the current data reception time may refer to the time when the audio acquisition device sends the currently acquired audio data.

[0153] In one specific embodiment, upon receiving the currently acquired audio data sent by the audio acquisition device, the current time can be obtained, and correspondingly, the current time can be used as the current data reception time.

[0154] In one specific embodiment, the current data reception time can be used as a timestamp and written into the currently acquired audio data.

[0155] In one specific embodiment, the current mixing start time can characterize the time required to perform mixing operations on the currently acquired audio data. Specifically, upon detecting a mixing command for the currently acquired audio data, the current time can be obtained, and accordingly, the current time can be used as the current mixing start time.

[0156] In one specific embodiment, the second acquisition delay data can be obtained by subtracting the current data reception time in the currently acquired audio data from the current mixing start time.

[0157] In the above embodiments, by obtaining the current data reception time upon receiving the currently acquired audio data sent by the audio acquisition device and writing the current data reception time into the currently acquired audio data, and by obtaining the current mixing start time upon detecting a mixing command for the currently acquired audio data, and performing a difference analysis on the current mixing start time and the current data reception time in the currently acquired audio data to obtain second acquisition delay data, accurate prediction of the delay from obtaining the currently acquired audio data corresponding to the current accompaniment audio data to detecting the mixing command for the currently acquired audio data can be achieved, thereby enabling accurate alignment between the accompaniment audio data and the acquired audio data. It is understood that by writing the current data reception time into the currently acquired audio data, as the currently acquired audio data undergoes 3A and sound effects processing, the aforementioned current data reception time, as a timestamp, will be continuously transmitted along with the currently acquired audio data. Combined with the mixing start time, this allows for prediction of the delay from receiving the acquired audio data to the start of the accompaniment.

[0158] In one specific embodiment, the first acquisition delay data can be added to the second acquisition delay data to obtain the current acquisition delay data.

[0159] In the above embodiments, by acquiring the first acquisition delay data corresponding to the current accompaniment audio data and the second acquisition delay data corresponding to the current accompaniment audio data, and superimposing the first acquisition delay data and the second acquisition delay data to obtain the current acquisition delay data, accurate prediction of the delay during the acquisition process can be achieved, thereby achieving accurate alignment between the accompaniment audio data and the acquired audio data. This is applicable to a variety of different scenarios and can better cope with changes in the system environment, improving the stability of system operation.

[0160] S203: Based on the current accompaniment delay data, modify the current audio acquisition time to obtain the modified audio acquisition time corresponding to the current acquired audio data.

[0161] In one specific embodiment, modifying the audio acquisition time can refer to the modified current audio acquisition time.

[0162] In one specific embodiment, the above-mentioned modification of the audio acquisition time can be obtained by the following formula:

[0163] T_target=T_cap_ready-T_result

[0164] Where T_target is the modified audio capture time; T_cap_ready is the current audio capture time; and T_result is the current accompaniment delay data.

[0165] In a specific embodiment, after obtaining the current cumulative delay data and modifying the audio acquisition time, the current cumulative delay data and the modified audio acquisition time corresponding to the current acquisition delay data can be stored in a preset memory.

[0166] In the above embodiments, by combining the current accompaniment delay data and modifying the current audio acquisition time, the modified audio acquisition time corresponding to the current acquired audio data can be obtained. This can correct the audio acquisition time of the current acquired audio data, thereby achieving accurate alignment between the accompaniment audio data and the acquired audio data, and thus improving the user experience.

[0167] S205: Based on the modified audio acquisition time, read the target accompaniment audio data from the accompaniment audio queue.

[0168] In one specific embodiment, the target accompaniment audio data may refer to the accompaniment audio data in the accompaniment audio queue that matches the currently acquired audio data.

[0169] In one specific embodiment, the above-mentioned method of reading target accompaniment audio data from the accompaniment audio queue based on modifying the audio acquisition time may include:

[0170] Retrieve the head backing track audio data from the backing track audio queue;

[0171] The data reading time and modified audio acquisition time in the head accompaniment audio data are matched to obtain the target matching result;

[0172] If the target matching result indicates that the data reading time and the modified audio acquisition time in the head accompaniment audio data match, the head accompaniment audio data is used as the target accompaniment audio data, and the head accompaniment audio data is removed from the accompaniment audio queue.

[0173] In one specific embodiment, the head accompaniment audio data may refer to the accompaniment audio data that is first in the accompaniment audio queue.

[0174] In one specific embodiment, the first accompaniment audio data in the accompaniment audio queue can be obtained to get the head accompaniment audio data.

[0175] In one specific embodiment, the data reading time in the head accompaniment audio data can be used to indicate the time when the aforementioned head accompaniment audio data is read and played.

[0176] In one specific embodiment, the target matching result can be used to indicate whether there is a match between the data read time and the modified audio capture time in the head-accompaniment audio data. The target matching result can include a first matching result or a second matching result. The first matching result can be used to indicate that the data read time and the modified audio capture time in the head-accompaniment audio data match. The second matching result can be used to indicate that the data read time and the modified audio capture time in the head-accompaniment audio data do not match.

[0177] In one specific embodiment, a difference analysis is performed on the data reading time and modified audio acquisition time in the head-accompaniment audio data to obtain target difference index data. If the target difference index data is less than or equal to a preset difference index data, the target matching result can be determined as a first matching result; if the target difference index data is greater than the preset difference index data, the target matching result can be determined as a second matching result. The target difference index data can be used to indicate the degree of difference between the data reading time and the modified audio acquisition time in the head-accompaniment audio data. Specifically, the preset difference index data can be set according to actual application needs, and this disclosure does not limit it. Optionally, the preset difference index data can be 10ms or 20ms, etc. For example, the preset difference index data can be 10ms.

[0178] In one specific embodiment, if the target matching result is the first matching result, the head accompaniment audio data in the accompaniment audio queue can be extracted as the target accompaniment audio data, and the head accompaniment audio data can be removed from the accompaniment audio queue. It can be understood that after removing the head accompaniment audio data from the accompaniment audio queue, the positions of the remaining accompaniment audio data can be shifted one position forward, that is, the accompaniment audio data that was originally ranked second can become the head accompaniment audio data.

[0179] In one specific embodiment, the above method may further include:

[0180] If the target matching result indicates that the data reading time and the modified audio acquisition time in the head accompaniment audio data do not match, the head accompaniment audio data shall be retained.

[0181] In a specific embodiment, if the target matching result is the second matching result, it can be determined that the accompaniment audio data matching the currently acquired audio data has not yet arrived, and there is no need to mix the accompaniment. The head accompaniment audio data can be kept in the accompaniment audio queue. As the currently acquired audio data is updated, until the target matching result is the first matching result, it can be determined that the currently acquired audio data matches the head accompaniment audio data in the accompaniment audio queue. The head accompaniment audio data in the accompaniment audio queue can be read out and mixed. Correspondingly, the read head accompaniment audio data can be removed from the accompaniment audio queue.

[0182] In one specific embodiment, if the accompaniment audio queue is empty, it can be determined that the accompaniment has not yet been played, and no mixing processing is required at this time.

[0183] In the above embodiments, by acquiring the head accompaniment audio data in the accompaniment audio queue, matching the data reading time and modified audio acquisition time in the head accompaniment audio data to obtain the target matching result, and when the target matching result indicates that the data reading time and modified audio acquisition time in the head accompaniment audio data match, the head accompaniment audio data is used as the target accompaniment audio data and removed from the accompaniment audio queue. This can achieve the mixing of mutually matching accompaniment audio data and acquired audio data, realize the accurate alignment of accompaniment audio data and acquired audio data, and thus improve the user experience.

[0184] S207: Mix the target accompaniment audio data and the currently acquired audio data to obtain mixed audio data.

[0185] In one specific embodiment, the mixed audio data can be audio data that includes accompaniment and subject speech. Specifically, mixed audio data can refer to audio data obtained by mixing accompaniment audio data and acquired audio data.

[0186] In one specific embodiment, the target accompaniment audio data and the currently acquired audio data are fused to obtain mixed audio data.

[0187] In one specific embodiment Figure 4 This is a schematic diagram illustrating the architecture of an audio processing apparatus according to an exemplary embodiment. Specifically, as shown below... Figure 4 As shown, the audio processing device may include a delay prediction module, an accompaniment audio reading module, and a mixing module; wherein, the delay prediction module can be used to predict the delay of the accompaniment to obtain the current accompaniment delay data; the accompaniment audio reading module can be used to read the target accompaniment audio data based on the current accompaniment delay data; and the mixing module can be used to perform mixing processing of the target accompaniment audio data and the currently acquired audio data.

[0188] In the above embodiments, the currently acquired audio data, the corresponding current audio acquisition time, and the current accompaniment delay data are obtained. The current accompaniment delay data is determined based on the current playback delay data and the current acquisition delay data corresponding to the currently acquired audio data. This allows for accurate prediction of the time difference between sound acquisition and accompaniment. Combined with the current accompaniment delay data, the current audio acquisition time is modified to obtain the modified audio acquisition time corresponding to the current audio data. This avoids system-induced deviations that could affect the accuracy of the audio acquisition time of the currently acquired audio data. Next, based on the modified audio acquisition time, target accompaniment audio data is read from the accompaniment audio queue. This allows for more accurate reading of accompaniment audio data that matches the currently acquired audio data. Then, the target accompaniment audio data and the currently acquired audio data are mixed to obtain mixed audio data. This achieves accurate alignment between the accompaniment audio data and the acquired audio data, making it suitable for various scenarios and better able to cope with changes in the system environment, improving system stability and avoiding maintenance costs associated with system updates, thereby enhancing the user experience. Understandably, by modifying the current audio capture time using the aforementioned accompaniment delay data, the precise alignment of the accompaniment and the captured audio can be guaranteed regardless of factors such as device, load, routing, or application logic. Furthermore, the business does not need to constantly test and update fixed offset values, which can reduce maintenance costs.

[0189] Figure 5 This is a flowchart illustrating an audio processing method according to an exemplary embodiment. Specifically, as shown below... Figure 5As shown, upon detecting a backing track playback command, the current backing track audio data and current data read time can be obtained; the current data read time is written into the current backing track audio data; and the current backing track audio data is added to the backing track audio queue to achieve synchronization between playback and buffering. The audio transmission delay data and hardware playback delay data corresponding to the current backing track audio data can be obtained; these data can be superimposed to obtain the current playback delay data; correspondingly, the first acquisition delay data and the second acquisition delay data corresponding to the current backing track audio data can be obtained; these data can be superimposed to obtain the current acquisition delay data; the current playback delay data and the current acquisition delay data can be superimposed to obtain the current cumulative delay data corresponding to the current acquisition audio data; and the current backing track delay data can be determined based on the current cumulative delay data. Furthermore, the system can acquire the currently captured audio data, the corresponding current audio capture time, and the current accompaniment delay data. Based on the current accompaniment delay data, the current audio capture time is modified to obtain the modified audio capture time corresponding to the currently captured audio data. Based on the modified audio capture time, the target accompaniment audio data is read from the accompaniment audio queue. The target accompaniment audio data and the currently captured audio data are mixed to obtain mixed audio data. Correspondingly, the mixed audio data can be encoded to obtain mixed audio encoding information. The mixed audio encoding information can be sent to the server so that the target terminal can request the mixed audio encoding information from the server, decode the requested mixed audio encoding information, obtain mixed audio data, and play it. The second target object can be the user account of the resource content recipient.

[0190] Figure 6 This is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment. Specifically, as shown below... Figure 6 As shown, the device may include:

[0191] The first data acquisition module 610 can be used to acquire the currently acquired audio data, the current audio acquisition time corresponding to the currently acquired audio data, and the current accompaniment delay data; the current accompaniment delay data is determined based on the current playback delay data corresponding to the currently acquired audio data and the current acquisition delay data corresponding to the currently acquired audio data.

[0192] The modification module 620 can be used to modify the current audio acquisition time based on the current accompaniment delay data, so as to obtain the modified audio acquisition time corresponding to the current acquired audio data.

[0193] The target accompaniment reading module 630 can be used to read target accompaniment audio data from the accompaniment audio queue based on the modified audio acquisition time;

[0194] The mixing processing module 640 can be used to mix the target accompaniment audio data and the currently acquired audio data to obtain mixed audio data.

[0195] In one specific embodiment, the above-described apparatus may further include:

[0196] The first overlay processing module can be used to overlay the current playback delay data and the current acquisition delay data to obtain the current cumulative delay data corresponding to the current acquisition audio data;

[0197] The accompaniment delay acquisition module can be used to determine the current accompaniment delay data based on the current cumulative delay data.

[0198] In one specific embodiment, the above-described apparatus may further include:

[0199] The historical data acquisition module can be used to acquire the historical cumulative delay data corresponding to at least one historical accompaniment audio data within the current period; the historical cumulative delay data corresponding to each historical accompaniment audio data is obtained by superimposing the historical playback delay data and the historical acquisition delay data corresponding to each historical accompaniment audio data.

[0200] Accordingly, the aforementioned accompaniment delay acquisition module may include:

[0201] The data distribution analysis module can be used to perform data distribution analysis on the current cumulative delay data and the historical cumulative delay data corresponding to at least one historical accompaniment audio data to obtain the delay distribution index data corresponding to the current period; the delay distribution index data is used to indicate the central tendency of the historical cumulative delay data corresponding to the current cumulative delay data and at least one historical accompaniment audio data.

[0202] The first execution module can be used to take the delay distribution index data as the current accompaniment delay data.

[0203] In one specific embodiment, the above-mentioned data distribution analysis module may include:

[0204] The sorting module can be used to sort the current cumulative delay data and the historical cumulative delay data corresponding to at least one historical accompaniment audio data in descending order to obtain a cumulative delay data sequence.

[0205] The delayed data filtering module can be used to determine the target cumulative delayed data from the cumulative delayed data sequence based on a preset ratio; the ratio of the number of cumulative delayed data less than or equal to the target cumulative delayed data in the cumulative delayed data sequence to the total number of cumulative delayed data in the cumulative delayed data sequence meets the preset ratio.

[0206] The second execution module can be used to use the target cumulative delay data as delay distribution index data.

[0207] In one specific embodiment, the above-described apparatus may further include:

[0208] The second data acquisition module can be used to acquire the current accompaniment audio data and the current data reading time when an accompaniment playback command is detected.

[0209] The first-time write module can be used to write the current data reading time into the current accompaniment audio data;

[0210] The data addition module can be used to add the current accompaniment audio data to the accompaniment audio queue.

[0211] In one specific embodiment, the above-described apparatus may further include:

[0212] The third data acquisition module can be used to acquire the audio transmission delay data and the hardware playback delay data corresponding to the current accompaniment audio data; the audio transmission delay data represents the delay from reading the current accompaniment audio data to transmitting it to the audio player; the hardware playback delay data represents the hardware playback delay of the audio player playing the current accompaniment audio data.

[0213] The second overlay processing module can be used to overlay audio transmission delay data and hardware playback delay data to obtain the current playback delay data.

[0214] In one specific embodiment, the above-described apparatus may further include:

[0215] The sending time acquisition module can be used to acquire the data sending time corresponding to the audio data sending command when an audio data sending command for the current accompaniment audio data is detected; the audio data sending command is used to instruct the current accompaniment audio data to be sent to the audio player;

[0216] The first difference analysis module can be used to perform difference analysis on the data transmission time and the current data reading time in the current accompaniment audio data to obtain audio transmission delay data.

[0217] In one specific embodiment, the above-described apparatus may further include:

[0218] The fourth data acquisition module can be used to acquire the first acquisition delay data and the second acquisition delay data corresponding to the current accompaniment audio data; the first acquisition delay data represents the acquisition hardware delay of the audio acquisition device; the second acquisition delay data represents the delay from acquiring the current acquisition audio data corresponding to the current accompaniment audio data to detecting the mixing command for the current acquisition audio data;

[0219] The third overlay processing module can be used to overlay the first acquisition delay data and the second acquisition delay data to obtain the current acquisition delay data.

[0220] In one specific embodiment, the above-described apparatus may further include:

[0221] The receiving time acquisition module can be used to obtain the current data receiving time when the audio data currently being acquired is sent by the audio acquisition device.

[0222] The second time writing module can be used to write the current data reception time into the currently acquired audio data;

[0223] The mixing time acquisition module can be used to acquire the current mixing start time when a mixing command for the currently acquired audio data is detected.

[0224] The second difference analysis module can be used to perform difference analysis on the current mixing start time and the current data reception time in the currently acquired audio data to obtain the second acquisition delay data.

[0225] In one specific embodiment, the target accompaniment reading module 630 described above may include:

[0226] The head audio acquisition module can be used to acquire head accompaniment audio data in the accompaniment audio queue;

[0227] The matching processing module can be used to match the data reading time and modified audio acquisition time in the head accompaniment audio data to obtain the target matching result;

[0228] The third execution module can be used to remove the head accompaniment audio data from the accompaniment audio queue when the data reading time and the modified audio acquisition time in the target matching result indicate that they match.

[0229] In one specific embodiment, the above-described apparatus may further include:

[0230] The fourth execution module can be used to retain the head accompaniment audio data when the data reading time and modified audio acquisition time in the target matching result indicate a mismatch.

[0231] Regarding the apparatus in the above embodiments, the specific manner in which each module and unit performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0232] Figure 7 This is a block diagram illustrating an electronic device for mixing acquired audio data and its corresponding accompaniment audio data according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an audio processing method.

[0233] Figure 8 This is a block diagram illustrating another electronic device for mixing acquired audio data and its corresponding accompaniment audio data according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an audio processing method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0234] Those skilled in the art will understand that Figure 7 or Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0235] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the audio processing method as described in the embodiments of this disclosure.

[0236] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the audio processing method of the embodiments of this disclosure.

[0237] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the audio processing method of the embodiments of this disclosure.

[0238] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0239] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0240] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0241] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0242] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that, The method includes: Obtain the currently acquired audio data, the current audio acquisition time corresponding to the currently acquired audio data, and the current accompaniment delay data; the current accompaniment delay data is determined based on the current playback delay data corresponding to the currently acquired audio data and the current acquisition delay data corresponding to the currently acquired audio data; Based on the current accompaniment delay data, the current audio acquisition time is modified to obtain the modified audio acquisition time corresponding to the current acquired audio data. Based on the modified audio acquisition time, the target accompaniment audio data is read from the accompaniment audio queue; The target accompaniment audio data and the currently acquired audio data are mixed to obtain mixed audio data.

2. The method according to claim 1, characterized in that, The current accompaniment delay data is obtained in the following ways: The current playback delay data and the current acquisition delay data are superimposed to obtain the current cumulative delay data corresponding to the current acquisition audio data; Based on the current cumulative delay data, the current accompaniment delay data is determined.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the historical cumulative delay data corresponding to at least one historical accompaniment audio data within the current period; the historical cumulative delay data corresponding to each historical accompaniment audio data is obtained by superimposing the historical playback delay data corresponding to each historical accompaniment audio data and the historical acquisition delay data corresponding to each historical accompaniment audio data. The step of determining the current accompaniment delay data based on the current cumulative delay data includes: Data distribution analysis is performed on the historical cumulative delay data corresponding to the current cumulative delay data and the at least one historical accompaniment audio data respectively to obtain the delay distribution index data corresponding to the current period; the delay distribution index data is used to indicate the central tendency of the historical cumulative delay data corresponding to the current cumulative delay data and the at least one historical accompaniment audio data respectively. The delay distribution index data is used as the current accompaniment delay data.

4. The method according to claim 3, characterized in that, The step of performing data distribution analysis on the historical cumulative delay data corresponding to the current cumulative delay data and the at least one historical accompaniment audio data to obtain delay distribution index data corresponding to the current period includes: The historical cumulative delay data corresponding to the current cumulative delay data and the at least one historical accompaniment audio data are sorted in descending order to obtain a cumulative delay data sequence; Based on a preset ratio, a target cumulative delay data is determined from the cumulative delay data sequence; the ratio of the number of cumulative delay data less than or equal to the target cumulative delay data in the cumulative delay data sequence to the total number of cumulative delay data in the cumulative delay data sequence satisfies the preset ratio. The target cumulative delay data is used as the delay distribution index data.

5. The method according to claim 1, characterized in that, The method further includes: Upon detecting a accompaniment playback command, obtain the current accompaniment audio data and the current data reading time; Write the current data reading time into the current accompaniment audio data; Add the current accompaniment audio data to the accompaniment audio queue.

6. The method according to claim 5, characterized in that, The current playback delay data is obtained in the following ways: Obtain the audio transmission delay data corresponding to the current accompaniment audio data and the hardware playback delay data corresponding to the current accompaniment audio data; The audio transmission delay data represents the delay from reading the current accompaniment audio data to transmitting it to the audio player; the hardware playback delay data represents the hardware playback delay of the audio player playing the current accompaniment audio data. The audio transmission delay data and the hardware playback delay data are superimposed to obtain the current playback delay data.

7. The method according to claim 6, characterized in that, The audio transmission delay data is obtained in the following ways: If an audio data sending command for the current accompaniment audio data is detected, the data sending time corresponding to the audio data sending command is obtained; The audio data sending instruction is used to instruct the current accompaniment audio data to be sent to the audio player; The audio transmission delay data is obtained by performing a difference analysis on the data transmission time and the current data reading time in the current accompaniment audio data.

8. The method according to claim 1, characterized in that, The currently collected delay data is obtained in the following ways: Obtain the first acquisition delay data and the second acquisition delay data corresponding to the current accompaniment audio data; The first acquisition delay data represents the acquisition hardware delay of the audio acquisition device; The second acquisition delay data represents the delay from acquiring the currently acquired audio data corresponding to the current accompaniment audio data to detecting the mixing command for the currently acquired audio data; The first acquisition delay data and the second acquisition delay data are superimposed to obtain the current acquisition delay data.

9. The method according to claim 8, characterized in that, The second acquisition delay data is obtained in the following ways: Upon receiving the currently acquired audio data sent by the audio acquisition device, the current data reception time is obtained; Write the current data reception time into the currently acquired audio data; Upon detecting a mixing command for the currently acquired audio data, the current mixing start time is obtained; A difference analysis is performed on the current mixing start time and the current data reception time in the currently acquired audio data to obtain the second acquisition delay data.

10. The method according to any one of claims 1-9, characterized in that, The step of reading target accompaniment audio data from the accompaniment audio queue based on the modified audio acquisition time includes: Obtain the head accompaniment audio data from the accompaniment audio queue; The data reading time in the head accompaniment audio data and the modified audio acquisition time are matched to obtain the target matching result; If the target matching result indicates that the data reading time in the head accompaniment audio data matches the modified audio acquisition time, the head accompaniment audio data is used as the target accompaniment audio data, and the head accompaniment audio data is removed from the accompaniment audio queue.

11. The method according to claim 10, characterized in that, The method further includes: If the target matching result indicates that the data reading time in the head accompaniment audio data does not match the modified audio acquisition time, the head accompaniment audio data shall be retained.

12. An audio processing device, characterized in that, The device includes: The first data acquisition module is used to acquire the currently acquired audio data, the current audio acquisition time corresponding to the currently acquired audio data, and the current accompaniment delay data; the current accompaniment delay data is determined based on the current playback delay data corresponding to the currently acquired audio data and the current acquisition delay data corresponding to the currently acquired audio data. The modification module is used to modify the current audio acquisition time based on the current accompaniment delay data to obtain the modified audio acquisition time corresponding to the current acquired audio data; The target accompaniment reading module is used to read target accompaniment audio data from the accompaniment audio queue based on the modified audio acquisition time; The mixing processing module is used to perform mixing processing on the target accompaniment audio data and the currently acquired audio data to obtain mixed audio data.

13. An electronic device, characterized in that, include: processor; Memory used to store computer programs; The processor is configured to execute the computer program to implement the audio processing method according to any one of claims 1 to 11.

14. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the audio processing method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by an electronic device, it implements the audio processing method according to any one of claims 1 to 11.

Images