Audio processing method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an audio processing method, device and equipment and a computer readable storage medium, and the method comprises the steps: receiving audio data, decoding the audio data to generate decoded audio data, storing the decoded audio data in a preset data buffer area, and storing the decoded audio data in the preset data buffer area. And determining a delay state according to the storage time of the decoded audio data in the data buffer area, representing the delayed storage of the decoded audio data in the data buffer area in response to the delay state, and determining a delay level according to the storage time. Therefore, the delay state and the delay level of the audio data are determined according to the storage time of the decoded audio data stored in the data buffer area, and the audio delay analysis efficiency of the audio receiving end can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to an audio processing method, device, equipment and computer-readable storage medium. Background Art

[0002] In some audio and video synchronization scenarios, for example, when users use Bluetooth headphones to watch videos or play games, if audio delay and playback freeze occur, it will bring a very poor user experience to users. Therefore, the analysis and solution of audio delay problems are of great significance to achieve higher requirements for audio and video synchronization and improve user experience.

[0003] In Bluetooth audio data transmission, the audio data to be played is usually encoded by the sending device (Audio Source) and sent to the receiving device (Audio Sink). The receiving device decodes the received audio data to restore the audio signal and plays it through the audio subsystem. Therefore, it is necessary to analyze the audio delay on the receiving device side of the audio data transmission. Summary of the invention

[0004] In view of this, an object of the embodiments of the present invention is to provide an audio processing method, apparatus, device and computer-readable storage medium to improve the efficiency of audio delay analysis at an audio receiving end.

[0005] In a first aspect, an embodiment of the present invention is to provide an audio processing method, the method comprising:

[0006] Receive audio data;

[0007] Decoding the audio data to generate decoded audio data;

[0008] Storing the decoded audio data in a pre-set data buffer, wherein the data buffer corresponds to the encoding and decoding method of the audio data;

[0009] Determining a delay state according to a storage time of the decoded audio data into the data buffer;

[0010] In response to the delay status characterizing that the decoded audio data is delayed in being stored in the data buffer, a delay level is determined according to the storage time.

[0011] Furthermore, the method further comprises:

[0012] A timestamp is generated for each processing node in the decoding process of the audio data, so as to perform delay positioning according to each timestamp when audio delay occurs.

[0013] Further, the decoding process includes multiple processing stages, and the delay positioning according to each of the timestamps includes:

[0014] Determine the processing duration of each processing stage, wherein the processing duration is determined according to the timestamp of the processing node of the corresponding processing stage;

[0015] In response to the processing time being greater than a corresponding processing time threshold, it is determined that a delay occurs in the corresponding processing stage.

[0016] Further, determining the delay state according to the storage time of the decoded audio data into the data buffer includes:

[0017] The delay state is determined based on the decoding completion time of the audio data and the storage time.

[0018] Further, determining the delay state according to the decoding completion time of the audio data and the storage time includes:

[0019] In response to a difference between the storage time and the decoding completion time being greater than a transmission time threshold, it is determined that the decoded audio data is delayed in being stored in the data buffer.

[0020] Further, determining the delay level according to the deposit time includes:

[0021] In response to the difference between the storage time and the decoding completion time being greater than a first preset value, the delay level is determined to be a first level, and the first preset value is a maximum delay allowable time corresponding to the data buffer.

[0022] Further, determining the delay level according to the deposit time also includes:

[0023] In response to the difference between the storage time and the decoding completion time being greater than a second preset value and less than a first preset value, the delay level is determined to be a second level, and the second preset value is less than the first preset value.

[0024] Furthermore, the method further comprises:

[0025] Prompt information of a corresponding level is sent according to the delay level.

[0026] Furthermore, the method further comprises:

[0027] The processing stage corresponding to the delay positioning is optimized according to the log information generated during the processing of the audio data.

[0028] Furthermore, the method further comprises:

[0029] The audio data in the data buffer is played.

[0030] In a second aspect, an embodiment of the present invention is intended to provide an audio processing device, the device comprising:

[0031] A receiving unit, used for receiving audio data;

[0032] A decoding unit, used for decoding the audio data to generate decoded audio data;

[0033] A storage unit, used for storing the decoded audio data into a preset data buffer, wherein the data buffer corresponds to the encoding and decoding mode of the audio data;

[0034] a delay determination unit, configured to determine a delay state according to a storage time of the decoded audio data into the data buffer;

[0035] The delay prompt unit is used to store the decoded audio data in the data buffer in response to the delay state representation, and determine the delay level according to the storage time.

[0036] In a third aspect, an embodiment of the present invention is intended to provide an audio processing device, the audio processing device comprising:

[0037] a wireless communication module configured to receive audio data;

[0038] A processor is configured to execute any of the methods described above.

[0039] In a fourth aspect, an embodiment of the present invention aims to provide a computer program product, wherein the computer program product comprises a computer program / instructions, and the computer program / instructions implement any of the methods described above when executed by a processor.

[0040] In a fifth aspect, an embodiment of the present invention aims to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps as described in any one of the above items are implemented.

[0041] The technical solution of the embodiment of the present invention receives audio data, decodes the audio data to generate decoded audio data, stores the decoded audio data in a pre-set data buffer, determines a delay state according to the storage time of the decoded audio data in the data buffer, stores the decoded audio data in the data buffer in response to the delay state, and determines a delay level according to the storage time. Thus, by determining the delay state and delay level of the audio data according to the storage time of the decoded audio data in the data buffer, audio delay analysis at the audio receiving end is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0043] Figure 1 It is a schematic diagram of an audio receiving device with an HCI structure in the prior art;

[0044] Figure 2 is a schematic diagram of an audio processing device according to an embodiment of the present invention;

[0045] Figure 3 is an architecture diagram of an audio processing system according to an embodiment of the present invention;

[0046] Figure 4 is an audio data processing flow chart of an audio processing device according to an embodiment of the present invention;

[0047] Figure 5 is a flowchart of an audio processing method according to an embodiment of the present invention;

[0048] Figure 6 is an example diagram of the encoding method of an embodiment of the present invention;

[0049] Figure 7 is a flowchart of an audio delay optimization method according to an embodiment of the present invention;

[0050] Figure 8 is a flow chart of a delay positioning method according to an embodiment of the present invention;

[0051] Fig. 9 is a schematic diagram of an audio processing device according to an embodiment of the present invention;

[0052] Fig.10 is another schematic diagram of an audio processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0054] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0055] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0056] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0057] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0058] When the audio and video are out of sync, adding audio delay detection at the audio receiving end can greatly facilitate solving the audio delay problem. Figure 1 The HCI structure shown, the hardware composition includes a Bluetooth chip 1 (also known as a controller), an HCI interface 2 and a host chip 3 (also known as a host). The HCI interface is a standard Bluetooth interface, which is used to enable the host chip host to send commands and data to the Bluetooth chip controller, and the host chip host receives events (events) and data from the Bluetooth chip controller. However, the prior art lacks audio delay analysis of the audio receiving end, or only based on the HCI structure, the chip manufacturer performs frame packet delay monitoring at the Bluetooth chip level, including air interface transmission, coexistence, scheduling and other delay detection, but the frame packet arrives at the host chip side on the other side of the HCI interface without any detection. When the audio quality problem of the audio receiving end needs to be tested, there is no other auxiliary detection means, and the audio delay analysis efficiency can only be low by manually reproducing and capturing the logs of the air interface and the host protocol stack. In view of this, an embodiment of the present invention aims to provide a data processing device and method to improve the audio delay analysis efficiency of the audio receiving end.

[0059] Figure 2 Schematic diagram of an audio processing device according to an embodiment of the present invention. Figure 2As shown, the audio processing device A in this embodiment is deployed at the audio receiving end, including a wireless communication module A1 and a processor A2. Optionally, the audio sending device B in this embodiment uses Bluetooth wireless communication technology to establish a communication connection with the audio processing device A. The audio sending device B can be a mobile phone, a computer or other multimedia device that can be used as an audio source, the audio processing device A can be a wireless communication electronic device such as a Bluetooth headset, a Bluetooth speaker, a car Bluetooth, etc., and the audio playback device C is a speaker built into the audio processing device. When performing audio transmission, the audio sending device B encodes the audio data to be played and sends it to the audio processing device A through the wireless communication module A1. After the audio processing device A receives the audio data, the processor A2 decodes the encoded audio data, restores the PCM byte stream corresponding to the audio data to be played, and then plays it through the audio playback device C.

[0060] Optionally, when performing audio delay analysis, the wireless communication module A1 in this embodiment is configured to receive audio data. The processor A2 is configured to decode the audio data to generate decoded audio data, store the decoded audio data into a pre-set data buffer, determine the delay state according to the storage time of the decoded audio data into the data buffer, store the decoded audio data into the data buffer in response to the delay state characterization, and determine the delay level according to the time. Among them, the data buffer corresponds to the encoding and decoding method of the audio data. Therefore, in this embodiment, by pre-setting a data buffer at the audio receiving end, and determining the delay state and delay level of the audio according to the storage time of the decoded audio data delayed into the data buffer, it is possible to realize the automated analysis of the audio delay of the audio receiving end, and provide a reliable basis for solving the subsequent audio delay problem.

[0061] Figure 3 is a diagram of the architecture of the audio transmission system according to an embodiment of the present invention. Figure 3The audio transmission system shown adopts HCI structure and is deployed in the audio processing device. From the hardware level, the audio transmission system mainly includes a Bluetooth chip (controller), a host chip (host) and an HCI interface connecting the Bluetooth chip and the host chip. The physical interface standard adopted by the HCI interface can be UART (Universal Asynchronous Receiver and Transmitter), and the HCI communication between the Bluetooth chip and the host chip is realized through the UART interface. At the software level, it includes a Bluetooth protocol stack 10, a host protocol stack 20, an application layer 30, and an HCI and peripheral layer 40. The Bluetooth protocol stack 10 is used to realize the Bluetooth communication between the audio processing device and the audio sending device, the host protocol stack 20 is used to manage the Bluetooth communication, the host protocol stack 20 and the Bluetooth protocol stack 10 communicate and interact through the HCI and the peripheral layer 40, and the application layer 30 is used to realize the Bluetooth application. Therefore, through the cooperation between the Bluetooth protocol stack 10, the host protocol stack 20 and the application layer 30, the audio sending device sends the audio data to be played to the audio processing device through Bluetooth communication, and then the audio processing device processes the received audio data, and then the audio signal is played through the audio playing device.

[0062] like Figure 3 As shown, the Bluetooth protocol stack 10 in this embodiment includes a Bluetooth baseband 11, a Bluetooth radio 12 and a Bluetooth protocol unit 13 (i.e., LMP shown in the figure). Among them, the Bluetooth baseband 11, as the bottom layer protocol, is responsible for managing the data link layer and physical layer of Bluetooth communication, and is specifically responsible for the encoding, decoding and transmission of data and the modulation and demodulation of the physical layer, as well as managing the establishment, maintenance and release of Bluetooth connections. As the physical layer in Bluetooth communication, the Bluetooth radio 12 is responsible for command communication with the remote audio sending device, converting digital signals into wireless signals, and transmitting them on the wireless channel. It is specifically responsible for the modulation, demodulation and amplification of wireless signals, the selection and management of wireless channels, and the management of parameters such as power and frequency of Bluetooth communication to ensure the quality of wireless communication between Bluetooth devices. The Bluetooth protocol unit 13 is responsible for managing the establishment, maintenance and release of Bluetooth connections. Through the collaborative work of LMP (a protocol in Bluetooth audio transmission), Bluetooth audio transmission can achieve high-quality and stable audio transmission effects and provide safe transmission guarantees.

[0063] Optionally, the host protocol stack 20 in this embodiment includes a protocol layer 21 (L2CAP), a protocol layer 22 (AVDTP), a decoding library 23, a protocol layer 24 (SDP) and a queue management layer 25. Among them, the protocol layer 21 uses the L2CAP protocol (Logical Link Control and Adaptation Layer Protocol), which can provide connection-oriented and connectionless data services to upper layer protocols with protocol multiplexing capabilities and segmentation and reorganization operations. The AVDTP protocol layer 22 includes an audio data unit (AVDTP Media), a channel unit (AVDTP Signaling) and a general data unit (AVDTP Data). The audio data unit is used to provide a data frame transmission path for Bluetooth audio data. AVDTP Signaling, as a control link of Bluetooth media A2DP, is responsible for controlling the playback, suspension, and shutdown of A2DP audio streams. AVDTP Data is responsible for the transmission of Bluetooth media data frames. AVDTP Signaling and AVDTPData transmit data via the L2CAP protocol. The decoding library 23 is used to decode audio data in different encoding formats into original data for subsequent playback, editing, processing, and other operations. The SDP protocol layer 24 is responsible for service discovery and description between Bluetooth devices, including service records, service descriptions, etc. It is specifically used to discover audio sending devices and establish connections under the L2CAP protocol for AVDTP Signaling and AVDTP Data in the protocol layer 22. The queue management layer 25 serves as the host protocol application interface layer and the Bluetooth media encoding frame packet queue and play queue logic processing. Among them, the application interface layer is mainly responsible for interacting and communicating with the application to realize the operation and control of the Bluetooth device. The Bluetooth media encoding frame packet queue is used to store and manage Bluetooth media data. The play queue logic processing is used to process the playback of audio data, specifically for obtaining media data from the Bluetooth media encoding frame packet queue, processing and decoding according to the playback rules, and finally outputting it to the audio device for playback.

[0064] Furthermore, in order to facilitate the playback and delay analysis of audio data, a data buffer corresponding to the encoding and decoding method is pre-set in this embodiment. The decoded audio data is stored in the data buffer, and the decoded audio data in the data buffer is managed and controlled in combination with the queue management layer 25 in the architecture, so that the audio playback can be achieved by extracting the audio data in the data buffer. At the same time, by detecting the decoding process of the audio data and the operation of storing the decoded data in the data buffer, a convenient and reliable basis is provided for the subsequent audio delay analysis.

[0065] Figure 4 : is an audio data processing flow chart of the audio processing device according to an embodiment of the present invention. Figure 4As shown, the flow of audio data in the audio processing device includes the following processes.

[0066] In step S01 , the Bluetooth chip receives Bluetooth media frame packet data.

[0067] In this embodiment, the audio data transmitted from the audio transmitting device is received via Bluetooth radio frequency, and the audio data is transmitted in the form of frame packets.

[0068] In step S02, the Bluetooth radio frequency is decoded and sent to the lower software layer.

[0069] In this embodiment, the audio data is sent to the lower software layer after being decoded by Bluetooth radio frequency.

[0070] In step S03, the Bluetooth chip performs internal scheduling processing.

[0071] In step S04, the data is transmitted via a low-speed UART or other peripheral bus.

[0072] In step S05, HCI peripheral layer transmission processing.

[0073] In this embodiment, the audio data processed by the Bluetooth chip is transmitted via the HCI peripheral layer and sent to the host chip.

[0074] In step S06, the host chip receives the Bluetooth media frame packet.

[0075] In step S07, it is added to the corresponding Bluetooth media frame queue.

[0076] In this embodiment, after receiving the Bluetooth media frame packet sent from the peripheral device, the host chip first adds the Bluetooth media frame packet to the corresponding Bluetooth media frame queue.

[0077] In step S08, dequeue and decode processing.

[0078] In this embodiment, the audio data in the Bluetooth media frame queue is dequeued and decoded in a predetermined order to generate a corresponding PCM data stream.

[0079] In step S09, the decoded audio data is stored in a data buffer.

[0080] In this embodiment, the PCM data stream generated by the decoding process of the host chip is stored in a pre-set data buffer, which corresponds to the encoding and decoding method of the audio data.

[0081] In step S10, flow to the audio subsystem.

[0082] In this embodiment, the decoded audio data in the data buffer is extracted, and the decoded audio data is transmitted to the playback device corresponding to the audio subsystem for playback, thereby realizing the transmission and wireless playback of the audio data.

[0083] In step S11, audio delay analysis is performed.

[0084] After receiving the Bluetooth media data frame from the RF hardware module, the corresponding audio data will pass through the Bluetooth chip software system, HCI host controller interface protocol layer, peripheral hardware transmission layer, Bluetooth media frame queue storage, Bluetooth decoding unit module, decoded playback stream queue, audio host system in sequence, and these paths are all processed asynchronously, so there is a possibility of scheduling delay problems, which will cause audio freeze or delay, making the played audio signal out of sync with the audio and video signal of the audio sending device, affecting the user experience. Therefore, in this embodiment, the audio delay during audio transmission will also be analyzed.

[0085] Optionally, in this embodiment, audio delay analysis of audio data is achieved by detecting the operations of decoding audio data and storing the audio data in a data buffer.

[0086] In the technical solution of this embodiment, the audio data transmitted by the audio transmitting device is received by the Bluetooth radio frequency, and the audio data is transmitted in the form of frame packets. After being decoded by the Bluetooth radio frequency, the audio data is sent to the low-level software layer, and after being processed by the internal scheduling of the Bluetooth chip, it is transmitted through the low-speed UART or other peripheral bus, and then transmitted through the HCI layer peripheral and sent to the host chip. After the host chip receives the Bluetooth media frame packet sent from the peripheral, it first adds the Bluetooth media frame packet to the corresponding Bluetooth media frame queue, and then dequeues and decodes it to generate the corresponding PCM data stream, and transmits the PCM data stream to the corresponding playback device of the audio subsystem for playback, thereby realizing the transmission and wireless playback of audio data. At the same time, by detecting the decoding of the audio data and the operation of storing it in the data buffer, the audio delay analysis of the audio data is realized.

[0087] Figure 5 Schematic diagram of an audio processing method according to an embodiment of the present invention. Figure 5 As shown, the audio processing method in this embodiment is applied to an audio receiving end, and audio delay analysis is implemented through the following steps.

[0088] In step S110, audio data is received.

[0089] In this embodiment, the audio data is the audio data to be played transmitted from the audio sending device. The audio data is transmitted in the form of a data frame packet. Further, in order to analyze the audio delay analysis on the host chip side of the audio receiving end, the audio data in this embodiment is the audio data received by the host chip and transmitted via the Bluetooth chip and the HCI interface.

[0090] In step S120, the audio data is decoded to generate decoded audio data.

[0091] In this embodiment, the received audio data is decoded by a preset coding and decoding method, and decoded audio data is generated. Optionally, the preset coding and decoding method in this embodiment can be used as follows: Figure 6 The encoding method shown includes any one of SBC, AAC, APTX and other encoding and decoding methods or other audio data encoding and decoding methods, which are not limited here.

[0092] Optionally, the decoding process in this embodiment includes multiple processing stages. In the process of decoding the audio data, a timestamp will be generated for each processing node in the decoding process of the audio data, so as to perform delay positioning according to each timestamp when audio delay occurs. Furthermore, the processing process on the host chip side in this embodiment includes at least the following processing nodes: receiving data from the Bluetooth host stack corresponding to the HCI peripheral, entering the received Bluetooth media frame into the queue to be encoded, dequeuing from the queue to be encoded, starting encoding, ending encoding, entering the queue of the media to be played, and the data stream to be written to the audio interface.

[0093] Furthermore, to facilitate the subsequent audio delay analysis, in this embodiment, a timestamp is generated for each processing node in the decoding process of the audio data, so that the delay location is performed according to each timestamp when the audio delay occurs. That is, starting from the reception of the audio data by the host chip, the timestamp T1 of the data received from the Bluetooth host stack corresponding to the HCI peripheral, the timestamp T2 of the received Bluetooth media frame entering the queue to be encoded, the timestamp T3 of the queue being dequeued from the queue to be encoded, the timestamp T4 of the start of encoding, the timestamp T5 of the end of encoding, the timestamp T6 of the queue of the media to be played, and the timestamp T7 of the data stream to be written to the audio interface are recorded.

[0094] In step S130, the decoded audio data is stored in a preset data buffer, where the data buffer corresponds to the encoding and decoding method of the audio data.

[0095] In this embodiment, usually, the decoded audio data will be first stored in a pre-set data buffer, and after the number of data frame packets stored in the data buffer reaches a predetermined number, they will be transferred in batches to the audio subsystem for playback. Figure 6 As shown in , different encoding and decoding methods have corresponding maximum delay times. If the audio data is still not stored in the data buffer after the maximum delay time has elapsed, it will cause audio quality problems such as audio and video synchronization and audio delay. That is, when the audio delay is stored in the data buffer for a period longer than the maximum delay time, it will cause serious problems with audio quality, affecting the synchronous playback of audio and video.

[0096] Furthermore, in this embodiment, in order to facilitate delay analysis, a data buffer is set in advance according to the codec method used in the audio transmission process. The playback duration corresponding to the data buffer corresponds to the maximum delay allowable time corresponding to the adopted codec method. For example, when the SBC codec method is adopted, the corresponding data buffer size is 220ms.

[0097] Optionally, the size of the data buffer in this embodiment is configured based on the following manner.

[0098] Data buffer size = (f×l×n)×(t max +v) / 1000

[0099] Among them, f is the sampling rate, also known as sampling speed or sampling rate, which refers to the number of samples extracted from the continuous signal and composed of discrete signals per unit time, and the unit is Hertz (Hz); l is the number of sampling channels, and the sampling channels include mono (MONO), stereo (Stereo), 5.1 channels, etc.; n is the number of sampling bits, which refers to how many levels the continuous intensity of the sound can be divided into after being digitally represented; t max is the maximum allowed delay time corresponding to the encoding and decoding method; v is the margin, and the purpose of setting the margin is to maintain the correspondence between the data buffer size and the maximum allowed delay.

[0100] In step S140, the delay state is determined according to the storage time of the decoded audio data into the data buffer.

[0101] In this embodiment, the delay state is determined according to the decoding completion time and the storage time of the audio data. Optionally, in this embodiment, the delay state of the corresponding audio data is determined according to the difference between the storage time and the decoding completion time, and when the difference between the storage time and the decoding completion time is greater than the transmission time threshold, it is determined that the decoded audio data is delayed and stored in the data buffer. Correspondingly, the audio transmission method in this embodiment also includes, in response to the difference between the storage time and the decoding completion time of the audio data being greater than the transmission time threshold, determining that the decoded audio data is delayed and stored in the data buffer.

[0102] It should be understood that the transmission time threshold in this embodiment is set according to the actual usage scenario, and can also be adjusted according to parameters such as the audio channel used for audio transmission and the amount of transmitted audio data, which is not limited here.

[0103] In step S150, in response to the delay status characterizing that the decoded audio data is delayed in being stored in the data buffer, a delay level is determined according to the storage time.

[0104] In this embodiment, different delay states correspond to different delay levels. Optionally, three delay levels are pre-set in this embodiment. When the difference between the deposit time and the decoding completion time is greater than the first preset value, the corresponding delay level is determined to be the first level; when the difference between the deposit time and the decoding completion time is greater than the second preset value, the delay level is determined to be the second level; when the difference between the deposit time and the decoding completion time is greater than the second preset value and less than the first preset value, the delay level is determined to be the second level; when the difference between the deposit time and the decoding completion time is greater than the third preset value and less than the second preset value, the delay level is determined to be the third level. Among them, the first preset value in this embodiment is the maximum delay allowable time corresponding to the data buffer, the second preset value is a value less than the first preset value, and the third preset value is less than the second preset value.

[0105] For example, in the process of using the SBC codec method for audio transmission, the corresponding maximum delay allowed time is 220ms. Assuming that the decoding completion time of the audio data is t0, the storage time of the decoded audio data in the data buffer is t1, and the difference t1-t0 between the storage time t1 and the decoding completion time is recorded as Δt, when Δt reaches 220ms (that is, Δt≥220ms), it indicates that the audio and video synchronization problem or the quality problem caused by the audio and video has occurred, and the audio delay level is determined to be the first level; when Δt is between 110-220ms (that is, 110≤Δt<220ms), it indicates that the audio and video synchronization problem or the quality problem caused by the audio and video is likely to occur, and the audio delay level is determined to be the second level; when Δt is between 45-110ms (that is, 45≤Δt<110ms), it indicates that the audio and video synchronization problem or the quality problem caused by the audio and video is likely to occur, and the audio delay level is determined to be the third level.

[0106] Optionally, in order to facilitate intuitive understanding of the audio delay situation, the present embodiment will also set corresponding prompt information for different delay states. At the same time, in order to distinguish the degree of delay, each delay level is set with corresponding reminders of different degrees. Correspondingly, after determining the delay level of the audio data, the audio processing method in the present embodiment will also send prompt information of the corresponding level according to the delay level. Continuing with the above example, when the delay level is the first level, add the reminder content of "audio and video synchronization problems or quality problems caused by audio delay have occurred, and need to be analyzed and optimized"; when the delay level is the second level, add the reminder content of "the audio delay is large, and there is a high probability of audio and video synchronization or audio quality problems, which need to be paid special attention to"; when the delay level is the third level, add the reminder content of "there is a delay in Bluetooth audio, which requires attention".

[0107] It should be understood that the number of delay levels in this embodiment can be processed in a finer or coarser grading manner according to actual usage requirements, and the preset values ​​corresponding to the time differences at different delay levels can also be adjusted according to the actual scenario. This is only an example, but not a limitation.

[0108] The technical solution of this embodiment realizes the transmission and playback of audio data by receiving audio data, decoding the audio data to generate decoded audio data, storing the decoded audio data in a data buffer pre-set to correspond to the encoding and decoding mode of the audio data, and playing the audio data in the data buffer. At the same time, in this embodiment, by determining the delay state according to the storage time of the decoded audio data stored in the data buffer, storing the decoded audio data in the data buffer in response to the delay state characterization, and determining the delay level according to the storage time, that is, by analyzing the storage time of the decoded audio data stored in the data buffer to determine the delay state and delay level of the audio data, it is possible to realize the automatic analysis of audio delay at the audio receiving end and improve the efficiency of audio delay analysis.

[0109] Optionally, after determining the delay level corresponding to the audio transmission process, in order to promptly resolve or avoid the problem of audio and video asynchrony or poor audio quality caused by audio delay, this embodiment also provides an audio delay optimization method, which can be integrated into the aforementioned audio processing method.

[0110] Figure 7 is a flow chart of the audio delay optimization method according to an embodiment of the present invention. Figure 7 As shown, the audio delay optimization method in this embodiment includes the following steps.

[0111] In step S210, the delay analysis result is determined.

[0112] In this embodiment, the delay analysis result corresponds to the processing result of the aforementioned audio processing method. The method for determining the delay analysis result can refer to the aforementioned audio processing method, which will not be repeated here.

[0113] In step S160, delayed positioning is performed.

[0114] In this embodiment, delay positioning is performed through the timestamps corresponding to each processing node generated during the decoding process. By analyzing the timestamps of each processing node, the processing stage that causes audio delay during the audio decoding process can be determined, and then the decoding processing stage that causes the delay can be optimized to solve or avoid the problem of audio and video asynchrony or poor audio quality caused by audio delay.

[0115] Alternatively, if Figure 8 As shown, in this embodiment, delayed positioning is performed through the following steps.

[0116] In step S161, the processing duration of each processing stage is determined.

[0117] In this embodiment, the processing duration of each processing stage is determined according to the timestamp of the processing node of the corresponding processing stage. For example, the timestamp content recorded in the above content is used, including the timestamp T1 of receiving data from the Bluetooth host stack corresponding to the HCI peripheral, the timestamp T2 of entering the received Bluetooth media frame into the queue to be encoded, the timestamp T3 of dequeuing from the queue to be encoded, the timestamp T4 of starting encoding, the timestamp T5 of ending encoding, the timestamp T6 of entering the queue to be played, and the timestamp T7 of the data stream to be written to the audio interface. The processing stages corresponding to the above timestamps and the processing durations corresponding to each processing stage are respectively: the processing duration of audio data transmission to the queue to be encoded is T2-T1, the processing duration of audio data waiting in the queue to be encoded is T3-T2, the processing duration of audio data decoding waiting is T4-T3, the processing duration of data decoding is T5-T4, and the processing duration of data transmission after decoding is completed is T6-T5.

[0118] In step S162, in response to the processing time being greater than the corresponding processing time threshold, it is determined that a delay occurs in the corresponding processing stage.

[0119] In this embodiment, the processing time threshold of each processing stage can be determined based on factors such as the amount of data processed and the hardware conditions on which the data processing relies. By comparing the actual processing time of each data stage and the corresponding time processing threshold, it is possible to determine whether the corresponding processing stage has a delay or the degree of delay, and to achieve delay positioning in the audio decoding process, so that subsequent developers can perform local optimization based on the actual delay corresponding to the delay positioning, and avoid problems such as audio and video asynchrony or poor audio quality caused by delays in the decoding process. At the same time, developers can also perform full-link analysis of audio delays through information such as the chips used in the audio processing process, recorded air interface logs, host protocol stack logs, and audio subsystem logs, and optimize the audio transmission process as a whole.

[0120] Furthermore, the audio processing method in this embodiment also includes the following steps.

[0121] In step S170, the processing stage corresponding to the delay positioning is optimized according to the log information generated during the processing of the audio data.

[0122] The technical solution of this embodiment receives audio data, decodes the audio data to generate decoded audio data, stores the decoded audio data in a data buffer pre-set to correspond to the encoding and decoding mode of the audio data, and plays the audio data in the data buffer to achieve the transmission and playback of audio data. At the same time, in this embodiment, by determining the delay state according to the storage time of the decoded audio data in the data buffer, responding to the delay state characterization, the decoded audio data is delayed and stored in the data buffer, and determining the delay level according to the storage time, it is possible to realize the automatic analysis of audio delay and improve the efficiency of audio delay analysis. In addition, by sending the corresponding level of prompt information according to the delay level, it is convenient to timely discover the audio delay problem and solve the problem of audio and video synchronization or poor audio quality caused by audio delay, and by optimizing the processing stage corresponding to the delay positioning according to the log information generated during the processing of the audio data, it is possible to solve the problem of audio and video synchronization or poor audio quality caused by audio delay, thereby improving the user's audio experience.

[0123] Fig. 9 Schematic diagram of an audio processing device according to an embodiment of the present invention. Fig. 9 As shown, the audio processing device in this embodiment includes a receiving unit 81, a decoding unit 82, a storage unit 83, a delay determination unit 84 and a delay prompt unit 85. Among them, the receiving unit 81 is used to receive audio data. The decoding unit 82 is used to decode the audio data to generate decoded audio data. The storage unit 83 is used to store the decoded audio data into a pre-set data buffer. Among them, the data buffer corresponds to the encoding and decoding method of the audio data, and the size of the data buffer under different encoding and decoding methods is different. The delay determination unit 84 is used to determine the delay state according to the storage time of the decoded audio data into the data buffer. The delay prompt unit 85 is used to respond to the delay state characterization that the decoded audio data is delayed and stored in the data buffer, and determine the delay level according to the storage time.

[0124] Optionally, the decoding unit 82 in this embodiment is also used to generate a timestamp for each processing node in the decoding process of the audio data, so as to perform delay positioning according to each timestamp when audio delay occurs. Wherein, the decoding process includes multiple processing stages, and the decoding unit 82 is specifically used to determine the processing time of each processing stage when performing delay positioning, and in response to the processing time being greater than the corresponding processing time threshold, determine that the corresponding processing stage is delayed. Wherein, the processing time is determined according to the timestamp of the processing node of the corresponding processing stage.

[0125] Optionally, the delay determination unit 84 in this embodiment is specifically used to determine the delay state according to the decoding completion time and the storage time of the audio data when determining the delay state according to the storage time of the decoded audio data into the data buffer, and in response to the difference between the storage time and the decoding completion time being greater than the transmission time threshold, determine that the decoded audio data is delayed and stored in the data buffer.

[0126] Alternatively, if Fig.10 As shown, when determining the delay level according to the storage time, the delay prompt unit 85 in this embodiment is specifically used to determine the delay level as the first level in response to the difference between the storage time and the decoding completion time being greater than the first preset value; and to determine the delay level as the second level in response to the difference between the storage time and the decoding completion time being greater than the second preset value and less than the first preset value. The first preset value is the maximum delay allowable time corresponding to the data buffer, and the second preset value is less than the first preset value. Further, the delay prompt unit 85 in this embodiment is also used to send prompt information of the corresponding level according to the delay level.

[0127] Optionally, the audio processing device in this embodiment further includes a delay optimization unit 86, and the delay optimization unit 86 is used to optimize the processing stage corresponding to the delay positioning according to the log information generated during the processing of the audio data.

[0128] Optionally, the audio processing device in this embodiment further includes an audio playing unit 87, and the audio playing unit 87 is used to play the audio data in the data buffer.

[0129] The technical solution of this embodiment receives audio data through a receiving unit, a decoding unit decodes the audio data to generate decoded audio data, a storage unit stores the decoded audio data in a data buffer corresponding to a pre-set encoding and decoding mode of the audio data, and plays the audio data in the data buffer through an audio playback unit, thereby realizing the transmission and playback of audio data. At the same time, in this embodiment, the delay determination unit determines the delay state according to the storage time of the decoded audio data in the data buffer, the delay prompt unit stores the decoded audio data in the data buffer in response to the delay state characterization, and determines the delay level according to the storage time, which can realize the automatic analysis of audio delay and improve the efficiency of audio delay analysis. In addition, the delay prompt unit sends the prompt information of the corresponding level according to the delay level, which is convenient for timely discovery of audio delay problems and solving the problems of audio and video asynchronism or poor audio quality caused by audio delay. The delay optimization unit can also optimize the processing stage corresponding to the delay positioning according to the log information generated during the processing of the audio data, which can solve the problems of audio and video asynchronism or poor audio quality caused by audio delay, thereby improving the audio experience of users.

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0132] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0133] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0134] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0135] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0136] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An audio processing method, characterized in that: The method comprises: Receive audio data; Decoding the audio data to generate decoded audio data; Storing the decoded audio data in a pre-set data buffer, wherein the data buffer corresponds to the encoding and decoding method of the audio data; Determining a delay state according to a storage time of the decoded audio data into the data buffer; In response to the delay status characterizing that the decoded audio data is delayed in being stored in the data buffer, a delay level is determined according to the storage time.

2. The method according to claim 1, characterized in that The method further comprises: A timestamp is generated for each processing node in the decoding process of the audio data, so as to perform delay positioning according to each timestamp when audio delay occurs.

3. The method according to claim 2, characterized in that The decoding process includes multiple processing stages, and the delay positioning according to each of the timestamps includes: Determine the processing duration of each processing stage, wherein the processing duration is determined according to the timestamp of the processing node of the corresponding processing stage; In response to the processing time being greater than a corresponding processing time threshold, it is determined that a delay occurs in the corresponding processing stage.

4. The method according to claim 1, characterized in that: Determining the delay state according to the storage time of the decoded audio data into the data buffer includes: The delay state is determined based on the decoding completion time of the audio data and the storage time.

5. The method according to claim 4, characterized in that Determining the delay state according to the decoding completion time of the audio data and the storage time comprises: In response to a difference between the storage time and the decoding completion time being greater than a transmission time threshold, it is determined that the decoded audio data is delayed in being stored in the data buffer.

6. The method according to claim 1, characterized in that Determining the delay level according to the deposit time includes: In response to the difference between the storage time and the decoding completion time being greater than a first preset value, the delay level is determined to be a first level, and the first preset value is a maximum delay allowable time corresponding to the data buffer.

7. The method according to claim 6, characterized in that Determining the delay level according to the deposit time further comprises: In response to the difference between the storage time and the decoding completion time being greater than a second preset value and less than a first preset value, the delay level is determined to be a second level, and the second preset value is less than the first preset value.

8. The method according to claim 1, characterized in that The method further comprises: Prompt information of a corresponding level is sent according to the delay level.

9. The method according to claim 3, characterized in that: The method further comprises: The processing stage corresponding to the delay positioning is optimized according to the log information generated during the processing of the audio data.

10. The method according to claim 1, characterized in that The method further comprises: The audio data in the data buffer is played.

11. An audio processing device, characterized in that: The device comprises: A receiving unit, used for receiving audio data; A decoding unit, used for decoding the audio data to generate decoded audio data; A storage unit, used for storing the decoded audio data into a preset data buffer, wherein the data buffer corresponds to the encoding and decoding mode of the audio data; a delay determination unit, configured to determine a delay state according to a storage time of the decoded audio data into the data buffer; The delay prompt unit is used to store the decoded audio data in the data buffer in response to the delay state representation, and determine the delay level according to the storage time.

12. An audio processing device, characterized in that: The audio processing device comprises: a wireless communication module configured to receive audio data; A processor configured to execute the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The computer program product comprises a computer program / instructions, and when the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 10 are implemented.

Images