Audio data processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to an audio data processing method, device, electronic device, and storage medium. The method includes: obtaining first audio data to be processed and a storage status of an ear-return buffer area, wherein the ear-return buffer area is used to cache audio data; when the storage status satisfies a preset status, obtaining second audio data corresponding to the first audio data from the stored audio data; generating target audio data based on the first audio data and the second audio data; and processing the target audio data. Since the present disclosure generates target audio data based on the first audio data and the second audio data when the ear-return buffer area satisfies a preset status, and then processes the target audio data, there is no need to insert silent frames or discard audio data, thereby avoiding the introduction of ear-return noise.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to an audio data processing method, device, electronic device, and storage medium. Background Art

[0002] With the rapid development of Internet and terminal technologies, many applications with karaoke functions have emerged. When using karaoke applications on mobile phones and other devices, earphone monitoring delay is a major factor affecting the user experience.

[0003] In the related art, JAVA API (Application Programming Interface) can be used to implement software ear return. In order to improve the problem of large delay of software ear return, an ear return buffer can be instantiated, and a collection thread can be started to store the collected audio data into the ear return buffer, and then a playback thread can be started to read the audio data from the ear return buffer and play it. Due to CPU jitter, the collection and playback callbacks do not arrive in order, and the following situations need to be handled: when the playback thread needs to get data from the ear return buffer, the ear return buffer is empty. At this time, the playback thread needs to insert a silent frame and play the silent data; when the collection thread needs to put the data into the ear return buffer, the ear return buffer is full. At this time, the problem can be solved by discarding the audio data.

[0004] However, the above strategy of inserting silent frames and discarding audio data will inevitably introduce noise, which will eventually affect the user's ear monitoring experience. Summary of the Invention

[0005] The present disclosure provides an audio data processing method, device, electronic device, and storage medium to at least address the problem of ear-return noise introduced in related technologies. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an audio data processing method, comprising:

[0007] Obtaining first audio data to be processed and a storage status of an earphone feedback buffer area, where the earphone feedback buffer area is used to buffer audio data;

[0008] When the storage state satisfies a preset state, obtaining second audio data corresponding to the first audio data from the stored audio data;

[0009] generating target audio data according to the first audio data and the second audio data;

[0010] The target audio data is processed.

[0011] Optionally, generating target audio data according to the first audio data and the second audio data includes:

[0012] The first audio data and the second audio data are combined to obtain the target audio data.

[0013] Optionally, when the first audio data is remaining audio data obtained from the ear-return buffer area through a playback thread, merging the first audio data and the second audio data to obtain the target audio data includes:

[0014] performing a fade-in process on the second audio data to obtain first intermediate data;

[0015] performing a fade-out process on the first audio data to obtain second intermediate data;

[0016] The first intermediate data and the second intermediate data are merged to obtain the target audio data.

[0017] Optionally, when the first audio data is audio data collected by an acquisition thread, merging the first audio data and the second audio data to obtain the target audio data includes:

[0018] performing a fade-out process on the second audio data to obtain third intermediate data;

[0019] performing a fade-in process on the first audio data to obtain fourth intermediate data;

[0020] The third intermediate data and the fourth intermediate data are combined to obtain the target audio data.

[0021] Optionally, when the first audio data is remaining audio data obtained from the earphone feedback buffer through the playback thread, the preset state is that the number of frames of the remaining audio data in the earphone feedback buffer is a first target frame number;

[0022] When the storage state satisfies a preset state, obtaining second audio data corresponding to the first audio data from the stored audio data includes:

[0023] When the storage state is that the number of frames of the remaining audio data in the ear return buffer is the first target number of frames, the second audio data corresponding to the first audio data is obtained from the history buffer through the playback thread.

[0024] Optionally, after processing the target audio data, the method further includes:

[0025] The target audio data is stored in the history buffer area through the playback thread.

[0026] Optionally, when the first audio data is audio data collected by a collection thread, the preset state is that the ear return buffer area is insufficient to cache the first audio data collected by the collection thread;

[0027] The acquiring, when the storage state satisfies a preset state, second audio data corresponding to the first audio data from the stored audio data includes: acquiring, by the acquisition thread, the second audio data corresponding to the first audio data from the earphone feedback buffer area when the storage state indicates that the earphone feedback buffer area is insufficient to buffer the first audio data;

[0028] The processing of the target audio data includes: caching the target audio data to a location where the second audio data is located in the ear return buffer area through the acquisition thread.

[0029] Optionally, acquiring, by the acquisition thread, the second audio data corresponding to the first audio data from the ear-return buffer area includes:

[0030] The acquisition thread obtains the audio data of the second target frame number cached last from the ear return buffer area as the second audio data.

[0031] Optionally, also include:

[0032] When the storage status satisfies the preset status and the playback duration of the audio data cached in the ear-return buffer area is less than the ear-return delay threshold, the capacity of the ear-return buffer area is expanded, wherein the playback duration of the audio data cached in the ear-return buffer area after the capacity expansion is less than or equal to the ear-return delay threshold.

[0033] According to a second aspect of an embodiment of the present disclosure, there is provided an audio data processing apparatus, comprising:

[0034] A first acquisition module is configured to acquire first audio data to be processed and a storage status of an ear-return buffer area, where the ear-return buffer area is used to cache audio data;

[0035] a second acquisition module configured to acquire, when the storage state satisfies a preset state, second audio data corresponding to the first audio data from the stored audio data;

[0036] a data generating module, configured to generate target audio data according to the first audio data and the second audio data;

[0037] The data processing module is configured to process the target audio data.

[0038] Optionally, the data generation module is configured to execute:

[0039] The first audio data and the second audio data are combined to obtain the target audio data.

[0040] Optionally, when the first audio data is remaining audio data obtained from the ear-return buffer area through a playback thread, the data generation module includes:

[0041] a first fade-in processing unit, configured to perform fade-in processing on the second audio data to obtain first intermediate data;

[0042] a first fade-out processing unit, configured to perform fade-out processing on the first audio data to obtain second intermediate data;

[0043] The first data merging unit is configured to merge the first intermediate data and the second intermediate data to obtain the target audio data.

[0044] Optionally, when the first audio data is audio data collected by an acquisition thread, the data generation module includes:

[0045] a second fade-out processing unit, configured to perform fade-out processing on the second audio data to obtain third intermediate data;

[0046] a second fade-in processing unit, configured to perform fade-in processing on the first audio data to obtain fourth intermediate data;

[0047] The second data merging unit is configured to merge the third intermediate data and the fourth intermediate data to obtain the target audio data.

[0048] Optionally, when the first audio data is remaining audio data obtained from the earphone feedback buffer through the playback thread, the preset state is that the number of frames of the remaining audio data in the earphone feedback buffer is a first target frame number;

[0049] The second acquisition module is configured to execute:

[0050] When the storage state is that the number of frames of the remaining audio data in the ear return buffer is the first target number of frames, the second audio data corresponding to the first audio data is obtained from the history buffer through the playback thread.

[0051] Optionally, the device further includes:

[0052] The target data cache module is configured to execute the storage of the target audio data into the history cache area through the playback thread.

[0053] Optionally, when the first audio data is audio data collected by a collection thread, the preset state is that the ear return buffer area is insufficient to cache the first audio data collected by the collection thread;

[0054] The second acquisition module is configured to execute: when the storage status indicates that the ear-return buffer area is insufficient to cache the first audio data, acquiring, through the acquisition thread, second audio data corresponding to the first audio data from the ear-return buffer area;

[0055] The data processing module is configured to execute: caching the target audio data to a location where the second audio data is located in the ear return buffer area through the acquisition thread.

[0056] Optionally, the second acquisition module includes:

[0057] The second data acquisition unit is configured to execute, through the acquisition thread, acquiring the audio data of the second target number of frames cached last from the ear return buffer area as the second audio data.

[0058] Optionally, the device further includes:

[0059] The capacity expansion module is configured to expand the capacity of the ear return buffer area when the storage state satisfies the preset state and the playback duration of the audio data cached in the ear return buffer area is less than the ear return delay threshold, wherein the playback duration of the audio data cached in the ear return buffer area after the capacity expansion is less than or equal to the ear return delay threshold.

[0060] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0061] processor;

[0062] a memory for storing instructions executable by the processor;

[0063] The processor is configured to execute the instructions to implement the audio data processing method as described in the first aspect.

[0064] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the audio data processing method as described in the first aspect.

[0065] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program or computer instructions, which, when executed by a processor, implements the audio data processing method described in the first aspect.

[0066] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0067] The embodiment of the present disclosure obtains the first audio data to be processed and the storage status of the ear return buffer area, and when the storage status meets the preset status, obtains the second audio data corresponding to the first audio data from the stored audio data, generates target audio data based on the first audio data and the second audio data, and processes the target audio data. Since the target audio data is generated based on the first audio data and the second audio data when the ear return buffer area meets the preset status, and then the target audio data is processed, there is no need to insert silent frames or discard audio data, which can avoid introducing ear return noise and improve the ear return audio effect.

[0068] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0070] Figure 1 is a flowchart of a method for processing audio data according to an exemplary embodiment;

[0071] Figure 2 is a flowchart of a method for processing audio data according to an exemplary embodiment;

[0072] Figure 3 is a flowchart of a method for processing audio data according to an exemplary embodiment;

[0073] Figure 4 is a block diagram of an audio data processing device according to an exemplary embodiment;

[0074] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0075] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0076] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0077] When solving the problem of ear-return delay, some manufacturers will implement hardware ear-return and provide a customized manufacturer ear-return API (Application Programming Interface). When using the manufacturer ear-return API, the delay of the ear-return is optimized by the manufacturer for a specific device, and the delay is lower than that of the software ear-return. If the device supports it, the manufacturer ear-return is generally preferred. However, if the device does not support the manufacturer ear-return, it is necessary to implement the ear-return effect through software, which is also called software ear-return. The implementation principle of software ear-return is to collect the user's pronunciation through the microphone in the device and play the collected data as soon as possible, so that the user can hear his own pronunciation in time and adjust his subsequent pronunciation accordingly.

[0078] In the related art, Java API can be used to implement software ear return. During the implementation process, media recorder (Media Recorder) and media player (Media Player) are mainly used to collect and play audio. However, this method has a large ear return delay, which reduces the user experience. In order to improve the problem of large delay in software ear return, an audio development framework (such as OpenSL ES API, AAudio API, Oboe API) can be used to implement software ear return. When implementing software ear return, these APIs instantiate an ear return buffer area for storing ear return data, start a collection thread to store the collected audio data in the ear return buffer area, and then start a playback thread to read the audio data from the ear return buffer area for playback. Due to CPU jitter, the collection and playback callbacks do not arrive in order, which will lead to the introduction of noise in extreme cases, affecting the user experience. In order to solve this technical problem, the present disclosure provides the following technical solutions.

[0079] Figure 1 is a flowchart of an audio data processing method according to an exemplary embodiment. Figure 1 As shown, the audio data processing method is used in electronic devices such as mobile phones and tablet computers, and includes the following steps.

[0080] In step S11, first audio data to be processed and a storage status of an ear-return buffer area are obtained. The ear-return buffer area is used to cache audio data.

[0081] The in-ear audio buffer is used to cache in-ear audio data. The audio data collected by the acquisition thread is cached in the in-ear audio buffer. The playback thread retrieves and plays the audio data from the in-ear audio buffer. The storage state can be full, nearly empty, or normal. Normal state indicates that there is sufficient space to store the audio data to be collected by the acquisition thread and sufficient audio data for the playback thread to obtain.

[0082] The audio data processing method in the disclosed embodiments can be applied, but is not limited to, to processing audio data when implementing software earphone monitoring using the OpenSL ES API, AAudio API, and Oboe API, to avoid introducing earphone noise in extreme situations where the earphone buffer is full or empty. OpenSL ES, supported since Android 2.3, offers high performance and low latency. AAudio is a new API introduced in Android O (Android 8.1), designed specifically for low-latency, high-performance audio applications. Oboe, supported since Android 4.1, uses the AAudio API for devices running Android 8.1 and above, and OpenSL ES for compatibility with devices running earlier versions.

[0083] When implementing software in-ear feedback, the first audio data to be processed can be audio data retrieved from the in-ear feedback buffer by the playback thread for playback, or audio data just collected by the acquisition thread and needing to be cached in the in-ear feedback buffer. Before the playback thread needs to retrieve the audio data to be played from the in-ear feedback buffer, the storage status of the in-ear feedback buffer is obtained; alternatively, when the acquisition thread has collected audio data and needs to cache it in the in-ear feedback buffer, the storage status of the in-ear feedback buffer is obtained.

[0084] In step S12, when the storage state meets the preset state, second audio data corresponding to the first audio data is obtained from the stored audio data.

[0085] The stored audio data may be audio data stored in the in-ear return buffer, or audio data played in the history buffer. The preset state may be a full state or a state about to be empty.

[0086] When the in-ear feedback buffer is full, second audio data corresponding to the first audio data is retrieved from the in-ear feedback buffer. When the in-ear feedback buffer is about to be empty, the second audio data corresponding to the first audio data is retrieved from the history buffer. The number of frames of the second audio data can be the same as the number of frames of the first audio data.

[0087] In step S13 , target audio data is generated based on the first audio data and the second audio data.

[0088] The first audio data and the second audio data are combined to generate new audio data to obtain target audio data.

[0089] In an exemplary embodiment, generating target audio data according to first audio data and second audio data includes: merging the first audio data and the second audio data to obtain the target audio data.

[0090] The number of frames of the first audio data can be the same as the number of frames of the second audio data. The first audio data and the second audio data are merged so that the number of frames of the target audio data obtained after the merger is the same as the number of frames of the first audio data and the number of frames of the second audio data. For example, when the first audio data is one frame and the second audio data is one frame, the number of frames of the target audio data obtained after the merger is also one frame. In this way, the target audio data and the first audio data occupy the same space to obtain the second audio data and have the same playback duration. In this way, the second audio data can be overwritten when the ear return buffer is full, and can be played when the ear return buffer is about to be empty. Crossfade processing can also be performed during the merging process so that the front of the generated target audio data can be connected with the previous and next audio data, ensuring data continuity, avoiding discarding audio data or inserting silent frames, and avoiding the introduction of noise.

[0091] In step S14, the target audio data is processed.

[0092] After the target audio data is generated, the target audio data can be played, or the target audio data can be cached in the earphone return buffer area.

[0093] When the target audio data is generated by the playback thread and the ear-return buffer is about to be empty, the playback thread directly plays the target audio data and ensures that the first audio data is retained in the ear-return buffer. When the target audio data is generated by the acquisition thread and the ear-return buffer is full, the acquisition thread caches the target audio data to the location of the second audio data in the ear-return buffer, that is, the target audio data is used to overwrite the second audio data in the ear-return buffer.

[0094] The audio data processing method provided by this exemplary embodiment obtains the first audio data to be processed and the storage status of the ear return buffer area. When the storage status meets the preset status, the second audio data corresponding to the first audio data is obtained from the stored audio data, and the target audio data is generated based on the first audio data and the second audio data, and the target audio data is processed. Since the target audio data is generated based on the first audio data and the second audio data when the ear return buffer area meets the preset status, and then the target audio data is processed, there is no need to insert silent frames or discard audio data, which can avoid introducing ear return noise and improve the ear return audio effect.

[0095] In an exemplary embodiment, when the first audio data is the remaining audio data obtained from the earphone feedback buffer through the playback thread, the preset state is that the number of frames of the remaining audio data in the earphone feedback buffer is the first target frame number;

[0096] When the storage status meets the preset status, the second audio data corresponding to the first audio data is obtained from the stored audio data, including: when the storage status is that the number of frames of the remaining audio data in the ear return buffer area is the first target frame number, the second audio data corresponding to the first audio data is obtained from the history buffer area through the playback thread.

[0097] Among them, the first target frame number is a smaller number of frames, which is an integer greater than 0 and close to 0, that is, the preset state is that the number of frames of remaining audio data in the ear return buffer is small and almost empty. The first target frame number can be 1, for example, to avoid the ear return buffer from being empty.

[0098] When the first audio data is the remaining audio data obtained from the ear-return buffer by the playback thread, before the playback thread obtains the first audio data from the ear-return buffer, if the storage status of the ear-return buffer is that the number of frames of audio data remaining in the ear-return buffer is the first target number of frames, that is, the ear-return buffer is about to become empty, the audio data currently cached in the ear-return buffer is relatively small, with only the first target number of frames of audio data remaining. In this case, when the playback thread obtains the first audio data from the ear-return buffer, it still retains the first audio data in the ear-return buffer. After obtaining the first audio data, it does not directly play it. Instead, it obtains second audio data corresponding to the first audio data from the historical buffer, that is, obtains the second audio data of the first target number of frames from the historical buffer, and combines the first audio data and the second audio data to generate target audio data. The playback thread can play the target audio data and retain the first audio data in the ear-return buffer. This ensures that the ear-return buffer will not become empty and at least the first target number of frames of remaining audio data will be retained.

[0099] The extreme case where the ear return buffer is empty is generally caused by CPU jitter or when the acquisition and playback callbacks do not arrive in order. This is a short-term situation. At this time, when the playback thread plays the target audio data and then obtains audio data from the ear return buffer, the ear return buffer generally will not only have the first target number of frames of audio data left. At this time, the playback thread can directly take out a frame of audio data from the ear return buffer and play this frame of data. Since the generated target audio data is generated by combining the first audio data in the ear return buffer and the second audio data played historically, the target audio data being played can be connected to the first audio data in the ear return buffer to be played later, and can also be connected to the second audio data played historically. No noise can be seen from the frequency spectrum of the audio data, and no noise can be heard subjectively, thus avoiding the problem of introducing noise when the ear return buffer is empty.

[0100] On the basis of the above technical solution, after processing the target audio data, it also includes: storing the target audio data in a history buffer area through a playback thread.

[0101] The history buffer area is used to store audio data played in history. For example, the audio data of the most recent frame played in history can be buffered to generate target audio data together with the first audio data.

[0102] After generating and playing the target audio data, the playback thread stores the target audio data just played in the historical buffer area, so that if the ear return buffer area still only has the first audio data of the first target frame number, new audio data can be generated again based on the audio data historically played in the historical buffer area to play the role of audio transition.

[0103] Based on the above technical solution, when the first audio data is the remaining audio data obtained from the ear return buffer area through the playback thread, the first audio data and the second audio data are merged and processed to obtain the target audio data, including: fading in the second audio data to obtain the first intermediate data; fading out the first audio data to obtain the second intermediate data; merging the first intermediate data and the second intermediate data to obtain the target audio data.

[0104] When the first audio data is the remaining audio data obtained from the ear return buffer area through the playback thread, the second audio data is faded in, that is, the volume of the second playback data changes from the lowest volume to the highest volume, and the first intermediate data is obtained; at the same time, the first audio data is faded out, that is, the volume of the first audio data changes from the highest volume to the lowest volume, and the second intermediate data is obtained; the first intermediate data and the second intermediate data are merged to obtain the target audio data.

[0105] For example, the first target frame number is one frame, that is, the first audio data is the remaining frame of audio data in the in-ear feedback buffer (curr_frame), and the second audio data is the frame of audio data just played before the current time (last_frame). In this way, according to the frame size of one frame of audio data (frame_cnt), the second audio data is faded in and the first audio data is faded out, that is, the first and second audio data are cross-faded. This can be achieved through the following code:

[0106]

[0107]

[0108] In the above code, alpha is the adjustment weight during the crossfade process, and step is the adjustment step size of the adjustment weight. When the loop variable is less than the frame size (frame_cnt), for each channel of audio data (channel), the loop reduces the weight of the first audio data (curr_frame) in the target audio data, that is, it achieves the fade-out processing of the first audio data, and increases the weight of the second audio data (last_frame) in the target audio data, that is, it achieves the fade-in processing of the second audio data.

[0109] By fading in the second audio data to obtain the first intermediate data, and fading out the first audio data to obtain the second intermediate data, the first intermediate data and the second intermediate data are merged to generate the target audio data. The target audio data generated in this way can be connected to the second audio data in the front and the first audio data in the ear return buffer in the back, ensuring the continuity of the data and avoiding the problem of using silent frames to cause jamming and introduce noise in the extreme case that the ear return buffer is empty.

[0110] In another exemplary embodiment, when the first audio data is audio data collected by a collection thread, the preset state is that the ear return buffer is insufficient to cache the first audio data collected by the collection thread;

[0111] When the storage state satisfies a preset state, obtaining second audio data corresponding to the first audio data from the stored audio data, including: when the storage state is that the earphone return buffer is insufficient to cache the first audio data, obtaining the second audio data corresponding to the first audio data from the earphone return buffer by a collection thread;

[0112] Processing the target audio data includes: caching the target audio data to a location where the second audio data is located in the ear return buffer area through an acquisition thread.

[0113] When the acquisition thread collects the first audio data and needs to cache the collected first audio data into the ear return buffer, if the storage space of the ear return buffer is about to be full or is full and is insufficient to cache the collected first audio data, the second audio data corresponding to the first audio data can be obtained from the ear return buffer, that is, the second audio data with the same frame number as the first audio data is obtained from the end of the ear return buffer, and the target audio data is generated by combining the collected first audio data and the second audio data last cached in the ear return buffer. The target audio data is cached to the location of the second audio data in the ear return buffer, that is, the target audio data is used to overwrite the second audio data in the ear return buffer.

[0114] The extreme case where the ear return buffer is full is generally caused by CPU jitter or when the acquisition and playback callbacks do not arrive in order. This is a short-term situation. At this time, after the acquisition thread caches the target audio data to the location of the second audio data in the ear return buffer, when it needs to cache the next frame of audio data to the ear return buffer, the ear return buffer will generally no longer be full. At this time, the acquisition thread can directly cache the newly collected frame of audio data in the ear return buffer. Since the generated target audio data is generated by combining the second audio data and the first audio data in the ear return buffer, the target audio data and the audio data before the second audio data in the ear return buffer can be connected, and can be connected with the audio data after the first audio data. No noise can be seen from the audio data spectrum, and no noise can be heard subjectively, thus avoiding the problem of introducing noise when the ear return buffer is full. There is no need to expand the ear return buffer, avoiding the delay caused by expanding the ear return buffer.

[0115] Based on the above technical solution, second audio data corresponding to the first audio data is obtained from the ear return buffer area through the acquisition thread, including: obtaining the audio data of the second target frame number of the last cache from the ear return buffer area through the acquisition thread as the second audio data.

[0116] The second target frame number is a smaller number of frames, for example, one frame, two frames, etc.

[0117] When the acquisition thread collects the first audio data and needs to cache the collected first audio data in the ear return buffer area, if the storage space of the ear return buffer area is about to be full or is full and is insufficient to cache the collected first audio data, the audio data of the second target frame number cached last can be obtained from the ear return buffer area as the second audio data corresponding to the first audio data. This facilitates combining the first audio data and the second audio data to generate the target audio data, and the target audio data can be cached in the ear return buffer area at the location of the second audio data. Moreover, the second audio data is the audio data of the second target frame number cached last in the ear return buffer area, and is audio data adjacent to the first audio data. At this time, the synthesized target audio data can be connected with the audio data before the second audio data, and can also be connected with the audio data collected after the first audio data. There is no need to directly discard the audio data, thereby avoiding the introduction of noise.

[0118] On the basis of the above technical solution, the method further includes:

[0119] When the storage status meets the preset status and the playback duration of the audio data cached in the ear return buffer area is less than the ear return delay threshold, the capacity of the ear return buffer area is expanded, wherein the playback duration of the audio data cached in the ear return buffer area after the capacity is expanded is less than or equal to the ear return delay threshold.

[0120] The in-ear feedback delay threshold is the tolerable in-ear feedback delay.

[0121] When the acquisition thread needs to cache the collected first audio data into the ear return buffer area, if the ear return buffer area is not sufficient to cache the collected first audio data, it can be determined whether the playback duration of the audio data cached in the ear return buffer area is less than the ear return delay threshold. If the playback duration of the audio data cached in the ear return buffer area is less than the ear return delay threshold, the capacity of the ear return buffer area can be expanded. However, the capacity of the ear return buffer area cannot be expanded indefinitely. After the ear return buffer area is expanded, the playback duration of all audio data cached must be less than or equal to the ear return delay threshold, so as to avoid excessive ear return delay.

[0122] Based on the above technical solution, when the first audio data is audio data collected through the acquisition thread, the first audio data and the second audio data are merged and processed to obtain the target audio data, including: fading out the second audio data to obtain third intermediate data; fading in the first audio data to obtain fourth intermediate data; merging the third intermediate data and the fourth intermediate data to obtain the target audio data.

[0123] When the first audio data is audio data collected by the collection thread, the second audio data is faded out, that is, the volume of the second audio data changes from large to small until the volume reaches the lowest, thereby obtaining third intermediate data; at the same time, the first audio data is faded in, that is, the volume of the collected first audio data changes from the lowest volume to small to large until the volume reaches the highest, thereby obtaining fourth intermediate data; the third intermediate data and the fourth intermediate data are merged to obtain target audio data.

[0124] For example, the second target frame number is one frame, that is, the second audio data is the last frame of audio data (last_frame) taken from the in-ear feedback buffer, and the first audio data is the audio data just collected by the collection thread. In this way, according to the frame size of one frame of audio data (frame_cnt), the second audio data (last_frame) is faded out, and the first audio data (curr_frame) is faded in. That is, the first and second audio data are cross-faded. This can be achieved through the following code:

[0125]

[0126] In the above code, alpha is the adjustment weight during the crossfade process, step is the adjustment step size of the adjustment coefficient, and when the loop variable is less than the frame size (frame_cnt), for each channel of audio data (channel), the loop increases the weight of the first audio data (curr_frame) in the target audio data, that is, fades the first audio data in, and reduces the weight of the second audio data (last_frame) in the target audio data, that is, fades the second audio data out.

[0127] By fading out the second audio data last cached in the ear return buffer and fading in the first audio data collected by the acquisition thread, the third intermediate data and the fourth intermediate data obtained after the processing are merged to generate target audio data. The target audio data generated in this way includes the second audio data and the first audio data. The beginning of the target audio data can be connected with the audio data before the second audio data in the ear return buffer, and the end of the target audio data can be connected with the audio data collected by the acquisition thread next time, avoiding the discarding of the audio data, ensuring the continuity of the data, and avoiding the problem of introducing noise due to the discarding of audio data in the extreme case that the ear return buffer is full.

[0128] Figure 2 is a flowchart of an audio data processing method according to an exemplary embodiment. Figure 2As shown, the audio data processing method is mainly a process in which the playback thread processes the audio data, and includes the following steps.

[0129] In step S21, when the remaining audio data is obtained from the ear-return buffer area through the playback thread, the storage state of the ear-return buffer area is obtained, and the obtained remaining audio data is used as the first audio data to be processed.

[0130] In step S22, when the storage status is that the number of frames of remaining audio data in the ear return buffer is the first target number of frames, the second audio data corresponding to the first audio data is obtained from the history buffer through the playback thread.

[0131] In step S23, fade-in processing is performed on the second audio data to obtain first intermediate data.

[0132] In step S24, the first audio data is faded out to obtain second intermediate data.

[0133] In step S25 , the first intermediate data and the second intermediate data are merged to obtain target audio data.

[0134] In step S26, the target audio data is played through the playing thread.

[0135] The specific process of each of the above steps can be found in the above exemplary embodiments and will not be repeated here.

[0136] The audio data processing method provided by this exemplary embodiment generates target audio data based on the first audio data and the second audio data and plays the target audio data when the playback thread obtains the first audio data from the ear return buffer and the storage state of the ear return buffer is that the number of frames of the remaining audio data in the ear return buffer is the first target number of frames. In this way, since the target audio data is combined with the remaining audio data in the ear return buffer and the second audio data played historically, the continuity of the data can be maintained, and the problem of introducing noise due to the insertion of silent frames in the extreme case of an empty ear return buffer is avoided.

[0137] Figure 3 is a flowchart of an audio data processing method according to an exemplary embodiment. Figure 3 As shown, the audio data processing method is mainly a process in which the acquisition thread processes the audio data, including the following steps.

[0138] In step S31, audio data is collected through the collection thread, and the collected audio data is used as the first audio data to obtain the storage status of the earphone feedback buffer area.

[0139] In step S32, when the storage status is that the earphone return buffer area is insufficient to buffer the first audio data, the second audio data corresponding to the first audio data is obtained from the earphone return buffer area through the acquisition thread.

[0140] In step S33, fade-out processing is performed on the second audio data to obtain third intermediate data.

[0141] In step S34, fade-in processing is performed on the first audio data to obtain fourth intermediate data.

[0142] In step S35 , the third intermediate data and the fourth intermediate data are merged to obtain target audio data.

[0143] In step S36, the target audio data is cached in the ear return buffer area at the location of the second audio data through the acquisition thread.

[0144] The specific process of each of the above steps can be found in the above exemplary embodiments and will not be repeated here.

[0145] The audio data processing method provided by this exemplary embodiment generates target audio data based on the first audio data and the second audio data last cached in the ear return buffer when the acquisition thread caches the collected first audio data into the ear return buffer. If the ear return buffer is insufficient to cache the first audio data, the target audio data is cached to the location of the second audio data in the ear return buffer. In this way, since the target audio data combines the second audio data in the ear return buffer and the newly collected first audio data, the continuity of the data can be maintained, avoiding the problem of discarding audio data and introducing noise due to the extreme situation that the ear return buffer is full, and there is no need to expand the ear return buffer, thus avoiding the problem of large delay.

[0146] Figure 4 FIG. 1 is a block diagram of an audio data processing apparatus according to an exemplary embodiment. Figure 4 The device includes a first acquisition module 41, a second acquisition module 42, a data generation module 43 and a data processing module 44.

[0147] The first acquisition module 41 is configured to acquire first audio data to be processed and a storage status of an ear-return buffer area, where the ear-return buffer area is used to cache audio data;

[0148] The second acquisition module 42 is configured to acquire second audio data corresponding to the first audio data from the stored audio data when the storage state meets the preset state;

[0149] The data generating module 43 is configured to generate target audio data according to the first audio data and the second audio data;

[0150] The data processing module 44 is configured to process the target audio data.

[0151] Optionally, the data generation module 43 is configured to execute:

[0152] The first audio data and the second audio data are combined to obtain target audio data.

[0153] Optionally, when the first audio data is remaining audio data obtained from the ear-return buffer area through the playback thread, the data generation module 43 includes:

[0154] a first fade-in processing unit, configured to perform fade-in processing on the second audio data to obtain first intermediate data;

[0155] a first fade-out processing unit, configured to perform fade-out processing on the first audio data to obtain second intermediate data;

[0156] The first data merging unit is configured to merge the first intermediate data and the second intermediate data to obtain target audio data.

[0157] Optionally, when the first audio data is audio data collected by an acquisition thread, the data generation module 43 includes:

[0158] a second fade-out processing unit, configured to perform fade-out processing on the second audio data to obtain third intermediate data;

[0159] a second fade-in processing unit, configured to perform fade-in processing on the first audio data to obtain fourth intermediate data;

[0160] The second data merging unit is configured to merge the third intermediate data and the fourth intermediate data to obtain target audio data.

[0161] Optionally, when the first audio data is remaining audio data obtained from an earphone feedback buffer through a playback thread, the preset state is that the number of frames of the remaining audio data in the earphone feedback buffer is the first target frame number;

[0162] The second acquisition module 42 is configured to execute:

[0163] When the storage status is that the number of frames of the remaining audio data in the ear return buffer is the first target number of frames, the second audio data corresponding to the first audio data is obtained from the history buffer through the playback thread.

[0164] Optionally, the device further comprises:

[0165] The target data cache module is configured to execute the storage of target audio data into the history cache area through the playback thread.

[0166] Optionally, when the first audio data is audio data collected by a collection thread, the preset state is that the ear return buffer is insufficient to cache the first audio data collected by the collection thread;

[0167] The second acquisition module 42 is configured to execute: when the storage status is that the earphone feedback buffer is insufficient to buffer the first audio data, acquiring second audio data corresponding to the first audio data from the earphone feedback buffer through the acquisition thread;

[0168] The data processing module 44 is configured to execute: caching the target audio data to the location of the second audio data in the ear return buffer area through the acquisition thread.

[0169] Optionally, the second obtaining module 42 includes:

[0170] The second data acquisition unit is configured to execute, through an acquisition thread, acquiring the audio data of the second target number of frames cached last from the ear return buffer area as the second audio data.

[0171] Optionally, the device further comprises:

[0172] The capacity expansion module is configured to expand the capacity of the ear return buffer area when the storage state meets the preset state and the playback duration of the audio data cached in the ear return buffer area is less than the ear return delay threshold, wherein the playback duration of the audio data cached in the ear return buffer area after the capacity expansion is less than or equal to the ear return delay threshold.

[0173] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0174] Figure 5 5 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0175] Reference Figure 5 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0176] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 502 may include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.

[0177] The memory 504 is configured to store various types of data to support operations on the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0178] The power supply assembly 506 provides power to the various components of the electronic device 500. The power supply assembly 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0179] The multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0180] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0181] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0182] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500. The sensor assembly 514 can also detect changes in the position of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and temperature changes of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0183] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0184] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned audio data processing method.

[0185] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions. The instructions can be executed by the processor 520 of the electronic device 500 to perform the above-mentioned audio data processing method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0186] In an exemplary embodiment, a computer program product is further provided, comprising a computer program or computer instructions, wherein the computer program or computer instructions implement the above-mentioned audio data processing method when executed by a processor.

[0187] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0188] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for processing audio data, characterized in that: include: Obtaining first audio data to be processed and a storage status of an earphone feedback buffer area, where the earphone feedback buffer area is used to buffer audio data; When the storage state satisfies a preset state, obtaining second audio data corresponding to the first audio data from the stored audio data; generating target audio data according to the first audio data and the second audio data; processing the target audio data; When the first audio data is the remaining audio data obtained from the earphone feedback buffer through the playback thread, the preset state is that the number of frames of the remaining audio data in the earphone feedback buffer is the first target frame number; When the storage state satisfies a preset state, obtaining second audio data corresponding to the first audio data from the stored audio data includes: When the storage state is that the number of frames of the remaining audio data in the ear return buffer is the first target number of frames, the second audio data corresponding to the first audio data is obtained from the history buffer through the playback thread.

2. The method according to claim 1, characterized in that Generating target audio data according to the first audio data and the second audio data includes: The first audio data and the second audio data are combined to obtain the target audio data.

3. The method according to claim 2, characterized in that When the first audio data is remaining audio data obtained from the ear-return buffer area through the playback thread, the merging of the first audio data and the second audio data to obtain the target audio data includes: performing a fade-in process on the second audio data to obtain first intermediate data; performing a fade-out process on the first audio data to obtain second intermediate data; The first intermediate data and the second intermediate data are merged to obtain the target audio data.

4. The method according to claim 2, characterized in that When the first audio data is audio data collected by an acquisition thread, the merging of the first audio data and the second audio data to obtain the target audio data includes: performing a fade-out process on the second audio data to obtain third intermediate data; performing a fade-in process on the first audio data to obtain fourth intermediate data; The third intermediate data and the fourth intermediate data are combined to obtain the target audio data.

5. The method according to claim 1, wherein After processing the target audio data, the method further includes: The target audio data is stored in the history buffer area through the playback thread.

6. The method according to claim 1 or 2, characterized in that When the first audio data is audio data collected by a collection thread, the preset state is that the ear return buffer area is insufficient to cache the first audio data collected by the collection thread; When the storage state satisfies a preset state, obtaining second audio data corresponding to the first audio data from the stored audio data includes: When the storage status indicates that the earphone feedback buffer area is insufficient to buffer the first audio data, acquiring, by the acquisition thread, the second audio data corresponding to the first audio data from the earphone feedback buffer area; The processing of the target audio data includes: caching the target audio data to a location where the second audio data is located in the ear return buffer area through the acquisition thread.

7. The method according to claim 6, characterized in that The acquiring, through the acquisition thread, the second audio data corresponding to the first audio data from the ear return buffer area includes: The acquisition thread obtains the audio data of the second target frame number cached last from the ear return buffer area as the second audio data.

8. The method according to claim 6, characterized in that Also includes: When the storage status satisfies the preset status and the playback duration of the audio data cached in the ear-return buffer area is less than the ear-return delay threshold, the capacity of the ear-return buffer area is expanded, wherein the playback duration of the audio data cached in the ear-return buffer area after the capacity expansion is less than or equal to the ear-return delay threshold.

9. An audio data processing device, characterized in that: include: A first acquisition module is configured to acquire first audio data to be processed and a storage status of an ear-return buffer area, where the ear-return buffer area is used to cache audio data; a second acquisition module configured to acquire, when the storage state satisfies a preset state, second audio data corresponding to the first audio data from the stored audio data; a data generating module, configured to generate target audio data according to the first audio data and the second audio data; A data processing module is configured to process the target audio data; When the first audio data is the remaining audio data obtained from the earphone feedback buffer through the playback thread, the preset state is that the number of frames of the remaining audio data in the earphone feedback buffer is the first target frame number; The second acquisition module is configured to execute: When the storage state is that the number of frames of the remaining audio data in the ear return buffer is the first target number of frames, the second audio data corresponding to the first audio data is obtained from the history buffer through the playback thread.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the audio data processing method according to any one of claims 1 to 8. 11 . A computer-readable storage medium, wherein when instructions in the computer storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the audio data processing method according to claim 1 .

12. A computer program product comprising a computer program or computer instructions, characterized in that When the computer program or computer instruction is executed by a processor, the audio data processing method according to any one of claims 1 to 8 is implemented.

Images