Audio playback delay AI correction method and device

Through dynamic adjustment of audio parameters based on machine learning, the audio playback delay problem of Android mobile devices is solved, and the user experience is improved, especially audio synchronization problems in musical instruments, DJs, games and virtual reality applications.

CN112735463BActive Publication Date: 2024-11-29HANGZHOU XIAOBANXIONG TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202011484031.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-16
Publication Date
2024-11-29
Estimated Expiration
2040-12-16

AI Technical Summary

Technical Problem

The delayed audio playback problem in Android mobile devices leads to poor user experience, affecting instrument performance, DJ beat matching, game audio synchronization, VoIP application and virtual reality experience.

Method used

Dynamically adjust audio parameters through a neural network based on machine learning, control the audio chip to measure and play analog audio stream signals, convert them into digital signals, and correct them at the audio hardware abstraction layer to reduce audio latency.

Benefits of technology

It effectively improves the audio playback delay of Android mobile devices and improves users' gaming and video experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112735463B_ABST
    Figure CN112735463B_ABST
Patent Text Reader

Abstract

The present invention relates to an AI correction method for audio playback delay of an Android mobile terminal device, comprising: controlling an audio chip to measure an incoming analog audio stream signal at a predetermined time interval; converting the analog audio stream signal into a digital audio stream signal and inputting the signal into a buffer; wherein the size of the buffer varies with different bus settings; monitoring the data volume of the buffer, using a neural network based on machine learning, and dynamically adjusting audio parameters based on the data volume in the buffer; and correcting and playing the audio.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of voice signal processing, and in particular to an AI correction method and device for audio playback delay of an Android mobile device. Background Art

[0002] Audio latency can be defined as "the time it takes for a signal to pass through a system". In mobile devices, audio latency is related to how long it takes for a user to receive audio feedback after clicking on the screen. If the audio latency is too long, it will make the user feel a sense of lag. Even if the technical picture is smooth, the sound is slow, which is still unpleasant.

[0003] The main problems caused by low latency in various application scenarios are: Musical instrument applications and sound effects applications: Audio artists can't play on stage, and performers using Android devices are always half a beat behind others. Even practice is impossible. DJs can't match beats because the signal they hear in their headphones is far behind the main signal played to the audience. It is also very difficult to use effects such as loop rolling or echo. Games, sound effects, such as explosions and gunshots will be delayed by several frames, making the game audio "detached" from the visual effects. This poor user experience cannot bring an immersive gaming experience. VoIP applications, such as Skype: If the user is using an Android phone, the overall audio delay is higher than the network delay. In other words, it takes longer to pass through the Android system than it takes for a data packet to travel between continents. Virtual reality VR: When the user turns his head, the audio "follows" the change too late, destroying the three-dimensional audio experience. Try Paul McCartney's Google Cardboard glasses app as an example. Google will hand over a billion dollars in revenue to Apple in the field of virtual reality.

[0004] It can be seen that low sound latency has become an indispensable basic function in modern operating systems. However, on Android devices, the latency problem is very prominent due to the inconsistent standards set by various manufacturers and the different hardware solutions used. Summary of the invention

[0005] In view of the above analysis, the present invention aims to provide an AI correction method and device for audio playback delay of an Android mobile device, which provides a method and device to overcome the above problems or at least partially solve the above problems.

[0006] According to one aspect of the present invention, an AI correction method for audio playback delay of an Android mobile device is provided.

[0007] Controlling the audio chip to measure the incoming analog audio stream signal at a predetermined time interval;

[0008] Convert the analog audio stream signal into a digital audio stream signal and input it into a buffer, wherein the size of the buffer is set according to the bus type;

[0009] Monitor the amount of data in the buffer, input the current audio parameters into a neural network based on machine learning, and obtain dynamic audio parameters generated according to the amount of data in the buffer;

[0010] The audio is modified and played according to the dynamic audio parameters.

[0011] Optionally, the audio parameters include sampling rate, channels and bit rate.

[0012] Optionally, dynamically adjusting audio parameters resides in an audio hardware abstraction layer (HAL).

[0013] Optionally, a channel is established at the AudioFlinger layer. If the user program uses native code and sets the audio buffer queue with the native hardware sampling rate and cycle size, no resampling will be performed and the buffer size will not be modified. If the buffer sampling rate requested by the program is different from the native sampling rate, resampling is required.

[0014] Optionally, where AudioRecord does not request a new buffer from the AudioFlinger layer.

[0015] Optionally, the audio stream signal may be transmitted from the audio chip to the GPU via a bus.

[0016] Optionally, the bus includes: USB, PCI, and FireWire.

[0017] According to one aspect of the present invention, an AI correction device for audio playback delay of an Android mobile device is provided.

[0018] It includes an audio import module 101, which is used to control the audio chip to measure the incoming analog audio stream signal at a predetermined time interval, convert the analog audio stream signal into a digital audio stream signal and then input it into a buffer, wherein the size of the buffer is set according to the bus type;

[0019] The AI ​​audio analysis module 102 is used to monitor the amount of data in the buffer, and input the current audio parameters into a neural network based on machine learning to obtain dynamically adjusted audio parameters based on the amount of data in the buffer;

[0020] The audio correction module 103 is used to correct and play the audio according to the dynamic audio parameters.

[0021] Optionally, the audio parameters include sampling rate, channel and bit rate;

[0022] Optionally, the AI ​​audio analysis module 102 dynamically adjusts audio parameters located in the audio hardware abstraction layer (HAL).

[0023] Optionally, a channel is established at the AudioFlinger layer. If the user program uses native code and sets the audio buffer queue with the native hardware sampling rate and cycle size, no resampling will be performed and the buffer size will not be modified. If the buffer sampling rate requested by the program is different from the native sampling rate, resampling is required.

[0024] Optionally, where AudioRecord does not request a new buffer from the AudioFlinger layer.

[0025] Optionally, the AI ​​audio analysis module 102 may transmit the audio stream signal from the audio chip to the GPU via a bus.

[0026] Optionally, the bus includes: USB, PCI, and FireWire.

[0027] The beneficial effects of the present invention are as follows: through AI correction of audio playback delay of Android mobile devices, the audio playback delay of Android mobile devices can be effectively improved, thereby improving the user's gaming and audio-visual experience.

[0028] Other features and advantages of the present invention will be described in the following description, and part of them will become obvious from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of an AI correction method for audio playback delay of an Android mobile device of the present invention;

[0030] Figure 2 This is a structural diagram of the AI ​​correction device for audio playback delay of an Android mobile device of the present invention; DETAILED DESCRIPTION

[0031] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] The specific embodiment of the present invention discloses an audio playback delay AI correction method, such as Figure 1 As shown: The method comprises the following steps:

[0033] Step 100: Control the audio chip to measure the incoming analog audio stream signal at a predetermined time interval; convert the analog audio stream signal into a digital audio stream signal and input it into a buffer; the size of the buffer varies with different bus settings; the audio chip is connected to the CPU via a bus, such as USB, PCI, FireWire, etc. The delay of different data buses depends on the size and number of its internal buffers, and the delay here is usually less than 1 millisecond.

[0034] Step 110: Monitor the input buffer so that when the amount of data in the monitored buffer reaches the resampling threshold, resample the data in the input buffer of the resampling module. The amount of cached data is reduced by resampling the data, which is equivalent to accelerating the playback on the voice receiving device, thereby achieving the purpose of reducing latency. Monitor the amount of data in the buffer, use a neural network based on machine learning, dynamically adjust the audio parameters based on the amount in the buffer, and input the current audio parameters into the neural network based on machine learning. The purpose of the present disclosure is to provide a mechanism for inferring an optimal data transmission rate that is as close as possible to the achievable data transmission rate. Therefore, the present disclosure provides a mechanism to avoid the following situation: attempt to transmit data at the highest possible data transmission rate, recognize that the highest data transmission rate is unsustainable, and downgrade the data transmission rate to a more sustainable value. In contrast, the mechanism disclosed in the present disclosure uses neural network technology to directly "guess" the achievable data transmission rate.

[0035] The audio parameters include sampling rate, channel and bit rate; the audio parameters reflect the amount of data that the Android device can process per second in this voice playback, so the size of the input buffer can be set to accommodate the amount of data processed in N seconds of this audio playback. Among them, the value of N can be inferred based on the AI ​​audio analysis module, for example, 5 seconds. Assuming that the audio processing parameters are specifically: 16K sampling rate, mono channel, 16bit bit rate, N value is 5 seconds, the size of the input buffer of the resampling module is 16 / 8*1*16000*5=156KB.

[0036] During the learning phase, the neural network training is trained with audio parameters corresponding to the audio playback as a plurality of inputs and an output of a plurality of audio parameters corresponding to the optimal data transmission rate. As is well known in the field of neural networks, during the training phase, the neural network implemented by the neural network training adjusts its weights. In addition, during the learning phase, the number of layers and the number of nodes per layer of the neural network can be adjusted to improve the accuracy of the model. At the end of the training phase, the prediction model generated by the neural network training includes the number of layers, the number of nodes per layer, and the weights.

[0037] It should be noted that the size of the input buffer is adjustable. For example, when the audio processing parameters of the voice receiving device in the current audio playback change, the size of the input buffer of the resampling module can be adaptively adjusted.

[0038] Specifically, the dynamic adjustment of audio parameters is located in the audio hardware abstraction layer (HAL); the audio hardware abstraction layer (HAL) acts as a middleman between Android's media services and the Linux audio driver. HAL is implemented by mobile device manufacturers to "port" Android to Android devices. Its implementation is open, and manufacturers are free to create any type of HAL code to communicate with media services using predefined structures. Media services read HAL and request to create input or output streams with optional optimization parameters such as sampling rate, buffer size or sound effects.

[0039] Optionally, the audio chip can be connected to the GPU via a bus, and the GPU can be used to obtain greater computing power, thereby improving the system computing performance; the bus includes but is not limited to: USB, PCI, and FireWire.

[0040] Through AI's control of the buffer, the consumption time in the ring buffer can be reduced, the unit usage frequency can be increased, and resource consumption can be reduced.

[0041] Step 130: Then, the audio is corrected and played according to the obtained audio parameters, wherein a channel is established at the AudioFlinger layer. If the user program uses native code and sets the audio buffer queue with the native hardware sampling rate and cycle size, no resampling will be performed and the buffer size will not be modified. Native means AOSP, which is an Android system that does not include manufacturer or operator customization programs and any third-party modifications, and AudioRecord will not request a new buffer from the AudioFlinger layer. If the buffer sampling rate requested by the program is different from the native sampling rate, resampling is required.

[0042] Establish a "fast mixing" channel for Audio Flinger. If the user program uses native [Android NDK] code and sets the audio buffer queue with the native hardware sampling rate and cycle size, no resampling, buffer increase and mixing [MixerThread] will occur in this step. For example, the native hardware sampling rate is 48000Hz.

[0043] Generally, in the process of the user program, AudioRecord implements the application of the audio input terminal, which is a client function library [for example, through OpenGL ES]. AudioRecord will periodically run a thread to request new buffers from Audio Flinger with the concept of push mechanism.

[0044] In this technical solution, AudioRecord does not request a new buffer from the AudioFlinger layer. Through the above mechanism, only one set of buffers is used during the transmission process, and no delay is generated in the audio channel.

[0045] The embodiment of the present invention also provides an audio playback delay AI correction device, such as Figure 2 As shown, it includes an audio import module 101, an AI audio analysis module 102, and an audio correction module 103; wherein the audio import module 101 controls the audio chip to measure the incoming analog audio stream signal at a predetermined time interval; the analog audio stream signal is converted into a digital audio stream signal and then input into a buffer; wherein the size of the buffer varies with different bus settings; the audio chip is connected to the CPU via a bus, such as USB, PCI, FireWire, etc. The delay of different data buses depends on the size and number of its internal buffers, and the delay here is usually less than 1 millisecond.

[0046] The AI ​​audio analysis module 102 monitors the input buffer so that when the amount of data in the monitored buffer reaches the resampling threshold, the data in the input buffer of the resampling module is resampled. The amount of cached data is reduced by resampling the data, which is equivalent to accelerating the playback on the voice receiving device, thereby achieving the purpose of reducing latency. The AI ​​audio analysis module 102 monitors the amount of data in the buffer and uses a neural network to dynamically adjust the audio parameters based on the amount in the buffer. The purpose of the present disclosure is to provide a mechanism for inferring an optimal data transmission rate that is as close as possible to the achievable data transmission rate. Therefore, the present disclosure provides a mechanism to avoid the following situation: attempting to transmit data at the highest possible data transmission rate, recognizing that the highest data transmission rate is unsustainable, and downgrading the data transmission rate to a more sustainable value. In contrast, the mechanism disclosed in the present disclosure uses neural network technology to directly "guess" the achievable data transmission rate.

[0047] The audio parameters include sampling rate, channel and bit rate; the audio parameters reflect the amount of data that the Android device can process per second in this voice playback, so the size of the input buffer can be set to accommodate the amount of data processed in N seconds of this audio playback. Among them, the value of N can be inferred based on the AI ​​audio analysis module, for example, 5 seconds. Assuming that the audio processing parameters are specifically: 16K sampling rate, mono channel, 16bit bit rate, N value is 5 seconds, the size of the input buffer of the resampling module is 16 / 8*1*16000*5=156KB.

[0048] During the learning phase, the neural network training is trained with audio parameters corresponding to the audio playback as multiple inputs and multiple outputs corresponding to the optimal data transmission rate. As is well known in the field of neural networks, during the training phase, the neural network implemented by the neural network training adjusts its weights. In addition, during the learning phase, the number of layers and the number of nodes per layer of the neural network can be adjusted to improve the accuracy of the model. At the end of the training phase, the prediction model generated by the neural network training includes the number of layers, the number of nodes per layer, and the weights.

[0049] It should be noted that the size of the input buffer is adjustable. For example, when the audio processing parameters of the voice receiving device in the current audio playback change, the size of the input buffer of the resampling module can be adaptively adjusted.

[0050] Specifically, the AI ​​audio analysis module 102 dynamically adjusts the audio parameters located in the audio hardware abstraction layer (HAL); the audio hardware abstraction layer (HAL) acts as a middleman between Android's media services and the Linux audio driver. HAL is implemented by mobile device manufacturers to "port" Android to Android devices. Its implementation is open, and manufacturers are free to create any type of HAL code and use predefined structures to communicate with media services. Media services read HAL and request to create input or output streams with optional optimization parameters such as sampling rate, buffer size, or sound effects.

[0051] Optionally, the AI ​​audio analysis module 102 can use the audio chip to connect to the GPU via a bus, and use the GPU to obtain greater computing power, thereby improving the system computing performance; the bus includes but is not limited to: USB, PCI, and FireWire.

[0052] The AI ​​audio analysis module 102 can reduce the consumption time in the ring buffer, increase the unit usage frequency, and reduce resource consumption.

[0053] Subsequently, the audio correction module 103 corrects and plays the audio, wherein a channel is established at the AudioFlinger layer. If the user program uses native code and sets the audio buffer queue with the native hardware sampling rate and cycle size, no resampling will be performed and the buffer size will not be modified. Among them, AudioRecord will not request a new buffer from the AudioFlinger layer. Native is AOSP, which is an Android system that does not include manufacturer or operator customized programs and any third-party modifications, wherein AudioRecord will not request a new buffer from the AudioFlinger layer. If the buffer sampling rate requested by the program is different from the native sampling rate, resampling is required.

[0054] The audio correction module 103 establishes a "fast mixing" channel for Audio Flinger. If the user program uses native [Android NDK] code and sets the audio buffer queue with the native hardware sampling rate and cycle size, no resampling, buffer increase and mixing [MixerThread] will occur in this step.

[0055] Generally, in the process of the user program, AudioRecord implements the application of the audio input terminal, which is a client function library [for example, through OpenGL ES]. AudioRecord will periodically run a thread to request new buffers from Audio Flinger with the concept of push mechanism.

[0056] In this technical solution, under the control of the audio correction module 103, AudioRecord will not request a new buffer from the AudioFlinger layer. Through the above mechanism, only one set of buffers is used during the transmission process, and no delay will be generated in the audio channel.

[0057] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and set in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into multiple sub-modules or sub-units or sub-components. All features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device so disclosed may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0058] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0059] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the robot according to an embodiment of the present invention. The present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0060] Those skilled in the art will appreciate that all or part of the processes of the above embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0061] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0062] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0063] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An AI correction method for audio playback delay of an Android mobile device, characterized in that: Controlling the audio chip to measure the incoming analog audio stream signal at a predetermined time interval; Convert the analog audio stream signal into a digital audio stream signal and input it into a buffer, wherein the size of the buffer is set according to the bus type; Monitor the input buffer, and when the amount of data in the buffer reaches the resampling threshold, resample the data in the input buffer of the resampling module; Monitor the amount of data in the buffer, input the current audio parameters into a neural network based on machine learning, and obtain dynamic audio parameters generated according to the amount of data in the buffer; The audio is modified and played according to the dynamic audio parameters.

2. The AI ​​correction method according to claim 1, characterized in that: The audio parameters include sampling rate, channels and bit rate.

3. The AI ​​correction method according to claim 1, characterized in that: A channel is established at the AudioFlinger layer. If the user program uses native code and sets the audio buffer queue with the native hardware sampling rate and cycle size, no resampling is performed and the buffer size is not modified. If the buffer sampling rate requested by the program is different from the native sampling rate, resampling is required.

4. The AI ​​correction method according to claim 1, characterized in that: The AudioRecord does not request a new buffer from the AudioFlinger layer.

5. The AI ​​correction method according to claim 1, characterized in that: The audio stream signal can be transmitted from the audio chip to the GPU via the bus.

6. An AI correction device for audio playback delay of an Android mobile device, characterized in that: The device includes: An audio import module (101) is used to control the audio chip to measure the incoming analog audio stream signal at a predetermined time interval, convert the analog audio stream signal into a digital audio stream signal and then input it into a buffer, wherein the size of the buffer is set according to the bus type; The AI ​​audio analysis module (102) is used to: Monitor the input buffer, and when the amount of data in the buffer reaches the resampling threshold, resample the data in the input buffer of the resampling module; Monitor the amount of data in the buffer, input the current audio parameters into a neural network based on machine learning, and obtain dynamic audio parameters based on the amount of data in the buffer; The audio correction module (103) is used to correct and play the audio according to the dynamic audio parameters.

7. The AI ​​correction device according to claim 6, characterized in that: The audio parameters include sampling rate, channels and bit rate; The AI ​​audio analysis module (102) dynamically adjusts audio parameters located in the audio hardware abstraction layer (HAL).

8. The AI ​​correction device according to claim 6, characterized in that: A channel is established at the AudioFlinger layer. If the user program uses native code and sets the audio buffer queue with the native hardware sampling rate and cycle size, no resampling is performed and the buffer size is not modified. If the buffer sampling rate requested by the program is different from the native sampling rate, resampling is required.

9. The AI ​​correction device according to claim 8, characterized in that: The AudioRecord will not request a new buffer from the AudioFlinger layer.

10. The AI ​​correction device according to claim 8, characterized in that: The AI ​​audio analysis module (102) can transmit the audio stream signal from the audio chip to the GPU via the bus.

Citation Information

Patent Citations

  • Audio data pushing method and system

    CN105681883A

  • Method, apparatus, medium and device for reducing audio delay

    CN109062537A