Audio frame data processing method and device, computer equipment and storage medium

Abstract

The invention discloses an audio frame data processing method and device, computer equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: creating an audio player and a buffer queue; modifying an enqueue function pointer of the buffer queue from pointing to a native enqueue function to pointing to a proxy enqueue function; in response to a calling instruction for the enqueue function pointer, acquiring an agent enqueue function pointed by the enqueue function pointer and audio frame data indicated by the calling instruction, the length of the audio frame data being equal to the original length; and filling the audio frame data with the target length into the buffer queue through the proxy enqueue function when the length of the acquired and unfilled audio frame data is not less than the target length. According to the method and the device, the original code logic does not need to be modified, and on the basis of not invading the original code logic, the length of the single transmission audio frame data is flexibly adjusted, so that the development cost of adjusting the single transmission length is reduced, and the operability and the flexibility are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an audio frame data processing method, apparatus, computer device, and storage medium. Background Technology

[0002] In applications, the processing and playback of audio frame data is a common requirement. For example, in games, multimedia applications, and live streaming, the processing and playback of audio frame data is of paramount importance. Playing audio frame data typically involves a secondary transmission of the audio frame data, that is, transmitting the captured audio frame data from the underlying layer to the audio player for playback.

[0003] In related technologies, the length of a single audio frame transmitted to an audio player is specified by the upper-layer audio service provider through fixed code logic. Modifying the length of a single audio frame transmission requires additional modifications to the relevant code logic by the audio service provider, resulting in high implementation costs and limited operability, thus limiting the flexibility of audio frame data processing. Summary of the Invention

[0004] This application provides an audio frame data processing method, apparatus, computer device, and storage medium, which enables flexible adjustment of the length of a single audio frame data fill without intruding on the original code logic, reducing development costs and improving operability and flexibility. The technical solution is as follows.

[0005] On the one hand, an audio frame data processing method is provided, the method comprising: Create an audio player and a buffer queue, wherein the audio player is used to read audio frame data from the buffer queue; The enqueue function pointer of the buffer queue is changed from pointing to the native enqueue function to pointing to the proxy enqueue function. The native enqueue function is used to fill the buffer queue with audio frame data. In response to a call instruction to the enqueue function pointer, the proxy enqueue function pointed to by the enqueue function pointer and the audio frame data indicated by the call instruction are obtained, wherein the length of the audio frame data is equal to the original length; The proxy enqueue function fills the buffer queue with audio frame data of the target length when the length of the acquired and unfilled audio frame data is not less than the target length.

[0006] On the other hand, an audio frame data processing apparatus is provided, the apparatus comprising: A creation module is used to create an audio player and a buffer queue, wherein the audio player is used to read audio frame data from the buffer queue; The creation module is also used to modify the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function. The native enqueue function is used to fill the audio frame data into the buffer queue. The acquisition module is used to acquire, in response to a call instruction to the enqueue function pointer, the proxy enqueue function pointed to by the enqueue function pointer and the audio frame data indicated by the call instruction, wherein the length of the audio frame data is equal to the original length; The filling module is used to fill the buffer queue with audio frame data of the target length when the length of the acquired and unfilled audio frame data is not less than the target length, through the proxy enqueue function.

[0007] Optionally, the creation module is used for: A redirection unit is used to redirect the call instruction to the native engine acquisition function to the proxy engine acquisition function, wherein the native engine acquisition function is used to acquire the audio engine interface; A creation unit is used to obtain the audio engine interface through the proxy engine's function, and to create the audio player and the buffer queue based on the audio engine interface.

[0008] Optionally, the creation unit is used for: The audio engine interface is obtained through the proxy engine retrieval function. The instance creation function pointer of the audio engine interface is changed from pointing to the native instance creation function to pointing to the proxy instance creation function. The native instance creation function is used to create an audio engine instance. In response to a call instruction to the instance creation function pointer, the proxy instance creation function pointed to by the instance creation function pointer is obtained; An audio engine instance is created using the proxy instance creation function, and the audio player and the buffer queue are created based on the audio engine instance.

[0009] Optionally, the proxy engine acquisition function includes a first call subfunction and a first modification subfunction; the creation unit is used for: The first calling sub-function calls the native engine acquisition function to obtain the audio engine interface; The first modification sub-function changes the instance creation function pointer of the audio engine interface from pointing to the native instance creation function to pointing to the proxy instance creation function.

[0010] Optionally, the creation unit is used for: The audio engine instance is created through the proxy instance creation function. The player creation function pointer of the audio engine instance is changed from pointing to the native player creation function to pointing to the proxy player creation function. The native player creation function is used to create an audio player. In response to a call instruction to the player creation function pointer, the proxy player creation function pointed to by the player creation function pointer is obtained; The audio player is created using the proxy player creation function, and the buffer queue is created based on the audio player.

[0011] Optionally, the proxy instance creation function includes a second calling subfunction and a second modification subfunction; the creation unit is used for: The audio engine instance is created by calling the native instance creation function through the second call sub-function; The second modification sub-function changes the player creation function pointer of the audio engine instance from pointing to the native player creation function to pointing to the proxy player creation function.

[0012] Optionally, the creation unit is used for: The audio player is created through the proxy player creation function, and the queue creation function pointer of the audio player is changed from pointing to the native queue creation function to pointing to the proxy queue creation function. The native queue creation function is used to create a cache queue. In response to a call instruction to the queue creation function pointer, the proxy queue creation function pointed to by the queue creation function pointer is obtained; The buffer queue is created using the proxy queue creation function.

[0013] Optionally, the proxy player creation function includes a third calling subfunction and a third modification subfunction; the creation unit is used for: The audio player is created by calling the native player creation function through the third sub-function. The third modification sub-function changes the pointer of the audio player's queue creation function from pointing to the native queue creation function to pointing to the proxy queue creation function.

[0014] Optionally, the proxy queue creation function includes a fourth calling subfunction and a fourth modification subfunction; the creation unit is used for: The buffer queue is created by calling the native queue creation function through the fourth sub-function. The fourth modification sub-function changes the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function.

[0015] Optionally, the filling module is used for: The proxy enqueue function adds the audio frame data indicated by the calling instruction to the data buffer pool. If the length of the audio frame data in the data buffer pool is not less than the target length, then the audio frame data of the target length in the data buffer pool is filled into the buffer queue.

[0016] Optionally, the filling module is used for: When the length of the acquired and unfilled audio frame data is not less than the target length, the native enqueue function is called to fill the buffer queue with the audio frame data of the target length.

[0017] Optionally, the device further includes a message return module for: After filling the buffer queue with audio frame data of the target length through the proxy enqueue function, the call success message of the call instruction is returned. If the length of the acquired but unfilled audio frame data is less than the target length, the proxy enqueue function returns a success message for the call instruction.

[0018] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the audio frame data processing method as described above.

[0019] On the other hand, a computer-readable storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to perform the operations performed by the audio frame data processing method described above.

[0020] On the other hand, a computer program product is provided, including a computer program loaded and executed by a processor to perform the operations performed by the audio frame data processing method described above.

[0021] The method, apparatus, computer device, and storage medium provided in this application modify the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function. Therefore, calls to the native enqueue function can be intercepted and redirected to the proxy enqueue function. The proxy enqueue function controls the audio frame data filling process. When the length of the acquired and unfilled audio frame data is not less than the target length, audio frame data of the target length is filled into the buffer queue, thereby adjusting the length of the audio frame data filled in a single transmission from the native length to the target length. Therefore, this application does not require modification of the native code logic. Without intruding on the native code logic, it achieves flexible adjustment of the length of audio frame data transmitted in a single transmission, reducing the development cost of adjusting the length of a single transmission and improving operability and flexibility. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application; Figure 2 This is a flowchart of an audio frame data processing method provided in an embodiment of this application; Figure 3 This is a flowchart of another audio frame data processing method provided in the embodiments of this application; Figure 4 This is a flowchart of an audio frame data filling method provided in an embodiment of this application; Figure 5 This is a flowchart of another audio frame data processing method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of an audio frame data processing device provided in an embodiment of this application; Figure 7 This is a schematic diagram of another audio frame data processing device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0025] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first calling subfunction may be referred to as a second calling subfunction, and similarly, a second calling subfunction may be referred to as a first calling subfunction.

[0026] In this context, "at least one" means one or more functions. For example, at least one function can be one function, two functions, three functions, or any integer number of functions greater than or equal to one. "Multiple" means two or more functions. For example, multiple functions can be two functions, three functions, or any integer number of functions greater than or equal to two. "Each" means each one of the at least one functions. For example, each function means each of the multiple functions. If the multiple functions are three functions, then each function means each of the three functions.

[0027] The term "in response to" as used in this application is used to indicate the conditions or states on which the operation is performed. When the conditions or states on which the operation is performed are met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0028] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have all been fully authorized by the user or relevant parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0029] For example, the audio frame data involved in this application are all fully authorized by the user or relevant parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0030] For ease of understanding, the following concepts involved in this application are explained.

[0031] (1) Android System: An open-source operating system based on the Linux kernel, used for mobile devices. Android has many features, such as a vast application ecosystem, a user-friendly interface, and powerful development tools and resources. Among these, Android is open source, which means that developers can view, modify, and distribute the source code. This openness makes Android very flexible and allows it to be widely customized and applied to various devices.

[0032] (2) OpenSL ES (Open Sound Library for Embedded Systems): This is a cross-platform audio processing library optimized for embedded systems, designed to provide high-performance, low-latency, and standardized audio development interfaces for mobile devices and embedded systems. The OpenSL ES interface has the following features: cross-platform support (suitable for embedded systems such as Android and iOS), hardware acceleration (can directly call underlying audio hardware resources, avoiding data copying between the Java layer and the Native layer, improving processing efficiency, such as replacing Android's Audio Track), and low latency (suitable for real-time audio processing scenarios, such as game sound effects and voice calls).

[0033] (3) C++ Method Interception (Hook): This refers to the ability to intercept calls to C++ methods within a process through technical means, supporting the observation and modification of the execution results of C++ methods. Based on different implementation principles, hooks can be divided into PLT / GOT Hook and Inline Hook. Among them, PLT / GOT Hook is a dynamic linking hook technology for ELF files, which intercepts dynamically linked functions by modifying the function addresses in the program's Global Offset Table (GOT) or Procedure Link Table (PLT). Inline Hook, on the other hand, is an inline hook that inserts jump instructions to jump to a user-defined function by overriding the first few machine instructions of the target function. The C++ method interception used in the embodiments of this application can be implemented using either the PLT / GOT Hook framework or the Inline Hook framework.

[0034] (4) PCM (Pulse-Code Modulation): This is a digital audio encoding format used to convert analog audio signals into digital signals. PCM encoding samples the analog audio signal at a certain sampling rate and quantizes the amplitude of each sample point into a digital value. These digital values ​​are then encoded according to a certain encoding method to generate digital audio data. PCM encoding is a lossless compression encoding method that can preserve all the information of the original audio signal.

[0035] The sampling rate refers to the frequency at which an analog audio signal is sampled, that is, the number of times the analog audio signal is sampled per second, usually expressed in Hertz (Hz). The sampling rate determines the quality and data volume of the digital audio data. A higher sampling rate can represent the original audio signal more accurately, and the quality of the digital audio data is better, but the data volume is also larger.

[0036] Sampling precision refers to the quantization precision of each sample point in digital audio data, usually expressed in bits. In digital audio data, sampling precision determines the range and accuracy of the digital value at each sample point, thus affecting the quality and volume of the digital audio data. Common sampling precisions include 8-bit, 16-bit, 24-bit, and 32-bit. Higher sampling precision allows for a more accurate representation of the original audio signal, resulting in better digital audio data quality, but also a larger data volume. When performing digital audio recording, editing, encoding, and decoding operations, it is necessary to select an appropriate sampling precision based on the actual situation to achieve optimal audio quality and data compression effects.

[0037] (5) Audio PCM data: Essentially based on PCM encoding, it converts analog audio signals into digital signals to obtain digital audio data. Audio PCM data is widely used in the field of digital audio, such as CD audio, digital audio recording, and digital audio transmission. In the process of audio processing and encoding / decoding, it is necessary to perform operations such as sampling, quantization, encoding, and decoding on audio PCM data to realize the processing and transmission of audio signals.

[0038] (6) Audio frame data: This refers to a short segment of audio PCM data (typically tens of milliseconds). When processing audio PCM data, whether playing or recording audio, the audio PCM data is presented in the form of audio frame data. Taking audio playback as an example, the data transmitted to the audio player in a single transmission is a pointer to the audio frame data and the length of that audio frame data. The data pointed to by the pointer is one audio frame data. By saving the audio frame data sequentially into a file or transmitting it to the server, and in conjunction with the preceding audio data format specification, audio data acquisition can be completed.

[0039] In applications, the processing and playback of audio frame data are common requirements. In audio playback scenarios, the length of audio frame data is specified by the upper-layer audio business logic through fixed code logic. However, in scenarios such as cross-process audio playback or uploading audio to a live stream, which require secondary transmission of the acquired audio frame data, it is necessary to adjust the length of the audio frame data transmitted to the audio player in a single transmission to achieve a balance between audio playback quality and transmission efficiency.

[0040] Based on this, this application provides a non-intrusive method for real-time adjustment of audio frame data length. Through C++ hooking, it intercepts in-process calls to OpenSL ES to create OpenSL engine instances. By proxying the OpenSL engine instance, it further intercepts calls to create audio players and buffer queues. By proxying the buffer queue, it obtains the timing of audio frame data filling, i.e., the timing of calls to the enqueue function. Then, by proxying the enqueue function, it buffers the currently input audio frame data. When the accumulated audio frame data length reaches the target length, it calls the native enqueue function to perform the filling and enqueueing operation, thereby adjusting the length of the audio frame data filled in a single instance. This successfully achieves real-time dynamic adjustment of audio frame data length without intruding on upper-layer audio business logic, providing an operational basis for balancing audio playback quality and transmission efficiency.

[0041] For a detailed implementation of the method provided in this application, please refer to the following embodiments.

[0042] The audio frame data processing method provided in this application can be used in computer devices. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. This application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0043] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application. See also... Figure 1 The computer system includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0044] Terminal 101 has an application 111 installed and running that supports audio frame data processing. This application 111 can be a social application, online payment application, online shopping application, game application, medical service application, video application, or similar type. When terminal 101 runs application 111, the user interface of application 111 is displayed on the screen of terminal 101. Terminal 101 is the terminal used by user 121.

[0045] Optionally, terminal 101 can refer to one of a plurality of terminals. As those skilled in the art will know, the number of terminals can be more or less. For example, there may be only one terminal, or there may be six, eight or more terminals. This application does not limit the number of terminals or the type of device.

[0046] Figure 1 Only one terminal is shown in the diagram. In different embodiments, multiple other terminals 103 can access the server 102. Optionally, one or more terminals 103 may also be terminals corresponding to developers, on which an application development and editing platform is installed. Developers can edit and update applications on terminals 103, and transmit the updated application installation package to the server 102 via wired or wireless network. Terminal 101 downloads the application installation package from the server 102 to update the application.

[0047] Terminal 101 and other terminals 103 are connected to server 102 via wired or wireless networks.

[0048] Server 102 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 102 is used to provide background services for applications. Optionally, server 102 undertakes the primary computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the primary computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0049] In some embodiments, server 102 sends audio frame data to terminal 101. Terminal 101 receives the audio frame data sent by server 102 and, using the audio frame data processing method provided in this application, fills the received audio frame data into the buffer queue of an audio player to play the received audio frame data. For example, in scenarios such as making phone calls, live streaming, or playing music online, server 102 provides audio frame data to terminal 101. In other embodiments, terminal 101 collects audio frame data through a microphone and, using the audio frame data processing method provided in this application, fills the collected audio frame data into the buffer queue of an audio player to play the collected audio frame data. For example, in scenarios such as recording audio or shooting video, terminal 101 collects audio frame data.

[0050] It should be noted that the above computer system is only an example. The method provided in this application embodiment can also be executed by terminal 101 or server 102 alone, or by other computer devices. This application embodiment does not limit this.

[0051] It should be noted that the method provided in this application embodiment can be applied to any type of operating system, such as Android system, iOS system, etc., and this application embodiment does not limit it.

[0052] Figure 2 This is a flowchart of an audio frame data processing method provided in an embodiment of this application. This embodiment is executed by a computer device. See also... Figure 2 The method includes the following steps.

[0053] 201. The computer device creates an audio player and a buffer queue, and the audio player is used to read audio frame data from the buffer queue.

[0054] When a computer device initiates the audio playback process, it creates an audio player and a buffer queue. Subsequently, the audio frame data is played through the collaborative work between the audio player and the buffer queue.

[0055] The buffer queue is a storage area used to store audio frame data to be played. Audio frame data passed in from the upper-layer audio service is first stored in this buffer queue. The audio player reads the audio frame data from this buffer queue sequentially for playback.

[0056] Filling the buffer queue with incoming audio frame data takes time and resources. The length of the audio frame data sent by the upper-layer audio service is fixed. According to the original code logic, the length of the audio frame data sent each time needs to be filled into the buffer queue, which means that the length of the audio frame data filled into the buffer queue each time is fixed. However, in this embodiment, in order to balance audio playback effect and transmission efficiency, the length of the audio frame data filled into the buffer queue each time is flexibly controlled by executing the following steps 202-204.

[0057] 202. The computer device changes the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function. The native enqueue function is used to fill the buffer queue with audio frame data.

[0058] The buffer queue has an enqueue function pointer, which is the address of a native enqueue function provided by the OpenSL ES framework. This native enqueue function is used to fill the buffer queue with the incoming audio frame data. In this embodiment, if it is desired to adjust the length of the audio frame data filled into the buffer queue at one time, the process of filling the buffer queue with audio frame data needs to be delegated.

[0059] Therefore, the computer device modifies the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function. The native enqueue function is a function provided by the OpenSL ES framework for filling audio frame data, and the proxy enqueue function is the function provided in this application for filling audio frame data. By modifying the enqueue function pointer to point to the proxy enqueue function, calls to the native enqueue function can be transformed into calls to the proxy enqueue function, thereby intercepting calls to the native enqueue function and controlling the audio frame data filling process through the proxy enqueue function provided in this application.

[0060] 203. In response to a call instruction to the enqueue function pointer, the computer device obtains the proxy enqueue function pointed to by the enqueue function pointer and the audio frame data indicated by the call instruction. The length of the audio frame data is equal to the original length.

[0061] The call instruction is used to call the original enqueuing function that the enqueuing function pointer originally pointed to. Since the enqueuing function pointer has been modified to point to the proxy enqueuing function, the proxy enqueuing function is obtained in response to the call instruction, thereby intercepting the call to the original enqueuing function.

[0062] The call instruction also carries the address of the audio frame data. Based on the address carried by the call instruction, the audio frame data is obtained. The audio frame data is the audio frame data passed in by the upper-layer audio service provider in a single instance. The length of the audio frame data is equal to the original length, which represents the length of the audio frame data filled into the buffer queue in a single instance as specified by the upper-layer audio service provider.

[0063] Optionally, the invocation instruction also carries the native length.

[0064] 204. The computer device, through the proxy enqueue function, fills the target length of audio frame data into the buffer queue when the length of the acquired and unfilled audio frame data is not less than the target length.

[0065] The target length and the native length are two independent lengths. The native length is the fixed length specified by the upper-layer service for a single filling into the buffer queue, while the target length is the expected length for a single filling into the buffer queue, determined to balance audio playback effect and transmission efficiency.

[0066] After receiving the proxy enqueue function, the computer device uses this function to determine if the length of the acquired but unfilled audio frame data is less than the target length. If the length is less than the target length, the operation of filling the audio frame data into the buffer queue is not performed at this time. Instead, it waits for the next call instruction to the enqueue function pointer to accumulate audio frame data. If the length is not less than the target length, it means that the length of the currently accumulated audio frame data has reached the target length, which is sufficient to perform a filling operation. In this case, the target length of audio frame data is filled into the buffer queue through the proxy enqueue function, thereby realizing the real-time dynamic adjustment of the length of a single filling from the original length to the target length.

[0067] The method provided in this application modifies the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function. Therefore, calls to the native enqueue function can be intercepted and redirected to the proxy enqueue function. The proxy enqueue function controls the audio frame data filling process. When the length of the acquired and unfilled audio frame data is not less than the target length, audio frame data of the target length is filled into the buffer queue, thereby adjusting the length of the audio frame data filled in a single transmission from the native length to the target length. Therefore, this application does not require modification of the native code logic. Without intruding on the native code logic, it achieves flexible adjustment of the length of audio frame data transmitted in a single transmission, reducing the development cost of adjusting the length of a single transmission and improving operability and flexibility.

[0068] The above Figure 2The embodiments described herein are merely brief illustrations of the audio frame data processing method. This audio frame data processing method also involves steps such as proxying the audio engine interface, an audio engine instance, an audio player, and a buffer queue; detailed processes are described below. Figure 3 Examples of implementations. Figure 3 This is a flowchart of another audio frame data processing method provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 3 The method includes the following steps.

[0069] 301. The computer device redirects the call instruction to the native engine acquisition function to the proxy engine acquisition function, which is used to obtain the audio engine interface.

[0070] The native engine retrieval function is provided by OpenSL ES and is used to obtain the audio engine interface, which is used to create audio engine instances. The proxy engine retrieval function is provided in this application and can be written by developers.

[0071] This proxy engine acquisition function is used to obtain the audio engine interface and implement other functionalities. For example, within the proxy engine acquisition function, the native engine acquisition function can be called to obtain the audio engine interface. In addition, the proxy engine acquisition function can also perform other operations on the obtained audio engine interface, such as modifying the function pointer of the audio engine interface.

[0072] In this embodiment, the call to the native engine acquisition function is redirected to a proxy engine acquisition function. This proxy function takes over the audio engine interface during the audio engine initialization phase. After acquiring the audio engine interface, the proxy function can further take over the audio player and buffer queue without changing the native interface structure and usage. This allows the entire audio frame data processing chain to be proxyed during initialization. Therefore, this application can take over and control the process of acquiring the audio engine interface without modifying the native code logic, improving convenience and flexibility.

[0073] In one possible implementation, the native engine's acquisition function could be slCreateEngine, whose function declaration is as follows: SLresult slCreateEngine(SLObjectItf *pEngine,SLuint32 numOptions,const SLEngineOption *pEngineOptions, SLuint32 numInterfaces, constSLInterfaceID *pInterfaceIds, const SLboolean *pInterfaceRequired) Here, pEngine refers to the audio engine interface. After the call to slCreateEngine, pEngine is initialized into a valid audio engine interface.

[0074] In one possible implementation, the computer device uses a C++ hook method to intercept the call instructions to the native engine's fetch function, thereby redirecting the call instructions to the proxy engine's fetch function, thus changing the direct call to the native engine's fetch function to a call to the proxy engine's fetch function.

[0075] 302. The computer device obtains the audio engine interface through the proxy engine function, and modifies the instance creation function pointer of the audio engine interface from pointing to the native instance creation function to pointing to the proxy instance creation function. The native instance creation function is used to create the audio engine instance.

[0076] After redirecting the call instruction to the native engine retrieval function to the proxy engine retrieval function, the computer device obtains the audio engine interface through the proxy engine retrieval function. Since the audio engine interface has been proxied, bypassing the native code logic, additional operations can be performed on the obtained audio engine interface, such as modifying the instance creation function pointer of the audio engine interface in this embodiment.

[0077] The audio engine interface includes an instance creation function pointer, which is the address of the native instance creation function provided by the OpenSL ES framework. This pointer points to the native instance creation function, which is used to create the audio engine instance. In this embodiment, if the length of the audio frame data filled into the buffer queue at a time is to be adjusted, the creation process of the audio engine instance needs to be proxied from the upstream of the entire audio frame data processing chain. After proxiing the creation process, the audio frame data filling process can be intercepted through layer-by-layer proxies, thereby enabling dynamic adjustment of the length of the audio frame data filled into the buffer queue at a time.

[0078] Therefore, the computer device modifies the instance creation function pointer of the audio engine interface from pointing to the native instance creation function to pointing to the proxy instance creation function, so that the call to the native instance creation function can be transformed into a call to the proxy instance creation function, thereby intercepting the call to the native instance creation function and controlling the process of creating an audio engine instance through the proxy instance creation function provided in this application.

[0079] The proxy instance creation function, provided in this application, can be written by developers. This function is used to create an audio engine instance and implement other functionalities. For example, the proxy instance creation function can call the native instance creation function to create an audio engine instance. Furthermore, the proxy instance creation function can also perform other operations on the created audio engine instance, such as modifying its function pointer.

[0080] In this embodiment, since the proxy engine acquisition function has taken over the audio engine interface, the instance creation function pointer of the audio engine interface can be modified to point to the proxy instance creation function through the proxy engine acquisition function. This makes the creation process of the audio engine instance controllable and ensures that the creation process of the audio engine instance can run under the proxy logic. Therefore, this application does not require modification of the original code logic and can proxy the creation process of the audio engine instance without affecting the original code logic, thus improving convenience and flexibility.

[0081] In one possible implementation, the native instance creation function can be pEngine's GetInterface, and the proxy instance creation function can be proxy_GetInterface.

[0082] In one possible implementation, the computer device obtains the function through a proxy engine. After acquiring the audio engine interface, it returns a success message indicating that the native engine's function has been successfully invoked. This prevents upper-layer business logic from being interfered with or affected by the proxy process, thus improving the stability of audio frame data processing.

[0083] In one possible implementation, the proxy engine acquisition function includes a first call subfunction and a first modification subfunction. Therefore, step 302 includes: using the first call subfunction to call the native engine acquisition function to obtain the audio engine interface; and using the first modification subfunction to change the instance creation function pointer of the audio engine interface from pointing to the native instance creation function to pointing to the proxy instance creation function.

[0084] The proxy engine retrieval function can be divided into two parts: one part is used to call the native engine retrieval function to implement the logic of retrieving the audio engine interface, and the other part is used to modify the function pointer of the audio engine interface, thereby decoupling the process of calling the native engine retrieval function from the process of modifying the instance creation function pointer.

[0085] In this implementation, the proxy engine retrieval function includes a first invocation subfunction and a first modification subfunction. The first invocation subfunction inherits the native functionality of retrieving the audio engine interface by calling the native engine retrieval function. This not only ensures that the native interface retrieval function is not broken and guarantees the accuracy of the obtained audio engine interface, but also eliminates the need to develop additional proxy code logic for retrieving the audio engine interface, improving convenience. After taking over the audio engine interface, the first modification subfunction replaces the function pointer of the audio engine interface to take over the next step of creating an audio engine instance, improving the clarity of the processing logic.

[0086] In this embodiment of the application, by executing the above steps 301-302, the interception of the native engine acquisition function is achieved. The purpose of intercepting the native engine acquisition function is to proxy the native instance creation function of the audio engine interface.

[0087] It should be noted that in this embodiment, by executing steps 302-308, the creation of an audio player and a buffer queue based on the audio engine interface is achieved. In another embodiment, other methods can also be used to create an audio player and a buffer queue based on the audio engine interface.

[0088] 303. In response to a call instruction to an instance creation function pointer, the computer device obtains the proxy instance creation function pointed to by the instance creation function pointer.

[0089] The instruction calls the native instance creation function that the instance creation function pointer originally pointed to. Since the instance creation function pointer has been modified to point to the proxy instance creation function, the proxy instance creation function is obtained in response to the instruction, thus intercepting the call to the native instance creation function.

[0090] In one possible implementation, the instruction for creating a function pointer to an instance is as follows: (*pEngine) ->GetInterface(pEngine, SL_IID_ENGINE,&engineEngine); Here, pEngine is the audio engine interface, engineEngine is an audio engine instance of type SLEngineItf, and ->GetInterface refers to the instance creation function pointer. Since the instance creation function pointer has been changed from pointing to the native instance creation function to pointing to the proxy instance creation function, the proxy instance creation function is called in response to this call instruction, instead of the native instance creation function.

[0091] 304. The computer device creates an audio engine instance through the proxy instance creation function, and modifies the player creation function pointer of the audio engine instance from pointing to the native player creation function to pointing to the proxy player creation function. The native player creation function is used to create the audio player.

[0092] After obtaining the proxy instance creation function, the computer device creates an audio engine instance through that function. Since the native code logic has been bypassed to proxy the audio engine instance, additional operations can be performed on it, such as modifying the player creation function pointer of the audio engine instance in this embodiment.

[0093] The audio engine instance includes a player creation function pointer, which is the address of the native player creation function provided by the OpenSL ES framework. This pointer points to the native player creation function, which is used to create the audio player. In this embodiment, if the length of the audio frame data filled into the buffer queue at one time is to be adjusted, the audio player creation process needs to be proxied in the layer-by-layer proxy.

[0094] Therefore, the computer device modifies the player creation function pointer of the audio engine instance from pointing to the native player creation function to pointing to the proxy player creation function, so that the call to the native player creation function can be converted into a call to the proxy player creation function, thereby intercepting the call to the native player creation function and controlling the process of creating an audio player through the proxy player creation function provided in this application.

[0095] The proxy player creation function, provided in this application, can be written by developers. This function is used to create an audio player and implement other functionalities. For example, the proxy player creation function can call the native player creation function to create an audio player. Furthermore, the proxy player creation function can perform other operations on the created audio player, such as modifying its function pointer.

[0096] In this embodiment, since the proxy instance creation function has taken over the audio engine instance, the player creation function pointer of the audio engine instance can be modified to point to the proxy player creation function through the proxy instance creation function. This makes the audio player creation process controllable and ensures that the audio player creation process can run under the proxy logic. Therefore, this application does not require modification of the original code logic and can proxy the audio player creation process without affecting the original code logic, improving convenience and flexibility.

[0097] In one possible implementation, the computer device creates an audio engine instance through a proxy instance creation function and then returns a success message indicating that the original instance creation function has been successfully invoked. This prevents upper-layer business logic from being interfered with or affected by the proxy process, thus improving the stability of the audio frame data processing.

[0098] In one possible implementation, the proxy instance creation function includes a second call subfunction and a second modification subfunction. Therefore, step 304 includes: calling the native instance creation function through the second call subfunction to create an audio engine instance; and modifying the player creation function pointer of the audio engine instance from pointing to the native player creation function to pointing to the proxy player creation function through the second modification subfunction.

[0099] The proxy instance creation function can be divided into two parts: one part is used to call the native instance creation function to implement the logic of creating an audio engine instance, and the other part is used to modify the function pointer of the audio engine instance, thereby decoupling the process of calling the native instance creation function from the process of modifying the player creation function pointer.

[0100] In this implementation, the proxy instance creation function includes a second calling subfunction and a second modification subfunction. The second calling subfunction inherits the native functionality for creating the audio engine instance by calling the native instance creation function. This not only ensures that the native instance creation functionality is not broken and guarantees the accuracy of the obtained audio engine instance, but also eliminates the need to develop additional proxy code logic for creating the audio engine instance, thus improving convenience. After taking over the audio engine instance, the second modification subfunction replaces the function pointer of the audio engine instance to take over the next step of creating the player, improving the clarity of the processing logic.

[0101] In this embodiment of the application, by executing the above steps 303-304, the interception of the native instance creation function is achieved. The purpose of intercepting the native instance creation function is to proxy the native player creation function of the audio engine instance.

[0102] It should be noted that in this embodiment, by executing steps 304-308, an audio player and a buffer queue are created based on an audio engine instance. In another embodiment, other methods can also be used to create an audio player and a buffer queue based on an audio engine instance.

[0103] 305. In response to a call instruction to the player creation function pointer, the computer device obtains the proxy player creation function pointed to by the player creation function pointer.

[0104] The call instruction is used to invoke the native player creation function that the player creation function pointer originally pointed to. Since the player creation function pointer has been modified to point to the proxy player creation function, the proxy player creation function is obtained in response to the call instruction, thereby intercepting the call to the native player creation function.

[0105] In one possible implementation, the instruction to create a function pointer for the player is as follows: (*engineEngine)->CreateAudioPlayer(engineEngine,&playerObject,audioSrc, nullptr, 1, ids, req); Here, `engineEngine` is an instance of the audio engine, and `playerObject` is an audio player of type `SLObjectItf`. `->CreateAudioPlayer` refers to a pointer to the player creation function. Since the pointer has been changed from pointing to the native player creation function to pointing to the proxy player creation function, this call instruction invokes the proxy player creation function, not the native player creation function.

[0106] 306. The computer device creates an audio player through the proxy player creation function. Based on the audio player, the queue creation function pointer of the audio player is modified from pointing to the native queue creation function to pointing to the proxy queue creation function. The native queue creation function is used to create a cache queue.

[0107] After obtaining the proxy player creation function, the computer device creates an audio player using that function. Since the native code logic has been bypassed to proxy the audio player, additional operations can be performed on it, such as modifying the queue creation function pointer of the audio player in this embodiment.

[0108] The audio player includes a queue creation function pointer, which is the address of a native queue creation function provided by the OpenSL ES framework. This pointer points to the native queue creation function, which is used to create a buffer queue. In this embodiment, if the length of the audio frame data filled into the buffer queue at one time is to be adjusted, the buffer queue creation process needs to be delegated in the layer-by-layer proxy.

[0109] Therefore, the computer device modifies the queue creation function pointer of the audio player from pointing to the native queue creation function to pointing to the proxy queue creation function, so that the call to the native queue creation function can be transformed into a call to the proxy queue creation function, thereby intercepting the call to the native queue creation function and controlling the process of creating the buffer queue through the proxy queue creation function provided in this application.

[0110] The proxy queue creation function provided in this application can be written by developers. This function is used to create a buffer queue and implement other functionalities. For example, the proxy queue creation function can call the native queue creation function to create a buffer queue. In addition, the proxy queue creation function can perform other operations on the created buffer queue, such as modifying the buffer queue's function pointer.

[0111] In this embodiment, since the proxy player creation function has taken over the audio player, the pointer of the audio player's queue creation function can be modified to point to the proxy queue creation function. This makes the creation process of the buffer queue controllable and ensures that the creation process of the buffer queue can run under the proxy logic. Therefore, this application does not require modification of the original code logic and can proxy the creation process of the buffer queue without affecting the original code logic, thus improving convenience and flexibility.

[0112] In one possible implementation, the computer device creates an audio player through a proxy player creation function. After creation, it returns a success message indicating that the native player creation function has been successfully invoked. This prevents upper-layer business logic from being interfered with or affected by the proxy process, thus improving the stability of audio frame data processing.

[0113] In one possible implementation, the proxy player creation function includes a third calling subfunction and a third modification subfunction. Therefore, step 306 includes: calling the native player creation function to create an audio player via the third calling subfunction; and modifying the audio player's queue creation function pointer from pointing to the native queue creation function to pointing to the proxy queue creation function via the third modification subfunction.

[0114] The proxy player creation function can be divided into two parts: one part is used to call the native player creation function to implement the logic of creating an audio player, and the other part is used to modify the function pointer of the audio player, thereby decoupling the process of calling the native player creation function from the process of modifying the queue creation function pointer.

[0115] In this implementation, the proxy player creation function includes a third calling subfunction and a third modifying subfunction that retrieves the proxy engine. The third calling subfunction inherits the native audio player creation functionality by calling the native player creation function. This not only ensures that the native player creation functionality is not compromised and guarantees the accuracy of the obtained audio player, but also eliminates the need for additional proxy code logic for player creation, improving convenience. After taking over the audio player, the third modifying subfunction replaces the audio player's function pointer to take over the next step of creating the buffer queue, improving the clarity of the processing logic.

[0116] In this embodiment of the application, by executing the above steps 305-306, the native player creation function is intercepted. The purpose of intercepting the native player creation function is to proxy the native queue creation function of the audio player.

[0117] It should be noted that in this embodiment, steps 306-308 are used to create a buffer queue based on an audio player. In another embodiment, other methods can also be used to create a buffer queue based on an audio player.

[0118] 307. In response to a call instruction to a queue creation function pointer, the computer device obtains the proxy queue creation function pointed to by the queue creation function pointer.

[0119] The instruction calls the native queue creation function that the queue creation function pointer originally pointed to. Since the queue creation function pointer has been modified to point to the proxy queue creation function, the proxy queue creation function is obtained in response to the instruction, thus intercepting the call to the native queue creation function.

[0120] In one possible implementation, the instruction to call the function pointer to create the queue is as follows: (*playerObject)->GetInterface(playerObject,SL_IID_BUFFERQUEUE,&pcmBufferQueue); Here, playerObject refers to the created audio player, pcmBufferQueue is a buffer queue of type SLBufferQueueItf, and ->GetInterface is a pointer to the queue creation function. Since the pointer to the queue creation function has been changed from pointing to the native queue creation function to pointing to the proxy queue creation function, the proxy queue creation function is called in response to this call instruction, instead of the native queue creation function.

[0121] 308. The computer device creates a buffer queue through a proxy queue creation function, and modifies the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function. The native enqueue function is used to fill the buffer queue with audio frame data.

[0122] After obtaining the proxy queue creation function, the computer device creates a buffer queue through this function. Since the native code logic has been bypassed to proxy the buffer queue, additional operations can be performed on it, such as modifying the enqueue function pointer of the buffer queue in this embodiment.

[0123] The buffer queue includes an enqueue function pointer, which is the address of a native enqueue function provided by the OpenSL ES framework. This native enqueue function is used to fill the buffer queue with audio frame data. The computer device modifies the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to a proxy enqueue function. This allows calls to the native enqueue function to be transformed into calls to the proxy enqueue function, thereby intercepting calls to the native enqueue function and controlling the process of filling audio frame data through the proxy enqueue function provided in this application.

[0124] The proxy enqueue function, provided in this application, can be written by developers. This proxy enqueue function is used to populate audio frame data and implement other functionalities. For example, the native enqueue function can be called within the proxy enqueue function to populate audio frame data. In addition, other operations can be performed on the passed-in audio frame data within the proxy enqueue function.

[0125] In one possible implementation, the computer device creates a buffer queue through a proxy queue creation function and returns a success message indicating that the native queue creation function has been successfully invoked. This prevents upper-layer business logic from being interfered with or affected by the proxy process, thus improving the stability of audio frame data processing.

[0126] In one possible implementation, the proxy queue creation function includes a fourth call subfunction and a fourth modification subfunction. Therefore, step 308 includes: calling the native queue creation function to create a buffer queue via the fourth call subfunction; and modifying the buffer queue's enqueue function pointer from pointing to the native enqueue function to pointing to the proxy enqueue function via the fourth modification subfunction.

[0127] The proxy queue creation function can be divided into two parts: one part calls the native queue creation function to implement the logic of creating a buffer queue, and the other part modifies the function pointer of the buffer queue, thereby decoupling the process of calling the native queue creation function from the process of modifying the enqueue function pointer.

[0128] In this implementation, the proxy queue creation function includes a fourth calling subfunction and a fourth modification subfunction. The fourth calling subfunction inherits the native queue creation functionality by calling the native queue creation function. This not only preserves the native queue creation functionality and ensures the accuracy of the obtained buffer queue, but also eliminates the need for additional proxy code logic for creating the buffer queue, improving convenience. After taking over the native queue, the fourth modification subfunction replaces the function pointer of the native queue to take over the next step of filling the buffer queue with audio frame data. This allows for flexible adjustment of the length of a single incoming audio frame data, improving the clarity of the processing logic.

[0129] In this embodiment of the application, by executing the above steps 307-308, the native queue creation function is intercepted. The purpose of intercepting the native queue creation function is to proxy the native enqueue function of the buffer queue.

[0130] 309. In response to a call instruction to an enqueue function pointer, the computer device obtains the proxy enqueue function pointed to by the enqueue function pointer and the audio frame data indicated by the call instruction. The length of the audio frame data is equal to the original length.

[0131] The call instruction is used to call the original enqueuing function that the enqueuing function pointer originally pointed to. Since the enqueuing function pointer has been modified to point to the proxy enqueuing function, the proxy enqueuing function is obtained in response to the call instruction, thereby intercepting the call to the original enqueuing function.

[0132] In one possible implementation, the instruction to call the enqueued function pointer carries the enqueued function pointer, the address of the audio frame data, and the original length.

[0133] For example, the instruction for calling an enqueued function pointer is as follows: (*pcmBufferQueue)->Enqueue(pcmBufferQueue, pBuffer, size); Here, `pcmBufferQueue` refers to the buffer queue, `pBuffer` is the address of the audio frame data, and `size` is the original length of the audio frame data. `->Enqueue` refers to the enqueue function pointer. Since the enqueue function pointer has been changed from pointing to the original enqueue function to pointing to the proxy enqueue function, the proxy enqueue function is called instead of the original enqueue function in response to this call instruction.

[0134] 310. The computer device, through the proxy enqueue function, fills the audio frame data of the target length into the buffer queue when the length of the acquired and unfilled audio frame data is not less than the target length.

[0135] The target length can be adjusted in real time via code. The target length can be greater than or less than the native length. For example, the target length is twice the native length. By increasing the length of audio frame data filled into the buffer queue at a time, the number of transmissions can be reduced, thus lowering the performance overhead of transmitting audio frame data.

[0136] Alternatively, the target length can be equal to the original length. If the target length is equal to the original length, it means that there is no need to adjust the length of a single fill.

[0137] In one possible implementation, the computer device adds the audio frame data indicated by the calling instruction to the data buffer pool through a proxy enqueue function. If the length of the audio frame data in the data buffer pool is not less than the target length, then the audio frame data of the target length in the data buffer pool is filled into the buffer queue.

[0138] The data buffer pool stores acquired audio frame data that has not yet been added to the buffer queue. The computer device first adds the audio frame data indicated by the current call instruction to the data buffer pool sequentially via a proxy enqueue function. Then, it checks if the length of all audio frame data in the data buffer pool is less than the target length. If it is not less than the target length, audio frame data of the target length is added to the buffer queue, thus dynamically adjusting the length of each filling operation from the original length to the target length. If the length is less than the target length, there is no need to perform the operation of adding audio frame data to the buffer queue in this instance.

[0139] In this implementation, the audio frame data is temporarily stored in a data buffer pool. This eliminates the need to fill the audio frame data one at a time. Instead, the audio frame data is accumulated, which allows for flexible control over the length of the audio frame data filled into the buffer queue at one time, thus improving the flexibility of filling audio frame data.

[0140] In one possible implementation, the computer device uses a proxy enqueue function to call the native enqueue function to fill the buffer queue with audio frame data of the target length when the length of the acquired and unfilled audio frame data is not less than the target length.

[0141] In this implementation, the proxy enqueue function inherits the native function of filling audio frame data by calling the native enqueue function. This not only ensures the accuracy of the audio frame data filling process without breaking the native audio frame data filling function, but also eliminates the need to develop additional proxy code logic for filling audio frame data, thus improving convenience.

[0142] In one possible implementation, the computer device uses a proxy enqueue function to fill the buffer queue with audio frame data of the target length and then returns a success message for the call instruction. Alternatively, the computer device uses a proxy enqueue function to return a success message for the call instruction if the length of the acquired and unfilled audio frame data is less than the target length. This success message indicates that the original enqueue function has been successfully called.

[0143] In this implementation, regardless of whether the target length is met, the proxy enqueue function will return a success message to the caller. This means that the upper-level business logic does not need to distinguish whether the audio frame data is actually filled into the queue, ensuring that the upper-level business logic can run normally and will not be interfered with or affected by the process of adjusting the data length by the proxy code logic, thus improving the stability of the audio frame data processing.

[0144] Figure 4 This is a flowchart of an audio frame data filling method provided in an embodiment of this application, such as... Figure 4 As shown, the method includes the following steps 401-404.

[0145] 401. The upper layer of the audio service triggers a call to the native enqueue function, passing in audio frame data.

[0146] 402. Add the audio frame data passed from the upper layer to the data buffer pool.

[0147] 403. Determine if the length of the audio frame data in the data buffer pool is less than the target length. If yes, return a success message to the upper layer of the audio service and wait for the next new audio frame data to be passed in, that is, return to step 401 above; if no, execute step 404 below.

[0148] 404. Call the native enqueue function to fill the target length of audio frame data in the data buffer pool into the buffer queue, and then return to step 403 above.

[0149] The method provided in this application modifies the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function. Therefore, calls to the native enqueue function can be intercepted and redirected to the proxy enqueue function. The proxy enqueue function controls the audio frame data filling process. When the length of the acquired and unfilled audio frame data is not less than the target length, audio frame data of the target length is filled into the buffer queue, thereby adjusting the length of the audio frame data filled in a single transmission from the native length to the target length. Therefore, this application does not require modification of the native code logic. Without intruding on the native code logic, it achieves flexible adjustment of the length of audio frame data transmitted in a single transmission, reducing the development cost of adjusting the length of a single transmission and improving operability and flexibility.

[0150] Figure 5 This is a flowchart of an audio frame data processing method provided in an embodiment of this application, such as... Figure 5 As shown, the method includes the following steps.

[0151] 1. Hook the native engine retrieval function sICreateEngine within the process.

[0152] 2. Intercept the native engine's sICreateEngine function. Replace the native engine's sICreateEngine function with the proxy engine's proxy_sICreateEngine function, and call the native engine's sICreateEngine function within the proxy_sICreateEngine function.

[0153] 3. Obtain the audio engine interface through the proxy engine's `proxy_sICreateEngine` function and intercept the native engine creation function of the audio engine interface. Replace the native engine creation function `GetLnterface` with the proxy engine creation function `proxy_GetLnterface`, and call the native engine creation function `GetLnterface` within the proxy engine creation function `proxy_GetLnterface`.

[0154] 4. Create an audio engine instance using the proxy engine creation function `proxy_GetInterface`, and intercept the native player creation function of the audio engine instance. Replace the native player creation function `CreateAudioPlayer` with the proxy player creation function `proxy_CreateAudioPlayer`, and call the native player creation function `CreateAudioPlayer` within the proxy player creation function `proxy_CreateAudioPlayer`.

[0155] 5. Create an audio player using the proxy_CreateAudioPlayer function, intercepting the native queue creation function of the audio player. Replace the native queue creation function Getlnterface with the proxy_Getlnterface function, and call the native queue creation function Getlnterface within the proxy_Getlnterface function.

[0156] 6. Create a buffer queue using the proxy queue creation function `proxy_GetInterface`, and intercept the native enqueue function of the buffer queue. Replace the native enqueue function `Enqueue` with the proxy enqueue function `proxy_Enqueue`, and call the native enqueue function `Enqueue` within the proxy enqueue function `proxy_Enqueue`.

[0157] 7. The frame filling timing can be obtained by creating the proxy queue based on the proxy_GetInterface function.

[0158] In related technologies, audio frame data is filled into a buffer queue using an enqueue function. When the audio service provider calls this function to fill audio frame data, the length of the passed-in audio frame data is a preset fixed value. Modifying the length of the audio frame data filled in a single instance requires the audio service provider to modify the code logic related to obtaining and filling the audio frame data, which necessitates intrusion into the audio business logic, resulting in high development costs and low operability. Furthermore, the length of the audio frame data is stored in a configuration format, lacking an interface for independent adjustment at the code level, making it inconvenient to adjust the length of the audio frame data in real-time during application runtime.

[0159] In the embodiments of this application, such as Figure 5As shown, at process startup, a C++ hook is used to register the proxy engine retrieval function `proxy_sICreateEngine`, thus determining the timing of calls to the native engine retrieval function `sICreateEngine` within the process. Subsequently, when the native engine retrieval function `sICreateEngine` is triggered, the call can be redirected to the proxy engine retrieval function `proxy_sICreateEngine`. Then, by proxying the audio engine instance, audio player, and buffer queue layer by layer, the frame filling timing of the buffer queue is determined. Finally, based on the frame filling timing, the incoming audio frame data is buffered through a data buffer pool, enabling flexible adjustment of the audio frame data length. Therefore, by introducing C++ hook technology and analyzing the OpenSL ES-based audio playback logic layer by layer, and by hooking key functions used during audio playback, non-intrusive real-time adjustment of the audio frame data length is achieved. This solves the problems of adjusting the audio frame data length depending on modifying the native code logic and the inconvenience of real-time modification, improving the flexibility of adjusting the audio frame data length, reducing the scope of modification, and adapting to various audio business scenarios.

[0160] Figure 6 This is a schematic diagram of the structure of an audio frame data processing device provided in an embodiment of this application. See also... Figure 6 The device includes: Create module 601 to create an audio player and a buffer queue. The audio player is used to read audio frame data from the buffer queue. Module 601 is also used to change the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function. The native enqueue function is used to fill the buffer queue with audio frame data. The acquisition module 602 is used to acquire the proxy enqueue function pointed to by the enqueue function pointer and the audio frame data indicated by the call instruction in response to the call instruction. The length of the audio frame data is equal to the original length. The filling module 603 is used to fill the target length audio frame data into the buffer queue by means of the proxy enqueue function, when the length of the acquired and unfilled audio frame data is not less than the target length.

[0161] The audio frame data processing apparatus provided in this application modifies the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function. Therefore, calls to the native enqueue function can be intercepted and redirected to the proxy enqueue function. The proxy enqueue function controls the audio frame data filling process. When the length of the acquired but unfilled audio frame data is not less than the target length, audio frame data of the target length is filled into the buffer queue, thereby adjusting the length of the audio frame data filled in a single transmission from the native length to the target length. Therefore, this application does not require modification of the native code logic. Without intruding on the native code logic, it achieves flexible adjustment of the length of audio frame data transmitted in a single transmission, reducing the development cost of adjusting the length of a single transmission and improving operability and flexibility.

[0162] Optionally, see Figure 7 Create module 601 for: Redirection unit 611 is used to redirect the call instruction to the native engine acquisition function to the proxy engine acquisition function, which is used to obtain the audio engine interface; Create unit 621 to obtain functions through the proxy engine, obtain the audio engine interface, and create an audio player and buffer queue based on the audio engine interface.

[0163] Optionally, see Figure 7 Create unit 621 for: The audio engine interface is obtained through the proxy engine function. The instance creation function pointer of the audio engine interface is changed from pointing to the native instance creation function to pointing to the proxy instance creation function. The native instance creation function is used to create the audio engine instance. In response to a call instruction to an instance creation function pointer, retrieve the proxy instance creation function pointed to by the instance creation function pointer; The function creates an audio engine instance through a proxy instance, and then creates an audio player and a buffer queue based on the audio engine instance.

[0164] Optionally, see Figure 7 The proxy engine obtains functions including a first call sub-function and a first modification sub-function; creation unit 621 is used for: The first sub-function call retrieves the audio engine interface by calling the native engine retrieval function. The first modified sub-function changes the instance creation function pointer of the audio engine interface from pointing to the native instance creation function to pointing to the proxy instance creation function.

[0165] Optionally, see Figure 7 Create unit 621 for: The audio engine instance is created by creating a proxy instance. The player creation function pointer of the audio engine instance is changed from pointing to the native player creation function to pointing to the proxy player creation function. The native player creation function is used to create the audio player. In response to a call to the player creation function pointer, retrieve the proxy player creation function pointed to by the player creation function pointer; An audio player is created using a proxy player creation function, and a buffer queue is created based on the audio player.

[0166] Optionally, see Figure 7 The proxy instance creation function includes a second calling subfunction and a second modification subfunction; creation unit 621 is used for: The second sub-function call invokes the native instance creation function to create an audio engine instance; The second modified sub-function changes the player creation function pointer of the audio engine instance from pointing to the native player creation function to pointing to the proxy player creation function.

[0167] Optionally, see Figure 7 Create unit 621 for: An audio player is created by using a proxy player creation function. The pointer of the audio player's queue creation function is changed from pointing to the native queue creation function to pointing to the proxy queue creation function. The native queue creation function is used to create the cache queue. In response to a call instruction to the queue creation function pointer, retrieve the proxy queue creation function pointed to by the queue creation function pointer; Create a buffer queue using the proxy queue creation function.

[0168] Optionally, see Figure 7 The proxy player creation function includes a third calling subfunction and a third modification subfunction; creation unit 621 is used for: The third sub-function calls the native player creation function to create an audio player. By modifying the third sub-function, the function pointer for creating the audio player's queue is changed from pointing to the native queue creation function to pointing to the proxy queue creation function.

[0169] Optionally, see Figure 7 The proxy queue creation function includes a fourth calling subfunction and a fourth modifying subfunction; creation unit 621 is used for: The buffer queue is created by calling the native queue creation function through the fourth sub-function call; The fourth modification sub-function changes the enqueue function pointer of the buffer queue from pointing to the native enqueue function to pointing to the proxy enqueue function.

[0170] Optionally, see Figure 7 Fill module 603, used for: The audio frame data indicated by the calling instruction is added to the data buffer pool through the proxy enqueue function. If the length of the audio frame data in the data buffer pool is not less than the target length, then the audio frame data of the target length in the data buffer pool is filled into the buffer queue.

[0171] Optionally, see Figure 7 Fill module 603, used for: By using a proxy enqueue function, when the length of the acquired and unfilled audio frame data is not less than the target length, the native enqueue function is called to fill the buffer queue with audio frame data of the target length.

[0172] Optionally, see Figure 7 The device also includes a message return module 604, used for: By using the proxy enqueue function, after filling the buffer queue with audio frame data of the target length, a success message for the call instruction is returned. By using the proxy enqueue function, if the length of the acquired and unfilled audio frame data is less than the target length, a success message for the call instruction is returned.

[0173] It should be noted that the audio frame data processing device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the audio frame data processing device and the audio frame data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0174] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed in the audio frame data processing method of the above embodiments.

[0175] Optionally, the computer device is provided as a terminal. Figure 8 A schematic diagram of the structure of a terminal 800 provided in an exemplary embodiment of this application is shown.

[0176] Terminal 800 includes a processor 801 and a memory 802.

[0177] Processor 801 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0178] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 802 are used to store at least one computer program, which is used by the processor 801 to implement the audio frame data processing method provided in the method embodiments of this application.

[0179] In some embodiments, the terminal 800 may also optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 803 via a bus, signal line, or circuit board. Optionally, the peripheral device includes at least one of: a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, and a power supply 808.

[0180] Peripheral device interface 803 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 801 and memory 802. In some embodiments, processor 801, memory 802 and peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 801, memory 802 and peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0181] The radio frequency (RF) circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 804 can communicate with other devices through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 804 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0182] Display screen 805 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 805 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 801 for processing. In this case, display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 805, disposed on the front panel of terminal 800; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 800 or in a folded design; in other embodiments, display screen 805 may be a flexible display screen, disposed on a curved or folded surface of terminal 800. Furthermore, display screen 805 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 805 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0183] The camera assembly 806 is used to acquire images or videos. Optionally, the camera assembly 806 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal 800, and the rear-facing camera is disposed on the back of the terminal 800. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash may be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0184] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 801 for processing, or input to the radio frequency circuit 804 to achieve voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 800. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0185] Power supply 808 is used to supply power to the various components in terminal 800. Power supply 808 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 808 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0186] In some embodiments, the terminal 800 further includes one or more sensors 809. The one or more sensors 809 include, but are not limited to, an accelerometer 810, a gyroscope 811, a pressure sensor 812, an optical sensor 813, and a proximity sensor 814.

[0187] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on terminal 800 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0188] Optionally, the computer device is provided as a server. Figure 9 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 900 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 901 and one or more memories 902. The memories 902 store at least one computer program, which is loaded and executed by the processor 901 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0189] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the audio frame data processing method of the above embodiments.

[0190] This application also provides a computer program product, including a computer program loaded and executed by a processor to perform the operations performed by the audio frame data processing method of the above embodiments.

[0191] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0192] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A method of processing audio frame data, the method comprising: The method comprises: creating an audio player and a buffer queue, the audio player being used to read audio frame data from the buffer queue; modifying a function pointer of the buffer queue from pointing to a native enqueue function to pointing to a proxy enqueue function, the native enqueue function being used to fill audio frame data to the buffer queue; in response to a call instruction to the enqueue function pointer, obtaining the proxy enqueue function pointed to by the enqueue function pointer and audio frame data indicated by the call instruction, the length of the audio frame data being equal to a native length; through the proxy enqueue function, filling audio frame data of a target length to the buffer queue when the length of the obtained and unfilled audio frame data is not less than the target length.

2. The method of claim 1, wherein, The creating an audio player and a buffer queue comprises: redirecting a call instruction to a native engine obtaining function to a proxy engine obtaining function, the native engine obtaining function being used to obtain an audio engine interface; through the proxy engine obtaining function, obtaining an audio engine interface, and based on the audio engine interface, creating the audio player and the buffer queue.

3. The method of claim 2, wherein, The through the proxy engine obtaining function, obtaining an audio engine interface, and based on the audio engine interface, creating the audio player and the buffer queue comprises: through the proxy engine obtaining function, obtaining the audio engine interface, and modifying a function pointer of an instance creation function of the audio engine interface from pointing to a native instance creation function to pointing to a proxy instance creation function, the native instance creation function being used to create an audio engine instance; in response to a call instruction to the instance creation function pointer, obtaining the proxy instance creation function pointed to by the instance creation function pointer; through the proxy instance creation function, creating an audio engine instance, and based on the audio engine instance, creating the audio player and the buffer queue.

4. The method of claim 3, wherein, The proxy engine obtaining function comprises a first call sub-function and a first modification sub-function; and the through the proxy engine obtaining function, obtaining an audio engine interface, and modifying a function pointer of an instance creation function of the audio engine interface from pointing to a native instance creation function to pointing to a proxy instance creation function comprises: through the first call sub-function, calling the native engine obtaining function to obtain the audio engine interface; through the first modification sub-function, modifying the function pointer of the instance creation function of the audio engine interface from pointing to the native instance creation function to pointing to the proxy instance creation function.

5. The method of claim 3, wherein, The through the proxy instance creation function, creating an audio engine instance, and based on the audio engine instance, creating the audio player and the buffer queue comprises: through the proxy instance creation function, creating the audio engine instance, and modifying a function pointer of a player creation function of the audio engine instance from pointing to a native player creation function to pointing to a proxy player creation function, the native player creation function being used to create an audio player; in response to a call instruction to the player creation function pointer, obtaining the proxy player creation function pointed to by the player creation function pointer; The audio player is created through the proxy player creation function, and the buffer queue is created based on the audio player.

6. The method of claim 5, wherein, The proxy instance creation function comprises a second calling sub-function and a second modifying sub-function; the audio engine instance is created through the proxy instance creation function, and a player creation function pointer of the audio engine instance is modified from a native player creation function to a proxy player creation function, which comprises: The audio engine instance is created through the second calling sub-function; The player creation function pointer of the audio engine instance is modified from the native player creation function to the proxy player creation function through the second modifying sub-function.

7. The method of claim 5, wherein, The audio player is created through the proxy player creation function, and the buffer queue is created based on the audio player, which comprises: The audio player is created through the proxy player creation function, and a queue creation function pointer of the audio player is modified from a native queue creation function to a proxy queue creation function, the native queue creation function being used to create a cache queue; In response to a calling instruction of the queue creation function pointer, the proxy queue creation function pointed by the queue creation function pointer is acquired; The buffer queue is created through the proxy queue creation function.

8. The method of claim 7, wherein, The proxy player creation function comprises a third calling sub-function and a third modifying sub-function; the audio player is created through the proxy player creation function, and a queue creation function pointer of the audio player is modified from a native queue creation function to a proxy queue creation function, which comprises: The audio player is created through the third calling sub-function; The queue creation function pointer of the audio player is modified from the native queue creation function to the proxy queue creation function through the third modifying sub-function.

9. The method of claim 7, wherein, The proxy queue creation function comprises a fourth calling sub-function and a fourth modifying sub-function; the buffer queue is created through the proxy queue creation function, which comprises: The buffer queue is created through the fourth calling sub-function; The enqueue function pointer of the buffer queue is modified from a native enqueue function to a proxy enqueue function, which comprises: The enqueue function pointer of the buffer queue is modified from the native enqueue function to the proxy enqueue function through the fourth modifying sub-function.

10. The method according to any one of claims 1 to 9, characterized in that, The target length of audio frame data is filled into the buffer queue when a length of acquired and unfilled audio frame data is not less than the target length through the proxy enqueue function, which comprises: The audio frame data indicated by the calling instruction is added to a data buffer pool through the proxy enqueue function, and the target length of audio frame data in the data buffer pool is filled into the buffer queue if a length of audio frame data in the data buffer pool is not less than the target length.

11. The method of claim 1, wherein, The target length is obtained through the proxy enqueue function, and when the length of the obtained and unfilled audio frame data is not less than the target length, the audio frame data of the target length is filled into the buffer queue, including: Through the proxy enqueue function, when the length of the obtained and unfilled audio frame data is not less than the target length, the native enqueue function is called to fill the audio frame data of the target length into the buffer queue.

12. The method of claim 1, wherein, The method further includes: After the audio frame data of the target length is filled into the buffer queue through the proxy enqueue function, a call success message of the call instruction is returned; When the length of the obtained and unfilled audio frame data is less than the target length, a call success message of the call instruction is returned through the proxy enqueue function.

13. An audio frame data processing apparatus, characterized by comprising: The device includes: A creation module is configured to create an audio player and a buffer queue, and the audio player is configured to read audio frame data from the buffer queue; The creation module is further configured to modify a pointer of an enqueue function of the buffer queue from pointing to a native enqueue function to pointing to a proxy enqueue function, and the native enqueue function is configured to fill audio frame data into the buffer queue; An obtaining module is configured to, in response to a call instruction for the pointer of the enqueue function, obtain the proxy enqueue function pointed to by the pointer of the enqueue function and audio frame data indicated by the call instruction, and the length of the audio frame data is equal to a native length; A filling module is configured to, through the proxy enqueue function, when the length of the obtained and unfilled audio frame data is not less than a target length, fill the audio frame data of the target length into the buffer queue.

14. A computer device, comprising: The computer device includes a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to implement the operations performed by the audio frame data processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, which is loaded and executed by the processor to implement the operations performed by the audio frame data processing method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program, characterized in that, The computer program is loaded and executed by the processor to implement the operations performed by the audio frame data processing method according to any one of claims 1 to 12.

Images