Real-time audio transmission method, system, medium, program product and terminal based on AVAudioEngine audio engine

Abstract

The application provides a real-time audio transmission method, system, medium, program product and terminal based on an AVAudioEngine audio engine. The AVAudioEngine audio engine is adopted to realize millisecond-level audio stream capture and transmission, and ensure low delay and high real-time performance of audio transmission. The application supports customized audio formats, can flexibly adapt to mainstream voice recognition service interfaces, and has strong compatibility. The application adopts a dynamic buffer area fragmentation mechanism for management, effectively reduces memory consumption, and significantly improves the running efficiency of an iOS device. The application introduces an exception handling mechanism, which can ensure stable operation of audio processing in the face of various abnormal conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of photon counting CT technology and self-supervised denoising, and particularly relates to a real-time audio transmission method, system, medium, program product and terminal based on an AVAudioEngine audio engine. BACKGROUND

[0002] At present, in the existing iOS system, an audio recording function is implemented through an AVAudioRecorder class. The AVAudioRecorder class provides an audio metering function (averagePowerForChannel) for obtaining volume information of audio, and can determine whether the audio is overloaded or silent. This function of the AVAudioRecorder class can be used in a simple audio monitoring scene, but obviously cannot meet the needs of an application scenario that needs to transmit a specific format audio stream in real time, that is, the AVAudioRecorder class cannot directly obtain real-time audio stream data in the audio recording process.

[0003] For example, in the application scenario of a real-time speech transcription (Real-time ASR) service of iFlytek, a developer needs to transmit a continuous audio stream in real time to obtain a corresponding text stream when using the service. However, the traditional audio processing method has the following defects:

[0004] (1) Real-time audio stream interception is not possible: the AVAudioRecorder class cannot provide a real-time data callback.

[0005] (2) Format adaptation is difficult: audio data needs to meet specific format requirements, and manual processing of audio format conversion and fragmented transmission is inefficient.

[0006] (3) High resource occupation: frequent audio data processing occupies a large amount of memory, which may cause a performance bottleneck. SUMMARY

[0007] In view of the above-mentioned defects of the prior art, the present application provides a real-time audio transmission method, system, medium, program product and terminal based on an AVAudioEngine audio engine, which is used to solve the problems of the prior art that the use of the AVAudioRecorder class cannot meet the needs of real-time audio stream acquisition, format adaptation and efficient transmission of real-time speech transcription services and the like.

[0008] To achieve the above object and other related objects, the first aspect of the present application provides a real-time audio transmission method based on an AVAudioEngine audio engine, characterized in that the method comprises: obtaining audio format information of a target server; initializing an AVAudioEngine audio engine based on the audio format information, and configuring the AVAudioEngine audio engine; obtaining audio data according to the configured AVAudioEngine audio engine, and processing the obtained audio data by using a dynamic buffer slicing mechanism, so as to transmit the processed audio data to the target server in real time.

[0009] In some embodiments of the first aspect of the present application, initializing the AVAudioEngine audio engine based on the audio format information and configuring the AVAudioEngine audio engine comprise: configuring audio parameters of audio recording of the AVAudioEngine audio engine based on the audio format information; defining an audio unit, setting an input format and a buffer size of the audio unit according to the audio parameters of the audio recording, and registering an audio callback function on the audio unit, so as to obtain a target audio unit.

[0010] In some embodiments of the first aspect of the present application, obtaining audio data according to the configured AVAudioEngine audio engine and processing the obtained audio data by using a dynamic buffer slicing mechanism comprise: the target audio unit obtains audio data in real time by using the audio callback function, and sends the audio data to a buffer of the target audio unit; the buffer of the target audio unit receives the audio data, and performs slicing processing on the audio data according to a preset slicing threshold.

[0011] In some embodiments of the first aspect of the present application, the audio format information of the target server comprises: a PCM format with a sampling rate of 16 kHz, a bit depth of 16 bit, and a single channel.

[0012] In some embodiments of the first aspect of the present application, initializing the AVAudioEngine audio engine further comprises creating a WAV file; the WAV file is used to locally store recorded audio data.

[0013] In some embodiments of the first aspect of the present application, the WebSocket protocol is used to transmit the processed audio data to the target server.

[0014] To achieve the above object and other related objects, the second aspect of the present application provides a real-time audio transmission system based on an AVAudioEngine audio engine, comprising: an audio format information acquisition module, configured to acquire audio format information of a target server; an audio engine configuration module, configured to initialize an AVAudioEngine audio engine based on the audio format information, and configure the AVAudioEngine audio engine; and a real-time transmission module, configured to acquire audio data according to the configured AVAudioEngine audio engine, and process the acquired audio data by using a dynamic buffer fragmentation mechanism, so as to transmit the processed audio data to the target server in real time.

[0015] To achieve the above object and other related objects, the third aspect of the present application provides a computer-readable storage medium, having a computer program stored thereon, wherein the computer program is executed by a processor to implement the real-time audio transmission method based on the AVAudioEngine audio engine.

[0016] To achieve the above object and other related objects, the fourth aspect of the present application provides a computer program product, comprising computer program code, wherein when the computer program code is executed on a computer, the computer is caused to implement the real-time audio transmission method based on the AVAudioEngine audio engine.

[0017] To achieve the above object and other related objects, the fifth aspect of the present application provides an electronic terminal, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the real-time audio transmission method based on the AVAudioEngine audio engine.

[0018] As described above, the real-time audio transmission method, system, medium, program product and terminal based on the AVAudioEngine audio engine provided by the present application have the following beneficial effects:

[0019] By using the AVAudioEngine audio engine, the present application realizes millisecond-level audio stream capture and transmission, ensures low latency and high real-time performance of audio transmission, supports customized audio formats, can flexibly adapt to mainstream voice recognition service interfaces, and has strong compatibility. By using the dynamic buffer fragmentation mechanism management, the present application effectively reduces memory consumption and significantly improves the running efficiency of iOS devices. The present application introduces an exception handling mechanism, which can ensure stable operation of audio processing in the face of various abnormal conditions. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1A flowchart of a real-time audio transmission method based on an AVAudioEngine audio engine is shown.

[0021] Figure 2 A schematic diagram of the working principle of a real-time audio transmission method based on an AVAudioEngine audio engine is shown.

[0022] Figure 3 A schematic diagram of the structure of a real-time audio transmission system based on an AVAudioEngine audio engine is shown.

[0023] Figure 4 A schematic diagram of the structure of an electronic terminal is shown. DETAILED DESCRIPTION

[0024] The embodiments of the present application will be described in detail below with specific reference being made to the drawings. Those skilled in the art can easily understand other advantages and purposes of the present application from the content disclosed in the specification. The present application can also be implemented or applied in other different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Before the present application is further described, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations:

[0025] <1>AVAudioEngine: an audio processing framework provided by Apple, is an audio engine framework provided by Apple, used for audio processing and audio synthesis. It is a powerful audio tool on iOS and macOS platforms, supporting real-time audio effect processing, audio mixing, audio playback, audio recording and audio synthesis functions. It is used for audio playback and processing in iOS, macOS and tvOS applications. It provides low-latency audio processing capabilities and supports real-time audio effects and mixing. AVAudioEngine is suitable for various application scenarios, including music players, game development, speech recognition, speech synthesis, etc. It provides flexible audio processing capabilities to meet the needs of different applications.

[0026] <2>PCM (Pulse Code Modulation) format: a coding method for digital audio and video signals, full name is Pulse Code Modulation. PCM data format converts analog signals into digital signals by sampling, quantizing and encoding analog signals for storage, transmission and processing on digital media. PCM data format is widely used in digital audio processing field, such as music production, audio editing, speech recognition, etc. In these applications, PCM data format can provide high-quality audio effect to meet the needs of professional users.

[0027] The application provides a real-time audio transmission method, system, medium, program product and terminal based on AVAudioEngine audio engine. The method captures audio data in real time through AVAudioEngine, and combines a dynamic buffer fragmentation mechanism to realize efficient and low-delay audio stream processing and transmission.

[0028] To facilitate understanding of the embodiments of the present application, first, the Figure 1 application is described in detail. Figure 1 A flowchart of a real-time audio transmission method based on AVAudioEngine audio engine in an embodiment of the application is shown. The method in this embodiment includes:

[0029] Step S11: Obtain the audio format information of the target server. The audio format information of the target server includes: PCM format with a sampling rate of 16 kHz, a bit depth of 16 bits, and a single channel.

[0030] It should be noted that in the existing iOS system, the AVAudioRecorder class is an audio recording class provided by the iOS system, mainly used for recording audio and saving it to a file. The AVAudioRecorder class does not support direct acquisition of real-time audio stream data, that is, it is not possible to extract audio data in real time during audio recording and transmit it to other systems or servers. However, in the scenarios of speech recognition, online education and real-time translation, it is necessary to transmit complete audio data to the server for processing, rather than just volume information. For example, speech recognition is the conversion of speech signals into text information, which is required in intelligent voice assistants, voice input methods, etc. to obtain audio data in real time and process it. Online education is the online classroom, where the speech communication between teachers and students needs to be accurately transmitted and recognized to achieve interactive teaching, voice answering, etc. Real-time translation is the real-time translation of speech in one language into another language, which requires fast acquisition and processing of audio data to output the translation result in time.

[0031] In this embodiment, the audio data is transmitted to the target server in real time. The target server is a server of a company, and the corresponding audio format information is a PCM format with a sampling rate of 16 kHz, a bit depth of 16 bits, and a single channel. In addition, 1280 bytes of data need to be sent every 40 ms. The target server can also be a server that needs to receive audio data in real time and process it. Different servers can be selected according to actual needs, and the corresponding audio format information can be obtained. In this embodiment, no limitation is made.

[0032] It should be explained that the PCM format audio data is not compressed and can restore the original audio signal well. The sampling rate is an important parameter, which represents the number of samples per second. The higher the sampling rate, the better the audio quality it can represent, but at the same time, the more storage space it occupies. A sampling rate of 16 kHz is usually used for voice recording and can meet the needs of general voice recognition and communication. Bit depth refers to the quantization accuracy of each sample, with bit as the unit. Single-channel audio has only one channel and is suitable for scenarios such as voice recording that do not require stereo sound effects. Compared with stereo sound (double-channel), single-channel audio has a smaller data volume.

[0033] Step S12: initializing an AVAudioEngine audio engine based on the audio format information and configuring the AVAudioEngine audio engine.

[0034] It should be noted that in this embodiment, the AVAudioEngine audio engine is used to transmit audio data in real time. Specifically, the AVAudioEngine audio engine is initialized based on the audio format information of the target server, and is appropriately configured, so as to obtain an initialized and configured AVAudioEngine audio engine. The AVAudioEngine audio engine can meet the needs of real-time audio data processing and transmission in scenarios such as voice recognition, online education, and real-time translation.

[0035] In some examples, the process of initializing the AVAudioEngine audio engine based on the audio format information and configuring the AVAudioEngine audio engine includes: configuring an audio parameter of audio recording of the AVAudioEngine audio engine based on the audio format information; defining an audio unit, setting an input format and a buffer size of the audio unit according to the audio parameter of the audio recording, and registering an audio callback function on the audio unit to obtain a target audio unit.

[0036] Specifically, the audio parameters of the audio recording of the AVAudioEngine audio engine are configured according to the audio format information, and the audio parameters include a sampling rate of 16 kHz and a number of bytes per frame of 2 (16 bits). Then the type of the audio session is configured as a recording mode (AVAudioSessionCategoryRecord class), which means that only audio recording is supported and playback is not supported, i.e. this class is mainly used for applications that need to record, such as a recorder application. On an iOS device, after setting the AVAudioSessionCategoryRecord class, the application can normally record, but other system sounds (such as incoming call ring, alarm, etc.) will not be played to ensure that the recording quality is not disturbed.

[0037] Further, an audio unit is defined, which is used to capture audio data collected by a microphone. The property of the audio unit is set to enable the audio input function, and the audio format is defined according to the audio parameters of the audio recording through AudioStreamBasicDescription, i.e. the audio format is PCM format, the sampling rate is 16 kHz, the bit depth is 16 bits, and it is a single channel. The defined audio format is set as the input format of the audio unit, and the buffer size in the audio unit is configured as 1280 bytes, i.e. 1280 bytes of audio data are sent every 40 ms. The audio callback function is registered on the audio unit, and the audio callback function is specified as AURenderCallback. The audio unit realizes real-time reception of audio data through the audio callback function, i.e. the audio callback function ensures that the audio data can be correctly collected and delivered to the subsequent processing logic.

[0038] It should be understood that after the above configuration, the target audio unit can be obtained, and the input format of the target audio unit is consistent with the audio format information of the target server, which can ensure the interface compatibility with the target server. For example, when the target server is a server of iFLYTEK, the input format of the target audio unit is set to PCM format, the sampling rate is 16 kHz, the bit depth is 16 bits, and it is a single channel. The target audio unit is compatible with the iFLYTEK interface, and the target audio unit can be used to transmit audio data to the iFLYTEK server in real time.

[0039] It should be noted that the PCM format has the advantages of good reducibility, strong compatibility, high flexibility and the like. Specifically, the PCM data format is a lossless compression method, which can accurately represent the original analog signal. Through sampling, quantization and encoding, the PCM data format can restore the same audio and video effects as the original analog signal on digital media. The PCM data format has good compatibility and can be widely used in various audio and video processing software. In addition, the PCM data format can be easily combined with other digital signal processing technologies, such as audio effect processing, video encoding, etc. The PCM data format has high flexibility, and users can adjust the sampling rate, quantization bit number and other parameters according to needs. This makes the PCM data format have strong adaptability in different application scenarios.

[0040] Step S13: Obtain audio data according to the configured AVAudioEngine audio engine, and process the obtained audio data by using a dynamic buffer slicing mechanism to transmit the processed audio data to the target server in real time.

[0041] In this embodiment, the audio data is processed by the configured AVAudioEngine audio engine. The specific processing process is as follows: during the audio recording process of the AVAudioEngine audio engine, the recorded audio data is appended to the buffer in real time, the audio data is dynamically cached in the buffer, and the size of the buffer is checked in a loop; if it is checked that the accumulated audio data in the buffer reaches a preset slicing threshold, the audio data is intercepted according to the preset slicing threshold to obtain sliced data, and the sliced data is sent to the target server. In this embodiment, millisecond-level audio stream capture and transmission can be realized by using AVAudioEngine, that is, efficient and low-delay audio stream processing and transmission can be realized by using the AVAudioEngine audio engine combined with the dynamic buffer slicing mechanism.

[0042] The dynamic buffer slicing mechanism is to slice the audio data in the buffer according to the preset slicing threshold, for example, the preset slicing threshold is 1280 bytes, and the slicing processing is performed every 40 ms and 1280 bytes. After the processing is completed, the processed audio data in the buffer is immediately released, that is, emptied, to avoid memory overflow.

[0043] In some examples, according to the configured AVAudioEngine audio engine, audio data is obtained, and the obtained audio data is processed by using a dynamic buffer slicing mechanism, including: the target audio unit obtains audio data in real time through the audio callback function, and sends the audio data to the buffer of the target audio unit; the buffer of the target audio unit receives the audio data, and slices the audio data according to a preset slicing threshold.

[0044] It needs to be explained that the target audio unit of starting the AVAudioEngine audio engine starts recording audio, at this time the target audio unit captures the audio data collected by the microphone device in real time. The target audio unit acquires audio data in real time through the audio callback function, renders the audio data in the callback, and sends the audio data to the buffer after successful rendering. The acquired audio data is processed by using a dynamic buffer fragmentation mechanism.

[0045] The process in which the target audio unit acquires audio data in real time through the audio callback function and sends the audio data to the buffer of the target audio unit is as follows: initializing the buffer of the target audio unit, the buffer size is 1280 bytes, and the buffer is used to store audio data; detecting the validity of the audio unit, if invalid, releasing the buffer memory and returning an error state, if valid, the audio unit acquires audio data and renders the audio data to the buffer, and subsequent processing is performed.

[0046] The processing process in the buffer is as follows: the acquired audio data is appended to the buffer of the target audio unit, and the data amount of the audio data accumulated in the buffer is cyclically judged; if the data amount of the audio data in the buffer is greater than or equal to a preset fragmentation threshold, the audio data in the buffer is intercepted and fragmented according to the preset fragmentation threshold to obtain fragmented data, and the fragmented data is sent to the target server; if the audio data byte in the buffer is less than the preset fragmentation threshold, the audio data continues to be accumulated.

[0047] Further, by dynamically caching audio data in the buffer, the audio data is fragmented according to the preset fragmentation threshold, and the processed audio data in the buffer is emptied, memory overflow can be avoided. Dynamic buffer management reduces memory consumption and improves the running efficiency of the iOS device.

[0048] In an embodiment, the initializing the AVAudioEngine audio engine further includes creating a WAV file; the WAV file is used to locally store the recorded audio data.

[0049] It needs to be explained that after the processed audio data is sent to the target server in this embodiment, a WAV file is also created locally for backup storage. The WAV file stores audio data in PCM format, which is a lossless audio format that can completely save the original audio data, and the WAV file format is widely supported and can be used in various audio players and editing software. The recorded audio data is saved as a backup to prevent data loss or problems during transmission, and can also be used for subsequent debugging and analysis.

[0050] In an embodiment, the processed audio data is transmitted to the target server using the WebSocket protocol. In this embodiment, the preset fragmentation threshold size and transmission protocol are adjusted according to the interface of the target server, which is not limited herein.

[0051] It should be explained that after the acquired audio data is processed using the dynamic buffer fragmentation mechanism, the fragmented data is obtained, and the fragmented data is transmitted to the target server in real time, wherein the fragmented data is sent to the target server using the WebSocket protocol. WebSocket is a network communication protocol used to establish a full-duplex communication channel between a client and a server. It allows bidirectional, real-time data transmission between the client and the server without the need to frequently establish and close connections like traditional HTTP protocols. Using the WebSocket protocol to establish a connection between the AVAudioEngine audio engine and the target server can achieve continuous transmission of processed audio data (fragmented data) to the target server without the need to re-establish a connection, reducing latency. The full-duplex communication mechanism of the WebSocket protocol ensures that audio data can be transmitted in real time, achieving low-latency network transmission, which is suitable for application scenarios that require real-time audio transmission.

[0052] In some examples, the process of processing the acquired audio data using the dynamic buffer fragmentation mechanism further includes detecting the validity of the target audio unit using an exception handling mechanism.

[0053] It should be noted that detecting the validity of the audio unit is an important step to ensure the normal operation of the audio processing flow. By introducing an exception handling mechanism to detect the validity of the audio unit, it can prevent null pointer exceptions and release invalid memory resources when an exception occurs. The validity detection of the target audio unit can ensure that the audio processing flow can run stably in the face of errors and exceptions.

[0054] To facilitate understanding of the real-time audio transmission method based on the AVAudioEngine audio engine provided in the present embodiment, the following specific embodiments are provided for illustration, as shown in Figure 2

[0055] Embodiment 1: Real-time speech-to-text system

[0056] The device configuration includes an iOS device (iPhone / iPad) with a system version ≥ iOS12, and an integrated AVAudioEngine audio engine.

[0057] ​Initialize the AVAudioEngine audio engine, and configure the AVAudioEngine audio engine, including audio format, register callback function, etc. The dynamic buffer fragmentation mechanism is used to realize the fragmentation and transmission of audio data in the buffer. The fragmented data is sent to the server of iFLYTEK through WebSocket.

[0058] The specific processing process is shown in Figure 2 The original input of the microphone of the iOS device is 44.1kHz. The configured AVAudioEngine real-time acquires audio data collected by the microphone. The audio format is 16kHz, 16bit, and PCM format. In the AVAudioEngine audio engine, the acquired audio data is appended to the buffer. When the data amount in the buffer is greater than or equal to 1280 bytes, the audio data is fragmented and sent to the target server (iFLYTEK server). If not, the buffer continues to accumulate. After the audio data is fragmented and sent to the target server, the remaining audio data in the buffer is updated, and it is determined whether to continue recording. If yes, the above steps of repeatedly acquiring audio data, appending to the buffer, and fragmenting and sending are continued. If not, the recording is ended.

[0059] The test uses the AVAudioEngine audio engine to transmit audio data to the iFLYTEK server in real time. The test result is that the audio stream transmission delay is less than or equal to 40ms, the recognition accuracy is improved by 15%, and the memory occupation is stable within 10MB.

[0060] It should be emphasized that the real-time audio transmission method based on the AVAudioEngine audio engine provided in the application realizes millisecond-level audio stream capture and transmission by using the AVAudioEngine audio engine, ensuring low delay and high real-time performance of audio transmission. The application supports customized audio format and can flexibly adapt to mainstream voice recognition service interfaces, with strong compatibility. The application effectively reduces memory consumption by using a dynamic buffer fragmentation mechanism, significantly improving the running efficiency of the iOS device. The application introduces an exception handling mechanism, which can ensure stable operation of audio processing in the face of various abnormal conditions.

[0061] In the embodiments of the application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", etc. The order is not limited. Those skilled in the art can understand that "first", "second", etc. do not limit the number and execution order, and "first", "second", etc. also do not necessarily mean different.

[0062] It should be noted that the words "exemplary" and "for example" are used herein to mean "an example of" rather than "an ideal". Any implementation described herein as "exemplary" or "for example" is not necessarily to be construed as preferred or advantageous over other implementations. Rather, the use of such terms is intended to present concepts in a concrete manner.

[0063] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c can represent a, b, c, a-b, a-c, b-c or a-b-c, wherein a, b and c can be single or multiple.

[0064] Figure 3 The embodiments of the present application provide a schematic block diagram of a real-time audio transmission system based on an AVAudioEngine audio engine. As shown in the figure, the system 300 comprises: Figure 3

[0065] An audio format information acquisition module 310 is configured to acquire audio format information of a target server.

[0066] An audio engine configuration module 320 is configured to initialize an AVAudioEngine audio engine based on the audio format information and configure the AVAudioEngine audio engine.

[0067] A real-time transmission module 330 is configured to acquire audio data according to the configured AVAudioEngine audio engine, process the acquired audio data by using a dynamic buffer fragmentation mechanism, and transmit the processed audio data to the target server in real time.

[0068] It should be understood that the specific process of each module performing the above corresponding steps has been described in detail in the above method embodiments, and for the sake of brevity, will not be repeated here.

[0069] ​It should also be appreciated that the division of tasks between the modules in the embodiments of the present application is illustrative only and that other divisions of tasks between the modules are possible. In addition, the various functional modules in the various embodiments of the present application can be integrated in one processor, or can be physically present separately, or two or more modules can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module.

[0070] Figure 4 is a schematic block diagram of an electronic terminal provided by an embodiment of the present application. As shown in Figure 4 , the electronic terminal includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the apparatus are coupled together by a bus system 404. It can be understood that the bus system 404 is used to realize the connection and communication between the components. In addition to including a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, in order to clearly illustrate the present application, all the buses are marked as the bus system in Figure 4 .

[0071] The user interface 405 can include a display, a keyboard, a mouse, a trackball, a click gun, a key, a button, a touchpad, or a touch screen, etc.

[0072] It can be understood that the memory 402 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), which is used as an external cache. By way of example but not by way of limitation, many forms of RAM can be used, such as static random access memory (SRAM, Static Random Access Memory), synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory). The memory described in the embodiments of the present application is intended to include but not limited to these and any other suitable categories of memory.

[0073] The memory 402 in the embodiment of the present application is configured to store various types of data to support the operation of the electronic terminal 400. Examples of the data include any executable programs for operating on the electronic terminal 400, such as an operating system 4021 and an application program 4022. The operating system 4021 includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks. The application program 4022 can include various application programs, such as a media player (Media Player), a browser (Browser), and the like, for implementing various application services. The application program 4022 can include the real-time audio transmission method based on the AVAudioEngine audio engine provided by the embodiment of the present application.

[0074] The method disclosed in the above embodiment of the present application can be applied to the processor 401 or implemented by the processor 401. The processor 401 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 401. The processor 401 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The processor 401 can implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application. The general-purpose processor 401 can be a microprocessor or any conventional processor, and the like. In combination with the steps of the accessory optimization method provided by the embodiment of the present application, the steps can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium, which is located in the memory. The processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0075] In the exemplary embodiment, the electronic terminal 400 can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), and the like, for executing the above method.

[0076] According to the method provided by the embodiment of the application, the application further provides a computer program product, which comprises computer program code, and when the computer program code runs on a computer, the computer is caused to execute the real-time audio transmission method based on the AVAudioEngine audio engine in any of the embodiments.

[0077] According to the method provided by the embodiment of the application, the application further provides a computer readable storage medium, which stores program code, and when the program code runs on a computer, the computer is caused to execute the real-time audio transmission method based on the AVAudioEngine audio engine in any of the embodiments.

[0078] The terms "component", "module", "system", and the like used in the present specification are used to represent computer-related entities, hardware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and / or thread of execution, and a component can be localized on one computer and / or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The components can communicate by way of local and / or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and / or across a network such as the Internet with other systems via the signal), software, and data, among others.

[0079] Those of ordinary skill in the art can realize that the various illustrative logical blocks and steps described in connection with the embodiments disclosed herein can be implemented or performed by electronic hardware, or a combination of computer software and electronic hardware. The functions described herein can be performed by hardware, software, or a combination thereof, depending on the particular application and design constraints. Those of ordinary skill in the art can realize the mechanisms, methods, and articles of manufacture described herein can be implemented in a variety of ways, and that the application encompasses all such variations.

[0080] Those of ordinary skill in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, device, and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0081] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0082] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0083] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit.

[0084] In the above embodiments, the functions of each unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, the functions can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the whole or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium, or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as high-density digital video disc (digital video disc, DVD), or semiconductor media (such as solid state disk (solid state disk, SSD), etc.

[0085] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0086] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0087] In summary, the real-time audio transmission method, system, medium, program product and terminal based on the AVAudioEngine audio engine provided by the present application include: obtaining audio format information of a target server; initializing the AVAudioEngine audio engine based on the audio format information, and configuring the AVAudioEngine audio engine; obtaining audio data according to the configured AVAudioEngine audio engine, and processing the obtained audio data by using a dynamic buffer fragmentation mechanism, to transmit the processed audio data to the target server in real time. By using the AVAudioEngine audio engine, the present application realizes millisecond-level audio stream capture and transmission, ensuring low latency and high real-time performance of audio transmission. The present application supports customized audio formats and can flexibly adapt to mainstream voice recognition service interfaces, with strong compatibility. By using the dynamic buffer fragmentation mechanism management, the present application effectively reduces memory consumption and significantly improves the running efficiency of iOS devices. The present application introduces an exception handling mechanism, which can ensure stable operation of audio processing in the face of various abnormal conditions. Therefore, the present application effectively overcomes the shortcomings of the prior art and has high industrial utilization value.

[0088] The above embodiments are only illustrative of the principles of the present application and its effects, and are not intended to limit the present application. Any modification or change made by any person skilled in the art without departing from the spirit and scope of the present application shall be covered by the claims of the present application.

Claims

1. A method for real-time audio transmission based on an AVAudioEngine audio engine, characterized in that, The method comprises the following steps: obtaining audio format information of a target server; initializing an AVAudioEngine audio engine based on the audio format information and configuring the AVAudioEngine audio engine; wherein the configuration process comprises: configuring audio parameters of audio recording of the AVAudioEngine audio engine based on the audio format information; defining an audio unit, setting input format and buffer size of the audio unit according to the audio parameters of the audio recording, and registering an audio callback function on the audio unit to obtain a target audio unit; obtaining audio data according to the configured AVAudioEngine audio engine, and processing the obtained audio data by using a dynamic buffer slicing mechanism to transmit the processed audio data to the target server in real time; the specific process is: appending the obtained audio data to the buffer of the target audio unit, and cyclically judging the data amount of the accumulated audio data in the buffer; if the data amount of the audio data in the buffer is greater than or equal to a preset slicing threshold, the audio data in the buffer is intercepted and sliced according to the preset slicing threshold to obtain sliced data, and the sliced data is sent to the target server; if the audio data byte in the buffer is less than the preset slicing threshold, the audio data is continuously accumulated.

2. The real-time audio transport method based on AVAudioEngine audio engine according to claim 1, characterized in that, initializing an AVAudioEngine audio engine based on the audio format information and configuring the AVAudioEngine audio engine, comprising: configuring audio parameters of audio recording of the AVAudioEngine audio engine based on the audio format information; defining an audio unit, setting input format and buffer size of the audio unit according to the audio parameters of the audio recording, and registering an audio callback function on the audio unit to obtain a target audio unit.

3. The real-time audio transport method based on AVAudioEngine audio engine according to claim 2, characterized in that, obtaining audio data according to the configured AVAudioEngine audio engine, and processing the obtained audio data by using a dynamic buffer slicing mechanism, comprising: the target audio unit obtains audio data in real time through the audio callback function, and sends the audio data to the buffer of the target audio unit; the buffer of the target audio unit receives the audio data, and processes the audio data by slicing according to a preset slicing threshold.

4. The real-time audio transport method based on AVAudioEngine audio engine according to claim 1, characterized in that, The audio format information of the target server comprises: PCM format with a sampling rate of 16 kHz, a bit depth of 16 bit, and a single channel.

5. The real-time audio transport method based on AVAudioEngine audio engine according to claim 1, characterized in that, The initialization of the AVAudioEngine audio engine further comprises creating a WAV file; the WAV file is used to locally store recorded audio data.

6. The real-time audio transport method based on AVAudioEngine audio engine according to claim 1, characterized in that, The processed audio data is transmitted to the target server by using a WebSocket protocol.

7. A real-time audio transport system based on an AVAudioEngine audio engine, characterized in that, The method comprises the following steps: an audio format information acquisition module is configured to obtain audio format information of a target server; An audio engine configuration module is configured to initialize and configure an AVAudioEngine audio engine based on audio format information; wherein the configuration process includes configuring audio parameters of audio recording of the AVAudioEngine audio engine based on the audio format information; defining an audio unit, setting an input format and a buffer size of the audio unit according to the audio parameters of the audio recording, and registering an audio callback function on the audio unit to obtain a target audio unit; A real-time transmission module is configured to obtain audio data based on the configured AVAudioEngine audio engine, and process the obtained audio data using a dynamic buffer slicing mechanism to transmit the processed audio data to the target server in real time; the specific process includes appending the obtained audio data to a buffer of the target audio unit, and cyclically judging a data volume of the accumulated audio data in the buffer; if the data volume of the audio data in the buffer is greater than or equal to a preset slicing threshold, the audio data in the buffer is intercepted and sliced according to the preset slicing threshold to obtain sliced data, and the sliced data is sent to the target server; if the audio data byte in the buffer is less than the preset slicing threshold, the audio data is continuously accumulated.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the real-time audio transmission method based on the AVAudioEngine audio engine in any one of claims 1 to 6.

9. A computer program product, characterised in that, The computer program product includes computer program code, and when the computer program code is run on a computer, the computer is caused to implement the real-time audio transmission method based on the AVAudioEngine audio engine in any one of claims 1 to 6.

10. An electronic terminal comprising a memory, a processor and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the real-time audio transmission method based on the AVAudioEngine audio engine in any one of claims 1 to 6.

Images