Audio data processing method, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an audio data processing method, electronic equipment and a storage medium. Based on the audio data processing method in the embodiment of the invention, initial audio data received by an entity sound card from each sound source device is allocated to a corresponding virtual sound card for processing according to a corresponding relationship between a pre-configured audio channel and the virtual sound card to obtain application audio data; each virtual sound card presents the sound source equipment type actually associated with the corresponding audio channel to the target application and sends the corresponding application audio data to the target application, and the target application can receive all audio data received by the entity sound card from each audio channel and flexibly select and use the audio data from all the audio data according to application needs of the target application. And the audio data use requirements of various applications can be met.

Description

Technical Field

[0001] This application relates to the field of audio technology, and more particularly to audio data processing methods, electronic devices, and storage media. Background Technology

[0002] With the continuous development and popularization of information technology, various types of electronic devices are constantly emerging. In electronic devices, using a sound card to connect to the audio source device and input the audio data to be played to the sound playback device is a common way for users to receive and send information when using electronic devices.

[0003] When the inventors studied the process of processing audio data provided by sound source devices received through sound cards at the application layer of existing Android-based electronic devices, they found that the application layer can only choose one of the audio data provided by multiple sound source devices, thus limiting the use of audio data at the application layer. Summary of the Invention

[0004] This invention provides an audio data processing method, an electronic device, and a storage medium to solve the technical problem that the application layer can only select one audio data provided by multiple sound source devices.

[0005] In a first aspect, embodiments of this application provide an audio data processing method, which includes:

[0006] The initial audio data received from each audio channel is obtained through a physical sound card;

[0007] Initial audio data from the same sound source device is distributed to the corresponding virtual sound card for processing to obtain application audio data. Each virtual sound card is configured with a corresponding audio channel and a corresponding sound source device type.

[0008] Each virtual sound card sends the corresponding application audio data to the target application, so that the target application can process the application audio data corresponding to at least one sound source device. The target application is the application that requested the initial audio data.

[0009] The above describes a process where the initial audio data received by the physical sound card from various sound source devices is allocated to the corresponding virtual sound cards for processing, based on the pre-configured correspondence between audio channels and virtual sound cards, to obtain application audio data. Each virtual sound card presents itself to the target application as the actual sound source device type associated with the corresponding audio channel and sends its own application audio data. The target application can receive all the audio data received by the physical sound card from each audio channel and flexibly select and use the audio data according to its own application needs, thus meeting the audio data usage requirements of various applications.

[0010] The corresponding audio channels and sound source device types configured in the virtual sound card are obtained by parsing the preset configuration file using a preset function.

[0011] As described above, the preset configuration file is parsed using a preset function to obtain the audio channels configured for each virtual sound card. The configuration of the audio channels in the configuration file can be flexibly adjusted as needed.

[0012] Specifically, the target application obtains the sound source device type corresponding to each virtual sound card by calling the sound source device type acquisition interface of the audio manager.

[0013] Based on the above, the audio data processing architecture of the system can be used to quickly obtain the sound source device type and accurately apply the audio data through the interface provided by the audio manager.

[0014] The application processing includes one or more of the following: mixing processing, remote transmission processing, playback processing, echo cancellation processing, noise reduction processing, and sound source localization processing.

[0015] Based on the above, once all audio data has been fully acquired, it can be processed flexibly in various ways as needed.

[0016] Specifically, each virtual sound card sends corresponding application audio data to the target application, enabling the target application to process the application audio data corresponding to at least one sound source device, including:

[0017] In the case of an exclusive request, the initial audio data corresponding to each virtual sound card is uniquely sent to the target application corresponding to the exclusive request, so that the target application corresponding to the exclusive request can perform application processing on the application audio data corresponding to at least one sound source device.

[0018] As mentioned above, audio data is provided specifically according to the type of request from the target application, ensuring the privacy and security of audio data transmission.

[0019] Each virtual sound card is configured with multiple audio channels, and different virtual sound cards have different audio channel configurations.

[0020] As described above, each virtual sound card is configured with multiple audio channels and receives audio data from the hardware corresponding to the multiple audio channels. The corresponding transmission and processing of audio data are achieved through physical association.

[0021] The types of sound source devices include microphones, projector interfaces, and at least one line input.

[0022] As mentioned above, the virtual sound card can be configured to support various existing sound source devices, enabling audio data input in various application scenarios.

[0023] The virtual sound card is configured as a plugin.

[0024] The virtual sound card, configured as a plug-in, allows for flexible configuration by selecting the appropriate plug-in to accommodate different combinations of sound source devices supported by various hardware configurations.

[0025] Secondly, embodiments of this application provide an electronic device, which includes:

[0026] One or more processors;

[0027] Memory, used to store one or more computer programs;

[0028] When one or more computer programs are executed by one or more processors, an electronic device enables the audio data processing method as described in the first aspect.

[0029] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio data processing method as described in the first aspect.

[0030] The electronic devices and computer-readable storage media of the second and third aspects can be used to perform the audio data processing methods provided in any of the above embodiments, and have corresponding functions and beneficial effects. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart of an audio data processing method provided in this application embodiment;

[0033] Figure 2 A schematic diagram of the architecture of an audio system for applying an audio data processing method provided in the embodiments of this application;

[0034] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the drawings, not all of the structures.

[0036] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art should be able to conceive after reading this application specification that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0037] The embodiments are described in detail below.

[0038] In electronic devices, using a sound card to receive audio data from a sound source device and input audio data to be played to a sound playback device is a common way for users to receive and send information while using electronic devices. For example, audio data can be collected through a microphone, processed by a sound card, and sent to another electronic device for playback. The sound card can also be used to play the audio data collected and sent by the other electronic device through headphones or speakers, thereby enabling two users to have direct voice conversations.

[0039] When the inventors studied the process of processing audio data provided by sound source devices received through sound cards at the application layer of existing Android-based electronic devices, they found that the sound routing mechanism built into the Android platform usually selects one of the multiple audio data received from the sound card driver according to a predetermined routing strategy and provides it to the application layer. The application layer can only choose to use one of the audio data provided by multiple sound source devices. This cannot meet the needs of some applications that require the use of all audio data for specific application scenarios, thus limiting the use of audio data at the application layer.

[0040] To address the aforementioned technical issues, this application proposes an audio data processing method. The method involves taking the initial audio data received by the physical sound card from various sound source devices, assigning it to the corresponding virtual sound card for processing according to a pre-configured correspondence between audio channels and virtual sound cards, thus obtaining application audio data. Each virtual sound card presents itself to the target application as the actual sound source device type associated with the corresponding audio channel and sends its own application audio data. The target application can receive all the audio data received by the physical sound card from each audio channel and flexibly select from all the audio data according to its own application needs, thereby meeting the audio data usage requirements of various applications.

[0041] Figure 1 This is a flowchart of an audio data processing method provided in an embodiment of this application, such as... Figure 1As shown, the audio data processing method includes, but is not limited to, steps S110-S130. This application provides an audio data processing method for an electronic device. The electronic device includes a sound card, which has multiple audio interfaces for receiving audio data from different sound source devices or sending audio data to be played to a sound playback device for playback.

[0042] like Figure 1 As shown, the audio data processing method includes:

[0043] Step S110: Obtain the initial audio data received from each audio channel through the physical sound card.

[0044] Please refer to Figure 2 This is a schematic diagram of the architecture of an audio system applying the audio data processing method in the embodiments of this application. The application layer includes various application software that primarily uses audio data; the architecture layer (i.e., the framework) mainly provides relevant processing interfaces for the application layer and links to the hardware abstraction layer (i.e., audio HAL); the hardware abstraction layer implements a large amount of audio data processing logic and links the architecture layer and the kernel layer (i.e., the Linux Kernel); the ALSA (Advanced Linux Sound Architecture) library in the tinyalsa layer is used to link the ALSA driver; the kernel layer contains the drivers for linking the hardware; the hardware layer can include various sound source devices and sound playback devices such as microphones and HDMI.

[0045] ALSA provides audio and MIDI (Musical Instrument Digital Interface) support on the Linux operating system. At the hardware layer, it exemplarily supports 2 XLR microphones, 2 linein inputs, 2 lineout inputs, 1 HDBase-T input, 2 lineout return signals, 1 AES67 PoE audio module, 1 HDMI in audio input, 2 HDMI out audio inputs, and 2 USB audio inputs. The inputs at the hardware layer can support a 48kHz sampling rate, 16–32bit, and dual-channel audio (the AES67 PoE audio module supports 8-channel input). The sound source devices in the hardware layer can include microphones, projector interfaces, and at least one line input. The virtual sound card can configure various existing sound source devices to achieve audio data input in various application scenarios.

[0046] exist Figure 2 Based on the audio system shown, which processes audio data, all initial audio data received from the sound source device through the audio channel is aggregated into the physical sound card.

[0047] Step S120: Distribute the initial audio data from the same sound source device to the corresponding virtual sound card for processing to obtain application audio data. Each virtual sound card is configured with a corresponding audio channel and a corresponding sound source device type.

[0048] The initial audio data received by the physical sound card, because the same sound source device corresponds to multiple audio channels for data transmission, can first be identified by the standard driver in the kernel layer corresponding to the sound card, based on the audio channel corresponding to the initial audio data received by the sound card, to determine whether the initial audio data comes from the same sound source device. When a virtual sound card is configured with corresponding audio channels and corresponding sound source device types, the initial audio data from the same sound source device can be distributed to the corresponding virtual sound card for processing to obtain application audio data. In this embodiment, the application audio data and the initial audio data sent by the corresponding sound source device may not be substantially different; this is only used to indicate that the initial audio data corresponding to each sound source device may be transmitted upwards independently.

[0049] In practice, virtual sound cards can be configured as plugins. By selecting the appropriate plugins to accommodate different combinations of sound source devices supported by various hardware configurations, virtual sound cards can be flexibly configured. Please refer to [reference needed]. Figure 2 The virtual sound card is set up as a plugin in the tinyalsa layer. The virtual sound card corresponding to each sound source device is enumerated as a corresponding hardware device in the hardware abstraction layer. That is, in addition to providing basic tinycap / tinyplay tools, the tinyalsa layer also needs to separate the sound source devices that were not clearly distinguished in the kernel layer, such as Mic, LINEIN1, and LINEIN2, into different hardware components and present them to the hardware abstraction layer. This allows the hardware abstraction layer to transmit the data provided by each hardware component upwards without being affected by other data, and complete the transmission to the application layer.

[0050] In another specific implementation, each virtual sound card is configured with multiple audio channels, and different virtual sound cards have different audio channel configurations. Based on the audio channel configuration, the sound card's standard driver layer can directly distribute the initial audio data to the corresponding virtual sound card according to the audio channel receiving the initial audio data. The audio channels configured for each virtual sound card can be 2, 3, or more, depending on the total number of lines used by the data transmission standard of different sound source devices. Based on the allocation of audio channels, each virtual sound card is configured with multiple corresponding audio channels, and receives audio data from the hardware corresponding to multiple audio channels accordingly, realizing the corresponding transmission and processing of audio data through physical association.

[0051] The specific configuration of a virtual sound card can be recorded using a dedicated configuration file, and the configuration is then completed based on this file. Specifically, the corresponding audio channels and sound source device types configured for each virtual sound card are obtained by parsing the preset configuration file using a preset function. As described above, the audio channels configured for each virtual sound card are obtained by parsing the preset configuration file using a preset function, and the audio channel configurations in the configuration file can be flexibly adjusted as needed.

[0052] Configuration files can be stored in a preset storage path with a preset filename and format. For example, a configuration file named asound.conf can be stored in / vendor / etc / alsa / . When an Android-based electronic device is running, the asound.conf file calls the snd_pcm_open API function, and the configuration parameters in asound.conf are loaded and parsed. The principle is that when the snd_pcm_open function is called, the asound.conf configuration file is parsed, and the physical sound card card0 is virtualized into a virtual sound card card99. For example, mic is split into / dev / snd / pcmC99d0, linein1 is split into / dev / snd / pcmC99d1, and linein2 is split into / dev / snd / pcmC99d2. These virtual sound cards are presented as independent hardware components to the hardware abstraction layer.

[0053] Step S130: Send the corresponding application audio data to the target application through each virtual sound card, so that the target application can perform application processing on the application audio data corresponding to at least one sound source device. The target application is the application that requested the initial audio data.

[0054] For virtual sound cards, in the native Android system, the Hardware Abstraction Layer (HAL) presents a physical sound card as a hardware component to the architecture layer. The multiple audio source devices actually connected to the physical sound card cannot all be presented to the application layer through the architecture layer. The architecture layer can only select one audio source from the multiple audio data streams that the physical sound card may connect to, according to a predetermined routing strategy, and provide it to the application layer. Consequently, the application layer can only obtain one audio data stream from the architecture layer. In this solution, a virtual sound card is used to present sound source devices such as mic, linein, hdmin, PoE Audio, and loopback acquisition as hardware components to the architecture layer. Figure 2In this architecture, mic, linein, hdmin, PoE Audio, and loopback each have their own audio HAL. By presenting an independent hardware component for each sound source device at the architecture layer, the audio routing strategy in the native Android system can be bypassed. Each virtual sound card, as an independent hardware component, can transmit audio data to the application layer, enabling the application layer to fully receive audio data from all sound source devices and selectively process it according to its own application needs.

[0055] The target application obtains the sound source device type corresponding to each virtual sound card by calling the sound source device type acquisition interface of the audio manager. The audio manager is usually located in the architecture layer. The target application can obtain available audio device information, i.e., the virtual sound cards corresponding to each sound source device, through the audio manager (AudioManager), and send a request to obtain initial audio data, specifically one or more channels. That is, each virtual sound card sends the corresponding application audio data to the target application according to the request. For example, a call application may only request microphone audio data, so only the virtual sound card corresponding to the microphone can send the corresponding application audio data; for example, a conferencing application may request all external input audio data and audio data captured by the microphone, so the conferencing application is the target application, and all virtual sound cards send the corresponding application audio data. Whether a virtual sound card sends application audio data can be determined by the architecture layer configuring the virtual sound card according to the request, and the virtual sound card choosing whether to transmit according to the settings; or the architecture layer confirms the audio data transmission requirement according to the request, and intercepts audio data not requested by the target application, not transmitting it to the target application. Based on the system's audio data processing architecture, the audio manager provides an interface that allows for the rapid acquisition of sound source device types and accurate application of audio data.

[0056] In the specific implementation process, application processing includes one or more of the following: audio mixing, remote transmission, playback, echo cancellation, noise reduction, and sound source localization. For the target application, based on all the comprehensively acquired audio data, various flexible processing methods can be applied to the audio data as needed. Of course, in the implementation process, some processing to optimize the output effect of audio data can be completed at a layer below the application layer. For example, audio mixing is equivalent to the architecture layer directly providing high-quality audio data to the application layer.

[0057] In another specific audio data processing scenario, such as when the target application is a call application, there are high requirements for the privacy of the audio data. In this case, each virtual sound card sends the corresponding application audio data to the target application, enabling the target application to process the application audio data corresponding to at least one sound source device. This includes: when the request type is an exclusive request, each virtual sound card uniquely sends the corresponding initial audio data to the target application corresponding to the exclusive request, allowing the target application to process the application audio data corresponding to at least one sound source device. That is, if the target application requests one or more types of audio data with an exclusive request type, this one or more types of audio data are only provided to that application and not to other applications until the application finishes acquiring the audio data. Of course, if multiple target applications send exclusive requests, the response is based on time priority, that is, the target application that sent the exclusive request first receives the audio data, until the user manually switches. Providing audio data specifically according to the type of request from the target application ensures the privacy and security of audio data transmission.

[0058] Overall, the system receives initial audio data from various sound source devices via the physical sound card. Based on the pre-configured correspondence between audio channels and virtual sound cards, the data is distributed to the corresponding virtual sound cards for processing to obtain application audio data. Each virtual sound card presents itself to the target application as the actual sound source device type associated with the corresponding audio channel and sends its own application audio data. The target application can receive all audio data received by the physical sound card from each audio channel and flexibly select and use all audio data according to its own application needs, thus meeting the audio data usage requirements of various applications.

[0059] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device includes a processor 310 and a memory 320. In one possible form of the electronic device, it may also include an input device 330, an output device 340, and a communication device 350. The number of processors 310 in the electronic device can be one or more. Figure 3 Taking a processor 310 as an example; the processor 310, memory 320, input device 330, output device 340, and communication device 350 in the electronic device can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0060] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the audio data processing method in the embodiments of this application. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 320, thereby implementing the aforementioned audio data processing method.

[0061] The memory 320 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 320 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include memory remotely located relative to the processor 310, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0062] Input device 330 can be used to receive network configuration information. Output device 340 may include a display device such as a screen.

[0063] The aforementioned electronic device can be used to execute any audio data processing method and has corresponding functions and beneficial effects.

[0064] This application also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform relevant operations in the audio data processing method provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0065] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0066] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0068] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0069] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0070] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. An audio data processing method, characterized in that, include: The initial audio data received from each audio channel is obtained through a physical sound card; Initial audio data from the same sound source device is distributed to the corresponding virtual sound card for processing to obtain application audio data. Each virtual sound card is configured with a corresponding audio channel and a corresponding sound source device type. Each virtual sound card sends corresponding application audio data to a target application, so that the target application processes the application audio data corresponding to at least one sound source device. The target application is the application that requested the initial audio data.

2. The audio data processing method according to claim 1, characterized in that, The corresponding audio channels and sound source device types configured in the virtual sound card are obtained by parsing a preset configuration file using a preset function.

3. The audio data processing method according to claim 1, characterized in that, The target application obtains the sound source device type corresponding to each virtual sound card by calling the sound source device type acquisition interface of the audio manager.

4. The audio data processing method according to claim 1, characterized in that, The application processing includes one or more of the following: audio mixing, remote transmission, playback, echo cancellation, noise reduction, and sound source localization.

5. The audio data processing method according to claim 1, characterized in that, The step of sending corresponding application audio data to the target application through each of the virtual sound cards, so that the target application can perform application processing on the application audio data corresponding to at least one sound source device, includes: When the type of the request is an exclusive request, the corresponding initial audio data is uniquely sent to the target application corresponding to the exclusive request through each virtual sound card, so that the target application corresponding to the exclusive request can perform application processing on the application audio data corresponding to at least one sound source device.

6. The audio data processing method according to claim 1, characterized in that, Each virtual sound card is configured with multiple audio channels, and different virtual sound cards have different audio channel configurations.

7. The audio data processing method according to claim 1, characterized in that, The types of sound source devices include microphones, projector interfaces, and at least one line input.

8. The audio data processing method according to claim 7, characterized in that, The virtual sound card is configured as a plugin.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the audio data processing method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the audio data processing method as described in any one of claims 1-8.

Images