Audio data processing method and device, electronic equipment and storage medium

Abstract

The invention provides an audio data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: triggering a first audio support end to analyze an operation into an audio control instruction in response to the operation of a first audio application, and transmitting the audio control instruction to a second audio support end through an inter-core communication channel; triggering a second audio support end to analyze the audio control instruction, at least obtaining a configuration parameter, and obtaining first audio data based on the configuration parameter; based on the configuration parameters and the attribute information of the output interface, performing output parameter configuration on the first audio data to obtain first playing data; and playing the first playing data through the selected output interface in response to meeting the playing condition. According to the invention, for the same hardware resource, different audio support schemes can be realized, and the flexibility of hardware adaptation is improved.

Description

Technical Field

[0001] The present disclosure relates to audio data processing technology, and in particular to an audio data processing method and device, an electronic device, and a storage medium. Background Art

[0002] At present, various audio output scenarios are usually set up in some vehicles, especially large vehicles, or considering the different audio output requirements of different drivers and passengers, it is often necessary to plan multiple audio resources to meet the output requirements of multiple sound zones. In order to achieve multi-sound zone audio processing, the sound system inside the vehicle is usually divided into multiple sound zones according to the internal space of the vehicle and the distribution of passengers. Each sound zone is responsible for playing audio signals by one or more speakers, and each speaker in the sound zone is connected to a hardware audio resource.

[0003] Under this design, if you want to adjust the audio output interface, you usually need to modify the hardware to achieve it. Obviously, this will not only increase the cost of adjusting the audio data output interface, but also cause inconvenience in use and poor user experience. Summary of the invention

[0004] The present disclosure provides an audio data processing method and device, an electronic device and a storage medium to at least solve the above technical problems existing in the prior art.

[0005] According to a first aspect of the present disclosure, there is provided an audio data processing method, which is applied to a system on chip (SoC), wherein the SoC includes at least two or more processor cores, and the two or more processor cores are connected through an inter-core communication channel; in the two or more processor cores, at least a first audio support end and a second audio support end are provided, the first audio support end can provide an access interface to multiple audio applications, and the second audio support end is connected to multiple audio interfaces; the method includes:

[0006] In response to the first audio application being operated, triggering the first audio support end to parse the operation into an audio control instruction, and sending the audio control instruction to the second audio support end through the inter-core communication channel;

[0007] Trigger the second audio support end to parse the audio control instruction, obtain at least configuration parameters of the first audio data, obtain the first audio data based on the configuration parameters, and determine at least one interface as an output interface for the first playback data among multiple audio interfaces;

[0008] Based on the configuration parameters and the attribute information of the output interface, configure the output parameters of the first audio data to obtain first playback data;

[0009] In response to the first playback data satisfying a playback condition, the first playback data is played through the selected output interface.

[0010] In some embodiments, the method further comprises:

[0011] In response to a homogeneous system or a heterogeneous system being loaded on the two or more processor cores, a virtual audio device node is created for each system in the homogeneous system or the heterogeneous system, and the virtual audio device node is used as the first audio support end of each system for access and control by an audio application of each system.

[0012] In some embodiments, the method further comprises:

[0013] Obtain relevant configuration parameters of the audio stream supported by the virtual audio device node, and trigger the first audio support end to send the relevant configuration parameters to the second audio support end through the inter-core communication channel;

[0014] The relevant configuration parameters include at least one of the following: number of channels, sampling rate, bit width, address of shared memory and single transmission length.

[0015] In some possible implementation modes, configuring the output parameters of the first audio data based on the configuration parameters and the attribute information of the output interface to obtain the first playback data includes:

[0016] Triggering the second audio support end to obtain the first audio data in the shared memory, and splitting the first audio data into corresponding voice endpoints in units of dual-channel data volume;

[0017] The second audio support end is triggered to perform at least one of resampling, mixing, and splicing of the audio data in the voice endpoint to obtain an output format supported by the output interface as the first playback data.

[0018] In some possible implementations, in response to the first playback data satisfying the playback condition, playing the first playback data through the selected output interface includes:

[0019] The output priority of the first playback data is determined based on the attribute information of the first playback data. When the priority order of the first playback data is the highest priority, the audio stream is output through the selected output interface.

[0020] In some possible implementations, a shared storage domain is provided between the two or more processor cores, and the audio data is stored in the shared storage domain;

[0021] The first audio support end and the second audio support end are both connected to the shared storage domain and access audio data through the shared storage domain.

[0022] According to a second aspect of the present disclosure, there is provided an audio data processing device, which is applied to a SoC, wherein the SoC includes at least two or more processor cores, and the two or more processor cores are connected through an inter-core communication channel; at least a first audio support terminal and a second audio support terminal are provided in the two or more processor cores, wherein the first audio support terminal can provide an access interface to multiple audio applications, and the second audio support terminal is connected to multiple audio interfaces; the device includes:

[0023] A first trigger unit, configured to trigger the first audio support end to parse the operation into an audio control instruction in response to the first audio application being operated, and to send the audio control instruction to the second audio support end through an inter-core communication channel;

[0024] A second trigger unit is used to trigger the second audio support end to parse the audio control instruction, obtain at least the configuration parameters of the first audio data, obtain the first audio data based on the configuration parameters, and determine at least one interface as an output interface for the first playback data among multiple audio interfaces;

[0025] A configuration unit, configured to configure output parameters of the first audio data based on the configuration parameters and the attribute information of the output interface to obtain first playback data;

[0026] The playing unit is configured to play the first playing data through the selected output interface in response to the first playing data satisfying a playing condition.

[0027] In some embodiments, the device further comprises:

[0028] A creation unit is used to create a virtual audio device node for each system in the homogeneous system or the heterogeneous system in response to the homogeneous system or the heterogeneous system being loaded on the two or more processor cores, and use the virtual audio device node as the first audio support end of each system for access and control by the audio application of each system.

[0029] In some embodiments, the device further comprises:

[0030] A third triggering unit, configured to obtain relevant configuration parameters of the audio stream supported by the virtual audio device node, and trigger the first audio support end to send the relevant configuration parameters to the second audio support end through an inter-core communication channel;

[0031] The relevant configuration parameters include at least one of the following: number of channels, sampling rate, bit width, address of shared memory and single transmission length.

[0032] In some possible implementation modes, the configuration unit is further configured to:

[0033] Triggering the second audio support end to obtain the first audio data in the shared memory, and splitting the first audio data into corresponding voice endpoints in units of dual-channel data volume;

[0034] The second audio support end is triggered to perform at least one of resampling, mixing, and splicing of the audio data in the voice endpoint to obtain an output format supported by the output interface as the first playback data.

[0035] In some possible implementation modes, the playback unit is further configured to:

[0036] The output priority of the first playback data is determined based on the attribute information of the first playback data. When the priority order of the first playback data is the highest priority, the audio stream is output through the selected output interface.

[0037] In some possible implementations, a shared storage domain is provided between the two or more processor cores, and the audio data is stored in the shared storage domain;

[0038] The first audio support end and the second audio support end are both connected to the shared storage domain and access audio data through the shared storage domain.

[0039] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0040] at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the audio data processing method described in the present disclosure.

[0043] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the steps of the audio data processing method described in the present disclosure.

[0044] The audio data processing method and device, electronic device and storage medium disclosed in the present invention, by setting multiple audio support terminals, the multiple audio support terminals are electrically connected to the same audio hardware and are controlled by the same audio hardware. In this way, no matter which operating system the audio application is called, the corresponding audio support terminal can be selected for the audio application as needed to achieve the output of the audio signal to a specific audio support terminal, and avoid hardware conflicts of the audio support terminal. The disclosed embodiment also supports the setting of attributes of different audio streams to complete the output priority management of multi-domain and multi-audio streams; by centrally managing the audio support terminal, unified management of audio resources is achieved, waste of audio resources is avoided, and audio resource utilization is improved. Through resource sharing and the audio stream processing capabilities built into the virtual audio backend, unnecessary hardware resources are reduced and hardware support costs are saved. The disclosed embodiment can implement different audio support schemes for the same hardware resources, improving the flexibility of hardware adaptation.

[0045] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0047] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0048] Figure 1 The following is a schematic diagram showing the implementation process of the audio data processing method of the present disclosure. Figure 1 ;

[0049] Figure 2 The following is a schematic diagram showing the implementation process of the audio data processing method of the present disclosure. Figure 2 ;

[0050] Figure 3 A schematic diagram of the audio data processing hardware structure in the SoC of the embodiment of the present disclosure is shown;

[0051] Figure 4 A schematic diagram showing the structure of dynamic audio data routing in a SoC according to an embodiment of the present disclosure is shown;

[0052] Figure 5 A schematic diagram showing the structure of an audio data processing device according to an embodiment of the present disclosure is shown;

[0053] Figure 6A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0054] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0055] Figure 1 The following is a schematic diagram showing the implementation process of the audio data processing method of the present disclosure. Figure 1 ,like Figure 1 As shown, the audio data processing method of the embodiment of the present disclosure is applied to SoC, which includes at least two or more processor cores, and the two or more processor cores are connected through an inter-core communication channel; SoC refers to the integration of multiple hardware components on a single chip, such as a central processing unit (CPU), a memory controller, a graphics processing unit (GPU), etc., to realize a complete computing system. The SOC may contain a die, or multiple identical or different wafers. Inter-core communication refers to the communication between different hardware domains inside the SOC, including asynchronous message passing through a mailbox mechanism, and data sharing across hardware domains through a shared memory mechanism. In the embodiment of the present disclosure, in more than two processor cores, at least a first audio support terminal and a second audio support terminal are provided, the first audio support terminal can provide an access interface to multiple audio applications, and the second audio support terminal is connected to multiple audio interfaces. As an example, the first audio support terminal is multiple, which is a processing front end of the audio signal, which mainly parses it into an audio signal control instruction for user operations, or directly receives control instructions for audio or audio and video files. The second audio support terminal mainly parses the audio file into an audio output signal according to its attributes or encoding method, and plays it through a set audio playback interface.

[0056] Taking a car as an example, the audio scene in the car often needs to plan multiple audio resources to meet the requirements of multiple audio zones, and the multiple audio zones need to support different audio zones through multiple second audio support terminals. Each audio zone is responsible for parsing the audio file by one or more second audio support terminals, and playing the audio signal through the corresponding audio output interface. In the embodiment of the present disclosure, all second audio support terminals in the audio zone are connected to the corresponding hardware audio resources.

[0057] like Figure 1 As shown, the audio data processing method of the embodiment of the present disclosure includes the following processing steps:

[0058] Step 101, in response to the first audio application being operated, triggering the first audio support end to parse the operation into an audio control instruction, and sending the audio control instruction to the second audio support end through an inter-core communication channel.

[0059] In the embodiment of the present disclosure, multiple audio applications are installed in the SoC system, and the multiple audio applications can be installed in different operating systems, or multiple audio applications are installed in the same operating system. These audio applications include but are not limited to multimedia playback applications such as audio playback applications, video playback applications, telephone ringtone and prompt tone support applications, system prompt tone support applications, etc.

[0060] The first audio application can be an audio application installed in the main operating system for the vehicle driver to operate, or it can be an audio application installed in the slave operating system, such as an audio application for passengers in the vehicle, such as the co-pilot or rear passengers. The operation of the first audio application can be initiated by the vehicle driver. For example, the vehicle driver can use the first audio application in the main operating system to select the audio output interface set in the rear passenger seat for the passenger through the second audio support terminal to output the audio or video file, so that the rear passengers can more conveniently listen to the played sound signal. Similarly, the co-pilot or rear passengers can also operate the first audio application installed in the slave operating system, and select the audio output interface set in the driver's seat for the user through the second audio support terminal to play the corresponding audio or video file, so that the co-pilot or rear passengers can more conveniently listen to the played sound signal. In this way, the driver and passengers in the vehicle can determine the output of the corresponding audio signal through the designated audio output interface determined by the second audio support terminal by operating the corresponding audio application.

[0061] In the disclosed embodiment, it is necessary to determine to which second audio support terminal the audio control instruction for the operation is to be output based on the operating system on which the first audio application is installed, and to notify the designated second audio support terminal to call the corresponding video file according to the audio control instruction and output the audio signal.

[0062] When the first audio application is operated, the first audio support terminal is triggered to parse the operation into an audio control instruction. The operation here can be a call to it by other applications, or a related operation based on the user's input device input, etc. The operation of the first audio application is parsed into an audio control instruction by the first audio support terminal of the corresponding operating system, and the audio control instruction is sent to the second audio support terminal through the inter-core communication channel. Here, the second audio support terminal is the output port of the audio signal, and the first audio support terminal is the processing front end that parses it into a control instruction for the operation.

[0063] In the disclosed embodiment, as an implementation method, the audio control instruction is sent to the second audio support terminal through a remote processor messaging (RPMsg) communication protocol mechanism, such as through a mailbox mechanism.

[0064] Step 102, triggering the second audio support terminal to parse the audio control instruction, obtain at least the configuration parameters of the first audio data, obtain the first audio data based on the configuration parameters, and determine at least one interface as the output interface for the first playback data among multiple audio interfaces.

[0065] In the embodiment of the present disclosure, after receiving the audio control instruction, the second audio support terminal parses the audio control instruction and obtains the configuration parameters of the first audio data, where the configuration parameters include at least one of the number of channels of the audio data, sampling parameters, noise parameters, encoding parameters, bit rate, compression parameters, volume, etc. In the embodiment of the present disclosure, the configuration parameters also include the storage address information of the first audio data, that is, the second audio support terminal reads and obtains the first audio data based on the audio control instruction, and determines its playback mode according to the relevant configuration parameters of the first audio data.

[0066] Step 103: Based on the configuration parameters and the attribute information of the output interface, configure the output parameters of the first audio data to obtain first playback data.

[0067] In the disclosed embodiment, the second audio support terminal configures the output parameters of the first audio data based on the configuration parameters of the first audio data, such as sampling the first audio data according to its sampling rate, and splitting it into channel-related parameters according to its audio channel number, and then decoding the first audio data according to the first audio data encoding parameters, bit rate, compression parameters, etc., determining its playback bit rate, and decompressing it according to its compression parameters to obtain the first playback data. And according to the attribute information of the first playback data, a playback interface is determined for the first playback data. The attribute information of the first playback data includes at least one of the encoding and decoding method, bit rate, and designated interface identifier.

[0068] Step 104: in response to the first playback data satisfying the playback condition, play the first playback data through the selected output interface.

[0069] In the disclosed embodiments, the playback data of different output interfaces need to be played according to the priority of the playback data, etc. The playback priority includes parameters such as the real-time playback of the audio signal and the playback delay requirement. As an example, the output priority of the first playback data is determined based on the attribute information of the first playback data. When the second audio support terminal determines that the priority order of the first playback data is the highest priority, the audio stream is output through the selected output interface. That is, when the first playback data is currently required to be played in the first order, the first playback data is played through the selected output interface, or, when there is no other audio playback data except the first playback data, the first playback data is played through the selected output interface.

[0070] Figure 2 The following is a schematic diagram showing the implementation process of the audio data processing method of the present disclosure. Figure 2 ,like Figure 2 As shown, the audio data processing method of the embodiment of the present disclosure is applied to a SoC, the SoC includes at least two or more processor cores, and the two or more processor cores are connected through an inter-core communication channel; the method includes the following processing steps:

[0071] Step 201, in response to a homogeneous system or a heterogeneous system being loaded on more than two processor cores, a virtual audio device node is created for each system in the homogeneous system or the heterogeneous system, and the virtual audio device node is used as the first audio support end of each system for access and control by an audio application of each system.

[0072] In the embodiments of the present disclosure, a multi-core heterogeneous system refers to a SoC that integrates multiple processing cores, and among these processing cores, at least two processing cores are different in structure, function or performance. A homogeneous system refers to a system in which the database type in each processing core in the SoC is the same, that is, each processing core supports the same data model, access method, optimization strategy, concurrency control algorithm, etc. The virtual audio device node of the embodiments of the present disclosure is equivalent to the driver of the virtual audio processor, providing a connection interface for audio applications in various operating systems installed in homogeneous systems or heterogeneous systems, and providing various supports such as installation for various audio applications.

[0073] Step 202, obtain relevant configuration parameters of the audio stream supported by the virtual audio device node, and trigger the first audio support end to send the relevant configuration parameters to the second audio support end through the inter-core communication channel.

[0074] In the disclosed embodiment, the relevant configuration parameters include at least one of the following: number of channels, sampling rate, bit width, address of shared memory and single transmission length. The relevant configuration parameters here are related to the configuration parameters of the first audio data, but the meanings are different. The relevant configuration parameters of the audio stream supported by the virtual audio device node mainly refer to the audio format that the virtual audio device node can support, its processing capability for audio data and other parameters, so as to ensure that the virtual audio device node can adapt to the operating system, and the audio application is adapted to the virtual audio device node.

[0075] The relevant configuration parameters are sent to the second audio support terminal through the RPMsg communication protocol mechanism. As an example, they can be sent through the mailbox mechanism.

[0076] Step 203, in response to the first audio application being operated, triggering the first audio support end to parse the operation into an audio control instruction, and sending the audio control instruction to the second audio support end through the inter-core communication channel.

[0077] In the embodiment of the present disclosure, multiple audio applications are installed in the SoC system, and the multiple audio applications can be installed in different operating systems, or multiple audio applications are installed in the same operating system. These audio applications include but are not limited to multimedia playback applications such as audio playback applications, video playback applications, telephone ringtone and prompt tone support applications, system prompt tone support applications, etc.

[0078] The first audio application can be an audio application installed in the main operating system for the vehicle driver to operate, or it can be an audio application installed in the slave operating system, such as an audio application for passengers in the vehicle, such as the co-pilot or rear passengers. The operation of the first audio application can be initiated by the vehicle driver. For example, the vehicle driver can use the first audio application in the main operating system to select the audio output interface set in the rear passenger seat for the passenger through the second audio support terminal to output the audio or video file, so that the rear passengers can more conveniently listen to the played sound signal. Similarly, the co-pilot or rear passengers can also operate the first audio application installed in the slave operating system, and select the audio output interface set in the driver's seat for the user through the second audio support terminal to play the corresponding audio or video file, so that the co-pilot or rear passengers can more conveniently listen to the played sound signal. In this way, the driver and passengers in the vehicle can determine the output of the corresponding audio signal through the designated audio output interface determined by the second audio support terminal by operating the corresponding audio application.

[0079] In the disclosed embodiment, it is necessary to determine to which second audio support terminal the audio control instruction for the operation is to be output based on the operating system on which the first audio application is installed, and to notify the designated second audio support terminal to call the corresponding video file according to the audio control instruction and output the audio signal.

[0080] When the first audio application is operated, the first audio support terminal is triggered to parse the operation into an audio control instruction. The operation here can be a call to it by other applications, or a related operation based on the user's input device input, etc. The operation of the first audio application is parsed into an audio control instruction by the first audio support terminal of the corresponding operating system, and the audio control instruction is sent to the second audio support terminal through the inter-core communication channel. Here, the second audio support terminal is the output port of the audio signal, and the first audio support terminal is the processing front end that parses it into a control instruction for the operation.

[0081] In the embodiment of the present disclosure, as an implementation method, the audio control instruction is sent to the second audio support terminal through the RPMsg communication protocol mechanism, such as through a mailbox mechanism.

[0082] Step 204, triggering the second audio support terminal to parse the audio control instruction, obtain at least the configuration parameters of the first audio data, obtain the first audio data based on the configuration parameters, and determine at least one interface as the output interface for the first playback data among multiple audio interfaces.

[0083] In the embodiment of the present disclosure, after receiving the audio control instruction, the second audio support terminal parses the audio control instruction and obtains the configuration parameters of the first audio data, where the configuration parameters include at least one of the number of channels of the audio data, sampling parameters, noise parameters, encoding parameters, bit rate, compression parameters, volume, etc. In the embodiment of the present disclosure, the configuration parameters also include the storage address information of the first audio data, that is, the second audio support terminal reads and obtains the first audio data based on the audio control instruction, and determines its playback mode according to the relevant configuration parameters of the first audio data.

[0084] Step 205, triggering the second audio support terminal to obtain the first audio data in the shared memory, and splitting the first audio data into corresponding voice endpoints in units of dual-channel data volume.

[0085] In the embodiment of the present disclosure, a shared storage domain is set between more than two processor cores in the SoC, and audio data is stored in the shared storage domain; the first audio support end and the second audio support end of the embodiment of the present disclosure are both connected to the shared storage domain, and access the audio data through the shared storage domain.

[0086] Step 206, triggering the second audio support end to perform at least one of resampling, mixing, and splicing on the audio data in the voice endpoint to obtain an output format supported by the output interface as the first playback data.

[0087] In the disclosed embodiment, the second audio support terminal configures the output parameters of the first audio data based on the configuration parameters of the first audio data. Specifically, the first audio data can be sampled according to its sampling rate, and divided into channel-related parameters according to its audio channel number, and then the first audio data is decoded and its playback bit rate is determined according to the first audio data encoding parameters, bit rate, compression parameters, etc., and the first audio data is decompressed according to its compression parameters to obtain the first playback data. And based on the attribute information of the first playback data, a playback interface is determined for the first playback data. The attribute information of the first playback data includes at least one of the encoding and decoding method, bit rate, and designated interface identifier.

[0088] Step 207: In response to the first playback data satisfying the playback condition, the first playback data is played through the selected output interface.

[0089] In the disclosed embodiments, the playback data of different output interfaces need to be played according to the priority of the playback data, etc. The playback priority includes parameters such as the real-time playback of the audio signal and the playback delay requirement. As an example, the output priority of the first playback data is determined based on the attribute information of the first playback data. When the second audio support terminal determines that the priority order of the first playback data is the highest priority, the audio stream is output through the selected output interface. That is, when the first playback data is currently required to be played in the first order, the first playback data is played through the selected output interface, or, when there is no other audio playback data except the first playback data, the first playback data is played through the selected output interface.

[0090] As an example, the selected output interface may be an Inter-IC Sound (I2S) interface.

[0091] The technical solution of the disclosed embodiment is to set up multiple audio support terminals, which are electrically connected to the same audio hardware and controlled by the same audio hardware. In this way, no matter which operating system the audio application is called, the corresponding audio support terminal can be selected for the audio application as needed to achieve the output of the audio signal to a specific audio support terminal, and avoid hardware conflicts of the audio support terminal. The disclosed embodiment also supports the setting of properties of different audio streams to complete the output priority management of multi-domain and multi-audio streams; by centrally managing the audio support terminal, unified management of audio resources is achieved, waste of audio resources is avoided, and audio resource utilization is improved. Through resource sharing and the built-in audio stream processing capabilities of the virtual audio backend, unnecessary hardware resources are reduced and hardware support costs are saved. The disclosed embodiment can implement different audio support schemes for the same hardware resources, thereby improving the flexibility of hardware adaptation.

[0092] The essence of the technical solution of the embodiment of the present disclosure is further illustrated by examples below.

[0093] Existing technologies are often easily limited by hardware resources when dealing with audio routing issues in automotive multi-domain systems, especially by the limitation on the number of I2S bus interfaces. Each domain needs to have exclusive use of one or more hardware I2S interfaces, which not only wastes hardware resources, but also limits the flexibility of using the I2S bus interface, and even affects the support of the output audio file format. When planning audio routing inside the audio digital signal processing chip (Digital Signal Processing, DSP), it is necessary to consider the problem of routing conflicts, which will increase the complexity of system design. In particular, when new domains and audio data streams need to be added, the hardware system needs to be modified and adjusted, which is not conducive to the expansion of the system and greatly increases the modification time and modification cost.

[0094] The technical solution of the disclosed embodiment utilizes the inter-domain communication and shared memory mechanism of SoC to realize the dynamic configurable mapping between virtual audio streams and hardware audio resources, and supports the centralized processing of audio data from different domains. It satisfies the realization of different audio solutions under the same set of hardware resources, and reduces the complexity of hardware adaptation and system design.

[0095] Figure 3 FIG. 4 shows a schematic diagram of the hardware structure of audio data processing in the SoC of the present disclosure. Figure 3 As shown, in the SoC of the embodiment of the present disclosure, operating systems such as Android and LINUX are installed, where LINUX can also be QNX, etc. By installing relevant applications in the processing core, the bottom layer of the operating system is divided into two parts: a virtual audio front end and a virtual audio back end that support audio data processing. The virtual audio front end and the virtual audio back end transmit audio control commands and configuration parameters through inter-domain communication, and use shared memory to share audio data and transmit between processing cores.

[0096] like Figure 3 As shown in the figure, the virtual audio front end exists in each application domain and is used to create virtual audio device nodes for audio applications to access and control. The application domains supported by the virtual audio front end include media audio, navigation audio, phone audio, prompt audio, sub-screen audio, alarm audio, etc. The virtual audio front end can also send the relevant configuration parameters of the audio stream, such as the number of channels, sampling rate, bit width, shared memory address, and single transmission length, to the virtual audio back end through RPMsg inter-domain communication. Each application domain corresponds to a virtual I2S node to realize the processing of audio application related parameters.

[0097] The virtual audio backend exists in the domain that controls the hardware I2S resources. It processes the corresponding audio data stream according to the configuration parameters received from each frontend, and outputs the corresponding data of the audio application in each operating system to the set output interface through dynamic audio routing, and plays it in the form of audio signals. The virtual audio backend obtains the audio data to be played from the shared memory between the processing cores, splits it into two-channel data as a basic unit, and caches it in the local memory unit (endpoint) respectively. Taking a four-channel audio data stream as an example, the backend copies it from the shared memory and splits it into two endpoints, each of which caches two channel data.

[0098] The virtual audio backend determines the output priority of different audio streams according to the attributes of different audio streams, such as media sound, phone call, system prompt sound, etc., and schedules the output of audio streams one by one according to the priority order of different audio streams. High-priority audio streams can obtain output permissions first, while low-priority audio streams may need to wait for the high-priority audio stream to finish playing before they can be played, thereby completing the output priority management of multi-domain multi-audio streams. In the disclosed embodiment, through limited hardware I2S interface resources, directional playback of different audio files in multiple operating systems is achieved, and the playback interface of audio signals can be flexibly deployed.

[0099] Figure 4 FIG. 4 shows a schematic diagram of a structure of dynamic audio data routing in a SoC according to an embodiment of the present disclosure. Figure 4 As shown, the virtual audio backend of the embodiment of the present disclosure resamples, mixes, splices, etc. the data in the endpoint into a data format supported by the hardware I2S output. During this processing, the processed data of different endpoints can be spliced ​​to the specified position of the output format data supported by the hardware I2S according to the software configuration, thereby realizing dynamically adjustable audio routing.

[0100] The disclosed embodiment sets up multiple audio support terminals, which are electrically connected to the same audio hardware and are controlled by the same audio hardware. In this way, no matter which operating system the audio application is called, the corresponding audio support terminal can be selected for the audio application as needed to achieve the output of the audio signal to a specific audio support terminal, and avoid hardware conflicts of the audio support terminal. The disclosed embodiment also supports setting the attributes of different audio streams to complete the output priority management of multi-domain and multi-audio streams; by centrally managing the audio support terminal, unified management of audio resources is achieved, waste of audio resources is avoided, and audio resource utilization is improved. Through resource sharing and the built-in audio stream processing capabilities of the virtual audio backend, unnecessary hardware resources are reduced and hardware support costs are saved. The disclosed embodiment can implement different audio support schemes for the same hardware resources, thereby improving the flexibility of hardware adaptation.

[0101] Figure 5 The schematic diagram of the structure of the audio data processing device according to the embodiment of the present disclosure is shown as follows: Figure 5 As shown, the audio data processing device of the embodiment of the present disclosure is applied to a SoC, and the SoC includes at least two or more processor cores, and the two or more processor cores are connected through an inter-core communication channel; in the two or more processor cores, at least a first audio support end and a second audio support end are provided, the first audio support end can provide an access interface to multiple audio applications, and the second audio support end is connected to multiple audio interfaces.

[0102] like Figure 5 As shown, the audio data processing device of the embodiment of the present disclosure includes:

[0103] A first trigger unit 50 is used for triggering the first audio support end to parse the operation into an audio control instruction in response to the first audio application being operated, and to send the audio control instruction to the second audio support end through the inter-core communication channel;

[0104] A second trigger unit 51 is used to trigger the second audio support end to parse the audio control instruction, obtain at least the configuration parameters of the first audio data, obtain the first audio data based on the configuration parameters, and determine at least one interface as an output interface for the first playback data among multiple audio interfaces;

[0105] A configuration unit 52, configured to configure output parameters of the first audio data based on the configuration parameters and the attribute information of the output interface to obtain first playback data;

[0106] The playing unit 53 is configured to play the first playing data through the selected output interface in response to the first playing data satisfying the playing condition.

[0107] exist Figure 5 Based on the audio data processing device shown in the figure, the audio data processing device of the embodiment of the present disclosure further includes:

[0108] Create a cell ( Figure 5 ), in response to a homogeneous system or a heterogeneous system being loaded on the two or more processor cores, creating a virtual audio device node for each system in the homogeneous system or the heterogeneous system, and using the virtual audio device node as the first audio support end of each system for access and control by the audio application of each system.

[0109] exist Figure 5 Based on the audio data processing device shown in the figure, the audio data processing device of the embodiment of the present disclosure further includes:

[0110] The third trigger unit ( Figure 5 (not shown), used to obtain relevant configuration parameters of the audio stream supported by the virtual audio device node, triggering the first audio support end to send the relevant configuration parameters to the second audio support end through the inter-core communication channel;

[0111] The relevant configuration parameters include at least one of the following: number of channels, sampling rate, bit width, address of shared memory and single transmission length.

[0112] In some possible implementation modes, the configuration unit 52 is further configured to:

[0113] Triggering the second audio support end to obtain the first audio data in the shared memory, and splitting the first audio data into corresponding voice endpoints in units of dual-channel data volume;

[0114] The second audio support end is triggered to perform at least one of resampling, mixing, and splicing of the audio data in the voice endpoint to obtain an output format supported by the output interface as the first playback data.

[0115] In some possible implementation modes, the playback unit 53 is further configured to:

[0116] The output priority of the first playback data is determined based on the attribute information of the first playback data. When the priority order of the first playback data is the highest priority, the audio stream is output through the selected output interface.

[0117] In some possible implementations, a shared storage domain is provided between the two or more processor cores, and the audio data is stored in the shared storage domain;

[0118] The first audio support end and the second audio support end are both connected to the shared storage domain and access audio data through the shared storage domain.

[0119] In an exemplary embodiment, the first trigger unit 50, the second trigger unit 51, the configuration unit 52, the playback unit 53, the creation unit, the third trigger unit, etc. can be implemented by one or more central processing units (CPU, Central Processing Unit), graphics processing units (GPU, Graphics Processing Unit), application specific integrated circuits (ASIC, Application Specific Integrated Circuit), DSP, programmable logic devices (PLD, Programmable Logic Device), complex programmable logic devices (CPLD, Complex Programmable Logic Device), field programmable gate arrays (FPGA, Field-Programmable Gate Array), general processors, controllers, microcontrollers (MCU, Micro Controller Unit), microprocessors (Microprocessors), or other electronic components.

[0120] Regarding the device in the above embodiment, the specific manner in which each module and unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0121] According to an embodiment of the present disclosure, the present disclosure also records an electronic device and a readable storage medium.

[0122] Figure 6 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0123] like Figure 6As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0124] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0125] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the audio data processing method. For example, in some embodiments, the audio data processing method of the present disclosure embodiment may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the audio data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the steps of the audio data processing method of the aforementioned embodiment by any other appropriate means (e.g., by means of firmware).

[0126] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0128] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0133] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0134] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An audio data processing method, applied to a system on chip (SoC), wherein the SoC comprises at least two processor cores, and the two processor cores are connected via an inter-core communication channel; characterized in that: In the two or more processor cores, at least a first audio support terminal and a second audio support terminal are provided, the first audio support terminal can provide an access interface to multiple audio applications, and the second audio support terminal is connected to multiple audio interfaces; the method comprises: In response to the first audio application being operated, triggering the first audio support end to parse the operation into an audio control instruction, and sending the audio control instruction to the second audio support end through the inter-core communication channel; Trigger the second audio support end to parse the audio control instruction, obtain at least configuration parameters of the first audio data, obtain the first audio data based on the configuration parameters, and determine at least one interface as an output interface for the first playback data among multiple audio interfaces; Based on the configuration parameters and the attribute information of the output interface, configure the output parameters of the first audio data to obtain first playback data; In response to the first playback data satisfying a playback condition, the first playback data is played through the selected output interface.

2. The method according to claim 1, characterized in that: The method further comprises: In response to a homogeneous system or a heterogeneous system being loaded on the two or more processor cores, a virtual audio device node is created for each system in the homogeneous system or the heterogeneous system, and the virtual audio device node is used as the first audio support end of each system for access and control by an audio application of each system.

3. The method according to claim 2, characterized in that The method further comprises: Obtain relevant configuration parameters of the audio stream supported by the virtual audio device node, and trigger the first audio support end to send the relevant configuration parameters to the second audio support end through the inter-core communication channel; The relevant configuration parameters include at least one of the following: number of channels, sampling rate, bit width, address of shared memory and single transmission length.

4. The method according to claim 1, characterized in that: The configuring output parameters of the first audio data based on the configuration parameters and the attribute information of the output interface to obtain first playback data includes: Triggering the second audio support end to obtain the first audio data in the shared memory, and splitting the first audio data into corresponding voice endpoints in units of dual-channel data volume; The second audio support end is triggered to perform at least one of resampling, mixing, and splicing on the audio data in the voice endpoint to obtain an output format supported by the output interface as the first playback data.

5. The method according to claim 1, characterized in that In response to the first playback data satisfying the playback condition, playing the first playback data through the selected output interface includes: The output priority of the first playback data is determined based on the attribute information of the first playback data. When the priority order of the first playback data is the highest priority, the audio stream is output through the selected output interface.

6. The method according to any one of claims 1 to 5, characterized in that: A shared storage domain is provided between the two or more processor cores, and the audio data is stored in the shared storage domain; The first audio support end and the second audio support end are both connected to the shared storage domain and access audio data through the shared storage domain.

7. An audio data processing device, applied to a system on chip (SoC), wherein the SoC comprises at least two processor cores, and the two processor cores are connected via an inter-core communication channel; characterized in that: In the two or more processor cores, at least a first audio support terminal and a second audio support terminal are provided, the first audio support terminal can provide an access interface to multiple audio applications, and the second audio support terminal is connected to multiple audio interfaces; the device comprises: A first trigger unit, configured to trigger the first audio support end to parse the operation into an audio control instruction in response to the first audio application being operated, and to send the audio control instruction to the second audio support end through an inter-core communication channel; A second trigger unit is used to trigger the second audio support end to parse the audio control instruction, obtain at least the configuration parameters of the first audio data, obtain the first audio data based on the configuration parameters, and determine at least one interface as an output interface for the first playback data among multiple audio interfaces; A configuration unit, configured to configure output parameters of the first audio data based on the configuration parameters and the attribute information of the output interface to obtain first playback data; The playing unit is configured to play the first playing data through the selected output interface in response to the first playing data satisfying a playing condition.

8. The device according to claim 7, characterized in that The device also includes: A creation unit is used to create a virtual audio device node for each system in the homogeneous system or the heterogeneous system in response to the homogeneous system or the heterogeneous system being loaded on the two or more processor cores, and use the virtual audio device node as the first audio support end of each system for access and control by the audio application of each system.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the audio data processing method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the steps of the audio data processing method according to any one of claims 1 to 6.

Images