Audio processing method, related device and storage medium

Abstract

The application discloses an audio processing method, related equipment and a storage medium, wherein the method comprises: obtaining, by a management domain in a multi-core heterogeneous system, audio data of M application domains in the multi-core heterogeneous system by using an inter-core communication mechanism of the multi-core heterogeneous system; the multi-core heterogeneous system comprises multiple hardware domains; each hardware domain in the multiple hardware domains is formed by hardware resources connected to multiple processor cores in the multi-core heterogeneous system; each hardware domain is isolated from each other; M application domains and the management domain correspond to at least part of the hardware domains in the multiple hardware domains; distributing each audio data to N first audio nodes of the management domain; each first audio node comprises at least part of channel data streams of at least one audio data; M and N are positive integers greater than or equal to 1; and performing audio processing on the channel data streams distributed to each first audio node to obtain target audio, which is used for playing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an audio processing method, apparatus, chip, electronic device and computer storage medium. Background Technology

[0002] In playback systems with two or more audio tracks to be played and a limited number of players (such as a single speaker), to avoid affecting the normal playback of each audio track at the same time, related technologies can address this by increasing hardware overhead. Further, in terms of hardware, audio processing hardware, such as a digital signal processing chip (DSP), is used to perform certain audio processing on these audio tracks (such as mixing, splicing, and / or sound effects processing) and then play them through a player. However, once the hardware interfaces are connected, the audio processing methods between the audio tracks cannot be easily changed, resulting in poor flexibility. Summary of the Invention

[0003] This application provides an audio processing method, apparatus, chip, electronic device, and computer storage medium to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this application, an audio processing method is provided, comprising:

[0005] An inter-core communication mechanism is employed in a multi-core heterogeneous system, whereby the management domain of the multi-core heterogeneous system obtains audio data from each of the M application domains within the system. The multi-core heterogeneous system comprises multiple hardware domains; each hardware domain consists of hardware resources connecting multiple processor cores to each processing core within the system, and these hardware domains are isolated from each other. The M application domains and the management domain correspond to at least a portion of the hardware domains within the multi-core heterogeneous system.

[0006] Each audio data is assigned to N first audio nodes in the management domain; wherein each first audio node includes at least a portion of the channel data stream of at least one audio data; M and N are both positive integers greater than or equal to 1;

[0007] The channel data streams allocated to each first audio node are processed to obtain target audio, which is used for playback.

[0008] In one possible implementation, each audio data stream includes at least one channel data stream;

[0009] The process of distributing each audio data point to the N first audio nodes of the management domain includes:

[0010] Each audio data stream, including all channels, is assigned to at least one of the N first audio nodes.

[0011] In one possible implementation, the step of performing audio processing on the channel data streams allocated to each first audio node to obtain target audio includes: at least one of the first audio nodes corresponds to an audio player.

[0012] Audio processing is performed on the channel data streams assigned to the first audio node corresponding to the same audio player;

[0013] Based on the audio processing results of the channel data streams of each first audio node corresponding to each audio player, the target audio for each audio data is obtained.

[0014] In one embodiment, the channel data stream of each audio data is sent from each of the M application domains by each of the first virtual audio driver units in each application domain; the management domain in the multi-core heterogeneous system includes a second virtual audio driver unit;

[0015] The step of obtaining audio data from each of the M application domains in the multi-core heterogeneous system from the management domain in the multi-core heterogeneous system includes:

[0016] The management domain receives all channel data streams of each audio data using the channel corresponding to each audio data through the second virtual audio driver unit, thereby obtaining audio data from M application domains.

[0017] In one possible implementation, the method further includes:

[0018] Based on audio recording requests from at least some of the M application domains, the recorded audio is allocated to a second audio node to send the recorded audio to the at least some application domains; wherein, the recorded audio is the audio requested to be recorded by the audio recording request;

[0019] The audio recording request is obtained based on the inter-core communication mechanism.

[0020] In one possible implementation, the audio processing of the channel data streams allocated to each first audio node to obtain the target audio includes:

[0021] The target audio is obtained by processing the channel data streams allocated to each first audio node and the recorded audio data streams of the second audio node.

[0022] In one possible implementation, each of the at least some application domains corresponds to recorded audio, the recorded audio including at least one recorded data stream;

[0023] The process of assigning the recorded audio to the second audio node includes:

[0024] Distribute the entire recorded data stream of each audio recording to at least one second audio node.

[0025] In one embodiment, the number of channels corresponding to each audio data may be the same as or different from the number of channel data streams included in each audio data.

[0026] According to a second aspect of this application, an audio processing apparatus is provided, comprising:

[0027] The acquisition unit is configured to obtain audio data from M application domains in a multi-core heterogeneous system via an inter-core communication mechanism of the multi-core heterogeneous system. The multi-core heterogeneous system comprises multiple hardware domains. Each hardware domain consists of hardware resources connecting multiple processor cores to each processing core in the multi-core heterogeneous system, and the hardware domains are isolated from each other. The M application domains and the management domain correspond to at least a portion of the hardware domains in the multi-core heterogeneous system.

[0028] The allocation unit is used to allocate each audio data to N first audio nodes of the management domain; wherein each first audio node includes at least a portion of the channel data stream of at least one audio data; M and N are both positive integers greater than or equal to 1;

[0029] The processing unit is used to perform audio processing on the channel data streams allocated to each first audio node to obtain target audio, which is used for playback.

[0030] According to a third aspect of this application, a chip is provided that includes the aforementioned audio processing device.

[0031] According to a fourth aspect of this application, an electronic device is provided, comprising:

[0032] At least one processor; and

[0033] A memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in this application.

[0035] According to a fifth aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this application.

[0036] The present application discloses an audio processing method, apparatus, chip, electronic device, and computer storage medium, wherein the method comprises: employing an inter-core communication mechanism of a multi-core heterogeneous system, obtaining audio data from M application domains in the multi-core heterogeneous system from a management domain in the multi-core heterogeneous system; allocating each audio data to N first audio nodes in the management domain; wherein each first audio node includes at least a portion of the channel data stream of at least one audio data; performing audio processing on the channel data stream allocated to each first audio node to obtain a target audio, the target audio being used for playback.

[0037] This management domain approach is a software-based solution, offering greater flexibility compared to related technologies that increase hardware overhead. It also avoids the cost increases associated with increased hardware requirements.

[0038] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0039] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which:

[0040] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0041] Figure 1 A schematic diagram of a multi-core heterogeneous system in an embodiment of this application is shown. Figure 1 ;

[0042] Figure 2 A schematic diagram of a multi-core heterogeneous system in an embodiment of this application is shown. Figure 2 ;

[0043] Figure 3 This document illustrates the implementation flow of the audio processing method in an embodiment of this application. Figure 1 ;

[0044] Figure 4 A schematic diagram illustrating an audio processing method based on a management domain in an embodiment of this application is shown;

[0045] Figure 5 The illustration shows a schematic diagram of the implementation of audio playback and audio recording in an embodiment of this application;

[0046] Figure 6 This document illustrates the implementation flow of the audio processing method in an embodiment of this application. Figure 2 ;

[0047] Figure 7 This document illustrates the implementation flow of the audio processing method in an embodiment of this application. Figure 3 ;

[0048] Figure 8 This illustration shows an example of an audio processing method implemented based on Android HAL in an embodiment of this application. Figure 1 ;

[0049] Figure 9 This illustration shows an example of an audio processing method implemented based on Android HAL in an embodiment of this application. Figure 2 ;

[0050] Figure 10 A schematic diagram of the composition structure of the audio processing device in an embodiment of this application is shown;

[0051] Figure 11 A schematic diagram of the composition structure of the electronic device in an embodiment of this application is shown. Detailed Implementation

[0052] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0055] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0057] It should be understood that in the various embodiments of this application, the sequence number of each implementation process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0058] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0059] The audio processing methods involved in this application include, but are not limited to, the following:

[0060] 1) Mixing is the process of integrating audio from two or more sources into a single stereo or mono track. During mixing, the frequency, dynamics, timbre, localization, reverberation, and soundstage of each audio signal are individually adjusted to optimize each track before being superimposed onto the final product. This method can produce a distinct audio output effect that is not audible to the average listener during live recording.

[0061] 2) Splicing refers to audio channel splicing, which involves redistributing one or more audio channels to another audio channel according to certain rules to create a completely new audio file to meet the needs of listeners or audio processing.

[0062] 3) Audio effects processing: This involves modifying parameters such as volume, timbre, and pitch, and / or removing human voices to achieve different audio output effects.

[0063] The audio processing logic of the method in this application is deployed in a multi-core heterogeneous system. In the embodiments of this application, the multi-core heterogeneous system includes a multi-core heterogeneous chip. A multi-core heterogeneous chip refers to a chip that integrates two or more processor cores within a single chip. For example, a single SOC (System-on-a-Chip) chip integrating two or more processor cores. Each processor core in a multi-core heterogeneous chip can act as an independent processor, independently executing the instructions required by each processor core and fulfilling the tasks required by each processor core. It can be understood that a multi-core heterogeneous chip is a chip with multiple processor cores. Compared to a single-core processor chip, the independent operation of each core's tasks can accelerate the running speed, improve multi-tasking capabilities, and thus bring the advantage of high performance. Furthermore, the multi-core processors being located on the same chip has the advantage of low cost.

[0064] like Figure 1 As shown, the multi-core heterogeneous chip includes multiple processor cores, namely a first processor core, a second processor core, ..., an Lth processor core. L is a positive integer greater than or equal to 2, and can be flexibly set according to actual needs. Each processor core is essentially a computing engine, and its type and / or number can differ. The types of processor cores include cores with high computing power and cores with high real-time performance (fast computation). In practical applications, the multiple processor cores can be of different types; thus, the multi-core heterogeneous chip is composed of multiple processor cores with different architectures. Differences in the type and / or number of processor cores can, to a certain extent, achieve architectural differences between the processor cores.

[0065] For example, since embedded processors (ARM) have the advantages of low cost and low power consumption, digital signal processors (DSP) have the advantages of dedicated digital processing, and programmable logic arrays (FPGAs) have the advantages of high-speed processing, each type of processor is used as a processor core and designed on the same SOC chip, thus obtaining a multi-core heterogeneous SOC chip.

[0066] Each processor core and the hardware resources connected to it, such as clock controllers, interrupt controllers, and memory space, constitute each hardware domain. That is, a multi-core heterogeneous chip includes multiple hardware domains. In a multi-core heterogeneous chip, each hardware domain is a collection of hardware resources. Different hardware domains are isolated from each other; this isolation can be considered a physical isolation. For example, hardware designs within the same hardware domain may be located close to each other on the multi-core heterogeneous chip, while hardware designs in different hardware domains may be located at different locations on the multi-core heterogeneous chip, achieving isolation through physical location. Of course, the mutual isolation between different hardware domains in this application embodiment may not be physical isolation, but logical isolation. This logical isolation can be reflected in the following: hardware resources within the same hardware domain need to use the same communication identifier to access each other within that hardware domain; that is, hardware resources within the same hardware domain can access each other based on the communication identifier within that hardware domain. Hardware resources in different hardware domains use different communication identifiers to access each other.

[0067] In practical applications, it is preferable to isolate different hardware domains as a form of logical isolation, which can at least save chip space.

[0068] In a multi-core heterogeneous chip system, an operating system is configured for each hardware domain. For example, a first operating system is configured for the first hardware domain, and a second operating system for the second hardware domain. The operating systems configured for different hardware domains can be the same or different, preferably different. For instance, the first operating system configured for the first hardware domain is Linux, and the second operating system configured for the second hardware domain is Android. Because Linux offers high security and Android is lightweight, tasks requiring high security in the multi-core heterogeneous system can be assigned to Linux, while tasks requiring fast execution can be assigned to Android. Therefore, different operating systems on different hardware domains can be used in a multi-core heterogeneous system to achieve efficient execution of various tasks.

[0069] To enable applications on multi-core heterogeneous systems, in this embodiment, the operating system corresponding to each hardware domain is considered as the operating system of each application domain. Through each application domain, applications on the multi-core heterogeneous system are implemented, such as those in audio and video output.

[0070] In some embodiments, to achieve better application, one or more application domains can be selected from the various application domains and used as the management domain. For example... Figure 2 As shown, one application domain is selected from each application domain as the management domain. If the management domain and application domain are used as a single domain, then the application domain in this application is the domain used for applications, and the management domain is the domain used for managing the application domain. Unlike related technologies that increase hardware overhead to achieve normal playback of various audio files simultaneously (hardware-level playback schemes), this application achieves normal playback of various audio files simultaneously based on the management of the management domain. This management can be considered a scheme that achieves normal playback of various audio files simultaneously through software-level management. Playback implemented at the software level offers excellent flexibility and can greatly avoid the inflexibility of hardware-level playback.

[0071] In this embodiment, communication needs also exist between hardware domains. When communication needs exist between different hardware domains, an inter-core communication mechanism can be used to achieve communication between hardware domains. Among these, inter-core communication mechanisms in multi-core heterogeneous systems include a mailbox mechanism suitable for instruction transmission and a memory sharing mechanism suitable for data sharing. Inter-core communication within a single SOC chip can ensure data transmission within the same chip, guaranteeing data security and transmission speed.

[0072] Typically, hardware resources differ across different hardware domains, and these differences may manifest in hardware type, model, quantity, and other aspects. This difference, to some extent, reflects the heterogeneity of multi-core heterogeneous systems. As discussed earlier, the multi-core heterogeneity in this application is a hardware-level concept and is unrelated to the software level.

[0073] like Figure 1 As shown, the multi-core heterogeneous chip in this embodiment also includes various types of control units. These control units include, but are not limited to, a power control unit, a non-volatile memory control unit, and a volatile memory control unit. The power control unit controls the power supply unit to provide power to the multi-core heterogeneous chip. The non-volatile memory control unit controls access to non-volatile memory units by at least one processor core in the multi-core heterogeneous chip. The volatile memory control unit controls access to volatile memory units by at least one processor core in the multi-core heterogeneous chip.

[0074] Among them, the power supply unit, non-volatile memory unit, and volatile memory unit, as hardware resources outside the multi-core heterogeneous chip, can be called upon when needed by the multi-core heterogeneous chip. In addition, hardware such as audio output units (such as speakers), audio acquisition units (such as microphones), and video output units (such as displays), as hardware resources outside the multi-core heterogeneous chip, can also be called upon when needed by the multi-core heterogeneous chip to achieve normal audio and video output.

[0075] The multi-core heterogeneous system in this application embodiment includes a multi-core heterogeneous chip and an operating system configured for different hardware domains. The audio processing method in this application embodiment is implemented on the multi-core heterogeneous system, and further, on the management domain of the multi-core heterogeneous system.

[0076] In this embodiment, a multi-core heterogeneous system can be applied to a driving device. The driving device includes at least one of private transportation vehicles and public transportation vehicles. Private transportation vehicles include, but are not limited to, self-balancing scooters, electric motorcycles, private cars, and private airplanes. Public transportation vehicles include, but are not limited to, buses, trains, subways, high-speed trains, and airplanes.

[0077] The following section provides a detailed description of the audio processing method, audio processing device, and other technical solutions of this application based on the multi-core heterogeneous system.

[0078] One embodiment of the audio processing method in this application is applied to a multi-core heterogeneous system, and further, to the management domain of the multi-core heterogeneous system. For example... Figure 3 As shown, the method includes:

[0079] S301: An inter-core communication mechanism of a multi-core heterogeneous system is adopted, wherein the management domain of the multi-core heterogeneous system obtains audio data from each of the M application domains in the multi-core heterogeneous system; wherein the multi-core heterogeneous system includes multiple hardware domains; each hardware domain in the multiple hardware domains is composed of hardware resources that connect multiple processor cores and each processing core in the multi-core heterogeneous system, and the hardware domains are isolated from each other; the M application domains and the management domain correspond to at least some of the hardware domains in the multiple hardware domains.

[0080] In this application, each application domain has one or more audio files (audio data) to be played. The management domain obtains the audio files to be played from each application domain by reading or receiving them. Each application domain may also have audio files that need to be recorded, and the management domain can send the recorded audio files to the application domains. Furthermore, the management domain uses the inter-core communication mechanism in a multi-core heterogeneous system to read or receive the audio files to be played from each application domain, and / or distribute the recorded audio files needed by each application domain to the application domains. In practical applications, the inter-core communication mechanism can be a mailbox mechanism suitable for instruction transmission. Communication between different domains (the management domain and each application domain) is achieved based on the mailbox mechanism. Since each domain is located within the same chip, inter-core communication within a single SOC chip can ensure the security and speed of data transmission within the same chip.

[0081] In this application, the processor cores of the application domain and the processor cores of the management domain may have the same or different architectures. The processor cores of different domains may also have the same or different architectures, depending on the specific circumstances.

[0082] S302: Distribute each audio data to N first audio nodes of the management domain; wherein each first audio node includes at least a portion of the data stream of at least one audio data; M and N are both positive integers greater than or equal to 1.

[0083] In this application, the management domain includes two or more audio nodes (Endpoints). In some extreme cases, the management domain may also include only one audio node. Considering that this application includes both playing audio from the application domain and recording audio, it is preferable that the management domain includes multiple audio nodes. For the multiple audio nodes in the management domain, some nodes are selected as nodes used in the audio playback scenario of the application domain (first audio nodes). Some nodes are selected as nodes used in the audio recording scenario (second audio nodes). Of course, the same audio node can be used as either the first audio node or the second audio node, depending on the specific circumstances.

[0084] It is understood that in the embodiments of this application, the audio data of the application domain can be mono data, in which case the audio data includes a mono data stream. The audio data of the application domain can be stereo data, in which case the audio data includes two channel data streams. The audio data of the application domain can be three-channel, four-channel, or more-channel data, in which case the audio data includes three-channel, four-channel, or more-channel data streams. Thus, each audio data in this application includes one, two, or more channel data streams.

[0085] For audio data from M application domains, where each audio data includes at least one channel data stream, the aforementioned scheme of allocating each audio data to N first audio nodes in the management domain can be implemented as follows: allocating all channel data streams included in each audio data to at least one of the N first audio nodes.

[0086] For each audio data stream, the entire data stream can be allocated to different audio nodes. Alternatively, a portion of the entire data stream can be allocated to the same audio node, while another portion can be allocated to different audio nodes. The entire data stream can also be allocated to the same audio node. This provides allocation flexibility. This flexibility reflects, to some extent, the flexibility of playing audio data implemented in software.

[0087] From the perspective of a single audio node, it may store one, two, or more data streams of the same audio data, or it may store one, two, or more data streams of different audio data. In other words, an audio node stores data streams of audio data. Data streams assigned to the same audio node may be data streams of the same audio data or data streams of different audio data. In this case, each audio node can be used as a storage unit to store one or more audio data streams from one or at least two channels.

[0088] S303: Perform audio processing on the channel data streams allocated to each first audio node to obtain target audio, which is used for playback.

[0089] In this step, audio processing includes, but is not limited to, at least one of the aforementioned mixing, splicing, and sound effects processing, demonstrating flexibility in audio processing. This flexibility in audio processing can be reflected in the flexibility of playing various audio data implemented in software.

[0090] If the audio processing is mixing, the target audio is the audio obtained by mixing audio data from M application domains. If the audio processing is splicing, the target audio is the audio obtained by splicing audio data from M application domains. If the audio processing is sound effects processing, the target audio is the audio obtained by applying sound effects to audio data from M application domains.

[0091] The execution entities of S301 to S303 are the management domains in a multi-core heterogeneous system. Based on the inter-core communication mechanism, the management domain distributes audio data obtained from each application domain to each audio node, and performs audio processing on the channel data streams distributed to each first audio node to obtain the target audio for each audio data. Based on the management domain's management of the audio data from each application domain, normal playback of each audio file to be played at the same time is achieved. This management domain management scheme is a software-based management scheme, offering flexibility in allocation and audio processing. Compared with related technologies that increase hardware overhead, it has strong flexibility and avoids the cost increase problem caused by increased hardware overhead.

[0092] In some embodiments, the channel data stream of each audio data is sent from each application domain by each first virtual audio driver unit in each of the M application domains; the management domain in the multi-core heterogeneous system includes a second virtual audio driver unit; wherein, obtaining the audio data of each of the M application domains in the multi-core heterogeneous system by the management domain in the multi-core heterogeneous system includes: the management domain receiving the full channel data stream of each audio data through the second virtual audio driver unit using the channel corresponding to each audio data, thereby obtaining the audio data of each of the M application domains.

[0093] Combination Figure 4 As shown, taking a multi-core heterogeneous chip as an example of a System-on-a-Chip (SoC), it includes three application domains and one management domain (virtual device management domain). The three application domains and one virtual device management domain correspond to different processor cores. For example, application domain 1 corresponds to processor core CPU (Central Processing Unit)-1, application domain 2 corresponds to processor core CPU-2, application domain 3 corresponds to processor core CPU-3, and the management domain corresponds to processor core CPU-4.

[0094] In this application, virtual audio driver units are installed in both the application domain and the management domain. These virtual driver units are installed to facilitate communication between different domains using inter-core communication mechanisms. For distinction, the virtual audio driver unit installed in the application domain is used as the first virtual audio driver unit or the front-end of the virtual audio driver (unit). The virtual audio driver unit installed in the management domain is used as the second virtual audio driver unit or the back-end of the virtual audio driver (unit).

[0095] It is understood that audio data in the application domain includes a mono, stereo, or multi-channel data stream. Taking a stereo data stream from application domain 1 as an example, the stereo data stream from application domain 1 is sent to the management domain through the first virtual audio driver unit of the application domain. The management domain receives the stereo data stream through the backend of the virtual audio driver unit using two channels. For example, channel data stream 1 in the stereo data stream is received through the backend of the virtual audio driver unit using channel 1, and channel data stream 2 in the stereo data stream is received through the backend of the virtual audio driver unit using channel 2. Taking a tri-channel data stream from application domain 2 as an example, the tri-channel data stream from application domain 2 is sent to the management domain through the first virtual audio driver unit of the application domain. The management domain receives the tri-channel data stream through the backend of the virtual audio driver using three channels. For example, channel data stream 1 in the tri-channel data stream is received through the backend of the virtual audio driver using channel 1, and channel data stream 2 in the tri-channel data stream is received through the backend of the virtual audio driver using channel 2. Channel data stream 3 in the three-channel data stream is received through the back end of the virtual audio driver unit using channel 3.

[0096] In practical applications, the virtual audio driver backend can distinguish which audio data originates from which application domain by using the identifier of each application domain. For two or more audio data from the same application domain, the audio stream identifier can confirm which audio stream within the same application domain each audio stream belongs to. The number of channels corresponding to each audio data may be the same as or different from the number of channel data streams included in each audio data. For example, for a two-channel data stream from application domain 1, the virtual audio driver backend receives it using channel 1. Channel 1 is in two-channel data format. For a three-channel data stream from application domain 2, the virtual audio driver backend receives it using channel 2. Channel 2 is in three-channel data format. For a mono data stream from application domain 3, the virtual audio driver backend receives it using channel 3. Channel 3 is in mono data format.

[0097] Combination Figure 4As shown, the management domain includes a software audio hub, used at least to distribute the channel data streams of each audio data source to N first audio nodes, and to perform audio processing on the channel data streams distributed to each first audio node. For example, data stream 1 from the two-channel data stream of application domain 1 is distributed to audio node 1, and data stream 2 from the two-channel data stream of application domain 1 is distributed to audio node 2. Alternatively, data stream 1 from the three-channel data stream of application domain 2 is distributed to audio node 1, and data streams 2 and 3 from the three-channel data stream are distributed to audio node 2 (first scheme). Or, data stream 2 from the three-channel data stream is distributed to audio node 2, and data stream 3 from the three-channel data stream is distributed to audio node 3 (second scheme). The distribution method of the channel data streams can be implemented according to pre-set configuration information. The configuration information includes at least which audio node each channel data stream of each application domain can be distributed to. In implementation, the configuration information can be flexibly configured according to the requirements of the audio data output effect. The configuration can be either a pre-assigned static configuration or a dynamic configuration based on application requirements. This allows for flexible allocation, and consequently, flexible software-based control over audio playback.

[0098] In the static configuration method, the data stream of audio node 1 is pre-configured. For example, it is pre-configured that the data stream of audio node 1 is output through audio output device 1, and the data stream of audio node 2 is output through audio output device 2. Thus, in implementation, the data stream configured to audio node 1 needs to be played through audio output device 1, and the data stream configured to audio node 2 needs to be played through audio output device 2.

[0099] Dynamic configuration is a flexible configuration method that can be adjusted according to the actual application scenario or situation. For example, a data stream that was originally to be allocated to audio node 1 is allocated to audio node 2, so that the output from audio output device 1 corresponding to audio node 1 is switched to audio output device 2 corresponding to audio node 2. Alternatively, a data stream that was originally to be allocated to audio node 1 is allocated to both audio node 1 and audio node 2, so that both audio output device 1 corresponding to audio node 1 and audio output device 2 corresponding to audio node 2 output data.

[0100] For example, if two-channel audio data from application domain 1 and three-channel audio data from application domain 2 are mixed and output, the channel data streams are allocated to the audio nodes according to the first scheme described above. If two-channel audio data from application domain 1 and three-channel audio data from application domain 2 are mixed and output after sound effects processing, the channel data streams are allocated to the audio nodes according to the second scheme described above.

[0101] exist Figure 4In a multi-core heterogeneous system, it also includes a DMA (Direct Memory Access) controller, an I2S (InterIC Sound) device, a codec, and audio output units such as speakers, an SPDIF digital interface (Sony / Philips Digital Interface), and an SPDIF device.

[0102] After audio processing of the channel data streams of audio nodes such as audio node 1 and audio node 2, the data is transmitted in bit form via DMA or CPU data copy and then transmitted to the codec via an I2S device. The codec converts the digital signal to an analog signal, and the resulting analog signal is played through the speaker. Similarly, after audio processing of the channel data streams of audio nodes such as audio node 3, the data is transmitted in bit form via DMA and then transmitted to the SPDIF device via the SPDIF bus interface for playback.

[0103] In the aforementioned solution, virtual audio driver units and software audio hubs within the management domain enable various audio playback schemes at the software level. At the software level, data streams from each channel can be flexibly allocated to audio nodes based on pre-set configuration information, offering significant flexibility compared to related technologies.

[0104] In some embodiments, the scheme of performing audio processing on the channel data streams allocated to each first audio node to obtain target audio includes: at least one of the first audio nodes corresponds to an audio player; performing audio processing on each channel data stream allocated to the first audio node corresponding to the same audio player; and obtaining target audio for each audio data based on the audio processing results of the channel data streams of each first audio node corresponding to each audio player.

[0105] In the aforementioned scheme, the audio player is an audio output unit, such as a speaker, that can be called upon by a multi-core heterogeneous system. The number of speakers is one or at least two. Which speakers(s) output the data from each audio node after audio processing can be pre-configured (static configuration) or dynamically configured during audio system operation (dynamic configuration). This configuration is flexible and can be tailored to specific needs. For details on static and dynamic configuration methods, please refer to the aforementioned explanations, which will not be repeated here.

[0106] For example, the speaker corresponding to audio node 1 is configured as speaker 1, data stream 1 from the dual-channel data stream of application domain 1 is assigned to audio node 1, data stream 1 from the dual-channel data stream of application domain 2 is assigned to audio node 1, audio processing is performed on the two data streams in audio node 1, and the audio-processed sound is output through speaker 1.

[0107] The speaker corresponding to audio node 2 is configured as speaker 2. Data stream 2 from the dual-channel data stream of application domain 1 is allocated to audio node 2, and data stream 2 from the dual-channel data stream of application domain 2 is also allocated to audio node 2. Audio processing is performed on the two data streams in audio node 2, and the processed sound is output through speaker 2. After software-level audio processing, the audio data from application domain 1 and the audio data from application domain 2 are output together through speaker 1 and speaker 2. The audio output by speaker 1 and speaker 2 together can be regarded as the target audio. Thus, normal playback of each audio data at the same time is achieved.

[0108] Combination Figure 5 As shown, the software audio hub includes multiple audio nodes. Each audio node stores the channel data stream of the audio data allocated to it. Because the software audio hub resides in the management domain, the channel data stream of audio data from the application domain can be used as a remote data stream for the management domain, while the channel data stream of audio data that the management domain itself needs to play can be used as a local data stream for the management domain. Figure 5 As you can see, audio nodes 1 and 2 are used to store remote data streams, while audio node 3 is used to store local data streams. Figure 5 Whether the audio node shown is used to store remote data streams or local data streams is merely one implementation method; any other reasonable solution is covered within the scope of this application. Each data stream assigned to an audio node corresponding to the same audio player undergoes audio processing to obtain data to be played for that same audio player. The set of data to be played for all audio players can be considered the target audio. Specifically, while data streams from audio nodes corresponding to the same audio player can be output through that same audio player after audio processing, data streams from audio nodes corresponding to other audio players can be output through those other audio players after audio processing. This allows listeners to hear audio output effects such as mixing, splicing, or volume adjustments. This achieves normal output of audio data for each application domain.

[0109] It is understood that the audio playback processing flow in the application domain of this application is mainly implemented on the management domain side. Therefore, the management domain is equipped with a virtual audio driver backend and a software audio hub. Both the virtual audio driver backend and the software audio hub are software modules. The audio playback process using the software modules is as follows: Figure 6 As shown. Initialize the software audio hub and virtual audio driver unit backend. For the virtual audio driver unit backend, create an audio driver backend service to determine if audio data from the application domain has been received. If audio data from the application domain is received, the audio data is transmitted to the software audio hub, which allocates audio nodes to process the channel data streams within those nodes. For the software audio hub, configure playback parameters and create a playback stream. Playback parameters include, but are not limited to, the playback format, audio sampling rate, and / or audio length during processing. The created playback stream is initially empty; after processing the channel data streams of each first audio node corresponding to each audio player, the processing results are loaded onto it to obtain the target audio. The software audio hub determines if there is audio to play or if audio data has been received. If yes, the target audio is obtained according to the aforementioned scheme for playback by the audio players.

[0110] In some embodiments, the audio processing method further includes:

[0111] Based on audio recording requests from at least some of the M application domains, the recorded audio is allocated to a second audio node to send the recorded audio to the at least some application domains; wherein, the recorded audio is the audio requested to be recorded by the audio recording request.

[0112] In addition to the need to play audio, application domains also have the need to record audio. Application domains with recording needs transmit the audio recording request to the management domain via inter-core communication mechanisms such as a mailbox mechanism. Further, the audio recording request is transmitted to the backend of the management domain's virtual audio driver unit through the application domain's virtual audio driver unit, thus realizing the transmission of the audio recording request from the application domain to the management domain. In response to the audio recording request, an audio acquisition unit, such as a microphone, is triggered to record the required audio. The management domain reads the audio recorded by the microphone and allocates the acquired recorded audio to one or more second audio nodes, which are used to store the recorded audio. When needed, the recorded audio stored in the second audio nodes can be sent to the application domain's virtual audio driver unit via the backend of the virtual audio driver unit, thus realizing the transmission of recorded audio from the management domain to the application domain, facilitating the application domain's use of the recorded audio it needs.

[0113] Combination Figure 4 As shown, during implementation, the microphone collects the audio to be recorded. The collected audio is converted from analog to digital signal by the codec. The converted digital signal is transmitted to the DMA in bit form via the I2S device. The DMA then transmits the digital signal to the second audio node to be assigned.

[0114] As seen from the preceding solutions, the audio processing method of this application includes both audio playback and audio recording scenarios, making it suitable for practical use and highly applicable. In the audio recording scheme, the management-domain-based approach to audio recording is simple and easy to implement.

[0115] In practical applications, each of the at least some application domains corresponds to recorded audio, and the recorded audio includes at least one recorded data stream. In simpler terms, for an application domain with recording requirements, the audio to be recorded can be recorded using mono, stereo, tri-channel, or more channels, and each recorded channel data can be considered as a recorded data stream. Thus, for a certain application domain with recording requirements, the audio to be recorded may include one recorded data stream, two recorded data streams, three recorded data streams, or more recorded data streams. Therefore, the aforementioned scheme for allocating recorded audio to second audio nodes can be: allocating all recorded data streams of each recorded audio to at least one second audio node.

[0116] That is, for all recorded data streams of the same audio, all recorded data streams can be assigned to the same second audio node. Alternatively, some recorded data streams can be assigned to the same second audio node, and the rest to another second audio node. Furthermore, different recorded data streams can be assigned to different second audio nodes, depending on the specific application.

[0117] During implementation, configuration information can be pre-set, and the allocation of recorded data streams to the second audio node can be based on this configuration information. This configuration information indicates which recorded data stream is allocated to which second audio node, thereby achieving the desired allocation of recorded data streams. Therefore, the technical solution of this application, through its audio recording management scheme within a management domain, can achieve flexible allocation, ensuring flexible output of recorded audio.

[0118] In the case where the recorded audio data stream is distributed to one or more second audio nodes, the aforementioned scheme of performing audio processing on the channel data streams distributed to each first audio node to obtain the target audio can be implemented in the following way: performing audio processing on the channel data streams distributed to each first audio node and the recorded audio data streams of the second audio nodes to obtain the target audio.

[0119] In the technical solution of this application, the purpose of recording audio mainly includes three types. The first type is for testing, the second type is for audio processing together with the channel data stream of the first audio node to obtain the target audio, and the third type is for storage, to be used when needed.

[0120] Combination Figure 5 As shown, taking the request to record audio 1 from application domain 1 as an example, audio 1 needs to be recorded using dual-channel audio, resulting in recording data stream 1 and recording data stream 2. Recording data stream 1 and recording data stream 2 are then assigned to a second audio node. For example, recording data stream 1 and recording data stream 2 are assigned to audio node 5, which is used as the second audio node. If the application domain does not currently have audio data to play, recording data stream 1 and recording data stream 2 assigned to the second audio node can be processed for sound effects and then output through the audio output unit. This scheme is equivalent to a scheme where the audio is captured and then output; this scheme is used for testing. For example, it is used to test whether the audio output unit and the audio acquisition unit are in normal working order. If the content output by the audio output unit is the same as the content captured by the audio acquisition unit, it indicates that both the audio output unit and the audio acquisition unit are in normal working order.

[0121] If the application domain has audio data that needs to be played, the audio data channel stream 1 is assigned to audio node 1, which is used as the first audio node, and the audio data channel stream 2 is assigned to audio node 2, which is also used as the first audio node. If audio node 1 and audio node 5 correspond to the same audio player 1, then the recording data streams 1 and 2 assigned to audio node 5, and the channel data stream 1 assigned to audio node 1, are processed for output by audio player 1. If audio node 2 corresponds to audio player 2, then the channel data stream 2 assigned to audio node 2 is processed for output by audio player 2. In implementation, audio player 1 and audio player 2 output together, and the audio output together can be used as the target audio. Thus, it is possible to achieve normal output of different audio, such as recorded audio and audio to be played by the application domain.

[0122] When recording streams 1 and 2 are assigned to audio node 5, they can also be assigned to other audio nodes, such as audio node 4, for storage and later use. For example, recording streams 1 and 2 can be sent to application domain 1 via the virtual audio driver unit backend. This enables the recordings to be used for various purposes.

[0123] like Figure 7As shown, the audio recording process using software modules such as the software audio hub and the virtual audio driver unit backend is as follows: Initialize the software audio hub and the virtual audio driver unit backend. For the virtual audio driver unit backend, create an audio driver backend service to determine if there is a recording request from the application domain. If there is a recording request from the application domain, request the software audio hub to record. For the software audio hub, configure its recording parameters and create a recording stream. It can be understood that recording parameters include, but are not limited to, the recording format, recording sampling rate, and / or recording length. When recording is required, a microphone is used to record the audio to be recorded, obtaining a recording stream, thus creating the recording stream. The recording stream is distributed to one or more second audio nodes, and the recording streams in the second audio nodes are sent to the virtual audio driver unit of the application domain via the virtual audio driver unit backend to realize the transmission of recorded audio from the management domain to the application domain, facilitating the use of the recorded audio by the application domain.

[0124] In the technical solution of this application, the management domain can adopt the Android HAL (Hardware Abstraction Layer) architecture to implement the audio playback scheme in the audio processing method. For example... Figure 8 As shown, Android HAL uses one or more mono, stereo, or TDM (time-division multiplexing) channels to transmit audio data. Each mono, stereo, or TDM channel has multiple slots. Figure 8 Taking TDM 8slot as an example. In practical applications, the audio that appears in a car, that is, the audio requested to be played by the application domain, includes media (such as music, video and other multimedia audio), system sounds, ring tones, phone tones, navigation tones, etc.

[0125] See Figure 8In Android Playback slots, taking multimedia audio such as music as a four-channel audio source as an example, the four-channel audio needs to be mapped to four slots. For instance, channel data stream 1 is mapped to slot 1, channel data stream 2 to slot 2, channel data stream 3 to slot 3, and channel data stream 4 to slot 4. The channel data streams mapped to each slot can be assigned to the corresponding audio nodes (Endpoints) according to the pre-configured static configuration information indicating which slot's data is assigned to which audio node. For example, slots 1 and 2 are assigned to Endpoint 2; slots 3 and 4 are assigned to Endpoint 3. The channel data stream assigned to Endpoint 2 undergoes audio processing and is output through the speaker corresponding to Endpoint 2. Similarly, the channel data stream assigned to Endpoint 3 undergoes audio processing and is output simultaneously through the speaker corresponding to Endpoint 3 and the speaker corresponding to Endpoint 2. The use of multiple speakers enables normal output of multimedia audio. The output of multiple speakers to the same audio will inevitably bring users a good listening experience.

[0126] Taking a telephone ringtone as an example of two-channel audio, the two-channel audio needs to be mapped to two slots. For instance, channel data stream 1 is mapped to slot 1, and channel data stream 2 is mapped to slot 2. Slots 1 and 2 are then assigned to Endpoint 4. Endpoint 4 and Endpoint 2 correspond to the same speaker. The channel data streams from Endpoint 2 and Endpoint 4 undergo audio processing for output from this same speaker. This achieves simultaneous output of the telephone ringtone and music.

[0127] Taking a mono audio call as an example, the mono audio needs to be mapped to one slot. For instance, channel data stream 1 is mapped to slot 1. Slot 1 is assigned to Endpoint 5. Endpoint 5 and Endpoint 2 correspond to the same speaker. The channel data streams of Endpoint 5 and Endpoint 2 are processed for audio output from this same speaker. Using the same speaker enables simultaneous output of both call and music audio.

[0128] Taking navigation audio as a two-channel audio source as an example, the two-channel audio needs to be mapped to two slots. For instance, channel data stream 1 is mapped to slot 1, and channel data stream 2 is mapped to slot 2. Slots 1 and 2 are then assigned to Endpoint 6. Endpoint 6 and Endpoint 2 correspond to the same speaker. The channel data streams from Endpoint 6 and Endpoint 2 undergo audio processing for output from this same speaker. This achieves simultaneous output of car navigation audio and music audio.

[0129] If the application domain's audio data originates from other operating systems such as Linux, see [link / reference]. Figure 8 The cluster playback slot (instrument sound playback slot) allows the management domain to map audio data from channel data streams to slots. For example, channel data stream 1 in a two-channel audio dataset can be mapped to slot 1, and channel data stream 2 in a two-channel audio dataset can be mapped to slot 2. Slots 1 and 2 are then assigned to Endpoint 1. Endpoint 1 and Endpoint 2 correspond to the same speaker, and the channel data streams from Endpoint 1 and Endpoint 2 undergo audio processing for output from this same speaker.

[0130] See Figure 8 In Local Playback, if the audio to be played exists locally in the management domain, to save resources, there is no need to map the audio data from the local device to the virtual audio backend. The audio to be played can simply be assigned to a pre-defined audio node, such as Endpoint 7. Endpoint 7 and Endpoint 2 correspond to the same speaker. The channel data streams of Endpoint 7 and Endpoint 2 are processed for audio output from this same speaker.

[0131] Thus, the audio processing method of this application is implemented based on the Android audio architecture and the Linux audio architecture. It is understood that in practical applications, the application domain can install the Android operating system or other operating systems such as Linux to implement the audio processing method of this application through Android applications or Linux applications.

[0132] Among them, Figure 8 In this context, a Virtual I2S Stream can be viewed as a mapping from a channel to a slot. Audio Hub is a software audio hub.

[0133] In the technical solution of this application, the management domain can adopt the Android HAL architecture to implement the audio recording scheme in the audio processing method. For example... Figure 9 As shown, taking TDM 4-slot as an example, when the microphone records four-channel audio requested by application domain 1, the recorded audio data stream 1 is mapped to slot 1, audio data stream 2 to slot 2, audio data stream 3 to slot 3, and audio data stream 4 to slot 4. Audio data streams 1 and 2 are assigned to Endpoint 8, and audio data streams 3 and 4 are assigned to Endpoint 9 for storage, thus fulfilling at least one of the aforementioned three purposes. When needed, the recorded audio streams assigned to Endpoint 8 and Endpoint 9 undergo their respective audio processing and are then output through their corresponding speakers.

[0134] Taking TDM 4-slot as an example, when the microphone records two-channel audio requested by application domain 2, the recorded audio data stream 1 is mapped to slot 1, recorded audio data stream 2 to slot 2, recorded audio data stream 3 to slot 3, and recorded audio data stream 4 to slot 4. Recorded audio data stream 1 and recorded audio data stream 2 are assigned to Endpoint 10 for storage, thus fulfilling at least one of the aforementioned three purposes. When needed, the recorded audio stream assigned to Endpoint 10 undergoes audio processing and is then output through the corresponding speaker.

[0135] Based on the Android HAL architecture, this approach employs software modules such as Audio Hub, representing a software-based audio processing method. This software implementation allows for flexible allocation of audio and recording data to audio nodes, as well as flexible playback capabilities, enabling data streams from audio nodes to be played through different audio output units. Compared to hardware-based methods in related technologies, this approach achieves greater flexibility in both audio processing and playback.

[0136] In practical applications, if a car has only one speaker, the technical solution of this application can be used to output audio data from different application domains through the same speaker, achieving normal output of each audio data and improving the user's listening experience.

[0137] In this application's technical solution, a software audio hub in the management domain manages relevant audio hardware devices such as codecs (digital-to-analog or analog-to-digital converters), audio output units, and audio acquisition units. Compared to hardware-based management, software-based management greatly simplifies the audio playback process for each application domain, enabling efficient audio playback.

[0138] By having the management domain perform audio processing in software, hardware overhead can be reduced, avoiding the resource consumption problems caused by increased hardware costs, such as additional hardware and interconnection interfaces, as in related technologies, thus significantly saving costs. Furthermore, in related technologies, audio data, after being processed by a hardware device, may be sent back to that device for further processing to achieve better audio playback, thereby increasing latency. In the technical solution of this application, audio processing in software by the management domain can significantly shorten latency and achieve timely playback of audio data.

[0139] This application also provides an audio processing device, such as... Figure 10 As shown, the device includes:

[0140] The acquisition unit 1001 is used to acquire audio data from M application domains in the multi-core heterogeneous system using an inter-core communication mechanism. The multi-core heterogeneous system includes multiple hardware domains. Each hardware domain is composed of hardware resources connecting multiple processor cores to each processing core in the multi-core heterogeneous system, and the hardware domains are isolated from each other. The M application domains and the management domain in the multi-core heterogeneous system correspond to at least a portion of the hardware domains in the multi-core heterogeneous system.

[0141] The allocation unit 1002 is used to allocate each audio data to N first audio nodes of the management domain; wherein each first audio node includes at least a portion of the channel data stream of at least one audio data; M and N are both positive integers greater than or equal to 1;

[0142] The processing unit 1003 is used to perform audio processing on the channel data streams allocated to each first audio node to obtain target audio, which is used for playback.

[0143] In some embodiments, each audio data includes at least one channel data stream; the allocation unit 1002 is configured to allocate all the channel data streams included in each audio data to at least one of the N first audio nodes.

[0144] In some embodiments, at least one of the first audio nodes corresponds to an audio player; the processing unit 1003 is configured to:

[0145] Audio processing is performed on the channel data streams assigned to the first audio node corresponding to the same audio player;

[0146] Based on the audio processing results of the channel data streams of each first audio node corresponding to each audio player, the target audio for each audio data is obtained.

[0147] In some embodiments, the channel data stream of each audio data is sent from each application domain by each first virtual audio driver unit in each of the M application domains; the management domain in the multi-core heterogeneous system includes a second virtual audio driver unit; the obtaining unit 1001 is used to obtain each audio data of the M application domains by receiving the full channel data stream of each audio data through the second virtual audio driver unit using the channel corresponding to each audio data.

[0148] In some embodiments, the allocation unit 1002 is configured to allocate recorded audio to a second audio node based on audio recording requests from at least some of the M application domains, so as to send the recorded audio to the at least some application domains; wherein the recorded audio is the audio requested to be recorded by the audio recording request; wherein the audio recording request is obtained based on the inter-core communication mechanism.

[0149] In some embodiments, the processing unit 1003 is configured to:

[0150] The target audio is obtained by processing the channel data streams allocated to each first audio node and the recorded audio data streams of the second audio node.

[0151] In some embodiments, each of the at least some application domains corresponds to a recorded audio, the recorded audio including at least one recorded data stream; the allocation unit 1002 is configured to allocate the entire recorded data stream of each recorded audio to at least one second audio node.

[0152] In some embodiments, the number of channels corresponding to each audio data may be the same as or different from the number of channel data streams included in each audio data.

[0153] It should be noted that the audio processing device in this application embodiment solves the problem in a similar way to the aforementioned audio processing method. Therefore, the implementation process and implementation principle of the audio processing device can be found in the description of the implementation process and implementation principle of the aforementioned method, and the repeated parts will not be repeated.

[0154] This application provides a chip, including as follows: Figure 10 The audio processing device shown.

[0155] This application also provides an electronic device and a readable storage medium.

[0156] Figure 11A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0157] like Figure 11 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0158] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0159] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as audio processing methods. For example, in some embodiments, the audio processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the audio processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform audio processing methods by any other suitable means (e.g., by means of firmware).

[0160] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0161] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0162] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0164] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0165] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0166] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio processing method, characterized in that, The method includes: An inter-core communication mechanism is employed in a multi-core heterogeneous system, whereby the management domain of the multi-core heterogeneous system obtains audio data from each of the M application domains within the system. The multi-core heterogeneous system comprises multiple hardware domains; each hardware domain consists of hardware resources connecting multiple processor cores to each processing core within the system, and these hardware domains are isolated from each other. The M application domains and the management domain correspond to at least a portion of the hardware domains within the multi-core heterogeneous system. Map one or more channel data streams of each audio data to at least one time slot; According to the preset configuration information, the channel data stream of each audio data mapped to each time slot is allocated to the corresponding first audio node, so as to allocate each audio data to N first audio nodes in the management domain; wherein, each first audio node includes at least a portion of the channel data stream of at least one audio data; M is a positive integer greater than 1, and N is a positive integer greater than 1. The channel data streams allocated to each first audio node are processed to obtain target audio, which is used for playback.

2. The method according to claim 1, characterized in that, Each audio data stream includes at least one channel data stream; The process of distributing each audio data point to the N first audio nodes of the management domain includes: Each audio data stream, including all channels, is assigned to at least one of the N first audio nodes.

3. The method according to claim 2, characterized in that, The step of processing the channel data streams allocated to each first audio node to obtain the target audio includes: At least one of the first audio nodes corresponds to an audio player; Audio processing is performed on the channel data streams assigned to the first audio node corresponding to the same audio player; Based on the audio processing results of the channel data streams of each first audio node corresponding to each audio player, the target audio for each audio data is obtained.

4. The method according to claim 2 or 3, characterized in that, The channel data stream of each audio data is sent from each of the application domains by the first virtual audio driver units in each of the M application domains; the management domain in the multi-core heterogeneous system includes a second virtual audio driver unit; The step of obtaining audio data from each of the M application domains in the multi-core heterogeneous system from the management domain in the multi-core heterogeneous system includes: The management domain receives all channel data streams of each audio data using the channel corresponding to each audio data through the second virtual audio driver unit, thereby obtaining audio data from M application domains.

5. The method according to claim 1, characterized in that, Also includes: Based on audio recording requests from at least some of the M application domains, the recorded audio is allocated to a second audio node to send the recorded audio to the at least some application domains; wherein, the recorded audio is the audio requested to be recorded by the audio recording request; the second audio node is used to store the recorded audio; The audio recording request is obtained based on the inter-core communication mechanism.

6. The method according to claim 5, characterized in that, The step of processing the channel data streams allocated to each first audio node to obtain the target audio includes: The target audio is obtained by processing the channel data streams allocated to each first audio node and the recorded audio data streams of the second audio node.

7. The method according to claim 5, characterized in that, Each of the at least some of the application domains corresponds to recorded audio, and the recorded audio includes at least one recorded data stream; The process of assigning the recorded audio to the second audio node includes: Distribute the entire recorded data stream of each audio recording to at least one second audio node.

8. The method according to claim 4, characterized in that, The number of channels corresponding to each audio data may be the same as or different from the number of channel data streams included in each audio data.

9. An audio processing device, characterized in that, include: The acquisition unit is configured to obtain audio data from M application domains in a multi-core heterogeneous system via an inter-core communication mechanism of the multi-core heterogeneous system. The multi-core heterogeneous system comprises multiple hardware domains. Each hardware domain consists of hardware resources connecting multiple processor cores to each processing core in the multi-core heterogeneous system, and the hardware domains are isolated from each other. The M application domains and the management domain correspond to at least a portion of the hardware domains in the multi-core heterogeneous system. The allocation unit is used to map one or more channel data streams of each audio data to at least one time slot; according to preset configuration information, it allocates the channel data streams of each audio data mapped to each time slot to the corresponding first audio node, so as to allocate each audio data to N first audio nodes of the management domain; wherein, each first audio node includes at least a portion of the channel data streams of at least one audio data; M is a positive integer greater than 1, and N is a positive integer greater than 1; The processing unit is used to perform audio processing on the channel data streams allocated to each first audio node to obtain target audio, which is used for playback.

10. A chip, characterized in that, The chip includes the audio processing device as described in claim 9.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images