Heterogeneous System Audio Data Interaction Processing Method, Device and Storage Medium

Abstract

The present invention discloses a method, apparatus and storage medium for audio data interaction processing in a heterogeneous system. The method includes setting an inter-core communication software interface for the Linux system and the FreeRTOS system based on the ALSA audio framework; integrating an audio peripheral in the DSP processor of the FreeRTOS system, and setting an inter-core communication hardware interface between the DSP processor and the ARM processor of the Linux system; setting a data structure, where the data structure includes a communication code and a shared memory address of the data structure object; based on the inter-core communication software interface and the inter-core communication hardware interface, the ARM processor and the DSP processor perform audio data interaction processing. The embodiments of the present invention can improve the data transmission efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a method, device and storage medium for audio data interaction processing in a heterogeneous system. Background Art

[0002] Currently, some SOC chip designs integrate an external DSP chip into the same IC, sharing the system bus and memory. This eliminates the need for an external DSP chip, saving area and cost, and at the same time releases the computing power of the SOC.

[0003] Currently, most of the integrated DSPs are only used to improve computing power and only process data. There is no integration of the ADC and DAC of the Codec in the DSP system domain, nor are there other digital audio interfaces to implement playback and recording. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. For this purpose, the present invention proposes a method for audio data interaction processing in a heterogeneous system, which can improve the data transmission efficiency.

[0005] An embodiment of the present invention provides a method for audio data interaction processing in a heterogeneous system, including the following steps: based on the ALSA audio framework, setting an inter-core communication software interface for the Linux system and the FreeRTOS system; integrating audio peripherals in the DSP processor of the FreeRTOS system, and setting an inter-core communication hardware interface between the DSP processor and the ARM processor of the Linux system; setting a data structure, the data structure including a communication code and a shared memory address of the data structure object; the communication code is used to identify the data structure object structure, and the data structure object includes audio data parameter information and a data address; based on the inter-core communication software interface and the inter-core communication hardware interface, the ARM processor and the DSP processor perform audio data interaction processing, and the audio data interaction processing includes implementing communication between the ARM processor and the DSP processor through the data structure.

[0006] According to some embodiments of the present invention, the ARM processor and the DSP processor performing audio data interaction processing includes: the ARM processor writes an instruction and the data structure into a shared buffer, sends an interrupt notification to a specified DSP processor, and the thread running on the ARM processor enters a sleep state; the DSP processor receives the interrupt notification, reads the data in the shared buffer and parses the instruction, and performs corresponding processing according to the instruction; the DSP processor completes the instruction and returns the data structure to the ARM processor.

[0007] According to some embodiments of the present invention, setting the inter-core communication software interface for the Linux system and the FreeRTOS system includes: adapting the ALSA audio framework, including: modifying the interface for DMA to transfer PCM data in the ALSA audio framework into a heterogeneous cross-core interface; adding a communication framework in both the Linux system and the FreeRTOS system to implement the communication function between the ARM processor and the DSP processor.

[0008] According to some embodiments of the present invention, the method further includes: the FreeRTOS system docking with the ALSA framework to implement the DSP audio driver, specifically including: in the kernel state of the Linux system, the ALSA audio framework docks with the audio sound card driver, and the audio sound card driver is implemented by a heterogeneous cross-core interface, and the heterogeneous cross-core interface is used to communicate with the audio information service at the DSP processor end; the audio information service is used to parse the communication content and operate the audio peripherals for recording and / or playing.

[0009] According to some embodiments of the present invention, the ARM processor and the DSP processor perform audio data interaction processing, including: opening an audio playback device and setting parameters; the Linux system writes data to the shared memory, the audio service of the DSP processor receives the address of the shared memory, calls the relevant task interface to process the data according to the communication instruction code, and then starts the audio device output to transmit the data to the audio playback device; the Linux system writes data to the shared memory, waits for the DSP processor to finish processing the audio data and notifies the Linux system through an interrupt; the Linux system continues to send the address of the shared memory to the DSP processor.

[0010] According to some embodiments of the present invention, the audio data interaction processing between the ARM processor and the DSP processor includes: turning on the audio input device and setting parameters; the Linux system sending the address of the shared memory to the audio service of the DSP processor and waiting for notification from the audio service; after receiving the address of the shared memory, the audio service calls the interfaces of relevant tasks according to the communication instruction code to start device input, processes the input data one by one in the component order, writes the processed data to the shared memory, and then sends a message to notify the Linux system; when the Linux system reads a piece of data, it sends the address of the shared memory with empty data to the audio service of the DSP processor and waits for the DSP processor to finish filling. When the DSP processor writes a full amount of data, it notifies the Linux system through an interrupt, updates the data status, the Linux continues to send the address of the shared memory with empty data to the audio service of the DSP processor, and then updates the data in the kernel cache and waits for notification from the audio service of the DSP processor.

[0011] The heterogeneous system audio data interaction processing method according to the embodiments of the present invention has at least the following beneficial effects: for the audio playback stream, the ARM processor writes the command and notifies the DSP processor, and after the DSP processor finishes processing, it can directly output without passing back to the ARM processor for output. For the recording stream, after the DSP side finishes processing, it is passed back to the ARM processor, without first passing through the ARM processor for acquisition and then being passed back to the ARM processor after the DSP finishes processing. This reduces the intermediate links and improves the data transmission efficiency. For the application program of the PCM data of Linux audio, only the initialization settings need to be added to enable the function, without additional data operation adaptation, and the previous ALSA library can be used, simplifying the runtime operation of the application.

[0012] Another aspect of the embodiments of the present invention provides an audio data interaction processing device for a heterogeneous system, including: a first module for setting an inter-core communication software interface for the Linux system and the FreeRTOS system based on the ALSA audio framework; a second module for integrating audio peripherals for the DSP processor of the FreeRTOS system and setting an inter-core communication hardware interface between the DSP processor and the ARM processor of the Linux system; a third module for setting a data structure, the data structure including a communication code and a shared memory address of the data structure object; the communication code is used to identify the data structure object structure, and the data structure object includes audio data parameter information and a data address; a fourth module for performing audio data interaction processing between the ARM processor and the DSP processor based on the inter-core communication software interface and the inter-core communication hardware interface, the audio data interaction processing including communicating between the ARM processor and the DSP processor through the data structure.

[0013] Another aspect of the embodiments of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the heterogeneous system audio data interaction processing method described above is implemented.

[0014] The additional aspects and advantages of the present invention will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present invention. Description of the Drawings

[0015] The above and / or additional aspects and advantages of the present invention will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0016] Figure 1 It is a flowchart of a method for processing audio data in a heterogeneous system in the related art;

[0017] Figure 2 It is a flowchart of the method of the embodiments of the present invention;

[0018] Figure 3 It is a schematic diagram of the ALSA audio framework of the embodiments of the present invention;

[0019] Figure 4 It is a schematic diagram of implementing the DSP domain audio driver by the ALSA framework of the embodiments of the present invention;

[0020] Figure 5 It is a data flow diagram of the audio playback stream of the embodiments of the present invention;

[0021] Figure 6 It is a data flow diagram of the audio recording stream of the embodiments of the present invention;

[0022] Figure 7Schematic diagram of the audio algorithm component according to an embodiment of the present invention;

[0023] Figure 8 Block diagram of the modules of the device according to an embodiment of the present invention. Detailed implementation manners

[0024] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

[0025] In the description of the present invention, the meaning of "a number of" is one or more, the meaning of "a plurality of" is two or more, "greater than", "less than", "exceeding", etc. are understood as not including the present number, and "above", "below", "within", etc. are understood as including the present number. If there is a description of "first" and "second", it is only for the purpose of distinguishing technical features and should not be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features.

[0026] Alsa is the abbreviation of Advanced Linux Sound Architecture, that is, the Advanced Linux Sound Architecture, which provides support for audio and MIDI on the Linux operating system. After the kernel version of Linux 2.6, Alsa has currently become the mainstream audio architecture of linux. In addition to the alsa-driver, ALSA includes the alsa-lib function library in the user space, which has a more friendly programming interface and is fully compatible with OSS. Developers can use the driver through these advanced APIs without directly interacting with the kernel driver API.

[0027] User space: The Alsa Libray API mainly provides a unified API interface for application programs. Each APP application program can implement sound playback, recording, and control by calling the API interface provided by alsa-lib. Now two basic libraries are provided. Tinyalsa is a simplified alsa-lib library, which is mainly used in the Android system.

[0028] ALSA CORE: The alsa core layer provides logical devices (PCM / CTL / MIDI / TIMER / ...) system calls upward and drives hardware devices (Machine / I2S / DMA / CODEC) downward.

[0029] ASOC Core: It is the standard framework of ALSA and the core part of the ALSA-driver, providing general methods and data structures for various audio device drivers and offering the ALSA Driver API for the Audio driver.

[0030] Hardware Driver: The driver for audio hardware devices, which consists of three major parts: Machine, Platform, and Codec. It provides the ALSA Driver API and the initialization and working processes of the corresponding audio devices, implementing specific functional components, which is also the part that needs to be specifically implemented by driver developers.

[0031] RPMsg, whose full name is Remote Processor Messaging, defines the standard binary interface used for communication between cores in a heterogeneous multi-core processing system (AMP, Asymmetric Multiprocessing).

[0032] Refer to Figure 1 , and the methods for processing audio data in heterogeneous systems in the current related technologies are as follows.

[0033] Overall idea of the playback path:

[0034] The Linux system application sends the acquired data to the shared memory through the HiFi Driver driven by the Dsp audio framework, then gives it to the dsp system through the mailbox. The dsp performs corresponding data processing on the shared memory, such as decoding / EQ / reverb / surround sound, and then notifies the Linux side to retrieve the data from the shared memory. After the application retrieves it, it calls the driver through the alsa-lib / tinyalsa library interface to play the audio.

[0035] Overall idea of the recording path:

[0036] The Linux system application calls the driver through the alsa-lib / tinyalsa library interface to record audio. The acquired PCM data is sent to the shared memory through the HiFi Driver driven by the Dsp audio framework, then given to the dsp system through the mailbox. The dsp performs corresponding data processing on the shared memory, such as NS / AGC / AEC, and then notifies the Linux-side application to retrieve the data.

[0037] In the existing technical solution, the time consumed on the path is relatively long. The existing technical solution is only used to accelerate mathematical operations and improve the algorithm operation speed, and there is no connection to an audio peripheral on the IC. In the IC solution relied on by the present invention, since a peripheral is connected, the data source can be directly obtained on the DSP, eliminating the link on the path and reducing the delay of audio recording and playback.

[0038] The existing technical solution is as follows: for the recorded audio stream, the ARM core controls the audio peripheral to obtain voice data, and then sends the audio data to the DSP core. The DSP core performs voice pre-processing, and after processing, it returns to the ARM core. The ARM core saves the voice data or sends it to the cloud for recognition. For the played audio stream, the ARM core sends data to the DSP to decode it into PCM data and perform voice post-processing sound effects, and then returns the data to the ARM core. The ARM core transmits the data to the audio peripheral to output the audio.

[0039] The technical problem to be solved by the present invention is to implement audio PCM data interaction processing in a heterogeneous system, and provides a method for audio system information interaction in a Linux (ARM) + FreeRTOS (DSP) heterogeneous system, provides a software method for the Linux and FreeRTOS systems to send and receive audio data, and the FreeRTOS system to process audio data, and cooperates with the above audio system information interaction method to achieve Linux (ARM) to control FreeRTOS (DSP) playback and recording, as well as audio control operations. Provides an algorithm component mechanism for processing audio PCM data, which can access third-party algorithms for voice pre- and post-processing, audio encoding and decoding, etc.

[0040] The technical application background involved in the present invention is as follows:

[0041] (1) The DSP hardware system includes audio input and output devices;

[0042] (2) There is a message communication mechanism between the hardware system on the ARM core and the hardware system on the DSP core;

[0043] (3) The functional application needs to implement the playback and recording of audio PCM data and perform algorithm processing on the audio data.

[0044] In terms of the software system, the present invention needs to adapt a software framework for remote processing message communication. For the Linux system, there is already an RPMsg framework, and the software driver only needs to implement the docking framework for other software running on the Linux system to use; for the FreeRTOS system, a corresponding driver needs to be implemented for other software running on the FreeRTOS system to use. Its interaction design mainly includes the following two types:

[0045] (1) The design of heterogeneous system audio message interaction mainly includes the interactive use of shared memory and the design of data structures for transmitting information;

[0046] (2) The design of audio playback and recording stream control mainly includes the design of data structures required for audio stream control, the ALSA driver adaptation design on the Linux side, and the message receiving and sending process design for audio streams on the RTOS system side.

[0047] In the embodiment of the present invention, on an ARM processor (Android / Linux system), data structures including instructions are written into a shared buffer, and then an interrupt notification is sent to a specified DSP processor (FreeRTOS system). The thread running on the ARM processor enters a sleep state and waits for the interrupt notification from the DSP processor to wake up this thread. After receiving the interrupt, the DSP processor reads the data in the shared buffer and parses the instructions, and then the DSP processor completes the command function. After completing the function, the original data structure information is returned to the ARM processor. After receiving the interrupt, the called thread exits the wait state and proceeds to the next operation.

[0048] On the one hand, the DSP processor of the present invention uses a multitasking system, and different messages are managed by different tasks respectively. Then Linux can use one thread to correspond to one stream processing command to achieve parallel processing of different messages and data by multiple audio streams, maximizing the utilization rate of the DSP. In the application, only the corresponding task threads need to be created as needed for processing, and there is no need to perform related processing on the DSP scheduling on the Linux side.

[0049] On the other hand, for the audio playback stream, the ARM processor writes the command and notifies the DSP processor. After the DSP processor finishes processing, it can directly output without returning to the ARM processor for output again. For the recording stream, after the DSP side obtains and processes it, it is then returned to the ARM processor, without first passing through the ARM processor to obtain it and then returning it to the ARM processor after the DSP processes it. This reduces the intermediate links and improves the data transmission efficiency. For the application of Linux audio PCM data, only the initialization settings need to be added to enable the function, without additional data operation adaptation. The previous ALSA library can be used, simplifying the runtime operations of the application.

[0050] In the dependent IC solution of the present invention, for the recorded audio stream, the ARM core issues information to the DSP core. The DSP obtains voice data through the audio peripheral and performs voice pre-processing, and then sends it back to the ARM core. The ARM core saves the voice data or sends it to the cloud. Compared with the existing technical solutions, the link where the ARM sends the recording to the DSP is removed, reducing the delay on the communication path. This solution only targets PCM data. For the playback path, the ARM sends the PCM data to the DSP for voice post-processing sound effects and then directly outputs it to the audio peripheral. Compared with the existing technical solutions, the process where the DSP sends the data back to the ARM core is removed on the communication path, reducing the delay on the output path. The usage effect is as convenient as an SoC connecting to a DSP chip through I2S.

[0051] In the technical solution of the present invention, the audio native ALSA framework is adapted at the Linux kernel level for the DSP communication process. During the recording and playback operations, the application layer development does not need to pay attention to the communication with the DSP to achieve data synchronization for data input and output. Only the native ALSA audio library is used during operation, and the DSP will perform data processing according to the configuration. Since the algorithm processing configuration is set before the recording and playback programs run, after the audio stream starts, the DSP will automatically perform voice data processing during the operation of the audio stream. This is more friendly to application developers.

[0052] The method of the present invention mainly includes:

[0053] Hardware dependencies: The DSP domain system integrates audio peripherals, such as: codec (ADC / DAC), I2S, Dmic interface. There is an inter-core communication hardware interface between the DSP domain and the ARM domain.

[0054] Based on the inter-core communication interface and audio PCM data, a communication data structure is designed. This structure contains a communication code and the shared memory address of the PCM data structure object. The communication code is used to identify the PCM data structure object structure, where the PCM data structure object contains the basic parameter information and data address of the PCM data. The same audio stream holds the same object, and different streams hold different objects.

[0055] The inter-core communication data packet contains two parts, one is a 32-bit integer communication code, and the other is a 32-bit physical address (this physical address memory does not need to be pre-allocated by the system, but is allocated when needed).

[0056] The communication codes are divided into three categories, which respectively correspond to the audio processing object types on the DSP and the ARM systems. Adding the shared memory data (including audio stream parameters and audio data) facilitates both systems to accurately access the corresponding shared data.

[0057] Refer to Figure 2, the method of the embodiment of the present invention includes: based on the ALSA audio framework, setting an inter-core communication software interface for the Linux system and the FreeRTOS system; integrating an audio peripheral in the DSP processor of the FreeRTOS system, and setting an inter-core communication hardware interface between the DSP processor and the ARM processor of the Linux system; setting a data structure, the data structure includes a communication code and a shared memory address of the data structure object; the communication code is used to identify the data structure object structure, and the data structure object includes audio data parameter information and a data address; based on the inter-core communication software interface and the inter-core communication hardware interface, the ARM processor and the DSP processor perform audio data interaction processing, and the audio data interaction processing includes implementing communication between the ARM processor and the DSP processor through the data structure.

[0058] Referring to Figure 3 , according to the native audio framework of Linux, the ALSA audio framework on the Linux side is adapted. Each sound card's recording stream and playback stream are each independently responsible for processing by a task, without interference, and recording and playback can be processed in parallel. The sound card registered on Linux, such as audiocodec, corresponds to the audio peripheral sound card device drivers adc driver and dacdriver of the DSP, and related instructions are implemented on the DSP side.

[0059] The adaptation of the ALSA standard audio framework on the Linux side mainly involves the following parts:

[0060] 1. AlsaCore:

[0061] Modify the ALSA native interface to add an inter-core interface between the DSP and the ARM.

[0062] 2. Audio DSP Framework:

[0063] The main framework for communication between the DSP and the ARM, which realizes functions such as command interaction and data transfer between two heterogeneous cores.

[0064] Compared with the traditional ALSA audio driver, the embodiment of the present invention replaces the interface for DMA to transfer PCM data with a heterogeneous inter-core interface. In addition, the Audio DSP Framework is added to both the Linux / DSP systems to realize the communication function between heterogeneous cores.

[0065] Referring to Figure 4, the embodiment of the present invention realizes docking with the ALSA framework to implement the audio driver in the DSP domain, and this driver registers the audio device of the DSP as a sound card according to the configuration. Figure 4The left side represents the Linux system, and the right side represents the FreeRTOS system of the DSP. The upper left corner represents the user mode. The application layer can access the sound card using the standard alsa-lib interface without customized modification, which is convenient for customers to use. The lower left corner is the Linux kernel mode. Under the ALSA framework, it is docked with the audio sound card driver to implement the audio driver in the DSP domain. This driver will be implemented by RPMSG (cross-core interface), which can communicate with the AudioMsg service ( Figure 4 on the right side). After the service parses the communication content, it can operate the hardware device to perform functions such as recording and playing. Under this design, the entire process of recording and playing requests initiated from the Linux side and the DSP side executing the recording and playing actions is completed.

[0066] In some embodiments, the communication structure is designed as follows: The inter-core communication data packet contains two pieces of data. One is a 32-bit integer communication code, and the other is a 32-bit physical address (this physical address memory does not need to be pre-allocated by the system, but is allocated when in use). The communication codes are divided into three categories, which respectively correspond to the audio processing object types on the DSP and ARM systems. Adding the shared memory data (including audio stream parameters and audio data) facilitates both systems to accurately access the corresponding shared data.

[0067] Refer to Figure 5 , the embodiment of the present invention realizes the design of the audio stream control system. As Figure 5 shown, first, the PCM data is copied from the Linux user mode to the cache in the kernel mode. The kernel driver sends the address of the cache to the dsp. After receiving it, the dsp processes the data in the cache one by one according to the component order. After processing, the data is passed to the audio driver of the dsp and output from the DAC to the speaker. The following is the mechanism during operation.

[0068] For the playback stream:

[0069] 1) Open the audio output stream. The required audio device will be opened and parameters will be set. The parameters include the sound card device, audio channels, sampling rate, buffer size, etc.

[0070] 2) Transmit the playback stream:

[0071] Linux writes data to the shared memory (allocated to a continuous memory through the dma interface). After the AudioServer of the DSP receives the physical address of the shared memory, it calls the interfaces of relevant tasks one by one according to the communication instruction code to process the data, and then starts the device output to send the data to the device.

[0072] When Linux is full of data and waits for the audio system of the DSP to consume it. After the audio system of the DSP finishes consuming the data volume of one period, it notifies the Linux system through the interrupt of the msgbox, updates the data status, continues to send the data address to the DSP, and then updates the data in the kernel buffer and waits for the notification from the AudioServer of the DSP.

[0073] 3) When there is no data output, close the audio output stream and release the relevant resources.

[0074] Refer to Figure 6 , the embodiment of the present invention realizes audio recording. As Figure 6 shown in the data flow diagram, first obtain data from the ADC of the audio sound card of the DSP, process it one by one according to the component order, then copy the PCM data to the shared memory. The AudioServer of the DSP will notify the Linux system through the msgbox interrupt, and then update the status. The Linux user-mode program obtains the recorded data. The following is the mechanism during operation.

[0075] For the recording stream:

[0076] 1) Open the audio input stream, which will open the required audio device and set parameters. The parameters include the sound card device, audio channels, sampling rate, buffer format, etc.

[0077] 2) Transmit the playback stream:

[0078] Linux sends the shared data address (allocated to a continuous memory through the dma interface) to the AudioServer of the DSP and waits for the notification from the AudioServer of the DSP. After receiving the physical address of the shared memory, the AudioServer of the DSP calls the interface of the relevant task according to the communication instruction code to start the device input, processes the input data one by one according to the component order, writes the data to the shared memory after processing, and then sends a msgbox message to notify Linux.

[0079] After Linux reads a piece of data, it will send the shared memory address of the empty data to the AudioServer of the DSP and wait for the DSP system to fill it up. After the DSP finishes writing the data volume of one period, it notifies the Linux system through the interrupt of the msgbox, updates the data status, Linux continues to send the shared memory address of the empty data to the AudioServer of the DSP, and then updates the data in the kernel buffer and waits for the notification from the AudioServer of the DSP.

[0080] 3) When there is no need for data input, Linux actively closes the audio input stream and releases the relevant resources.

[0081] In some embodiments, the embodiments of the present invention also implement an audio mixer control design. On the Linux system side, the common extended get and put interfaces of the alsa mixer are adapted. When the application layer sets controls through alsa's amixer or tinyalsa's tinymix, it will call the mixer part in the hifi driver, and then remotely call the interface in the DSP's mixer task through msgbox to set it to the corresponding driver in the DSP's RTOS. In the design, all mixer interface calls of the sound card are uniformly managed by a mixer task, and this interface is designed as a synchronous interface.

[0082] Refer to Figure 7 , the embodiments of the present invention implement an audio algorithm component design. In the user space (Linux-APP), aw-rpaf-lib is provided for basic interfaces such as configuring algorithm components and obtaining algorithm-processed data; the aw-rpaf-alsa plugin is an audio plugin based on the open-source alsa-lib, which can more friendly configure the algorithm information of the audio stream; these control information are transmitted to the DSP through the RPAF framework (including component and alsa) of the kernel. When the DSP performs audio stream input / output (recording / playing), it will traverse and execute these algorithms and send the processed data back to Linux.

[0083] Refer to Figure 8 , the device of the embodiments of the present invention includes: a first module for setting an inter-core communication software interface for the Linux system and the FreeRTOS system based on the ALSA audio framework; a second module for integrating audio peripherals for the DSP processor of the FreeRTOS system and setting an inter-core communication hardware interface between the DSP processor and the ARM processor of the Linux system; a third module for setting a data structure, the data structure includes a communication code and a shared memory address of the data structure object; the communication code is used to identify the data structure object structure, and the data structure object includes audio data parameter information and data addresses; a fourth module for performing audio data interaction processing between the ARM processor and the DSP processor based on the inter-core communication software interface and the inter-core communication hardware interface, and the audio data interaction processing includes communicating between the ARM processor and the DSP processor through the data structure.

[0084] Although specific embodiments are described herein, those of ordinary skill in the art will recognize that many other modifications or alternative embodiments are also within the scope of the present disclosure. For example, any one of the functions and / or processing capabilities described in connection with a particular device or component can be performed by any other device or component. Additionally, although various illustrative implementations and architectures have been described in accordance with embodiments of the present disclosure, those of ordinary skill in the art will recognize that many other modifications to the illustrative implementations and architectures described herein are also within the scope of the present disclosure.

[0085] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments, and various changes can be made without departing from the spirit of the present invention within the knowledge of those of ordinary skill in the art to which the present invention pertains.

Claims

1. An audio data interaction processing method for a heterogeneous system, characterized in that Including the following steps: Based on the ALSA audio framework, set up an inter-core communication software interface for the Linux system and the FreeRTOS system; The DSP processor of the FreeRTOS system integrates audio peripherals, and an inter-core communication hardware interface is set up between the DSP processor and the ARM processor of the Linux system; Set up a data structure, which includes a communication code and the shared memory address of the data structure object; the communication code is used to identify the structure of the data structure object, and the data structure object includes audio data parameter information and data addresses; Based on the inter-core communication software interface and the inter-core communication hardware interface, the ARM processor and the DSP processor perform audio data interaction processing, and the audio data interaction processing includes communicating between the ARM processor and the DSP processor through the data structure; The audio data interaction processing between the ARM processor and the DSP processor includes: For the audio data of recording, the ARM core of the ARM processor sends information to the DSP core of the DSP processor, the DSP core obtains the voice data of the audio data through the audio peripheral and performs voice pre-processing, and after processing, it is sent to the ARM core, and the ARM core saves the voice data or sends it to the cloud.

2. The heterogeneous system audio data interaction processing method according to claim 1, wherein The audio data interaction processing between the ARM processor and the DSP processor includes: The ARM processor writes the instruction and the data structure into the shared buffer, sends an interrupt notification to the specified DSP processor, and the thread running on the ARM processor enters the sleep state; The DSP processor receives the interrupt notification, reads the data in the shared buffer and parses the instruction, and executes corresponding processing according to the instruction; The DSP processor completes the instruction and returns the data structure to the ARM processor.

3. The heterogeneous system audio data interaction processing method according to claim 1, wherein Setting up the inter-core communication software interface for the Linux system and the FreeRTOS system includes: Adapting the ALSA audio framework, including: Modify the interface for the DMA to transfer PCM data in the ALSA audio framework to a heterogeneous cross-core interface; Add a communication framework in both the Linux system and the FreeRTOS system to implement the communication function between the ARM processor and the DSP processor.

4. The method for processing heterogeneous system audio data interaction according to claim 1, wherein The method further includes: the FreeRTOS system docks with the ALSA audio framework to implement DSP audio driver, specifically including: In the kernel state of the Linux system, the ALSA audio framework docks with the audio sound card driver, and the audio sound card driver is implemented by a heterogeneous cross-core interface, and the heterogeneous cross-core interface is used to communicate with the audio information service at the DSP processor end; the audio information service is used to parse the communication content and operate the audio peripheral for recording and / or playing.

5. The method for processing heterogeneous system audio data interaction according to claim 1, wherein The audio data interaction processing between the ARM processor and the DSP processor includes: Open the audio playback device and set parameters; The Linux system writes data to the shared memory. The audio service of the DSP processor receives the address of the shared memory, calls the relevant task interface according to the communication instruction code to process the data, and then starts the audio device output to transmit the data to the audio playback device; The Linux system writes data to the shared memory, waits for the DSP processor to finish processing the audio data and notifies the Linux system through an interrupt; the Linux system continues to send the address of the shared memory to the DSP processor.

6. The heterogeneous system audio data interaction processing method according to claim 1, wherein The audio data interaction processing between the ARM processor and the DSP processor includes: Open the audio input device and set parameters; The Linux system sends the address of the shared memory to the audio service of the DSP processor and waits for the audio service to notify; After receiving the address of the shared memory, the audio service calls the interface of the relevant task according to the communication instruction code to start the device input, processes the input data one by one according to the component order, writes the data to the shared memory after processing, and then sends a message to notify the Linux system; When the Linux system reads a piece of data, it sends the address of the shared memory with empty data to the audio service of the DSP processor and waits for the DSP processor to fill it up. When the DSP processor writes a full amount of data, it notifies the Linux system through an interrupt, updates the data status, and the Linux continues to send the address of the shared memory with empty data to the audio service of the DSP processor, and then updates the data in the kernel cache and waits for the audio service of the DSP processor to notify.

7. An audio data interaction processing device for a heterogeneous system, characterized in that, Including: The first module is used to set the inter-core communication software interface for the Linux system and the FreeRTOS system based on the ALSA audio framework; The second module is used to integrate the audio peripherals for the DSP processor of the FreeRTOS system and set the inter-core communication hardware interface between the DSP processor and the ARM processor of the Linux system; The third module is used to set a data structure, and the data structure includes a communication code and the shared memory address of the data structure object; the communication code is used to identify the structure of the data structure object, and the data structure object includes audio data parameter information and data address; The fourth module is used to perform audio data interaction processing between the ARM processor and the DSP processor based on the inter-core communication software interface and the inter-core communication hardware interface. The audio data interaction processing includes implementing communication between the ARM processor and the DSP processor through the data structure, and is also used for the audio data for recording. The ARM core of the ARM processor sends information to the DSP core of the DSP processor. The DSP core obtains the voice data of the audio data through the audio peripheral and performs voice pre-processing, and then sends it to the ARM core. The ARM core saves the voice data or sends it to the cloud.

8. A computer-readable storage medium having a computer program stored thereon, and the method according to any one of claims 1 to 6 is implemented when the computer program is executed by a processor.

Images