Audio processing device and method

Abstract

The present application relates to an audio processing device and method, belonging to the field of audio processing technology. The audio processing device includes: a recording program running in an application layer, which is used to initiate a recording request in response to a recording instruction and transmit the recording request to an aDSP; the aDSP is connected to an audio acquisition component and is used to obtain the audio to be processed collected by the audio acquisition component when a recording request is received; a preprocessing algorithm pre-configured in the aDSP is called to preprocess the audio to be processed to obtain preprocessed audio data; the preprocessed audio data is returned to the recording program; the recording program is further used to call a preset processing unit based on the preprocessed audio data to perform voice interaction processing to obtain a voice interaction result when the preprocessed audio data is received; the voice interaction result is sent to the recording program for voice interaction; the performance of the audio processing algorithm and the stability of the system operation can be improved.

Description

Technical Field

[0001] The present application relates to an audio processing device and method, and belongs to the technical field of audio processing. Background Art

[0002] With the development of artificial intelligence, most electronic devices now have the ability to interact with users through voice. During voice interaction, electronic devices use voice data processing algorithms to perform a series of processing on the audio data collected by the terminal. These algorithms typically include voice front-end signal processing algorithms and voice interaction processing algorithms. These algorithms include echo cancellation, sound source localization, beamforming, voice noise reduction, speech enhancement, and voice wake-up. Voice interaction processing algorithms include speech recognition, endpoint detection, semantic understanding, and dialogue management.

[0003] At present, the voice data processing algorithm is actually one or several encapsulated and integrated executable codes that run on the CPU of a specific architecture platform. By integrating this executable code into the terminal system, the effect of audio data processing in voice interaction and call scenarios can be achieved.

[0004] For example, taking the in-vehicle terminal in a smart car as an example, the audio processing flow of the in-vehicle terminal includes the following: First, the application layer collects multi-channel audio data through recording or call requests; then, the collected multi-channel audio is sent to the voice data processing algorithm according to the interface requirements for echo cancellation, noise reduction and separation; finally, the processed audio is used for voice endpoint detection, wake-up, recognition, wake-up assisted positioning, and other interactions, or after echo cancellation and noise reduction, it is sent to the other end of the call (call scenario). The entire audio processing flow is loaded into a single processing unit, such as a CPU, for load processing.

[0005] However, due to the characteristics of voice data processing algorithms, complex audio-intensive calculations must run continuously on a single processor (typically the CPU) during the recording process, while the front-end signal processing algorithm must remain in the background. This results in voice data processing occupying a large amount of processor resources. Simultaneous operation of other workloads on the same processing unit can easily affect system stability. For example, on the Qualcomm platform used in smart cars, one processor may be overloaded, affecting system operation, while other available processing units are idle.

[0006] In addition, due to the limitation of terminal hardware resources, when the resource usage exceeds the limit, it will make the optimization of the voice data processing algorithm difficult. The only way to achieve a preliminary balance between system resources and processing effects is to reduce the amount of calculation at the cost of castrating some effects. Summary of the Invention

[0007] This application provides an audio processing device and method that can solve the problem that when all audio processing algorithms are run on the same processing unit, the performance optimization of the audio processing algorithm is limited or the processing unit is unstable due to the limited computing resources of the processing unit. This application provides the following technical solutions:

[0008] In a first aspect, an audio processing device is provided, the audio processing device comprising:

[0009] A recording program running in the application layer is used to initiate a recording request in response to a recording instruction and transmit the recording request to the audio signal processing chip aDSP;

[0010] The aDSP is connected to the audio acquisition component and is used to obtain the audio to be processed collected by the audio acquisition component when receiving the recording request; call the pre-processing algorithm pre-configured in the aDSP to pre-process the audio to be processed to obtain pre-processed audio data; and return the pre-processed audio data to the recording program;

[0011] The recording program is also used to call a preset processing unit to perform voice interaction processing based on the preprocessed audio data when receiving the preprocessed audio data, to obtain a voice interaction result; and send the voice interaction result to the recording program for voice interaction.

[0012] Optionally, the pre-processing algorithm in the aDSP includes at least an echo cancellation algorithm and a noise reduction algorithm.

[0013] Optionally, the pre-processing algorithm in the aDSP includes at least an echo cancellation algorithm, and the audio processing device further includes a computing signal processing chip cDSP;

[0014] The recording program is further configured to transmit the pre-processed audio data to the cDSP;

[0015] The cDSP is configured to, upon receiving the pre-processed audio data, invoke a noise reduction algorithm pre-configured in the cDSP to perform noise reduction on the pre-processed audio data to obtain noise-reduced audio data; and return the noise-reduced audio data to the recording program;

[0016] The recording program is further used to call a preset processing unit to perform voice interaction processing based on the noise-reduced audio data to obtain a voice interaction result.

[0017] Optionally, the cDSP is further configured to:

[0018] Check whether noise reduction is successful;

[0019] If the noise reduction is successful, triggering the step of returning the noise-reduced audio data to the recording program;

[0020] In the event of a failure in noise reduction, a failure feedback is returned to the recording program so that the recording program can output the reason for the audio failure.

[0021] Optionally, the noise reduction algorithm includes a beamforming algorithm.

[0022] Optionally, the aDSP is further configured to:

[0023] Check whether preprocessing is successful;

[0024] If the preprocessing is successful, triggering the step of returning the preprocessed audio data to the recording program;

[0025] In the event of pre-processing failure, failure feedback is returned to the recording program so that the recording program can output the reason for the audio failure.

[0026] Optionally, the preset processing unit is configured to:

[0027] Perform blind source separation on the received audio data using a blind source separation algorithm;

[0028] After the blind source separation is successful, the multi-channel data is processed for voice interaction according to the preset channel order.

[0029] Optionally, the voice interaction processing includes voice wake-up processing, voice recognition processing, and executing interactive actions corresponding to the voice recognition results.

[0030] Optionally, the audio request carries recording parameters, and the calling of a pre-configured pre-processing algorithm in the aDSP to pre-process the audio to be processed to obtain pre-processed audio data includes:

[0031] The audio to be processed is streamed into the preprocessing algorithm frame by frame according to the recording parameters for preprocessing to obtain preprocessed audio data.

[0032] In a second aspect, an audio processing method is provided, the method comprising:

[0033] Initiating a recording request in response to a recording instruction;

[0034] Calling aDSP based on the recording request to trigger the aDSP to obtain the audio to be processed collected by the audio collection component; and calling a preprocessing algorithm pre-configured in the aDSP to preprocess the audio to be processed to obtain preprocessed audio data;

[0035] Based on the preprocessed audio data, a preset processing unit is called to perform voice interaction processing to obtain a voice interaction result for voice interaction.

[0036] The beneficial effects of the present application include at least: by transplanting part of the processing of the audio data to aDSP, since the preset processing unit only needs to process the audio data processed by aDSP, the occupation of the computing power resources of the preset processing unit by the audio processing algorithm can be reduced. At this time, the part of the algorithm running in the preset processing unit can use more computing power resources, so this part of the algorithm can be further optimized; it can solve the problem that when all audio processing algorithms are run on the same processing unit, the performance optimization of the audio processing algorithm is limited due to the limited computing power resources of the processing unit, and the performance of the audio processing algorithm can be improved. At the same time, since the occupation of the computing power resources of the preset processing unit is reduced, other programs running on the preset processing unit are more stable, thereby solving the problem that when all audio processing algorithms are run on the same processing unit, the processing unit is unstable, and improving the stability of system operation.

[0037] In addition, by further porting part of the audio processing algorithm to cDSP for operation, not only can the computing power occupied by the preset processing unit be reduced, but also the computing power occupied by aDSP can be reduced, which can further improve system stability and algorithm performance.

[0038] In addition, by detecting the preprocessing process and the noise reduction process, prompts can be output in time when abnormalities occur in the preprocessing process and the noise reduction process, so that users can solve the abnormalities in time, which can improve the timeliness of abnormal prompts and solutions.

[0039] In addition, by performing blind source separation on the audio data, the target human voice audio and background noise can be separated to achieve further speech noise reduction effect and extract purer target audio data.

[0040] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and to implement it in accordance with the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a structural diagram of an audio processing device provided by an embodiment of the present application;

[0042] Figure 2 This is a schematic diagram of the computing power usage of the pre-processing algorithm in the case of four microphones provided in one embodiment of the present application;

[0043] Figure 3This is a schematic diagram of the computing power usage of the preprocessing algorithm in the case of 6 microphones provided in one embodiment of the present application;

[0044] Figure 4 This is a flowchart of an audio processing method provided by an embodiment of the present application;

[0045] Figure 5 This is a flowchart of an audio processing method provided by another embodiment of the present application. DETAILED DESCRIPTION

[0046] The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0047] In traditional voice data processing algorithm solutions, developers focus more on the differences in the operating systems and instruction sets during the packaging and integration process, with an emphasis on code portability and reducing platform dependence. The encapsulated calling logic determines that the final output general algorithm processing module can only run on a single processor.

[0048] Taking the increasingly stringent performance requirements and multi-zone voice interaction scenarios pursued by smart cars as an example, traditional noise reduction algorithms have reached bottlenecks in some areas of most voice data algorithm processing and are difficult to optimize. Even in some complex solutions, the computing power requirements of traditional noise reduction algorithms are above the indicators. Therefore, better algorithm solutions will be difficult to introduce due to the relatively large resource consumption.

[0049] At present, after the voice data processing algorithm is deployed on the platform of the audio processing device, a large number of test sets are collected to continuously adjust and optimize the parameters of the voice data processing algorithm to improve the processing effect of the voice data processing algorithm. At the same time, the optimization and balance of performance and resource usage are achieved by trimming and discarding some processing logic of the voice data processing algorithm that does not concern the effect, or reducing the number of channels for input audio collection.

[0050] However, while current algorithm optimization methods can generally meet most voice interaction needs, they cannot guarantee both performance and effectiveness. Specifically, while pruning the algorithm or reducing the amount of data input can reduce the computing resources used, even using a large test set to optimize algorithm performance will still affect the accuracy of the algorithm to some extent.

[0051] Based on the above technical problem, this embodiment provides an audio processing device. Figure 1This is a structural diagram of an audio processing device provided by an embodiment of the present application. The audio processing device refers to an electronic device with audio processing capabilities, including but not limited to: an on-board terminal installed in a vehicle, or a mobile phone, tablet computer, or laptop computer, etc. This embodiment does not limit the device type of the audio processing device. Figure 1 The audio processing device provides audio services based on the Qualcomm audio architecture, which includes at least: an application layer 101, a hardware abstraction layer (HAL) 102, a kernel layer 103 and hardware 104.

[0052] The application layer 101 runs an audio application to provide audio services to users. In one example, the application layer 101 runs an audio recording program (Audio Recorder) to provide audio services to users. The recording program can be a voice assistant or a call program, and this embodiment does not limit the type of recording program.

[0053] Because audio processing devices may also be running other programs, the recording program may run alongside other system software and services. Considering both system fluency and resource allocation, the resource allocation for the recording program is generally limited within a system. For example, in the smart car sector, users are increasingly demanding the playability, convenience, and richness of their voice interaction experience. More and more users are opting for interaction in more audio zones to achieve a richer experience, while also ensuring that the audio processing quality in each zone meets the desired specifications.

[0054] In order to ensure the effectiveness of the audio service provided by the recording program, in this embodiment, by deploying the calculation processes of different stages of the audio processing algorithm in different computing units, the problem of excessive CPU resource occupation can be reduced. At the same time, the CPU resources released can be used for other processing optimizations or the operation of more other system services.

[0055] Specifically, this embodiment introduces an audio signal processing chip (audio Digital Singal Processor, aDSP) into the hardware 104. A preprocessing algorithm runs in the aDSP. Since the preprocessing algorithm is computationally intensive, by placing some audio-intensive operations in Qualcomm's low-power audio-specific processor aDSP, the CPU resource occupation of these operations can be reduced.

[0056] Hardware 104 also includes a codec (codec) connected to the aDSP. The codec includes at least one audio input interface and at least one audio output interface. The audio input interface is connected to the audio acquisition component and is used to collect at least audio data output by the microphone and / or re-acquired data. The audio output interface is connected to a speaker to play audio sent by the aDSP.

[0057] aDSP includes an analog front end (Analog Front End, AFE), an audio device manager (Audio Device Manager, ADM) and an audio stream manager (Audio Stream Manager, ASM).

[0058] In this embodiment, by pre-configuring the DSP parameter configuration file acdb file, the audio data is processed by the pre-processing algorithm of the processing path flow inside the aDSP before reaching the recording program.

[0059] Specifically, the AFE includes a transmitter (TX) and a receiver (RX). The TX sends collected audio data to the ADM. The ADM invokes a preprocessing algorithm to preprocess the audio data sent by the TX and sends the preprocessed audio data to the ASM for further processing. Schematically, the preprocessing algorithm is integrated into the ASM's Common-Object Pre Processing (COPreP) module.

[0060] ASM sends the audio stream under the ASM module to the recording program through the registered APR BUS channel through the kernel layer 103 and the hardware abstraction layer 102. At this time, since the audio data has been processed by the pre-processing algorithm, the subsequent recording program can reduce CPU usage when calling the CPU for further processing.

[0061] Specifically, the audio processing flow includes: a recording program for initiating a recording request in response to a recording instruction and transmitting the recording request to the aDSP;

[0062] aDSP is used to obtain the audio to be processed collected by the audio collection component when receiving a recording request; call the pre-processing algorithm pre-configured in aDSP to pre-process the audio to be processed to obtain pre-processed audio data; and return the pre-processed audio data to the recording program;

[0063] The recording program is also used to call the preset processing unit to perform voice interaction processing based on the preprocessed audio data when receiving the preprocessed audio data, and obtain the voice interaction result; and send the voice interaction result to the recording program for voice interaction.

[0064] In addition to voice interaction processing, the preset processing unit also executes other applications on the audio processing device, such as system services and applications, etc. The preset processing unit can be a CPU.

[0065] The audio request carries recording parameters, and the aDSP inputs the audio to be processed into the preprocessing algorithm frame by frame according to the recording parameters for preprocessing to obtain preprocessed audio data.

[0066] Optionally, the aDSP is further configured to: detect whether preprocessing is successful; if so, trigger the step of returning the preprocessed audio data to the recording program; and if preprocessing fails, return failure feedback to the recording program, so that the recording program can output a reason for the audio failure. In this case, the reason for the audio failure includes a preprocessing failure.

[0067] In one example, the preprocessing algorithm in the aDSP includes at least an acoustic echo cancellation (AEC) algorithm and a noise reduction algorithm. Accordingly, the audio failure causes include echo cancellation failure and noise reduction failure. The noise reduction algorithm includes, but is not limited to, a beamforming (BF) algorithm. In other embodiments, the noise reduction algorithm may also be a linear filter noise reduction or artificial intelligence (AI) noise reduction. This embodiment does not limit the implementation of the noise reduction algorithm.

[0068] In one example, determining whether echo cancellation is successful includes: calculating an evaluation parameter of the audio signal after echo cancellation, and determining that the echo cancellation is successful if the evaluation parameter is within a preset range; and determining that the echo cancellation fails if the evaluation parameter is not within the preset range.

[0069] Among them, the evaluation parameter includes echo return loss gain. In this case, the preset range can be greater than the preset gain threshold; or, the evaluation parameter is a score obtained using a voice quality perception evaluation algorithm. In this case, the preset range can be greater than the preset score threshold (for example, 2). In other embodiments, the evaluation parameter can also be other values. This embodiment does not limit the evaluation method of echo cancellation.

[0070] Alternatively, it is determined whether the echo cancellation algorithm is successfully loaded into the aDSP. If so, the evaluation parameters of the echo-cancelled audio signal are calculated to further determine whether the echo cancellation is successful based on the evaluation parameters. If the loading fails, it is determined that the echo cancellation has failed.

[0071] In one example, detecting whether noise reduction is successful includes: calculating the signal-to-noise ratio and / or root mean square error of the audio data after noise reduction; determining that the noise reduction is successful when the signal-to-noise ratio is greater than a signal-to-noise ratio threshold and / or the root mean square error is less than the root mean square threshold; and determining that the noise reduction fails when the signal-to-noise ratio is less than or equal to the signal-to-noise ratio threshold and / or the root mean square error is greater than or equal to the root mean square threshold.

[0072] Alternatively, it is determined whether the noise reduction algorithm is successfully loaded into the aDSP. If so, the signal-to-noise ratio and / or root mean square error of the noise-reduced audio data are calculated to further determine whether the noise reduction is successful. If the loading fails, it is determined that the noise reduction has failed.

[0073] Because the pre-processing algorithm has high requirements for the margin in the aDSP. The high density of the voice data processing algorithm and the real-time rate requirements of audio data processing determine the high occupancy of the pre-processing algorithm. For example: when processing audio, it is necessary to feed a frame of audio data with a length of 32ms into the pre-processing algorithm every 32ms. The processing of each frame of audio data must be completed within 32ms to ensure that all input audio can be processed and the output audio is normal without frame loss. Based on this, the pre-processing algorithm has the following performance occupancy trend: in the process of processing each frame of audio, after receiving the audio data, it will complete the processing at full speed within 32ms with the highest computing power, and then the frequency clock will be reduced. Figure 2 The diagram shows the computing power usage per millisecond in the case of 4 microphones. Figure 2 It can be seen that the computing power usage of aDSP reaches its peak when each frame of audio signal is received, and decreases after the signal processing is completed. With the increase of audio acquisition components or the processing of audio data with higher sampling rates, the processing time of each frame will increase. For example, Figure 3 The computing power usage in the case of 6 microphones is shown. The computing power reaches the peak time for more than Figure 2 The duration of time when the computing power reaches its peak. At this time, it is necessary to ensure that too many other audio processing algorithms are not running inside the aDSP.

[0074] Based on the above problem, in another example, the audio processing device also includes a compute signal processing chip (compute DSP, cDSP), which is a hardware acceleration unit specifically used for general computing in the Qualcomm audio framework. The cDSP is used to execute algorithms with higher complexity in the front-end voice data processing algorithm. Schematically, the cDSP is used to deploy noise reduction algorithms. By transplanting more computationally intensive algorithm processing to the cDSP, the algorithm's computing power occupation of the aDSP can be reduced. At this time, the preprocessing algorithm in the aDSP includes at least an echo cancellation algorithm. The recording program is also used to transmit the preprocessed audio data to the cDSP; the cDSP is used to, upon receiving the preprocessed audio data, call the pre-configured noise reduction algorithm in the cDSP to perform noise reduction on the preprocessed audio data to obtain noise-reduced audio data; and return the noise-reduced audio data to the recording program; the recording program is also used to call the preset processing unit based on the noise-reduced audio data to perform voice interaction processing to obtain a voice interaction result.

[0075] In this example, AEC is independently loaded into the aDSP and runs as firmware. Noise reduction and post-processing can be integrated into the same executable file, with computationally intensive algorithms offloaded to the cDSP via the hexagonnn interface, effectively saving CPU usage. Voice interaction algorithms such as wake-up, recognition, conversation, and angle positioning, which are linked to computationally intensive algorithms, can be configured in the cDSP and / or the CPU.

[0076] In other embodiments, each algorithm processing unit involved in the voice data processing algorithm may be released separately and combined by the application layer, so that each part of the voice data processing algorithm cooperates to complete the entire voice data processing process.

[0077] Optionally, when the cDSP executes the noise reduction algorithm, the cDSP is further configured to: detect whether the noise reduction is successful; if so, trigger the step of returning the noise-reduced audio data to the recording program; and if the noise reduction fails, return failure feedback to the recording program so that the recording program can output a reason for the audio failure. In this case, the reason for the audio failure includes: noise reduction failure.

[0078] Specifically, after the recording program obtains the audio data after echo cancellation, it calls cDSP through the cDSP driver to execute the executable file of the noise reduction algorithm in cDSP, and returns the noise-reduced audio data to the recording program. At this time, the recording program calls the preset processing unit for voice interaction processing.

[0079] Optionally, after receiving the noise-reduced audio data, the preset processing unit uses a blind source separation (BSS) algorithm to perform blind source separation on the received audio data; after the blind source separation is successful, the multi-channel data is processed for voice interaction according to a preset channel order.

[0080] In this embodiment, the audio data collected by the audio collection component is multi-channel audio data. After a series of processing (including blind source separation) is performed on the multi-channel audio data, the data obtained is multi-channel data.

[0081] Optionally, the voice interaction processing includes voice wake-up processing, voice recognition processing, and executing an interactive action corresponding to the voice recognition result. In other embodiments, the voice interaction processing may also include only voice wake-up processing, or include voice wake-up processing and voice recognition processing. This embodiment does not limit the method of voice interaction processing.

[0082] The following example illustrates the audio processing process, using the echo cancellation algorithm deployed within the aDSP, the noise reduction algorithm deployed within the cDSP, and the BF algorithm as the noise reduction algorithm. Referring to 4, this process includes at least the following steps:

[0083] Step 41: In response to the recording instruction, the recording program in the application layer calls the recording interface to initiate recording.

[0084] Specifically, the recording request is initiated by configuring the recording parameters through a native interface such as the audio recorder and sent to the Qualcomm audio framework for processing.

[0085] In one example, a recording command is generated when the audio processing device is powered on. After the audio processing device is powered on, the recording program continues to run in the background of the audio processing device in response to the recording command. For example, in a smart car scenario, a voice assistant begins recording when the vehicle terminal is powered on and runs in the background, allowing the user to activate the voice assistant at any time by using a wake-up word to perform an action.

[0086] In another example, upon receiving a preset operation, the audio processing device generates a recording instruction to trigger a recording program to initiate recording. Optionally, the preset operation includes, but is not limited to, connecting a voice conversation or starting a recording program. This embodiment does not limit the implementation of the preset operation. For example, in a call scenario, a recording instruction is generally generated upon answering the call to trigger the recording program to initiate a recording request, and recording stops upon hanging up the call.

[0087] In other embodiments, the recording instruction may be generated in other ways, which are not listed here in this embodiment.

[0088] In step 42, the Qualcomm audio framework collects audio data from multiple microphones and echoes it back to the AEC. The aDSP in the Qualcomm audio framework creates a designated link and sends the audio data frame by frame to the AEC for echo cancellation processing.

[0089] In this step, the audio data collected by the Qualcomm audio framework first completes the echo cancellation processing during the recording process, that is, the echo audio recorded from the power amplifier is eliminated from the recorded multi-channel audio, and only the human voice and part of the background noise are retained.

[0090] Step 43: Check whether the loading and processing of the AEC in the aDSP are successful.

[0091] If the execution fails, the audio recording fails, the conversation is exited, and the process ends.

[0092] If the execution is successful, go to step 44.

[0093] Step 44: The aDSP passes the audio data processed by the echo cancellation algorithm to the upper layer through the Qualcomm audio framework processing logic, and further transmits it to the recording program of the application layer;

[0094] The recording program obtains the audio after echo cancellation, and the application layer passes the audio after echo cancellation as a parameter to the beamforming algorithm, and the processing interface initiates the calculation process.

[0095] In step 45, the recording program places the actual beamforming algorithm calculation in the cDSP by calling the cDSP RPC interface that relies on the cDSP to perform noise reduction processing on the audio data after beam adjustment.

[0096] In this step, for the audio data of multiple channels recorded by multiple microphones, beams with multiple sound zones are generated to suppress noise and interference directions and enhance the effect of signals in the target direction.

[0097] Step 46: Check whether the loading and operation processing of the cDSP is successful.

[0098] If it fails, throw out the failure reason, exit the dialogue, and the outflow ends.

[0099] If successful, proceed to step 47.

[0100] In step 47 , the multi-directional beam audio after noise reduction processing is transmitted back to the calling interface so as to be sent as an input parameter to the next processing unit Blind Source Separation (BSS) algorithm.

[0101] In step 48, BSS performs blind source separation on the input audio. After corresponding feature analysis and correlation analysis of the audio, the target human voice audio and background noise are separated from the input multi-channel audio to achieve further speech noise reduction effect and extract relatively pure target audio data.

[0102] In this embodiment, blind source separation is performed in a CPU as an example. In other embodiments, blind source separation may also be performed in a cDSP as part of a noise reduction algorithm. This embodiment does not limit the implementation method of blind source separation.

[0103] Step 49: Check whether the BSS processing is successful.

[0104] If it fails, the failure reason is thrown, the dialogue is exited, and the process ends.

[0105] If successful, execute step 410.

[0106] In step 410 , the recording program calls back the multi-channel audio data after blind source separation and sends it to subsequent voice interaction processing in a predetermined channel order.

[0107] Voice interaction can be processed in a variety of combinations. In one example, the input audio, after noise reduction, is sent to the wake-up process. Combined with the angle information of the awakened channel, the audio of the awakened channel is sent to perform subsequent single-channel recognition. The recognition results are fed into the dialogue and semantic analysis, and corresponding action processing and other operations are performed.

[0108] Refer to Table 1 below for the CPU usage percentage of each algorithm in a four-zone scenario under the same CPU model (e.g., total CPU computing power: 85k DMIPS (Dhrystone Million Instructions executed Per Second)). It has been verified that by porting the echo cancellation algorithm to the aDSP, 11.72% of the CPU resident usage can be saved. If the BF algorithm is further ported to the cDSP, a further 6.36% of the CPU resident usage can be saved. Compared with the CPU computing power occupied by all running on the CPU, the CPU computing power occupied by the audio processing solution based on the general audio framework will save nearly half of the CPU computing power.

[0109] Table 1:

[0110]

[0111]

[0112] In summary, the audio processing device provided by this embodiment, by transplanting part of the processing of the audio data to aDSP, since the preset processing unit only needs to process the audio data processed by aDSP, it can reduce the occupation of the computing power resources of the preset processing unit by the audio processing algorithm. At this time, some algorithms running in the preset processing unit can have more computing power resources to use, so this part of the algorithm can be further optimized; it can solve the problem that when all audio processing algorithms are run on the same processing unit, the performance optimization of the audio processing algorithm is limited due to the limited computing power resources of the processing unit, and the performance of the audio processing algorithm can be improved. At the same time, since the occupation of the computing power resources of the preset processing unit is reduced, other programs running on the preset processing unit are more stable, thereby solving the problem that when all audio processing algorithms are run on the same processing unit, the processing unit is unstable, and improving the stability of system operation.

[0113] In addition, by further porting part of the audio processing algorithm to cDSP for operation, not only can the computing power occupied by the preset processing unit be reduced, but also the computing power occupied by aDSP can be reduced, which can further improve system stability and algorithm performance.

[0114] In addition, by detecting the preprocessing process and the noise reduction process, prompts can be output in time when abnormalities occur in the preprocessing process and the noise reduction process, so that users can solve the abnormalities in time, which can improve the timeliness of abnormal prompts and solutions.

[0115] In addition, by performing blind source separation on the audio data, the target human voice audio and background noise can be separated to achieve further speech noise reduction effect and extract purer target audio data.

[0116] Figure 5 This is a flowchart of an audio processing method provided by an embodiment of the present application. Figure 1 The method is described using the audio processing device shown in FIG. 1 as an example. The method includes at least the following steps:

[0117] Step 501, initiating a recording request in response to a recording instruction;

[0118] Step 502: calling aDSP based on the recording request to trigger aDSP to obtain the audio to be processed collected by the audio collection component; and calling a pre-processing algorithm pre-configured in aDSP to pre-process the audio to be processed to obtain pre-processed audio data;

[0119] Step 503: Call a preset processing unit to perform voice interaction processing based on the preprocessed audio data to obtain a voice interaction result for voice interaction.

[0120] In summary, the audio processing method provided in this embodiment, by transplanting part of the processing of the audio data to aDSP, since the preset processing unit only needs to process the audio data processed by aDSP, it can reduce the occupation of the computing power resources of the preset processing unit by the audio processing algorithm. At this time, the part of the algorithm running in the preset processing unit can have more computing power resources to use, so this part of the algorithm can be further optimized; it can solve the problem that when all audio processing algorithms are run on the same processing unit, the performance optimization of the audio processing algorithm is limited due to the limited computing power resources of the processing unit, and the performance of the audio processing algorithm can be improved. At the same time, since the occupation of the computing power resources of the preset processing unit is reduced, other programs running on the preset processing unit are more stable, thereby solving the problem that when all audio processing algorithms are run on the same processing unit, the processing unit is unstable, and the system operation stability is improved.

[0121] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored. The program is loaded and executed by a processor to implement the audio processing method of the above method embodiment.

[0122] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, wherein the computer-readable storage medium stores a program, and the program is loaded and executed by a processor to implement the audio processing method of the above method embodiment.

[0123] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An audio processing device, characterized in that The audio processing device comprises: A recording program running in the application layer is used to initiate a recording request in response to a recording instruction and transmit the recording request to the audio signal processing chip aDSP; The aDSP is connected to the audio acquisition component and is used to obtain the audio to be processed collected by the audio acquisition component when receiving the recording request; call the pre-processing algorithm pre-configured in the aDSP to pre-process the audio to be processed to obtain pre-processed audio data; and return the pre-processed audio data to the recording program; The preprocessing algorithm in the aDSP includes at least: an echo cancellation algorithm and a noise reduction algorithm; the aDSP is further used to: detect whether the noise reduction is successful; if the noise reduction is successful, trigger the execution of the step of returning the noise-reduced audio data to the recording program; if the noise reduction fails, return failure feedback to the recording program so that the recording program can output the reason for the audio failure; the audio failure reason includes echo cancellation failure and noise reduction failure; or, the preprocessing algorithm in the aDSP includes at least an echo cancellation algorithm, and the audio processing device further includes a computing signal processing chip cDSP; the recording program is further used to transmit the preprocessed audio data to the cDSP; the cDSP is used to, when receiving the preprocessed audio data, call the noise reduction algorithm pre-configured in the cDSP to perform noise reduction on the preprocessed audio data to obtain noise-reduced audio data; and return the noise-reduced audio data to the recording program; The recording program is also used to call a preset processing unit to perform voice interaction processing based on the preprocessed audio data when receiving the preprocessed audio data, to obtain a voice interaction result; and send the voice interaction result to the recording program for voice interaction.

2. The audio processing device according to claim 1, characterized in that The cDSP is also used to: Check whether noise reduction is successful; If the noise reduction is successful, triggering the step of returning the noise-reduced audio data to the recording program; In the event of a failure in noise reduction, a failure feedback is returned to the recording program so that the recording program can output the reason for the audio failure.

3. The audio processing device according to claim 1, characterized in that The noise reduction algorithm includes a beamforming algorithm.

4. The audio processing device according to claim 1, wherein The aDSP is also used to: Check whether preprocessing is successful; If the preprocessing is successful, triggering the step of returning the preprocessed audio data to the recording program; In the event of pre-processing failure, failure feedback is returned to the recording program so that the recording program can output the reason for the audio failure.

5. The audio processing device according to claim 1, characterized in that The preset processing unit is used to: Perform blind source separation on the received audio data using a blind source separation algorithm; After the blind source separation is successful, the multi-channel data is processed for voice interaction according to the preset channel order.

6. The audio processing device according to claim 5, characterized in that The voice interaction processing includes voice wake-up processing, voice recognition processing and executing interactive actions corresponding to the voice recognition results.

7. The audio processing device according to any one of claims 1 to 6, characterized in that: The recording request carries recording parameters, and the calling of a preprocessing algorithm pre-configured in the aDSP to preprocess the audio to be processed to obtain preprocessed audio data includes: The audio to be processed is streamed into the preprocessing algorithm frame by frame according to the recording parameters for preprocessing to obtain preprocessed audio data.

8. An audio processing method, applied to the audio processing device according to any one of claims 1 to 7, characterized in that: The method comprises: Initiating a recording request in response to a recording instruction; Calling aDSP based on the recording request to trigger the aDSP to obtain the audio to be processed collected by the audio collection component; and calling a preprocessing algorithm pre-configured in the aDSP to preprocess the audio to be processed to obtain preprocessed audio data; Based on the preprocessed audio data, a preset processing unit is called to perform voice interaction processing to obtain a voice interaction result for voice interaction.

Images