Audio processing method and apparatus, electronic device, storage medium, program product

Abstract

The present disclosure provides an audio processing method and device, electronic equipment, storage medium and program product. The method comprises: in the case of detecting that an audio device is changed, obtaining device attribute information of the changed audio device; switching a first radio mode currently running to a second radio mode matched with the changed audio device; based on the device attribute information, collecting audio data based on the changed audio device in the second radio mode; and performing noise reduction processing corresponding to the second radio mode on the audio data to obtain target audio data. According to the embodiments of the present disclosure, in the case of changing the audio device, the current radio mode can be adaptively switched to the radio mode matched with the changed audio device, the collected audio data is processed by using the noise reduction processing mode corresponding to the switched radio mode, which is beneficial to the accuracy and efficiency of subsequent voice interaction between the target audio data and the user.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an audio processing method and apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Intelligent voice interaction is being used more and more widely, with its core goal being to achieve seamless and fluent dialogue between users and devices. This relies particularly on the real-time performance and robustness of the system's audio processing. To achieve this goal, real-time two-way voice interaction technology is typically used. This technology allows the system to simultaneously acquire and process the user's voice input while outputting audio signals, thereby simulating the continuity and responsiveness of natural human conversation.

[0003] At the technical implementation level, in order to ensure interaction quality in complex acoustic environments, voice interaction often relies on dedicated processing algorithms for specific audio hardware configurations (such as multi-microphone arrays). This tightly coupled hardware and software design pattern can fully utilize the characteristics of the device in a given hardware environment to achieve optimal audio front-end processing effects and provide users with a clear and stable dialogue experience.

[0004] However, in actual deployment and use, the audio channel between the user and the computing device is not static. Due to differences in usage scenarios, personal preferences, or privacy needs, users may dynamically switch between different audio input / output devices. When such a switch occurs, because the previous audio processing algorithm is deeply bound to the original hardware configuration, its processing performance may not be maintained under the new device configuration, resulting in unexpected changes or degradation in the acquisition and processing of audio data.

[0005] Therefore, it is crucial to reduce the impact on audio data acquisition and processing under dynamically changing audio hardware environments and maintain intelligent voice interaction. Summary of the Invention

[0006] This disclosure provides an audio processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0007] In a first aspect, this disclosure provides an audio processing method, including:

[0008] If an audio device change is detected, obtain the device attribute information of the changed audio device;

[0009] Switch the currently running first radio mode to a second radio mode that matches the changed audio device;

[0010] Based on the device attribute information, audio data is collected using the modified audio device in the second recording mode;

[0011] The audio data is subjected to noise reduction processing corresponding to the second recording mode to obtain the target audio data.

[0012] Secondly, this disclosure provides an audio processing apparatus, including:

[0013] The acquisition module is configured to acquire the device attribute information of the changed audio device when a change in audio device is detected.

[0014] The switching module is configured to switch the currently running first radio mode to a second radio mode that matches the changed audio device;

[0015] The acquisition module is configured to acquire audio data based on the changed audio device in the second recording mode, according to the device attribute information;

[0016] The noise reduction module is configured to perform noise reduction processing on the audio data corresponding to the second sound recording mode to obtain the target audio data.

[0017] Thirdly, this disclosure provides an electronic device comprising: at least one processor and an audio management interface; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to cause the at least one processor to perform the above-described audio processing method through the audio management interface.

[0018] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described audio processing method.

[0019] Fifthly, this disclosure provides a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described audio processing method.

[0020] The embodiments provided in this disclosure can detect audio devices in real time to determine whether the audio devices have changed. If a change is detected, the device attribute information of the changed audio device is obtained, and the currently running first recording mode is switched to a second recording mode that matches the changed audio device. Thus, based on the obtained device attribute information, audio data can be collected in the switched second recording mode based on the changed audio device, and the collected audio data can be further processed by the noise reduction processing method corresponding to the second recording mode to obtain the processed target audio data. During the audio acquisition phase, if the audio device changes, the current recording mode is adaptively switched to a mode that matches the new audio device. Audio data is then acquired based on the switched recording mode, ensuring uninterrupted audio data acquisition even after a device change, thus improving acquisition efficiency. In the audio processing phase, the noise reduction method corresponding to the switched recording mode is used to process the acquired audio data. This ensures that audio data acquired from different audio devices is processed using a matching method, improving the accuracy of audio processing and consequently the accuracy of the generated target audio data. This, in turn, benefits the accuracy and efficiency of subsequent voice interactions with users based on the target audio data.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0023] Figure 1 A flowchart of an audio processing method provided in this disclosure embodiment;

[0024] Figure 2 A flowchart illustrating an audio processing method provided in this embodiment of the disclosure;

[0025] Figure 3 A block diagram of an audio processing apparatus provided in an embodiment of this disclosure;

[0026] Figure 4 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0029] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0031] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0032] With the continuous development of artificial intelligence technology, digital human systems have gradually become an important direction for improving human-computer interaction. These systems typically possess a human-like appearance and intelligent dialogue capabilities, enabling natural communication with users through multimodal methods such as voice and vision. Among these, full-duplex voice interaction capability is key to achieving smooth, human-like dialogue, allowing the system to continuously collect and process user voice input while playing its own audio, thus achieving a seamless dialogue experience similar to human conversation.

[0033] To achieve high-quality full-duplex interaction, existing digital human systems often rely on specific hardware configurations and algorithmic collaboration. For example, they employ multi-microphone arrays (such as an eight-microphone array) as audio input devices, combined with dedicated hardware processing units and built-in acoustic echo cancellation (AEC) algorithms. This deep integration of hardware and software effectively suppresses ambient noise and echoes generated by the device's own audio playback, thus ensuring clear voice acquisition and real-time interaction even in complex acoustic environments.

[0034] In real-world applications, users may switch between different audio input / output devices as needed when interacting with computing devices. For example, in mobile work or privacy-protecting scenarios, users might switch from the built-in speaker and microphone to headphones. When this device switch occurs, the system's audio input source changes accordingly. Because algorithms previously optimized for high-quality full-duplex interaction (including noise reduction and echo cancellation) are typically deeply customized and bound to the hardware characteristics of multi-microphone arrays, these algorithms may not be directly compatible or continue to function when switching to devices with only a single microphone or other different acoustic characteristics.

[0035] According to the audio processing method of this disclosure, the audio device can be detected to determine whether the currently used audio device has changed. If the audio device has changed, the currently running first recording mode is switched to a second recording mode that matches the changed audio device, and the device attribute information of the changed audio device is obtained. Based on the device attribute information of the changed audio device, audio data is collected based on the changed audio device in the switched second recording mode, and the collected audio data is further subjected to noise reduction processing corresponding to the second recording mode to obtain target audio data. Thus, in subsequent voice interaction with the user, the processed target audio data can be used to conduct voice interaction. This achieves that different audio devices have matching recording modes to collect audio data in the audio acquisition stage, and different audio devices have matching noise reduction processing methods to process the audio data in the audio processing stage. This ensures that audio data acquisition and processing can continue even when the audio device changes, improving the robustness of audio data acquisition and processing and reducing the impact on user experience.

[0036] The audio processing method according to embodiments of this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, laptop computer, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0037] Figure 1 A flowchart illustrating an audio processing method provided in an embodiment of this disclosure. See also... Figure 1 The method specifically includes the following steps:

[0038] Step 102: If an audio device change is detected, obtain the device attribute information of the changed audio device.

[0039] In intelligent voice interaction scenarios, it is usually necessary to collect user voice and interact with an audio interaction system. In practical applications, the audio interaction system can be a digital human system, an interactive intelligent agent, or other intelligent system that can be used for human-computer interaction. The audio processing method provided in this disclosure can be applied to electronic devices that have deployed audio interaction systems. The types of electronic devices include, but are not limited to, the types mentioned above, and can be determined according to the actual application.

[0040] After the audio interaction system is started, it can interact with the user via voice. Simultaneously, the device deploying the audio interaction system can detect changes in the audio device. The audio device refers to the audio acquisition or playback device used for voice interaction with the audio interaction system, such as computer speakers, headphones, or other audio equipment. Device attribute information refers to the attribute information of the audio device, including the device serial number, the number of microphones in the device, and the microphone hardware identifier. Specifically, after the device runs the audio interaction system, during intelligent voice interaction with the user based on the audio interaction system, it can also monitor in real time whether the audio device used for interaction with the audio interaction system has changed. If a change is detected, the device attribute information of the changed audio device is obtained for subsequent audio data collection based on this information. For example, if a user interacts with a digital human system deployed in a laptop through the laptop's speakers, and then plugs in headphones, the audio device for intelligent voice interaction with the digital human system changes from the laptop's speakers to the headphones, and the headphone's device attribute information is obtained.

[0041] In this embodiment, two recording modes for intelligent voice interaction with the audio interaction system are provided: a multi-microphone recording mode and a single-microphone recording mode. The algorithm involved in the multi-microphone recording mode is bound to the hardware configuration of the electronic device itself, while the single-microphone recording mode can be applied to devices with other hardware configurations. Therefore, when a change in audio device is detected, the appropriate recording mode for the changed audio device can be determined based on the device attribute information of the changed audio device. Then, audio data is collected based on the changed audio device under the matching recording mode, and subsequent processing of the audio data is performed.

[0042] In practical applications, before the audio interaction system is started, the device cannot directly know the device attribute information of the audio device that is interacting with the audio interaction system for the first time. Therefore, after the audio interaction system is started, it is still necessary to obtain the device attribute information of the audio device when the audio interaction system is started, and determine the current recording mode to be run based on the device attribute information.

[0043] Based on this, in a specific embodiment provided in this disclosure, before detecting a change in the audio device, the method further includes: initializing an audio management interface; calling the audio management interface to obtain the current device attribute information of the current audio device; determining the device type of the current audio device based on the current device attribute information; and running the radio mode corresponding to the current audio device based on the device type.

[0044] The audio management interface refers to the operating system's underlying audio management interface, used to detect audio device change events and obtain audio device attribute information. The current audio device refers to the audio device initially used for intelligent voice interaction with the audio interaction system after startup. Current device attribute information refers to the device attribute information of the current audio device. Device types include single-microphone audio devices and multi-microphone audio devices, determined by the number of microphones in the device attribute information. A single-microphone audio device is an audio device with one microphone, such as headphones; a multi-microphone audio device is an audio device with more than one microphone, such as user equipment with an eight-microphone array hardware configuration.

[0045] Specifically, after the audio interaction system starts, it initializes the audio management interface and the software libraries (such as PyAudio, pulsectl, etc.) required for multi-microphone and single-microphone modes. It calls the audio management interface to obtain the current device attribute information of the current audio device and determines the device type based on this information. Therefore, it can run the corresponding recording mode according to the device type of the current audio device.

[0046] Taking a laptop as an example, if the user directly uses the laptop's speakers to interact with the audio interaction system after it starts up, the audio management interface is called to obtain the current device attribute information of the laptop's speakers. Based on the current device attribute information, the current audio device is determined to be the laptop's speakers, and the multi-microphone recording mode corresponding to the laptop's speakers is run. If the user plugs in headphones to interact with the audio interaction system after it starts up, the audio management interface is called to obtain the current device attribute information of the headphones. Based on the current device attribute information, the current audio device is determined to be the headphones, and the single-microphone recording mode corresponding to the headphones is run.

[0047] In this embodiment of the disclosure, after the audio interaction system is started, the current device attribute information of the current audio device can be obtained by calling the audio management interface, thereby determining the device type of the current audio device, and running the corresponding sound recording mode of the current audio device based on the device type. This enables different sound recording modes to be set for the current audio devices of different device types, improving the accuracy of audio data collection based on audio devices of different device types. At the same time, after the audio interaction system is started, the current audio device that is conducting intelligent voice interaction with the audio interaction system for the first time can be determined based on the current device attribute information, avoiding the direct running of the sound recording mode corresponding to the default audio device, and reducing the possibility of running the wrong sound recording mode.

[0048] Furthermore, as mentioned above, a single-microphone audio device refers to an audio device with one microphone, while a multi-microphone audio device refers to an audio device with more than one microphone. In other words, the device type of the current audio device is determined based on the number of microphones in the current device attribute information. Therefore, the specific implementation method for determining the device type of the current audio device based on the current device attribute information is as follows:

[0049] In one specific embodiment provided in this disclosure, determining the device type of the current audio device based on the current device attribute information includes: determining the number of microphones in the current device attribute information; if the number of microphones is equal to 1, determining the device type of the current audio device as a single-microphone audio device; if the number of microphones is greater than 1, determining the device type of the current audio device as a multi-microphone audio device.

[0050] Specifically, after obtaining the current device attribute information of the current audio device, the number of microphones contained in the current device attribute information is determined, and it is determined whether the number of microphones of the current audio device is greater than 1. If the number of microphones is not greater than 1 (i.e., the number of microphones is equal to 1), the device type of the current audio device can be determined to be a single-microphone audio device; if the number of microphones is greater than 1, the device type of the current audio device can be determined to be a multi-microphone audio device.

[0051] In this embodiment of the disclosure, the device type of the audio device is determined by determining the number of microphones in the audio device, which simplifies the process of determining the device type of the audio device and reduces the difficulty of determining the device type of the audio device.

[0052] Furthermore, as mentioned above, the recording modes include single-microphone recording mode and multi-microphone recording mode. After determining the device type of the audio device, the corresponding recording mode can be run according to the device type.

[0053] In one specific embodiment provided in this disclosure, the recording mode corresponding to the current audio device is run according to the device type, including: running the single-microphone recording mode when the device type is the single-microphone audio device; and running the multi-microphone recording mode when the device type is the multi-microphone audio device.

[0054] Specifically, if the current audio device is determined to be a single-microphone audio device, then the single-microphone recording mode matching the single-microphone audio device will be run; if the current audio device is determined to be a multi-microphone audio device, then the multi-microphone recording mode matching the multi-microphone audio device will be run.

[0055] In this embodiment, a recording mode matching the audio device is determined according to different device types, and the corresponding recording mode is run. This method is applicable to audio devices of various device types for audio data acquisition and processing.

[0056] Step 104: Switch the currently running first radio mode to a second radio mode that matches the changed audio device.

[0057] During the operation of the audio interaction system, the device can call the audio management interface to detect audio device change events in real time. When an audio device change event is detected, it indicates that an audio device change has occurred. At this time, the currently running first recording mode can be switched to a second recording mode that matches the changed audio device. It should be noted that the first and second recording modes can be either single-microphone or multi-microphone modes. That is, in step 104, the currently running single-microphone mode can be switched to a multi-microphone mode, or vice versa. The specific switching method is determined according to the actual application situation, and this disclosure does not limit it here.

[0058] This embodiment of the disclosure can monitor changes in audio devices in real time, and promptly switch to a matching radio mode when a change in audio devices is detected, so that different radio modes can be flexibly switched as audio devices change.

[0059] Step 106: Based on the device attribute information, collect audio data in the second recording mode based on the changed audio device.

[0060] After switching the currently operating first recording mode to a second recording mode that matches the changed audio device, audio data is collected based on the changed audio device in the second recording mode. The audio data includes user voice data, ambient noise, etc.

[0061] In one specific embodiment provided in this disclosure, audio data is collected based on the changed audio device in the second recording mode according to the device attribute information, including: determining the number of microphones and the device serial number in the device attribute information; determining the changed audio device according to the device serial number; if the device type of the changed audio device is determined to be a single-microphone audio device based on the number of microphones, audio data is collected based on the changed audio device in the single-microphone recording mode; if the device type of the changed audio device is determined to be a multi-microphone audio device based on the number of microphones, audio data is collected based on the changed audio device in the multi-microphone recording mode.

[0062] Specifically, the number of microphones and the device serial number are determined from the acquired device attribute information. Based on the device serial number, the changed audio device can be identified and specified for audio data collection, and subsequent processing of the audio data can be performed. As mentioned above, when the number of microphones is determined to be 1, the device type of the changed audio device is determined to be a single-microphone audio device. In this case, the second recording mode after switching is the single-microphone recording mode. Therefore, in the single-microphone recording mode, the user's voice data and environmental noise are collected based on the changed audio device (such as headphones plugged in by the user). When the number of microphones is determined to be greater than 1, the device type of the changed audio device is determined to be a multi-microphone device. In this case, the second recording mode after switching is the multi-microphone recording mode. Therefore, in the multi-microphone recording mode, the user's voice data and environmental noise are collected based on the changed audio device (such as the speakers of a laptop computer).

[0063] In this embodiment of the disclosure, after switching the changed audio device to a matching radio mode, audio data can be collected based on the changed audio device in the corresponding radio mode, thereby improving the flexibility of audio data collection while ensuring the usability of the changed audio device.

[0064] Furthermore, in a specific embodiment provided in this disclosure, acquiring audio data based on the modified audio device in the single-microphone recording mode includes: calling an audio input library and acquiring mono audio data based on the modified audio device.

[0065] Specifically, when the audio interaction system starts, all the software libraries required for different recording modes are initialized. When running the single-microphone recording mode, a general audio input library can be called, and its `open()` method can be further invoked to acquire mono audio data using a different audio device. In practical applications, the audio input library can be the PyAudio library or other audio acquisition libraries. The specific selection of the audio input library can be determined based on the actual application; this disclosure does not impose any limitations on it.

[0066] In this embodiment of the disclosure, by calling a general audio input library to collect audio data, the algorithm for collecting audio data in single-microphone recording mode can be applied to most audio devices, thereby improving audio device compatibility.

[0067] In one specific embodiment provided in this disclosure, the acquisition of audio data based on the changed audio device in the multi-microphone recording mode includes: calling a preset multi-channel audio acquisition tool to acquire audio data of multiple channels based on the changed audio device.

[0068] Specifically, when running the multi-microphone recording mode, a preset multi-channel audio acquisition tool can be invoked to acquire audio data from multiple channels using a different audio device. In practical applications, the multi-channel audio acquisition tool can be the preset parecord command-line tool. Through this parecord command-line tool, raw audio data from multiple independent channels can be acquired. The audio data acquired from these channels includes user voices, ambient noise, echoes generated by the device's own playback sound, and pure audio data from the sound card that only contains the device's own playback sound.

[0069] In this embodiment of the disclosure, by calling a preset multi-channel audio acquisition tool, audio data from multiple different channels are acquired, thereby improving the diversity and accuracy of the acquired audio data.

[0070] This embodiment of the disclosure achieves the following: determining the audio device needed to collect audio data based on device attribute information, and collecting audio data based on the determined audio device in the second recording mode matched by the audio device, thereby realizing the adaptation of the recording mode and the audio device and improving the accuracy of audio collection.

[0071] Step 108: Perform noise reduction processing on the audio data corresponding to the second recording mode to obtain the target audio data.

[0072] After the audio data is collected by the modified audio device, it is not pure human voice data; it also contains environmental noise, device echo, and other data. Therefore, further noise reduction processing is required to obtain the noise-reduced target audio data. The target audio data refers to the audio data used to generate the user's response audio after noise reduction processing; that is, the user's voice data contained within the collected audio data.

[0073] In this embodiment of the disclosure, different sound recording modes also correspond to different noise reduction processing methods. Therefore, for audio data collected by different audio devices in different sound recording modes, it is also necessary to perform corresponding noise reduction processing on the audio data based on the noise reduction processing method corresponding to the different sound recording modes, and obtain the target audio data after noise reduction processing.

[0074] Based on this, in a specific embodiment provided in this disclosure, the audio data is subjected to noise reduction processing corresponding to the second sound recording mode to obtain target audio data, including: inputting the mono audio data into a single-microphone audio processing unit; and performing noise reduction processing on the mono audio data based on the single-microphone audio processing unit to obtain target audio data.

[0075] The audio processing method provided in this embodiment includes an electronic device with a single-microphone audio processing unit and a multi-microphone audio processing unit. The single-microphone audio processing unit is used for noise reduction processing of mono audio data, and the multi-microphone audio processing unit is used for noise reduction processing of multi-channel audio data.

[0076] Specifically, after acquiring mono audio data using a single-microphone audio device, the mono audio data is input into a dedicated single-microphone audio processing unit. In this unit, noise reduction is performed on the mono audio data. In practical applications, when the single-microphone audio device is a headset, due to the physical structure of the headset, its microphone and speaker are close to the user's mouth, and its microphone and speaker are far from the device speaker of the audio interaction system. Therefore, the signal strength of the user's voice data acquired by the headset's microphone is much higher than the echo signal from the device speaker. Thus, when performing noise reduction on the mono audio data, only the mono audio data needs to be denoised, without echo cancellation. For example, spectral subtraction can be used to denoise the mono audio data, estimating and suppressing background noise in the frequency domain while preserving the main user voice components, or a noise threshold can be set for noise reduction. The algorithm for denoising the mono audio data can be determined based on the actual application, and this disclosure does not limit it. After noise reduction processing of the mono audio data, the target audio data can be obtained.

[0077] In this embodiment of the present disclosure, during the noise reduction processing of mono audio data based on the single-microphone audio processing unit, noise reduction processing can be performed only on mono audio data according to the physical structure characteristics of the single-microphone audio device itself, without performing more complex echo cancellation, reducing unnecessary processing steps, and improving the processing efficiency of noise reduction processing of mono audio data while ensuring the accuracy of the obtained target audio data.

[0078] In one specific embodiment provided in this disclosure, the audio data is subjected to noise reduction processing corresponding to the second sound recording mode to obtain target audio data, including: inputting the audio data of the multiple channels into a multi-microphone audio processing unit; performing noise reduction processing on the audio data of the multiple channels based on the multi-microphone audio processing unit, and removing echo data in the audio data of the multiple channels to obtain target audio data.

[0079] Specifically, after acquiring multi-channel audio data using a multi-microphone audio device, the multi-channel audio data is input into a dedicated multi-microphone audio processing unit. In this unit, noise reduction and echo cancellation are performed on the multi-channel audio data. Furthermore, as mentioned above, the acquired multi-channel audio data includes user voice data, environmental noise, device echo, and re-acquired clean audio data from the device. In practical applications, this audio data can be compared and adaptively filtered in the digital domain to remove echo data. Furthermore, noise reduction algorithms (such as spectral subtraction) are used to suppress environmental noise, thereby separating the user voice data from this audio data to obtain the target audio data.

[0080] In this embodiment, a multi-microphone audio data processing unit is used to perform noise reduction and echo cancellation on multi-channel audio data. This process removes noise and echoes mixed in with the multi-channel audio data, retaining only the user's voice and improving the accuracy of the obtained target audio data.

[0081] Regardless of whether the changed audio device is a single-microphone or multi-microphone audio device, after noise reduction processing of the audio data acquired by the audio device to obtain the target audio data, it is necessary to perform corresponding audio processing on the target audio data to generate a response audio corresponding to the target audio data. The response audio refers to the audio used to reply to the target audio data.

[0082] Based on this, in a specific embodiment provided in this disclosure, the method further includes: processing the target audio data based on an audio processing model to obtain a response audio; and outputting the response audio to the modified audio device so that the modified audio device plays the response audio.

[0083] The audio processing model refers to the model used to generate response audio for the target audio data. This model is deployed within the audio interaction system and can be a speech dialogue model or a speech recognition model in practical applications. Specifically, after obtaining the processed target audio data, the audio interaction system can input the target audio data into the audio processing model for processing and generate response audio to address the target audio data. Then, the response audio is output to a designated audio device and played through that device to complete the current round of voice interaction.

[0084] For example, after generating the response audio of the target audio data based on the audio processing model, if the changed audio device is the speaker of a laptop, the response audio will be output to the speaker and played through the speaker; if the changed audio device is headphones, the response audio will be output to the headphones and played through the headphones.

[0085] In this embodiment of the disclosure, after obtaining the target audio data, the target audio data is further processed based on the audio processing model to generate corresponding response data. The response data is then played through a modified audio device to complete intelligent voice interaction with the user and improve the user experience.

[0086] This embodiment of the disclosure implements real-time detection of audio devices via an audio management interface to determine if the audio device has changed. If a change is detected, the device attribute information of the changed audio device is obtained, and the currently running first recording mode is adaptively switched to a second recording mode matching the changed audio device. Therefore, based on the obtained device attribute information, audio data can be collected in the switched second recording mode according to the changed audio device. This ensures that audio data collection can continue even after a change in the audio device, guaranteeing uninterrupted audio data collection and improving audio data collection efficiency. Through the second... The noise reduction processing method corresponding to the sound reception mode performs noise reduction processing on the acquired audio data to obtain the processed target audio data. This ensures that audio data acquired from different audio devices is processed using a matching audio processing method, improving the accuracy of audio data processing, and thus improving the accuracy of the generated target audio data. The audio processing model generates a corresponding response audio for the target audio data and outputs the response audio to the changed audio device. The changed audio device plays the response audio, completing the intelligent voice interaction between the audio interaction system and the user, ensuring the accuracy and efficiency of voice interaction with the user, and improving the user experience.

[0087] The following is in conjunction with the appendix Figure 2 The audio processing method provided in the embodiments of this disclosure will be further explained and described below. Figure 2 A flowchart of an audio processing method provided in this disclosure embodiment is shown below. Figure 2 As shown, after the audio interaction system starts, the software library and audio management interface are initialized. Figure 2 (Not shown in the image) The audio management interface is called to detect whether the audio device has changed. If a change is detected, the device attribute information of the changed audio device is obtained, including the number of microphones, device serial number, and microphone hardware identifier. It is then determined whether the number of microphones of the changed audio device is greater than 1. If the number of microphones is greater than 1, multi-microphone recording mode is run for the changed audio device. Specifically, the audio device to be changed for subsequent audio acquisition and processing is determined based on the device serial number. A preset multi-channel audio acquisition tool (such as the Parecord command-line tool) is invoked to acquire audio data from multiple channels (including user voice, ambient noise, echoes generated by the device's own playback, and pure audio data from the sound card containing only the device's own playback). The acquired multi-channel audio data is input into the multi-microphone audio processing unit, where noise reduction and echo cancellation are performed to obtain the target audio data. The target audio data is then input into the audio processing model for processing to generate the response audio. The response audio is output to the changed audio device and played through the changed audio device, thus completing the current round of intelligent voice interaction with the user. The system continues to detect whether the audio device has changed until the intelligent voice interaction process ends.

[0088] Taking the example of a user plugging and unplugging headphones from a computer, an audio device change occurs when the user unplugs the headphones. The changed audio device is now the computer's speakers. The system acquires the number of microphones, device serial number, and microphone hardware identifier of the computer's speakers. It checks if the number of microphones on the computer's speakers is greater than one. Since the number is greater than one, a multi-microphone recording mode is activated. Specifically, based on the device serial number, the audio device for subsequent audio acquisition and processing is determined to be the computer's speakers. The preset `parecord` command-line tool is called to acquire audio data from multiple channels. This multi-channel audio data is input to the multi-microphone audio processing unit, where noise reduction and echo cancellation are performed to obtain the target audio data. This target audio data is then input into the digital human system's audio processing model for further processing, generating a response audio. This response audio is output to the computer's speakers and played through the speakers, completing the current round of intelligent voice interaction with the user. The system continues to detect audio device changes until the intelligent voice interaction process ends.

[0089] Furthermore, with only one microphone, a single-microphone recording mode is run for the changed audio device. Specifically, the changed audio device for subsequent audio acquisition and processing is determined based on the device serial number. An audio input library (e.g., PyAudio library) is used to acquire mono audio data. The acquired mono audio data is input into the single-microphone audio processing unit, where noise reduction is performed without echo cancellation to obtain the target audio data. This target audio data is then input into the audio processing model for processing, generating a response audio. The response audio is output to the changed audio device and played through the changed audio device, thus completing the current round of intelligent voice interaction with the user. The system continues to monitor for changes in the audio device until the intelligent voice interaction process ends.

[0090] Continuing with the example of a user plugging and unplugging headphones into a computer, when the user plugs in the headphones, an audio device change occurs, and the changed audio device is the headphones. The system retrieves the number of microphones, device serial number, and microphone hardware identifier of the headphones. It checks if the number of microphones on the headphones is greater than one. Since the number of microphones on the headphones is equal to one, the single-microphone recording mode is activated. Specifically, based on the device serial number, the audio device for subsequent audio acquisition and processing is determined to be the headphones. The PyAudio library is called to acquire mono audio data, which is then input into the single-microphone audio processing unit. Noise reduction processing is performed on the mono audio data in the single-microphone audio processing unit to obtain the target audio data. Then, based on the same process as the multi-microphone recording mode described above, the current round of intelligent voice interaction with the user is completed, and the system continues to detect whether the audio device has changed until the intelligent voice interaction process ends.

[0091] This embodiment implements the following: Real-time detection of audio devices is achieved by calling an audio management interface to determine if the audio device has changed. If a change is detected, the device attribute information of the changed audio device is obtained, and the recording mode is adaptively switched to improve flexibility in switching recording modes. Based on the obtained device attribute information, audio data is collected using the changed audio device in the switched recording mode, ensuring uninterrupted audio data collection even after a device change, thus improving the efficiency of audio data collection. The collected audio data is then denoised using a noise reduction method corresponding to the recording mode to obtain processed target audio data. This ensures that audio data collected from different audio devices is processed using a matching audio processing method, improving the accuracy of generating target audio data. Finally, a corresponding response audio is generated for the target audio data using an audio processing model, and the response audio is output to the changed audio device, ensuring the accuracy and efficiency of voice interaction with the user and improving the user experience.

[0092] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0093] In addition, this disclosure also provides audio processing apparatus, electronic devices, computer-readable storage media, and computer program products, all of which can be used to implement the audio processing method provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section, and will not be repeated here.

[0094] Figure 3 This is a block diagram of an audio processing apparatus provided in an embodiment of the present disclosure.

[0095] See Figure 3 This disclosure provides an audio processing apparatus, which includes:

[0096] The acquisition module 302 is configured to acquire the device attribute information of the changed audio device when a change in audio device is detected.

[0097] The switching module 304 is configured to switch the currently running first radio mode to a second radio mode that matches the changed audio device;

[0098] The acquisition module 306 is configured to acquire audio data based on the changed audio device in the second sound recording mode, according to the device attribute information.

[0099] The noise reduction module 308 is configured to perform noise reduction processing on the audio data corresponding to the second sound recording mode to obtain the target audio data.

[0100] Optionally, the device further includes an operating module configured to:

[0101] Initialize the audio management interface;

[0102] Call the audio management interface to obtain the current device attribute information of the current audio device;

[0103] Based on the current device attribute information, determine the device type of the current audio device;

[0104] Based on the device type, run the radio mode corresponding to the current audio device.

[0105] Optionally, the operating module is further configured to:

[0106] Determine the number of microphones in the current device attribute information;

[0107] When the number of microphones is equal to 1, the device type of the current audio device is determined to be a single-microphone audio device;

[0108] If the number of microphones is greater than 1, the device type of the current audio device is determined to be a multi-microphone audio device.

[0109] Optionally, the radio reception mode includes a single-microphone reception mode or a multi-microphone reception mode;

[0110] The running module is further configured as follows:

[0111] When the device type is the single-microphone audio device, the single-microphone recording mode is run;

[0112] When the device type is the multi-microphone audio device, the multi-microphone recording mode is run.

[0113] Optionally, the second microphone mode is a single-microphone microphone mode or a multi-microphone microphone mode;

[0114] The acquisition module 306 is further configured as follows:

[0115] Determine the number of microphones and the device serial number in the device attribute information;

[0116] The changed audio device is determined based on the device serial number;

[0117] If the device type of the changed audio device is determined to be a single-microphone audio device based on the number of microphones, audio data is collected based on the changed audio device in the single-microphone recording mode;

[0118] If the device type of the changed audio device is determined to be a multi-microphone audio device based on the number of microphones, audio data is collected based on the changed audio device in the multi-microphone recording mode.

[0119] Optionally, the acquisition module 306 is further configured to:

[0120] Call the audio input library and collect mono audio data based on the modified audio device;

[0121] The noise reduction module 308 is further configured as follows:

[0122] The mono audio data is input into the single-microphone audio processing unit;

[0123] The single-microphone audio processing unit performs noise reduction processing on the mono audio data to obtain the target audio data.

[0124] Optionally, the acquisition module 306 is further configured to:

[0125] The preset multi-channel audio acquisition tool is invoked to acquire audio data from multiple channels based on the changed audio device;

[0126] The noise reduction module 308 is further configured as follows:

[0127] The audio data of the multiple channels is input into the multi-microphone audio processing unit;

[0128] The multi-microphone audio processing unit performs noise reduction processing on the audio data of the multiple channels and removes echo data from the audio data of the multiple channels to obtain the target audio data.

[0129] Optionally, the device further includes an output module configured to:

[0130] The target audio data is processed based on an audio processing model to obtain the response audio;

[0131] The response audio is output to the modified audio device so that the modified audio device plays the response audio.

[0132] The audio processing apparatus provided in this embodiment includes: an acquisition module configured to acquire device attribute information of the changed audio device when a change in audio device is detected; a switching module configured to switch a currently running first recording mode to a second recording mode matching the changed audio device; an acquisition module configured to acquire audio data based on the changed audio device in the second recording mode according to the device attribute information; and a noise reduction module configured to perform noise reduction processing on the audio data corresponding to the second recording mode to obtain target audio data.

[0133] This embodiment of the disclosure implements a method that calls an audio management interface to detect audio devices in real time to determine if the audio devices have changed. If a change is detected, the device attribute information of the changed audio device is obtained, and the currently running first recording mode is adaptively switched to a second recording mode matching the changed audio device. Therefore, based on the obtained device attribute information, audio data can be collected in the switched second recording mode according to the changed audio device. This ensures that audio data collection can continue even after a change in the audio device, guaranteeing uninterrupted audio data collection and improving audio data collection efficiency. The system performs noise reduction on the acquired audio data using the noise reduction method corresponding to the second sound recording mode, resulting in processed target audio data. This ensures that audio data acquired from different audio devices is processed using a matching audio processing method, improving the accuracy of audio data processing and thus the accuracy of the generated target audio data. The system then generates corresponding response audio for the target audio data using an audio processing model and outputs the response audio to a different audio device. Playing the response audio through the different audio device completes the intelligent voice interaction between the audio interaction system and the user, ensuring the accuracy and efficiency of voice interaction and improving the user experience.

[0134] Figure 4 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0135] See Figure 4 This disclosure provides an electronic device 400, which includes: at least one processor 401 and an audio management interface (...). Figure 4 (not shown in the image); at least one memory 402 and one or more I / O interfaces 403 are connected between the processor 401 and the memory 402; wherein the memory 402 stores one or more computer programs that can be executed by at least one processor 401, and the one or more computer programs are executed by at least one processor 401 to cause at least one processor 401 to perform the above-described audio processing method through the audio management interface.

[0136] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the above-described audio processing method. The computer-readable storage medium may be volatile or non-volatile.

[0137] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described audio processing method.

[0138] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0139] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0140] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0141] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0142] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0143] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0144] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0145] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0147] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. An audio processing method, characterized in that, include: If an audio device change is detected, obtain the device attribute information of the changed audio device; Switch the currently running first radio mode to a second radio mode that matches the changed audio device; Based on the device attribute information, audio data is collected using the modified audio device in the second recording mode; The audio data is subjected to noise reduction processing corresponding to the second recording mode to obtain the target audio data.

2. The method as described in claim 1, characterized in that, Before detecting an audio device change, the method further includes: Initialize the audio management interface; Call the audio management interface to obtain the current device attribute information of the current audio device; Based on the current device attribute information, determine the device type of the current audio device; Based on the device type, run the radio mode corresponding to the current audio device.

3. The method as described in claim 2, characterized in that, Based on the current device attribute information, the device type of the current audio device is determined, including: Determine the number of microphones in the current device attribute information; When the number of microphones is equal to 1, the device type of the current audio device is determined to be a single-microphone audio device; If the number of microphones is greater than 1, the device type of the current audio device is determined to be a multi-microphone audio device.

4. The method as described in claim 3, characterized in that, The sound reception mode includes single-microphone reception mode or multi-microphone reception mode. Based on the device type, the corresponding radio mode for the current audio device is run, including: When the device type is the single-microphone audio device, the single-microphone recording mode is run; When the device type is the multi-microphone audio device, the multi-microphone recording mode is run.

5. The method as described in claim 1, characterized in that, The second microphone mode is either a single-microphone microphone mode or a multi-microphone microphone mode; Based on the device attribute information, in the second recording mode, audio data is collected based on the changed audio device, including: Determine the number of microphones and the device serial number in the device attribute information; The changed audio device is determined based on the device serial number; If the device type of the changed audio device is determined to be a single-microphone audio device based on the number of microphones, audio data is collected based on the changed audio device in the single-microphone recording mode; If the device type of the changed audio device is determined to be a multi-microphone audio device based on the number of microphones, audio data is collected based on the changed audio device in the multi-microphone recording mode.

6. The method as described in claim 5, characterized in that, In the single-microphone recording mode, audio data is collected based on the modified audio device, including: Call the audio input library and collect mono audio data based on the modified audio device; The audio data is subjected to noise reduction processing corresponding to the second recording mode to obtain target audio data, including: The mono audio data is input into the single-microphone audio processing unit; The single-microphone audio processing unit performs noise reduction processing on the mono audio data to obtain the target audio data.

7. The method as described in claim 5, characterized in that, In the multi-microphone recording mode, audio data is collected based on the changed audio device, including: The preset multi-channel audio acquisition tool is invoked to acquire audio data from multiple channels based on the changed audio device; The audio data is subjected to noise reduction processing corresponding to the second recording mode to obtain target audio data, including: The audio data of the multiple channels is input into the multi-microphone audio processing unit; The multi-microphone audio processing unit performs noise reduction processing on the audio data of the multiple channels and removes echo data from the audio data of the multiple channels to obtain the target audio data.

8. The method as described in claim 1, characterized in that, The method further includes: The target audio data is processed based on an audio processing model to obtain the response audio; The response audio is output to the modified audio device so that the modified audio device plays the response audio.

9. An audio processing device, characterized in that, include: The acquisition module is configured to acquire the device attribute information of the changed audio device when a change in audio device is detected. The switching module is configured to switch the currently running first radio mode to a second radio mode that matches the changed audio device; The acquisition module is configured to acquire audio data based on the changed audio device in the second recording mode, according to the device attribute information; The noise reduction module is configured to perform noise reduction processing on the audio data corresponding to the second sound recording mode to obtain the target audio data.

10. An electronic device, characterized in that, include: At least one processor and audio management interface; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more of the computer programs being executed by the at least one processor to cause the at least one processor to perform the method as described in any one of claims 1-8 through the audio management interface.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device performs the method as described in any one of claims 1-8.

Images