Voice processing system and method and vehicle

By performing echo cancellation and noise reduction processing in the vehicle cockpit, the problems of voice processing delay and resource waste in the prior art are solved, and efficient multi-user voice interaction and directional sound pickup playback are achieved.

CN119068896APending Publication Date: 2024-12-03SAIC GENERAL MOTORS +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411332537.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-24
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing vehicle cockpit voice processing system has problems such as delay, bandwidth usage and waste of computing power resources in multi-user voice interaction scenarios, especially in multi-tone sound pickup and echo cancellation processing.

Method used

An operating system-independent audio processing module is designed, including an echo cancellation and noise reduction module, directed sound pickup and playback through an array microphone and digital signal processor, and directly performs ECNR functions in Audio DSP, reducing transmission links and Improve processing speed.

Benefits of technology

It significantly shortens the voice processing delay, saves bandwidth and computing resources, improves the quality and processing speed of voice interaction, and supports targeted voice interaction in multi-user scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068896A_ABST
    Figure CN119068896A_ABST
Patent Text Reader

Abstract

The invention relates to a voice processing system, a voice processing method, a vehicle including the system, and a computer readable storage medium capable of performing the method. The speech processing system includes a speech pickup module for picking up audio from one or more pickup areas within a target space; the audio playing module is used for playing audio to one or more playing areas in the target space; the operating system is used for processing various tasks including a voice processing task; and the audio processing module is independent of the operating system. By setting an echo cancellation and / or noise reduction function on the audio processing module, the link transmission time delay is shortened, the bandwidth occupation is reduced, and the effect of voice communication in the cabin is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to speech processing technology, and more specifically to speech processing technology for vehicle cockpits. Background Art

[0002] With the development of smart cars, cockpit functions are becoming more and more diverse. In-car multi-screen display, multi-zone voice recognition, headrest speakers and other functions are becoming more and more common. The scene functions in the vehicle cockpit are becoming more and more abundant, and there are more possible experience scalability in scenes such as car phone and conference. Large-space vehicles, such as MPVs (multi-Purpose Vehicles) and other multi-seat models are particularly popular in the market. These models often involve multi-user voice interaction scenarios. How to optimize the interactive experience and improve the processing speed and voice interaction quality is a concern of technicians in this field.

[0003] The information provided in this section is for the purpose of generally presenting the background of the present application and therefore may contain information that does not constitute prior art in the art. Summary of the invention

[0004] The purpose of this application is to solve or at least alleviate one or more problems existing in the prior art.

[0005] One aspect of the present application relates to a speech processing system, comprising:

[0006] One or more voice pickup modules, which are used to pick up audio from one or more pickup areas in the target space,

[0007] One or more audio playback modules, which are used to play audio to one or more playback areas in a target space, where the target space may be a vehicle cabin,

[0008] An operating system that handles a variety of tasks, including speech processing tasks,

[0009] an audio processing module, which is independent of the operating system,

[0010] The audio processing module is used to perform echo cancellation and / or noise reduction and / or integrate the picked-up one or more audio channels into one audio channel, and send the processed audio to a subsequent processing unit.

[0011] The audio processing module processes the audio from one or more audio sources and sends the processed audio to the one or more audio playing modules.

[0012] In an embodiment of the present application, optionally, the subsequent processing unit includes the operating system and the Bluetooth module, and the audio processing module includes a first audio processing module and a second audio processing module, the first audio processing module is configured to perform echo elimination and / or noise reduction processing on one or more picked up audio channels, and the second audio processing module is configured to process the audio from the one or more audio sources for playback, and obtain a reference sound of the target space based on the audio from the one or more audio sources for echo elimination.

[0013] In an embodiment of the present application, optionally, the second audio processing module processes the audio from the one or more audio sources in one or more of the following ways: mixing, copying, and splitting.

[0014] In an embodiment of the present application, optionally, the audio obtained after processing by the first audio processing module is a human voice in the sound pickup area.

[0015] In an embodiment of the present application, optionally, the voice pickup module is an array microphone, the audio playback module is a headrest speaker or an in-car speaker, and the audio processing module is a digital signal processor.

[0016] In an embodiment of the present application, optionally, the audio processing module includes a beamforming module to perform directionally picking up sound in a sound pickup area in a target space.

[0017] In an embodiment of the present application, optionally, the operating system controls the audio processing module according to a set voice mode to select a pickup area / playback area to pick up / play audio.

[0018] Another aspect of the present application relates to a vehicle, comprising a speech processing system according to any one of the foregoing contents.

[0019] Another aspect of the present application relates to a speech processing method for a speech processing system as described in any one of the foregoing contents, comprising the following steps:

[0020] Determine one or more pickup zones and / or one or more playback zones based on the speech pattern,

[0021] Pick up audio from one or more pickup zones,

[0022] Perform echo cancellation and / or noise reduction and / or integrate the picked-up one or more audio channels into one audio channel.

[0023] Sending the processed audio to a subsequent processing unit;

[0024] Process audio from one or more audio sources and send it to one or more playback zones.

[0025] The present application also relates to a computer-readable storage medium on which computer instructions are stored, characterized in that the computer instructions implement the steps of the method described in the above content when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The disclosure of the present application will become more easily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. In addition, similar numbers in the drawings represent similar components, wherein:

[0027] Figure 1 An example of a speech processing system in the prior art is shown;

[0028] Figure 2 An example of a cockpit voice system configuration in the prior art is shown;

[0029] Figure 3 An embodiment of the speech processing system of the present application is shown;

[0030] Figure 4 An embodiment of the cockpit voice system configuration of the present application is shown;

[0031] Figure 5-6 Examples of application scenarios of the cockpit voice system configuration of the present application are shown respectively. DETAILED DESCRIPTION

[0032] First of all, it should be noted that the following will illustrate the composition, characteristics and advantages of the speech processing system, method and vehicle according to the present application by way of example, but it should be understood that all descriptions are given for illustration only and should not be construed as forming any limitation on the present application. In this article, the technical terms "first" and "second" are only used for the purpose of distinguishing expressions and are not intended to indicate their order and relative importance. The technical term "connection (or connection, etc.)" covers the direct connection of a specific component to another component and / or indirect connection to another component. In addition, unless otherwise clearly specified and limited, the dimensions, directions or positional relationships indicated by the technical terms "length", "width", "height", "upper", "top", "bottom" and the like are based on the dimensions, orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and cannot be understood as limitations on the present application.

[0033] In addition, for any single technical feature described or implied in the embodiments mentioned herein, or any single technical feature shown or implied in the drawings, the present application still allows any combination or deletion between these technical features (or their equivalents) without any technical obstacles, thereby obtaining more other embodiments of the present application that may not be directly mentioned herein. In addition, general matters that are already known to those skilled in the art will not be repeated herein.

[0034] With the continuous development of vehicle intelligence, the vehicle cockpit has become the third space of life, and its functions are becoming increasingly rich. The rear row can realize different meeting scenes in the cockpit through the rear screens such as seatback screens, ceiling screens, rear microphones, headrest speakers and other hardware configurations, combined with the network conferencing software installed in the vehicle operating system. In terms of voice function, it can provide Bluetooth calls, single or multi-person conferences. For models with four or even six-tone zone voice functions, in order to pick up the voices of rear passengers, generally 1 to 4 microphones are arranged in the second or even third row to help pick up the sound.

[0035] In the development of vehicle intelligent cockpits, the trend of integration is becoming increasingly obvious, and in the cockpit SoC (System on Chip, system-level chip), Qualcomm 8155, 8295 and other SoC chips that integrate Audio DSP (Audio Digital Signal Processor) chips have become mainstream. When picking up the voices of people in the cockpit, ECNR (Echo Cancellation & Noise Reduction) processing is required. In the prior art, the ECNR function is usually implemented at the operating system level (such as Android HAL, etc.), and the audio must be uploaded from the digital signal processor to the operating system. The transmission link is long and cannot meet some low-latency requirements. In addition, when multiple sound zones need to be picked up, the number of transmission links increases, which will occupy more resources such as bandwidth and computing power.

[0036] like Figure 1As shown, it is an example of a voice processing system in the prior art. The Hypervisor (virtual machine manager) implements the dual operating systems of QNX+Android in the ARM (Advanced RISC Machine) core, where the QNX operating system is mainly responsible for the instrument function, and Android is mainly responsible for IVI (In-Vehicle Infotainment), processing entertainment, navigation, network communication and other tasks, which also includes the function of conference APP (application), so that the passengers in the cockpit can conduct online voice and video conferences. The Audio DSP transmits the audio picked up from the car through the microphone and other hardware to the Android operating system, and the picked up audio is processed by the ECNR module in the Android operating system. When the cockpit passengers use the phone / conference APP to hold a meeting, the audio processed by the ECNR module is further processed by the conference APP; in the scenario of connecting the mobile phone Bluetooth (for example, for calls), the audio processed by the ECNR module can be sent to the user's mobile phone through the Bluetooth module. The Audio DSP also processes the remote audio from the conference APP and the sounds from various other media audio sources (such as music players, navigation prompts, etc.), such as mixing, copying, splitting, etc., and sends the processed audio downstream to the external / internal AMP power amplifier, and then plays the audio to the cockpit through hardware such as the in-car speakers. Among them, the Audio DSP copies the audio from each audio source to be played in the cockpit as a reference sound for the ECNR module to eliminate echoes.

[0037] Figure 2 An example of the configuration of the voice system in the vehicle cockpit in the prior art is shown. In the cockpit, there are only 1-2 microphones at the main driver and co-driver positions to undertake the function of single / dual-zone voice or telephone conversation. A certain number of microphones can be set up in the front and rear rows of the vehicle to play audio into the cockpit, and the rear row does not have a sound pickup device.

[0038] There are such deficiencies in the prior art. For example, when connecting a car’s Bluetooth to a mobile phone, the picked up audio must be uploaded to the operating system through the Audio DSP, and then echo cancellation and noise reduction are performed in the operating system. The human voice processed by the ECNR module is sent to the Bluetooth module to be sent to the user’s mobile phone via Bluetooth. This transmission link is long and the transmission delay is long. Taking the voice processing process of the conference APP as an example, the audio picked up from the microphone in the car (especially multi-channel audio) is sent to the ECNR module on the Android side via the Audio DSP for echo cancellation and noise reduction. The processed audio is then sent to, for example, the cloud via the conference APP. The conference APP further sends the remote audio from the cloud (such as audio from participants not in the cockpit) downlink to the Audio DSP for playback in the car. In addition, it should be noted that Figure 1 A simplified diagram of picking up one audio channel is schematically shown in FIG. 1 . When picking up audio from multiple locations, there will be multiple upload links in the upstream direction from the Audio DSP to the operating system, which means that more bandwidth will be occupied.

[0039] The present application proposes such a solution to solve or improve at least one of the problems existing in the prior art.

[0040] refer to Figure 3 In one embodiment of the speech processing system of the present application, an audio processing module 10 is included, which is specifically an Audio DSP, and the audio processing module 10 is a module independent of the operating system 20. The Audio DSP includes a first audio processing module 11, specifically an ECNR (echo cancellation & noise reduction) module, which is used to perform echo cancellation and noise reduction processing on the audio picked up by the voice pickup module 31. The Audio DSP also includes a second audio processing module 12, which processes the audio from various sound sources (such as the voices of non-cockpit participants in the phone / conference APP, music players, navigation prompts, and audio received when the Bluetooth connection is connected to the mobile phone). The second audio processing module 12 mixes, copies, splits, and processes the audio from the sound source and further sends it downlink to the audio playback module 32. In this embodiment, the audio playback module 32 includes a headrest speaker, which can broadcast audio to the occupants at a specified position (such as remote audio from the phone / conference APP). Of course, the audio playback module 32 may also include an in-car speaker. The second audio processing module 12 also copies and splits the audio from each audio source to obtain reference audio, which is used in the echo cancellation process in the first audio processing module 11 .

[0041] Among them, the voice pickup module 31 is Figure 3In the embodiment, it is specifically an array microphone, and may include one or more array microphones. Different array microphones can pick up audio at different positions in the cockpit, mainly human voices at the positions, such as the voices of passengers at different seats in the cockpit, and send these audios to the ECNR module in the Audio DSP for echo cancellation and noise reduction processing, and integrate them into one audio channel and further send them to the operating system 20. In one embodiment, people in the cockpit participate in the meeting through the operating system, and the audio from the ECNR module is sent to the Android operating system. They can participate in the meeting through the application in the Android operating system, and people at designated positions in the cockpit can be set to participate in the meeting. In this way, the array microphone can pick up the audio at these designated positions. For example, in Figure 4-6 As shown in the figure, the cockpit may be provided with two rows of screens for use by second-row passengers, such as a ceiling screen, a partition screen and a seatback screen, etc., wherein the seatback screen may be arranged behind the main driver's and / or co-driver's seat for use by passengers in the left and right positions of the second row respectively. Passengers may set the voice mode by operating each screen to select the position to pick up / play the audio. The operating system will send the set voice mode to the ECNR module through a parameter command to control the directional pickup and playback of the audio.

[0042] Each array microphone usually includes multiple microphone heads. The audio picked up by the multiple microphone heads is transmitted to the Audio DSP through the digital audio bus module. In the Audio DSP, the picked up audio is beamformed by the beamforming module to focus and locate the audio picked up by the array microphone. This can achieve a better directional sound reception effect for the designated sound pickup area in the cockpit and avoid interference from sounds at other locations.

[0043] After being processed by the Audio DSP / second audio processing module, the audio from the sound source is further sent downstream to the headrest speaker 32 via the digital audio bus module and the external AMP power amplifier, and can be sent to the headrest speaker 32 at a specified location for directional playback. In addition, the Audio DSP can also send audio from a specific sound source to the headrest speaker, for example, only sending the audio in the conference APP to the headrest speaker, while excluding the audio from the music player and the navigation prompt sound.

[0044] Although not shown in the figure, it should be noted that the Audio DSP can also process the remote audio from the Bluetooth module and send it to the cabin space, such as the headrest speakers or the speakers in the car, so that the passengers in the cabin can make mobile phone calls through the Bluetooth module. The remote audio in this article generally refers to the audio sent to the cabin through the network or other communication methods other than the passengers in the cabin, such as the audio of other participants who are not members of the car and are holding an online conference with the passengers in the car through the conference APP, and the audio of the person on the other end of the mobile phone who is making a mobile phone Bluetooth call with the passengers in the car.

[0045] exist Figure 5 and Figure 6 In the figure, the gray filled area indicates the activated pickup area and playback area. Figure 5 The middle sound pickup and playback area is the left and right positions of the second row. Audio is played through the headrest speakers at these two positions. Figure 6 In the scenario shown, the sound pickup area and playback area are the front driver's seat, the co-driver's seat, and the left side of the back row, and audio is played through the headrest speakers at these three locations. It can be seen that the voice processing system of the present application can achieve directional voice interaction in one or more designated areas in the cabin space. Of course, in addition to the headrest speakers, directional or non-directional audio playback can also be performed through the in-car speaker 33.

[0046] It can be seen that there are multiple audio channels picked up in the cockpit at the same time. Since the multiple audio channels are processed in the ECNR module and integrated into one channel for uploading, the bandwidth usage can be significantly reduced, saving algorithm computing resources. In addition, since the ECNR function is directly executed in the Audio DSP, the link is shortened, the delay is reduced, and the processing speed is improved. Figure 1 The prior art in Figure 2 In the solution, the acquisition of reference sound and echo cancellation are both performed in the Audio DSP, eliminating the link of uploading the reference sound to the operating system, further improving the processing speed.

[0047] The operating system of the vehicle chip is divided into two independent operating systems by the virtual machine manager, the first operating system 21 (specifically the QNX operating system) and the second operating system 22 (specifically the Android operating system), wherein the QNX operating system has advantages in real-time, stability and security, while the Android operating system can provide more abundant applications. According to the characteristics of the two operating systems, the operating system can be selected for task processing according to the type of voice processing requirements. For example, voice processing such as vehicle rescue calls that require a higher level of reliability can be processed on the QNX side, while functions such as video conferencing and voice calls in normal conditions can be processed on the Andoid side. In the traditional solution, the ECNR module is located on the Android side, which cannot perform echo cancellation and noise reduction processing on the voice processing task on the QNX side, or an ECNR module needs to be added on the QNX side to meet this function. In the solution of the present application, the data processed by the ECNR module can be sent to the Android side or used on the QNX side, and transmitted to the remote communication module on the QNX side through eAVB (Ethernet Audio Video Bridging) audio for voice interaction with the outside world, saving system resources and enabling the voice processing task on the QNX side to achieve better results.

[0048] The whole vehicle conference or call scene in the prior art is not designed for a variety of scenarios, and cannot meet the needs of users in various scenarios. The technical solution of this application can realize the following different scenarios: the main driver's single call does not affect other users, the co-driver or the back row can join the call, and the rear passengers can hold single and multi-person meetings. Since the Audio DSP can access multiple audio peripheral modules as an audio processing core, such as the A2B (Automotive Audio Bus) digital audio bus module, etc., it can access multiple array microphones and vehicle-mounted external AMP power amplifiers through the digital audio bus, which can expand the audio input and output ports in the cockpit, thereby realizing the pickup and playback of multiple audio channels. For multi-user voice interaction scenarios with multiple passengers, passengers in each seat can select the pickup / playback mode of the phone / conference APP through the central control screen, the ceiling / partition screen in the back row, and the back seat back screen. After obtaining the mode selected by the user, the parameter command can be issued and set through the specific interface defined between Android and Ausio DSP to activate the tuning parameters of the specific scene preset in the ECNR module. For example, in a scenario where only the back row attends the meeting, the ECNR module only picks up sound from the two array microphones in the back row, ignores the sound from the other two array microphones, and after processing, selects the headrest speakers in the back row to produce sound.

[0049] The specific embodiments described above in this application are only for more clearly describing the principles of this application, wherein each component is clearly shown or described to make the principles of this application easier to understand. Without departing from the scope of this application, those skilled in the art can easily make various modifications or changes to this application. Therefore, it should be understood that these modifications or changes should be included in the scope of patent protection of this application.

Claims

1. A speech processing system, comprising: One or more voice pickup modules, which are used to pick up audio from one or more pickup areas in the target space, One or more audio playback modules, which are used to play audio to one or more playback areas in the target space, An operating system that handles a variety of tasks, including speech processing tasks, an audio processing module, which is independent of the operating system, The audio processing module is used to perform echo cancellation and / or noise reduction and / or integrate the picked-up one or more audio channels into one audio channel, and send the processed audio to a subsequent processing unit. The audio processing module processes the audio from one or more audio sources and sends the processed audio to the one or more audio playing modules.

2. The speech processing system according to claim 1, wherein: The subsequent processing unit includes the operating system and the Bluetooth module. The audio processing module includes a first audio processing module and a second audio processing module. The first audio processing module is configured to perform echo cancellation and / or noise reduction processing on one or more picked-up audio channels. The second audio processing module is configured to process the audio from the one or more audio sources for playback, and obtain reference sound of the target space based on the audio from the one or more audio sources for echo cancellation.

3. The speech processing system according to claim 2, wherein: The second audio processing module processes the audio from the one or more audio sources in one or more of the following ways: mixing, copying, and splitting.

4. The speech processing system according to claim 2, wherein: The audio obtained after processing by the first audio processing module is a human voice in the sound pickup area.

5. The speech processing system according to claim 1, wherein: The voice pickup module is an array microphone, the audio playback module is a headrest speaker or an in-car speaker, and the audio processing module is a digital signal processor.

6. The speech processing system according to claim 1, wherein: The audio processing module includes a beamforming module to carry out directionally picking up sound in a sound picking area in a target space.

7. The speech processing system according to claim 1, wherein: The operating system controls the audio processing module according to the set voice mode to select a sound pickup area / playing area to pick up / play audio.

8. A vehicle comprising the speech processing system according to any one of claims 1-7.

9. A speech processing method for a speech processing system according to any one of claims 1 to 7, comprising the following steps: Determine one or more pickup zones and / or one or more playback zones based on the speech pattern, Pick up audio from one or more pickup zones, Perform echo cancellation and / or noise reduction and / or integrate the picked-up one or more audio channels into one audio channel. Sending the processed audio to a subsequent processing unit; Process audio from one or more audio sources and send it to one or more playback zones.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to claim 9 are implemented.

Citation Information

Cited By

  • Audio transmission system and method, related apparatus, and vehicle

    WO2026145824A1