Audio processing method and device, car machine and medium

Abstract

The application provides an audio processing method and device, a vehicle machine and a medium, and relates to the technical field of vehicle-mounted audio. The method comprises the following steps: performing wake-up word detection on at least two audio signals; the at least two audio signals comprise an out-of-vehicle audio signal and an in-vehicle audio signal; when the wake-up word is detected, performing interference cancellation on the audio signals to obtain a first target audio signal of the out-of-vehicle audio signal and a second target audio signal of the in-vehicle audio signal; determining a target sound source area by using a cross-attention mechanism according to the first target audio signal and the second target audio signal; and executing a vehicle control instruction according to the audio signal corresponding to the target sound source area. The application cooperatively performs wake-up word detection and interference cancellation on multiple audio signals inside and outside the vehicle, accurately locates the target sound source area by using the cross-attention mechanism, improves the anti-interference performance of wake-up word detection and the accuracy of sound source positioning in a complex vehicle-mounted acoustic environment, and ensures that the vehicle only responds to the wake-up instruction of the target sound source area.

Description

Technical Field

[0001] This invention relates to the field of in-vehicle audio technology, and more particularly to an audio processing method, device, in-vehicle system, and medium. Background Technology

[0002] As vehicles become increasingly intelligent, users expect to interact with them more naturally and conveniently via voice commands. To meet this need, intelligent voice control of vehicles is currently typically achieved through vehicle voice interaction systems.

[0003] However, current vehicle voice interaction systems are typically designed for a single vehicle environment, such as implementing only zoned voice interaction for the in-vehicle environment or only for the external environment. This makes it difficult for the system to achieve reliable sound zone isolation in complex scenarios where sound sources exist both inside and outside the vehicle, frequently leading to command recognition confusion or execution errors, severely impacting the user experience. Summary of the Invention

[0004] This invention provides an audio processing method, apparatus, vehicle-mounted system, and medium to address the shortcomings of current vehicle voice interaction systems in complex scenarios where sound sources exist both inside and outside the vehicle, making it difficult to achieve reliable sound zone isolation and frequently resulting in command recognition confusion or execution errors.

[0005] This invention provides an audio processing method, the method comprising:

[0006] Wake-up word detection is performed on at least two audio signals; the at least two audio signals include at least one external audio signal and at least one internal audio signal.

[0007] When a wake word is detected, interference cancellation is performed on the at least two audio signals to obtain a first target audio signal corresponding to the at least one external audio signal and a second target audio signal corresponding to the at least one internal audio signal.

[0008] Based on the first target audio signal and the second target audio signal, the target sound source region where the wake word is located is determined through a cross-attention mechanism;

[0009] Execute corresponding vehicle control commands based on the audio signal corresponding to the target sound source region.

[0010] According to an embodiment of the present invention, an audio processing method is provided in which the first target audio signal corresponding to at least one external audio signal is obtained in the following manner:

[0011] For each external audio signal, determine the other audio signal among the at least two audio signals besides the external audio signal as the first reference audio signal of the external audio signal;

[0012] The external audio signal and the first reference audio signal are input to the first adaptive filter to obtain the first target audio signal output by the first adaptive filter.

[0013] According to an embodiment of the present invention, an audio processing method is provided in which the second target audio signal corresponding to the at least one in-vehicle audio signal is obtained in the following manner:

[0014] For each in-vehicle audio signal, all external audio signals in the at least two audio signals are determined as the second reference audio signal of the in-vehicle audio signal;

[0015] The in-vehicle audio signal and the second reference audio signal are input to the second adaptive filter to obtain the second target audio signal output by the second adaptive filter.

[0016] According to an embodiment of the present invention, an audio processing method is provided, wherein determining the target sound source region where the wake word is located based on a first target audio signal and a second target audio signal using a cross-attention mechanism includes:

[0017] Extract a first acoustic feature from the first target audio signal, and extract a second acoustic feature from the second target audio signal;

[0018] Using the first acoustic feature as the query and the second acoustic feature as the key and value, the first fused feature is obtained;

[0019] Using the second acoustic feature as the query and the first acoustic feature as the key and value, the second fused feature is obtained;

[0020] Based on the first fusion feature and the second fusion feature, the target sound source region where the wake-up word is located is determined.

[0021] According to an embodiment of the present invention, an audio processing method is provided, wherein determining the target sound source region where the wake word is located based on the first fusion feature and the second fusion feature includes:

[0022] Based on the first fusion feature and the second fusion feature, obtain the third fusion feature;

[0023] Based on the third fusion feature, the probability of the wake word in each preset sound source region is determined;

[0024] The target sound source region where the wake word is located is determined based on the probability of the wake word in each preset sound source region.

[0025] According to an embodiment of the present invention, an audio processing method is provided, wherein determining the target sound source region where the wake word is located based on the probability of the wake word in each preset sound source region includes:

[0026] When the preset sound source area includes at least one preset functional area in the external area and at least one preset functional area in the internal area, the preset sound source area with the highest probability is determined as the target sound source area where the wake-up word is located.

[0027] When the preset sound source area only includes the outside area and the inside area of ​​the vehicle, the preset sound source area with the highest probability is determined as the candidate sound source area where the wake word is located.

[0028] If the candidate sound source region is an area outside the vehicle, the target sound source region where the wake-up word is located in the area outside the vehicle is determined based on the first target audio signal;

[0029] If the candidate sound source region is the in-vehicle region, the target sound source region where the wake-up word is located in the in-vehicle region is determined based on the second target audio signal.

[0030] According to an embodiment of the present invention, an audio processing method is provided, wherein executing a corresponding vehicle control command based on the audio signal corresponding to the target sound source region includes:

[0031] If the target sound source area only includes the vehicle interior area, the at least one external audio signal is muted, and the corresponding vehicle control command is executed according to the at least one internal audio signal.

[0032] If the target sound source area only includes the area outside the vehicle, the at least one in-vehicle audio signal is muted, and the corresponding vehicle control command is executed according to the at least one outside audio signal.

[0033] When the target sound source area includes both the in-vehicle area and the out-of-vehicle area, corresponding vehicle control commands are executed based on the at least one in-vehicle audio signal, and corresponding vehicle control commands are executed based on the at least one out-of-vehicle audio signal.

[0034] According to an embodiment of the present invention, an audio processing method further includes, before performing wake word detection on at least two audio signals:

[0035] Time synchronization correction is performed on the at least one external audio signal and the at least one internal audio signal.

[0036] This invention also provides an audio processing apparatus, comprising:

[0037] A wake-up detection module is used to detect wake-up words from at least two audio signals; the at least two audio signals include at least one external audio signal and at least one internal audio signal.

[0038] An interference cancellation module is used to cancel interference on the at least two audio signals when a wake-up word is detected, so as to obtain a first target audio signal corresponding to the at least one external audio signal and a second target audio signal corresponding to the at least one internal audio signal.

[0039] The sound source localization module is used to determine the target sound source region where the wake word is located based on the first target audio signal and the second target audio signal through a cross-attention mechanism;

[0040] The vehicle control module is used to execute corresponding vehicle control commands based on the audio signal corresponding to the target sound source area.

[0041] This invention also provides a vehicle infotainment system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the audio processing methods described above.

[0042] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the audio processing method described above.

[0043] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the audio processing methods described above.

[0044] The audio processing method provided in this invention simultaneously detects wake words in both in-vehicle and external audio signals. After detecting a wake word, interference is first eliminated to obtain a clean target audio signal. Then, a cross-attention mechanism is used to deeply analyze the inherent correlation between the in-vehicle and external target audio signals to determine the target sound source region. Finally, corresponding vehicle control commands are executed based on the determined target sound source region. This improves the anti-interference capability of wake word detection and the accuracy of sound source localization in complex in-vehicle acoustic environments, ensuring that the vehicle only responds to vehicle control commands from the target sound source region, effectively avoiding false triggering, and improving the reliability and user experience of in-vehicle voice interaction. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the audio processing method provided in an embodiment of the present invention.

[0047] Figure 2 This is one of the scenario diagrams of the audio processing method provided in the embodiments of the present invention.

[0048] Figure 3 This is a second scenario illustration of the audio processing method provided in this embodiment of the invention.

[0049] Figure 4 This is a schematic diagram of the audio processing device provided by the present invention.

[0050] Figure 5 This is a schematic diagram of the vehicle infotainment system provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] A vehicle voice interaction system with separate interior and exterior zones is a technology that allows drivers and passengers to interact with the vehicle via voice simultaneously inside and outside the vehicle. It is divided into in-vehicle voice interaction technology and out-of-vehicle voice interaction technology. In-vehicle voice interaction technology, also known as vehicle-mounted voice technology, is currently the mainstream technology, referring to the use of an in-vehicle microphone array to achieve voice localization, wake-up, and recognition. Out-of-vehicle voice interaction technology, as a new supplement to intelligent voice, allows drivers and passengers to interact with the car from outside by combining external microphones. This technology can improve the passenger experience and is an important component of automotive intelligence.

[0053] Currently, a simple dual-microphone array system is mainly used. For in-vehicle wake-up, only the in-vehicle microphone array is activated, while the external microphone array is ignored. For out-of-vehicle wake-up, only the external microphone array is activated, while the in-vehicle microphone array is ignored. When a wake word is detected, a simple energy threshold comparison is used to determine whether the wake-up person is inside or outside the vehicle, and then the system switches to the corresponding in-vehicle mode or out-of-vehicle mode.

[0054] In addition, there is a solution that uses seat detection for in-vehicle and out-of-vehicle wake-up and positioning. When someone is detected inside the vehicle, the external wake-up module is turned off. Although this solution can effectively avoid crosstalk between the in-vehicle and out-of-vehicle wake-up systems, it limits the application scenarios, such as when someone is inside the vehicle and the trunk cannot be opened by voice from outside.

[0055] To further enhance security, related technologies propose using external microphones for external voice enhancement and internal microphones for internal voice enhancement. The audio outputs of these two modules are compared with registered voiceprints, such as those of the driver, to determine whether to respond to the wake-up call. If the voiceprint meets the requirements of an authorized user, the energy of the internal and external voice outputs is compared to determine the location of the target person. While this solves the problem of malicious external wake-up and improves the security of the external wake-up system, this voiceprint scheme limits the users of external wake-up and reduces its versatility. Furthermore, because the collected voice contains noise and interference, especially in basements where there is significant reverberation, even voiceprints obtained through voice enhancement are usually of poor quality, limiting the application scenarios.

[0056] Related technologies also propose using a smart car key equipped with sensors to pinpoint the location of a target person. This smart car key can receive wireless signals from the vehicle. When the target person approaches the vehicle, the smart car key detects the signal emitted by the vehicle. At this point, the smart car key sends a request containing the target person's location information to the vehicle. Upon receiving the request, the vehicle uses its internal positioning system to obtain the target person's location information. However, this solution can only locate the position of the smart key carrier.

[0057] Based on this, this invention proposes an audio processing method that utilizes distributed microphone arrays inside and outside the vehicle simultaneously and analyzes the causal and non-causal relationships between signals to achieve a highly robust method for external vehicle zone voice wake-up and sound zone isolation.

[0058] Figure 1 This is a flowchart illustrating the audio processing method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps 110, 120, 130 and 140.

[0059] Step 110: Perform wake-up word detection on at least two audio signals; the at least two audio signals include at least one external audio signal and at least one internal audio signal.

[0060] In this embodiment, multiple microphones are deployed on the vehicle to collect audio signals, and the system will process at least one audio signal from outside the vehicle and one audio signal from inside the vehicle simultaneously.

[0061] Specifically, at least one external audio signal can be acquired by one or more microphones or microphone arrays located outside the vehicle. For example, these microphones can be deployed near key functional areas of the vehicle, such as [reference needed]. Figure 2 As shown, the microphones are deployed in the driver's side door (Area A), passenger side door (Area B), front of the vehicle (Area C), or trunk (Area D). In practical applications, a large distance is used between the external microphones, for example, maintaining a distance of more than 1 meter between adjacent microphones, to ensure that the sound source has a large time difference in reaching different microphones, thereby achieving accurate localization of the subsequent sound source.

[0062] Accordingly, at least one in-vehicle audio signal can be acquired by one or more microphones or microphone arrays located inside the vehicle. For example, see reference. Figure 2 As shown, these microphones can be deployed in the driver's seat (E1 area), passenger seat (E2 area), the area behind the driver's seat (E3 area), the area behind the passenger seat (E4 area), etc.

[0063] In this embodiment, at least one external audio signal and at least one internal audio signal are collected by microphones or microphone arrays deployed at different locations in the vehicle, and then wake-up word detection is performed on each collected audio signal.

[0064] Specifically, wake-up word detection refers to the audio processing system continuously identifying pre-set wake-up words, such as Xiao Ai (Xiaomi's AI assistant), in real time within each continuous audio stream. Once a pre-set wake-up word is detected in the audio signal of a certain audio stream, the system is triggered to switch from standby mode to working mode, complete the wake-up response, and trigger the subsequent audio processing flow.

[0065] In one example, each acquired audio signal can be preprocessed with pre-emphasis, frame-by-frame windowing, and adaptive noise suppression to filter out valid audio segments containing speech. Then, log-Mel spectrum and multi-scale differential features are extracted and input into a pre-trained wake word detection model for real-time inference, outputting wake word matching confidence. Finally, the confidence threshold and the verification results of multiple consecutive frames are used to determine whether the preset conditions are met. If the preset conditions are met, the system is triggered to switch from standby mode to working mode to complete the wake-up response.

[0066] Step 120: If a wake-up word is detected, interference cancellation is performed on the at least two audio signals to obtain a first target audio signal corresponding to the at least one external audio signal and a second target audio signal corresponding to the at least one internal audio signal.

[0067] In this embodiment, after a wake-up word is detected in any audio signal, interference cancellation processing is performed on all audio signals in order to improve the accuracy of subsequent sound source localization.

[0068] It should be noted that interference cancellation refers to separating the target audio signal from the interference signal in each of at least two simultaneously acquired audio signals using signal processing algorithms. Specifically, based on the differences in spatial characteristics and correlation between the at least two simultaneously acquired audio signals, the interference type is first identified through signal feature analysis, such as echo, environmental noise, and spatial sidelobe interference. Then, targeted algorithms such as adaptive filtering, adaptive beamforming, multi-channel spectral subtraction, or deep learning noise estimation are employed to enhance the effective signal by leveraging the strong synchronicity and consistent spatial directionality of the target signal across multiple channels, while suppressing random noise or modelable echo interference that is weakly correlated with the target signal.

[0069] After interference cancellation processing is performed on each audio signal, the target audio signal after interference cancellation is obtained for each audio signal. Specifically, for the first target audio signal corresponding to the external audio signal, its main component is sound from outside the vehicle, while the component of sound from inside the vehicle has been suppressed. For the second target audio signal corresponding to the internal audio signal, its main component is sound from inside the vehicle, while the component of sound from outside the vehicle has been suppressed.

[0070] Step 130: Based on the first target audio signal and the second target audio signal, determine the target sound source region where the wake word is located through a cross-attention mechanism.

[0071] Here, after obtaining the first target audio signal corresponding to at least one external audio signal and the second target audio signal corresponding to at least one internal audio signal after interference cancellation, the correlation features of the target sound source in the external audio signal and the internal audio signal are extracted through the cross-attention mechanism by utilizing the differences and correlations in the spatial distribution and temporal characteristics of the external audio signal and the internal audio signal, such as phase consistency and intensity ratio, thereby identifying the target sound source region where the wake-up word is located.

[0072] In one example, a pre-trained deep learning model based on a cross-attention mechanism can be used to synchronously input the first target audio signal and the second target audio signal into the model, and the model can directly output the target sound source region where the wake word is located.

[0073] Specifically, a batch of audio samples labeled with real sound source regions in an in-vehicle scenario can be used as training data. The model uses a cross-attention mechanism to model and analyze the temporal correlation and energy distribution characteristics of at least one external audio signal and at least one internal audio signal in the audio samples. It learns the mapping pattern between sound source regions and signal propagation characteristics in an in-vehicle scenario. That is, if a wake-up word first appears with high energy in the audio signal collected by the external microphone, and then appears with a preset delay and attenuation in the audio signal collected by the internal microphone, the model determines that the sound source is highly likely located in the external region. Similarly, if a sound first appears with high energy in the audio signal collected by the internal microphone, and then appears with a preset delay and attenuation in the audio signal collected by the external microphone, the model determines that the sound source is highly likely located in the internal region. Through this training, the model can capture the propagation difference characteristics of the two signals and achieve the identification of the target sound source region.

[0074] Here, the target sound source region refers to the area pre-defined by the system to distinguish the location of sound sources. In one example, the target sound source region can be divided into a relatively broad category, such as simply dividing it into the interior and exterior areas. In another example, the target sound source region can be divided into more specific locations, such as the left side of the exterior, the trunk area, the driver's seat, and the front passenger seat. The specific location can be flexibly set according to needs and there are no restrictions.

[0075] Step 140: Execute the corresponding vehicle control command based on the audio signal corresponding to the target sound source area.

[0076] After identifying the target sound source area, the system selects the audio signal corresponding to that area for further speech recognition. For example, if the target sound source area is determined to be outside the vehicle, the system will use at least one external audio signal collected by a microphone or microphone array deployed in that area to identify the user's specific command; conversely, if it is determined to be inside the vehicle, the system will use at least one internal audio signal collected by a microphone or microphone array deployed in that area to identify the user's specific command.

[0077] In one example, after recognizing the vehicle control command, the system can further verify whether the command matches the target sound source area. For instance, if the target sound source area is outside the vehicle and the command is to open the trunk, then the command is executed. If the target sound source area is outside the vehicle, but the command is to lower the air conditioning temperature, the system may choose not to execute the command, or it may notify the user to confirm the command a second time. Only after successful confirmation will the command be executed.

[0078] In one example, a vehicle control command library can be pre-set, storing at least one preset vehicle control command matching each preset sound source region. When the vehicle enters the wake-up vehicle control command recognition mode, after recognizing the vehicle control command based on the audio signal in the target sound source region, all preset vehicle control commands matching the target sound source region in the vehicle control command library are first filtered out. Then, it is determined whether any of the filtered preset vehicle control commands contains a preset vehicle control command that is identical to or semantically similar to the currently recognized vehicle control command. If so, the currently recognized vehicle control command is executed.

[0079] In one example, if no preset vehicle control commands are found that are identical to or semantically similar to the currently identified vehicle control commands, the currently identified vehicle control commands can be sent to the user for secondary confirmation. After the secondary confirmation is successful, the currently identified vehicle control commands can be associated and stored in the vehicle control command library to enable real-time updates of the vehicle control command library according to user needs.

[0080] The audio processing method provided in this invention simultaneously detects wake words in both in-vehicle and external audio signals. After detecting a wake word, interference is first eliminated to obtain a clean target audio signal. Then, a cross-attention mechanism is used to deeply analyze the inherent correlation between the in-vehicle and external target audio signals to determine the target sound source region. Finally, corresponding vehicle control commands are executed based on the determined target sound source region. This improves the anti-interference capability of wake word detection and the accuracy of sound source localization in complex in-vehicle acoustic environments, ensuring that the vehicle only responds to vehicle control commands from the target sound source region, effectively avoiding false triggering, and improving the reliability and user experience of in-vehicle voice interaction.

[0081] In some embodiments, the first target audio signal corresponding to the at least one external audio signal is obtained in the following manner:

[0082] For each external audio signal, determine the other audio signal among the at least two audio signals besides the external audio signal as the first reference audio signal of the external audio signal;

[0083] The external audio signal and the first reference audio signal are input to the first adaptive filter to obtain the first target audio signal output by the first adaptive filter.

[0084] In this embodiment, for ease of understanding, the external microphones M_out1 and M_out2 and the internal microphones M_in1 and M_in2 deployed on the vehicle are used as examples for explanation.

[0085] Specifically, when the system needs to process the external audio signal collected by microphone M_out1, it treats the external audio signal collected by microphone M_out1 as the main audio signal. This main audio signal includes the audio signal from the area where microphone M_out1 is expected to be deployed, as well as interference noise from other areas. At the same time, the system collectively determines all audio signals collected by all other microphones, namely microphone M_out2, microphone M_in1, and microphone M_in2, as the first reference audio signal.

[0086] It should be noted that, since there is a clear one-way propagation sequence between the external sound source and the microphones inside and outside the vehicle, that is, the external sound source first reaches the external microphone and then is transmitted to the internal microphone, this propagation delay makes the interference in the signal collected by the internal microphone and the main audio signal collected by the external microphone satisfy causality. Therefore, in this embodiment, all audio signals other than the external audio signal in the collected audio signal are determined to be the first reference audio signal of the external audio signal.

[0087] After determining the first reference audio signal, the system simultaneously inputs the external audio signal collected by microphone M_out1 and the determined first reference audio signal into a first adaptive filter for interference cancellation processing.

[0088] Here, the first adaptive filter is a digital signal processor whose internal parameters can be automatically adjusted according to the statistical characteristics of the input signal. Specifically, the first adaptive filter utilizes the characteristic that the interference components in the reference audio signal are highly correlated with those in the main audio signal to adaptively adjust its transfer function to generate an interference signal. This interference signal is then subtracted from the main audio signal to eliminate the interference, thus obtaining the desired first target audio signal.

[0089] In one example, the feature vectors of each audio signal in the first reference audio signal are extracted and then multiplied by the corresponding filter weights to obtain the interference components of each audio signal in the first reference audio signal. Finally, the interference components of all audio signals in the first reference audio signal are summed to obtain the corresponding interference signal.

[0090] In this embodiment, the first adaptive filter can be implemented using various filtering algorithms, including but not limited to the least mean square algorithm, the normalized least mean square algorithm, or the recursive least squares algorithm. For each external audio signal that needs to be processed, the above process is repeated to generate the corresponding first target audio signal, which will not be elaborated further here.

[0091] The audio processing method provided in this embodiment of the invention uses all other audio signals as the first reference audio signal of the external audio signal, and uses a first adaptive filter to eliminate interference between the external audio signal and the first reference audio signal, thereby achieving accurate purification of the external audio signal and improving the accuracy of subsequent target sound source area localization.

[0092] In some embodiments, the second target audio signal corresponding to the at least one in-vehicle audio signal is obtained in the following manner:

[0093] For each in-vehicle audio signal, all external audio signals in the at least two audio signals are determined as the second reference audio signal of the in-vehicle audio signal;

[0094] The in-vehicle audio signal and the second reference audio signal are input to the second adaptive filter to obtain the second target audio signal output by the second adaptive filter.

[0095] In this embodiment, for ease of understanding, we will continue to use the external microphones M_out1 and M_out2 and the internal microphones M_in1 and M_in2 deployed on the vehicle as examples for explanation.

[0096] Specifically, when the system needs to process the in-vehicle audio signal collected by microphone M_in1, it treats the in-vehicle audio signal collected by microphone M_in1 as the main audio signal. This main audio signal includes the audio signal from the area where microphone M_in1 is to be deployed, as well as interference noise from other areas. At the same time, the system collectively determines all audio signals collected by all other external microphones, namely external microphones M_out1 and M_out2, as the second reference audio signal.

[0097] It should be noted that, due to the dispersed distribution of in-vehicle microphones and the short propagation path from the in-vehicle sound source to each in-vehicle microphone, the time difference is negligible. The sound from the same in-vehicle sound source may reach different in-vehicle microphones at the same time, resulting in the signals between in-vehicle microphones not satisfying causality. However, the signals between in-vehicle microphones and external microphones satisfy causality. Therefore, in this embodiment, only the audio signals of all external microphones are used as the second reference audio signal for each in-vehicle audio signal.

[0098] After determining the second reference audio signal, the system simultaneously inputs the in-vehicle audio signal collected by microphone M_in1 and the determined second reference audio signal into a second adaptive filter for interference cancellation processing.

[0099] Here, the second adaptive filter is a digital signal processor whose internal parameters can be automatically adjusted according to the statistical characteristics of the input signal. Specifically, the second adaptive filter utilizes the characteristic that the interference components in the reference audio signal are highly correlated with those in the main audio signal, adaptively adjusting its transfer function to generate an interference signal. This interference signal is then subtracted from the main audio signal to eliminate the interference, thus obtaining the desired second target audio signal.

[0100] In one example, the feature vectors of each audio signal in the second reference audio signal are extracted, and then multiplied with the corresponding filter weights to obtain the interference components of each audio signal in the second reference audio signal. Finally, the interference components of all audio signals in the second reference audio signal are summed to obtain the corresponding interference signal.

[0101] In this embodiment, the second adaptive filter can be implemented using various filtering algorithms, including but not limited to the least mean square algorithm, the normalized least mean square algorithm, or the recursive least squares algorithm. For each in-vehicle audio signal that needs processing, the above process is repeated to generate the corresponding second target audio signal, which will not be elaborated further here.

[0102] The audio processing method provided in this embodiment of the invention uses all external audio signals as the second reference audio signal of the internal audio signal, and uses a second adaptive filter to eliminate interference between the internal audio signal and the second reference audio signal, thereby achieving accurate purification of the internal audio signal and improving the accuracy of subsequent target sound source area localization.

[0103] In some embodiments, the first adaptive filter and / or the second adaptive filter mentioned above perform interference cancellation using the following formula:

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] in, This represents the target audio signal of the m-th main microphone after interference cancellation. This represents the raw audio signal from the m-th master microphone at time n. This represents the interference signal predicted from the reference microphone signal in the m-th master microphone signal; Let L represent the signal vector of the k-th reference microphone at time n, with a length of L, which includes the signals at the current time and the previous (L-1) time. This represents the filter weight vector of the k-th reference microphone corresponding to the m-th master microphone at time n; This represents the filter weight vector of the k-th reference microphone corresponding to the m-th master microphone at time n+1. Let L represent the step size factor, L represent the filter length, M represent the total number of microphones, including M1 in-vehicle microphones and M2 out-of-vehicle microphones, and K represent the number of reference microphones. When the main microphone is an out-of-vehicle microphone, K = M-1, and when the main microphone is an in-vehicle microphone, K = M2.

[0110] After the above processing, the external microphones suppress sound sources inside the vehicle and sound sources outside the vehicle that are not in the local area, while the internal microphones suppress sound sources outside the vehicle, achieving the effect of preliminary zoning. At this point, the zoning characteristics are obvious, that is, preliminary shielding is achieved between the inside and outside of the vehicle.

[0111] In some embodiments, determining the target sound source region where the wake word is located using a cross-attention mechanism based on the first target audio signal and the second target audio signal includes:

[0112] Extract a first acoustic feature from the first target audio signal, and extract a second acoustic feature from the second target audio signal;

[0113] Using the first acoustic feature as the query and the second acoustic feature as the key and value, the first fused feature is obtained;

[0114] Using the second acoustic feature as the query and the first acoustic feature as the key and value, the second fused feature is obtained;

[0115] Based on the first fusion feature and the second fusion feature, the target sound source region where the wake-up word is located is determined.

[0116] First, the system needs to extract a first acoustic feature from the first target audio signal and a second acoustic feature from the second target audio signal. Here, the acoustic feature can be a combination of one or more features, including but not limited to: Mel frequency cepstral coefficients, log Mel spectrum, etc., without limitation.

[0117] After extracting the first acoustic features of each first target audio signal and the second acoustic features of each second target audio signal, we obtain the first acoustic features representing all external audio signals and the second acoustic features representing all internal audio signals. Then, we perform two attention calculations on the first and second acoustic features.

[0118] Specifically, all first acoustic features are used as queries in the attention mechanism, and all second acoustic features are used as both keys and values. The similarity between the query and the keys and values ​​is calculated to obtain the corresponding relevance weights. Then, all second acoustic features are weighted and summed according to their relevance weights to obtain the first fused feature. Here, the first fused feature represents the aggregation result of wake-up word features related to wake-up word audio in the in-vehicle audio and wake-up word audio in the out-of-vehicle audio.

[0119] In contrast to the first fusion feature, all second acoustic features are used as queries, and all first acoustic features are used as both keys and values. The similarity between the query and the keys and values ​​is calculated to obtain the corresponding relevance weights. Then, all first acoustic features are weighted and summed according to their relevance weights to obtain the second fusion feature. Here, the second fusion feature represents the aggregation result of wake-up word features related to the wake-up word audio in the external audio and the wake-up word audio in the internal audio.

[0120] After obtaining the first and second fusion features through the two calculations described above, the target sound source region where the wake-up word is located can be determined by combining the first and second fusion features. For example, if the wake-up word intensity of the first fusion feature is low and the wake-up word intensity of the second fusion feature is high, then the target sound source region is inside the vehicle; if the wake-up word intensity of the second fusion feature is low and the wake-up word intensity of the first fusion feature is high, then the target sound source region is outside the vehicle; if the wake-up word intensity of the first and second fusion features is not significantly different, then both the outside and inside of the vehicle are target sound source regions.

[0121] The audio processing method provided in this invention achieves accurate localization of sound source regions in complex in-vehicle environments through bidirectional cross-attention fusion of acoustic features inside and outside the vehicle.

[0122] In some embodiments, determining the target sound source region where the wake-up word is located based on the first fusion feature and the second fusion feature includes:

[0123] Based on the first fusion feature and the second fusion feature, obtain the third fusion feature;

[0124] Based on the third fusion feature, the probability of the wake word in each preset sound source region is determined;

[0125] The target sound source region where the wake word is located is determined based on the probability of the wake word in each preset sound source region.

[0126] In this embodiment, the first and second fusion features are further fused to obtain a more comprehensive fusion feature, namely the third fusion feature. There are various ways to obtain the third fusion feature, including but not limited to direct concatenation, element-wise addition, or weighted averaging after multiplication.

[0127] After obtaining the third fusion feature, the system inputs it into a classification head, which typically includes a fully connected layer and a softmax layer. The number of output nodes in the fully connected layer equals the number of preset sound source regions. The softmax layer converts the output features corresponding to the output nodes of the fully connected layer into a probability distribution, such that the sum of all output values ​​is 1. For example, if the preset sound source regions are the in-vehicle region and the out-of-vehicle region, the output might be (0.1, 0.9), indicating that the wake-up word has a 10% probability of originating from inside the vehicle and a 90% probability of originating from outside. Finally, the preset sound source region with the highest probability value is selected as the final judgment result for the target sound source region.

[0128] The audio processing method provided in this invention generates a third fusion feature by fusing the bidirectional interactive features of audio inside and outside the vehicle, accurately calculates the probability of wake-up words in each preset sound source region, and achieves reliable localization of wake-up word sound source regions in complex vehicle environments.

[0129] In some embodiments, determining the target sound source region where the wake-up word is located based on the probability of the wake-up word in each preset sound source region includes:

[0130] When the preset sound source area includes at least one preset functional area in the external area and at least one preset functional area in the internal area, the preset sound source area with the highest probability is determined as the target sound source area where the wake-up word is located.

[0131] When the preset sound source area only includes the area outside the vehicle and the area inside the vehicle, the preset sound source area with the highest probability is determined to be the candidate sound source area where the wake-up word is located.

[0132] If the candidate sound source region is an area outside the vehicle, the target sound source region where the wake-up word is located in the area outside the vehicle is determined based on the first target audio signal;

[0133] If the candidate sound source region is the in-vehicle region, the target sound source region where the wake-up word is located in the in-vehicle region is determined based on the second target audio signal.

[0134] In this embodiment, the target sound source region where the wake word is located is determined dynamically based on the positioning strategy of the target sound source region at each preset sound source region according to the granularity of the preset sound source region division, combined with the probability of the wake word in each preset sound source region.

[0135] Specifically, the granularity of the preset sound source region division in this embodiment is divided into the following two types:

[0136] Fine-grained division: The preset sound source area includes at least one preset functional area outside the vehicle, such as the front of the vehicle, the left side of the vehicle, and the right side of the vehicle, and at least one preset functional area inside the vehicle, such as the driver's seat, the passenger seat, the left side of the rear seat, and the right side of the rear seat;

[0137] Coarse-grained division: The preset sound source area only includes the external and internal areas of the vehicle, that is, it only distinguishes between the inside and outside of the vehicle, without specifying the specific location.

[0138] In one example, when the granularity of the preset sound source region is fine-grained, the preset sound source region with the highest probability is directly determined as the target sound source region where the wake-up word is located. For example, if the model's output probability is 0.1 for the driver's seat, 0.05 for the passenger seat, 0.8 for the trunk, and 0.05 for the front left door, the system will directly determine the area in the trunk as the final target sound source region.

[0139] In another example, when the preset sound source region is divided into coarse-grained regions, the preset sound source region with the highest probability is first determined as the candidate sound source region where the wake-up word is located. For example, if the model outputs a probability of 0.2 for the in-vehicle region and 0.8 for the out-of-vehicle region, then the out-of-vehicle region is selected as the candidate sound source region.

[0140] When the candidate sound source area is outside the vehicle, the system will call a special external sound source localization algorithm. This algorithm uses information such as the time difference and direction of arrival between multiple microphones, and combines all the first target audio signals through beamforming or acoustic imaging techniques to calculate the specific location of the sound in the external space and map it to the nearest preset functional area, such as the trunk.

[0141] Similarly, when the candidate sound source area is the in-vehicle area, the system will call an in-vehicle zoning algorithm specifically for in-vehicle sound source localization. This in-vehicle zoning algorithm uses information such as the time difference and direction of arrival between multiple microphones, and combines all the second target audio signals through beamforming or acoustic imaging techniques to calculate the specific location of the sound in the external space of the vehicle, and maps it to the nearest preset functional area, such as the driver's seat.

[0142] The audio processing method provided in this embodiment of the invention achieves accurate positioning of the target sound source region by dynamically adjusting the positioning strategy of the target sound source region according to the granularity of the preset sound source region division, and combining the probability of the wake word in each preset sound source region.

[0143] In some embodiments, executing the corresponding vehicle control command based on the audio signal corresponding to the target sound source region includes:

[0144] If the target sound source area only includes the vehicle interior area, the at least one external audio signal is muted, and the corresponding vehicle control command is executed according to the at least one internal audio signal.

[0145] If the target sound source area only includes the area outside the vehicle, the at least one in-vehicle audio signal is muted, and the corresponding vehicle control command is executed according to the at least one outside audio signal.

[0146] When the target sound source area includes both the in-vehicle area and the out-of-vehicle area, corresponding vehicle control commands are executed based on the at least one in-vehicle audio signal, and corresponding vehicle control commands are executed based on the at least one out-of-vehicle audio signal.

[0147] refer to Figure 3 As shown, when the wake-up word originates from inside the vehicle, it indicates that the external audio signal is irrelevant noise. Therefore, all external audio signals are muted to prevent external sounds from being mistakenly identified as part of the command. Then, the corresponding vehicle control command is executed based on the internal audio signal. For example, if the system determines that the sound is coming from the driver's seat, it ignores all signals from external microphones and only performs voice recognition on the signals collected by the internal microphone. If it recognizes the vehicle control command to open the sunroof, it then executes the action of opening the sunroof.

[0148] refer to Figure 3 As shown, when the wake-up word originates from outside the vehicle, it indicates that the in-vehicle audio signal is irrelevant noise. Therefore, all in-vehicle audio signals are muted to prevent in-vehicle sounds from being mistakenly identified as part of the command. Then, the corresponding vehicle control command is executed based on the external audio signal. For example, if the sound is determined to be coming from the trunk, the system ignores the in-vehicle sound and only performs voice recognition on the microphone signal near the trunk or microphone signals from all locations outside the vehicle. If a vehicle control command to open the trunk is recognized, the system then executes the action of opening the trunk.

[0149] Continue to refer to Figure 3 As shown, if the system detects valid voice interaction both inside and outside the vehicle, it will execute corresponding vehicle control commands based on the audio signal inside the vehicle and corresponding vehicle control instructions based on the audio signal outside the vehicle. If the two vehicle control instructions are the same, only one vehicle control instruction needs to be executed; if the two vehicle control instructions are different, the corresponding vehicle control instructions will be executed separately, achieving coordinated response of multi-area commands.

[0150] Here, when executing the corresponding vehicle control command based on the audio signal under the corresponding target sound source area, the vehicle control command can be identified based on the original audio signal collected, or the original audio signal collected can be subjected to interference cancellation, and the vehicle control command can be identified based on the target audio signal after interference cancellation. There is no restriction on this.

[0151] The audio processing method provided in this embodiment of the invention mutes audio signals in non-target areas to avoid interference signals affecting the execution of control commands, while accurately triggering corresponding functions based on signals in the target area, thus ensuring the reliability of in-vehicle voice interaction.

[0152] In some embodiments, prior to the wake-word detection of at least two audio signals, the method further includes:

[0153] Time synchronization correction is performed on the at least one external audio signal and the at least one internal audio signal.

[0154] Here, time synchronization correction refers to determining the time reference point of multiple audio signals, unifying all signals onto the same time axis, and eliminating relative time differences.

[0155] In one example, if the microphones inside and outside the vehicle can be synchronized in hardware, such as by sharing the same sampling clock or synchronizing via the PTP protocol, then the synchronization acquisition is triggered directly based on the hardware clock, ensuring that the original signal has no time deviation.

[0156] In one example, if the microphones inside and outside the vehicle cannot be synchronized by hardware, a common reference event can be extracted from multiple signals, such as a fixed sound effect when the vehicle starts, a microphone self-test signal, or a sudden sharp noise in the environment. The occurrence time of this reference event in each signal can be set as the time origin. Algorithms such as interpolation and delay compensation can also be used to adjust the time axis of one or more signals to ensure that the same event is consistent in time in all signals.

[0157] The audio processing method provided in this invention can align the audio signals of all microphones on the time axis through time synchronization correction, avoiding missed wake-up word detection, interference cancellation failure, and positioning errors caused by time misalignment.

[0158] Based on any of the above embodiments, the present invention also provides an audio processing apparatus, with reference to... Figure 4 The device includes:

[0159] The wake-up detection module 410 is used to detect wake-up words for at least two audio signals; the at least two audio signals include at least one external audio signal and at least one internal audio signal.

[0160] The interference cancellation module 420 is used to cancel interference on the at least two audio signals when a wake-up word is detected, so as to obtain a first target audio signal corresponding to the at least one external audio signal and a second target audio signal corresponding to the at least one internal audio signal.

[0161] The sound source localization module 430 is used to determine the target sound source region where the wake word is located based on the first target audio signal and the second target audio signal through a cross-attention mechanism;

[0162] The vehicle control module 440 is used to execute corresponding vehicle control commands based on the audio signal corresponding to the target sound source area.

[0163] The audio processing device provided in this invention simultaneously detects wake words from both in-vehicle and external audio signals. After detecting a wake word, it first performs interference cancellation to obtain a clean target audio signal. Then, it uses a cross-attention mechanism to deeply analyze the inherent correlation between the in-vehicle and external target audio signals to determine the target sound source region. Finally, it executes corresponding vehicle control commands based on the determined target sound source region. This improves the anti-interference capability of wake word detection and the accuracy of sound source localization in complex in-vehicle acoustic environments, ensuring that the vehicle only responds to vehicle control commands from the target sound source region, effectively avoiding false triggering, and improving the reliability and user experience of in-vehicle voice interaction.

[0164] Figure 5 An example is a schematic diagram of the physical structure of a vehicle infotainment system, such as... Figure 5 As shown, the vehicle-mounted system may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions from the memory 530 to execute an audio processing method, which includes:

[0165] Wake-up word detection is performed on at least two audio signals; the at least two audio signals include at least one external audio signal and at least one internal audio signal.

[0166] When a wake word is detected, interference cancellation is performed on the at least two audio signals to obtain a first target audio signal corresponding to the at least one external audio signal and a second target audio signal corresponding to the at least one internal audio signal.

[0167] Based on the first target audio signal and the second target audio signal, the target sound source region where the wake word is located is determined through a cross-attention mechanism;

[0168] Execute corresponding vehicle control commands based on the audio signal corresponding to the target sound source region.

[0169] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to perform the audio processing methods provided by the above methods, the methods comprising:

[0171] Wake-up word detection is performed on at least two audio signals; the at least two audio signals include at least one external audio signal and at least one internal audio signal.

[0172] When a wake word is detected, interference cancellation is performed on the at least two audio signals to obtain a first target audio signal corresponding to the at least one external audio signal and a second target audio signal corresponding to the at least one internal audio signal.

[0173] Based on the first target audio signal and the second target audio signal, the target sound source region where the wake word is located is determined through a cross-attention mechanism;

[0174] Execute corresponding vehicle control commands based on the audio signal corresponding to the target sound source region.

[0175] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the audio processing methods provided by the methods described above, the methods comprising:

[0176] Wake-up word detection is performed on at least two audio signals; the at least two audio signals include at least one external audio signal and at least one internal audio signal.

[0177] When a wake word is detected, interference cancellation is performed on the at least two audio signals to obtain a first target audio signal corresponding to the at least one external audio signal and a second target audio signal corresponding to the at least one internal audio signal.

[0178] Based on the first target audio signal and the second target audio signal, the target sound source region where the wake word is located is determined through a cross-attention mechanism;

[0179] Execute corresponding vehicle control commands based on the audio signal corresponding to the target sound source region.

[0180] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0181] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio processing method, characterized in that, The method includes: Wake-up word detection is performed on at least two audio signals; the at least two audio signals include at least one external audio signal and at least one internal audio signal. When a wake word is detected, interference cancellation is performed on the at least two audio signals to obtain a first target audio signal corresponding to the at least one external audio signal and a second target audio signal corresponding to the at least one internal audio signal. Using the first acoustic feature of the first target audio signal as a query and the second acoustic feature of the second target audio signal as a key and value, a first fusion feature is obtained; using the second acoustic feature as a query and the first acoustic feature as a key and value, a second fusion feature is obtained; based on the first fusion feature and the second fusion feature, the target sound source region where the wake word is located is determined; Execute corresponding vehicle control commands based on the audio signal corresponding to the target sound source region.

2. The audio processing method according to claim 1, characterized in that, The first target audio signal corresponding to the at least one external audio signal is obtained in the following way: For each external audio signal, determine the other audio signal among the at least two audio signals besides the external audio signal as the first reference audio signal of the external audio signal; The external audio signal and the first reference audio signal are input to the first adaptive filter to obtain the first target audio signal output by the first adaptive filter.

3. The audio processing method according to claim 1, characterized in that, The second target audio signal corresponding to the at least one in-vehicle audio signal is obtained in the following way: For each in-vehicle audio signal, all external audio signals in the at least two audio signals are determined as the second reference audio signal of the in-vehicle audio signal; The in-vehicle audio signal and the second reference audio signal are input to the second adaptive filter to obtain the second target audio signal output by the second adaptive filter.

4. The audio processing method according to claim 1, characterized in that, The step of determining the target sound source region where the wake-up word is located based on the first fusion feature and the second fusion feature includes: Based on the first fusion feature and the second fusion feature, obtain the third fusion feature; Based on the third fusion feature, the probability of the wake word in each preset sound source region is determined; The target sound source region where the wake word is located is determined based on the probability of the wake word in each preset sound source region.

5. The audio processing method according to claim 4, characterized in that, The step of determining the target sound source region where the wake-up word is located based on the probability of the wake-up word in each preset sound source region includes: When the preset sound source area includes at least one preset functional area in the external area and at least one preset functional area in the internal area, the preset sound source area with the highest probability is determined as the target sound source area where the wake-up word is located. When the preset sound source area only includes the area outside the vehicle and the area inside the vehicle, the preset sound source area with the highest probability is determined to be the candidate sound source area where the wake-up word is located. If the candidate sound source region is an area outside the vehicle, the target sound source region where the wake-up word is located in the area outside the vehicle is determined based on the first target audio signal; If the candidate sound source region is the in-vehicle region, the target sound source region where the wake-up word is located in the in-vehicle region is determined based on the second target audio signal.

6. The audio processing method according to claim 1, characterized in that, The step of executing corresponding vehicle control commands based on the audio signal corresponding to the target sound source region includes: If the target sound source area only includes the vehicle interior area, the at least one external audio signal is muted, and the corresponding vehicle control command is executed according to the at least one internal audio signal. If the target sound source area only includes the area outside the vehicle, the at least one in-vehicle audio signal is muted, and the corresponding vehicle control command is executed according to the at least one outside audio signal. When the target sound source area includes both the in-vehicle area and the out-of-vehicle area, corresponding vehicle control commands are executed based on the at least one in-vehicle audio signal, and corresponding vehicle control commands are executed based on the at least one out-of-vehicle audio signal.

7. The audio processing method according to claim 1, characterized in that, Before performing wake-word detection on at least two audio signals, the method further includes: Time synchronization correction is performed on the at least one external audio signal and the at least one internal audio signal.

8. An audio processing apparatus, characterized in that, include: A wake-up detection module is used to detect wake-up words from at least two audio signals; the at least two audio signals include at least one external audio signal and at least one internal audio signal. An interference cancellation module is used to cancel interference on the at least two audio signals when a wake-up word is detected, so as to obtain a first target audio signal corresponding to the at least one external audio signal and a second target audio signal corresponding to the at least one internal audio signal. The sound source localization module is used to take the first acoustic feature of the first target audio signal as a query and the second acoustic feature of the second target audio signal as a key and value to obtain the first fused feature; Using the second acoustic feature as the query and the first acoustic feature as the key and value, a second fusion feature is obtained; based on the first fusion feature and the second fusion feature, the target sound source region where the wake word is located is determined; The vehicle control module is used to execute corresponding vehicle control commands based on the audio signal corresponding to the target sound source area.

9. A vehicle infotainment system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the audio processing method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the audio processing method as described in any one of claims 1 to 7.

Images