Audio processing method and device, electronic equipment, vehicle and storage medium

Abstract

The invention provides an audio processing method and device, electronic equipment, a vehicle and a storage medium, and relates to the field of voice processing, and the method comprises the steps: playing an original audio signal through a loudspeaker array of the electronic equipment; the method comprises the following steps: picking up an environment audio signal in an environment where the electronic equipment is located through a microphone array of the electronic equipment; performing sound source direction estimation on the environment audio signal to obtain at least one target sound source direction; based on each target sound source direction, performing voice enhancement on the environment audio signal to obtain an enhanced audio signal in each target sound source direction; and generating a target audio signal based on each enhanced audio signal and the original audio signal. Therefore, a Karaoke entertainment function can be realized by multiplexing the built-in microphone array of the electronic equipment, and the built-in microphone array generally has lower delay, so that smoother and more natural hearing experience can be provided.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to an audio processing method, device, electronic device, vehicle and storage medium. Background Art

[0002] In the related art, for the in-car KTV (Karaoke Television) system, the car owner needs to buy a handheld microphone to sing or sing Karaoke in the car. However, the handheld microphone system has a high delay of about 50ms (milliseconds), which will cause some passengers with keen hearing to feel the time delay and poor hearing. In addition, in order to ensure the safety of vehicle driving, the car owner can only sing with the handheld microphone when stuck in traffic or parked, and cannot sing while driving, which greatly limits the use scenarios and user experience of the in-car KTV system. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the present application proposes an audio processing method, device, electronic device, vehicle and storage medium to realize the entertainment function of Karaoke by reusing the built-in microphone array of the electronic device. Since the built-in microphone array usually has lower latency, it can provide a smoother and more natural listening experience.

[0005] In one aspect, an embodiment of the present application provides an audio processing method, including:

[0006] Playing the original audio signal through the speaker array of the electronic device, and picking up the ambient audio signal in the environment where the electronic device is located through the microphone array of the electronic device;

[0007] Performing sound source direction estimation on the ambient audio signal to obtain at least one target sound source direction;

[0008] Based on the directions of the target sound sources, the ambient audio signal is speech enhanced to obtain enhanced audio signals in the directions of the target sound sources;

[0009] A target audio signal is generated based on each of the enhanced audio signals and the original audio signal.

[0010] Another aspect of the present application provides an audio processing device, including:

[0011] A playback and pickup module, used to play the original audio signal through the speaker array of the electronic device, and to pick up the ambient audio signal in the environment where the electronic device is located through the microphone array of the electronic device;

[0012] an estimation module, configured to estimate the sound source direction of the ambient audio signal to obtain at least one target sound source direction;

[0013] An enhancement module, configured to perform speech enhancement on the ambient audio signal based on the directions of the target sound sources to obtain enhanced audio signals in the directions of the target sound sources;

[0014] A generating module is used to generate a target audio signal based on each of the enhanced audio signals and the original audio signal.

[0015] Another aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the audio processing method as described in the aforementioned aspect is implemented.

[0016] On the other hand, an embodiment of the present application proposes a vehicle, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the audio processing method as described in the aforementioned aspect.

[0017] Another aspect of the present application provides a non-temporary computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the audio processing method as described in the aforementioned aspect is implemented.

[0018] Another aspect of the present application provides a computer program product on which a computer program is stored. When the program is executed by a processor, the audio processing method as described in the above aspect is implemented.

[0019] Another aspect of the present application provides a chip, wherein the chip includes an interface circuit and a processing circuit coupled to each other, the interface circuit is used to input or output signals, and the processing circuit is configured to execute the audio processing method as described in the aforementioned aspect.

[0020] The audio processing method, device, electronic device, vehicle and storage medium proposed in this application can realize the multiplexing of the built-in microphone array of electronic devices (such as vehicles, smart furniture devices, etc.) to realize the entertainment function of Karaoke. Since the built-in microphone array usually has lower latency, it can provide a smoother and more natural listening experience. For example, taking the electronic device as an example of a vehicle, the built-in microphone array does not need to be held by the user, which reduces the risk of distraction of the user during driving and improves driving safety. It can not only enhance the intelligent entertainment of the cockpit and enhance the user's riding experience, but also the user can sing while the vehicle is driving, not limited to traffic jams or parking scenarios, thereby expanding the use scenarios of the in-vehicle KTV system and improving the user's experience.

[0021] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flowchart of a first audio processing method provided in an embodiment of the present application;

[0024] Figure 2 A flowchart of a second audio processing method provided in an embodiment of the present application;

[0025] Figure 3 A flowchart of a third audio processing method provided in an embodiment of the present application;

[0026] Figure 4 A flowchart of a fourth audio processing method provided in an embodiment of the present application;

[0027] Figure 5 A flowchart of a fifth audio processing method provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of multiple musical instrument sound sources separated from an original audio signal provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of the sound field position of each musical instrument sound source after adjustment by the user provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of an audio processing flow provided in an embodiment of the present application;

[0031] Fig. 9A schematic diagram of a selection interface for a karaoke mode provided in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of another audio processing flow provided in an embodiment of the present application;

[0033] Fig.11 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application;

[0034] Fig.12 is a block diagram of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0036] The present application proposes an audio processing method, device, electronic device, vehicle and storage medium.

[0037] The following describes the audio processing method, device, electronic device, vehicle and storage medium of the embodiments of the present application with reference to the accompanying drawings.

[0038] Figure 1 A flowchart of the first audio processing method provided in an embodiment of the present application.

[0039] The embodiment of the present application takes the audio processing method being configured in an audio processing device as an example. The audio processing device can be applied to any electronic device with computing capabilities so that the electronic device can perform audio processing functions.

[0040] The electronic device may be a vehicle, a smart home device, a mobile terminal, etc. The mobile terminal may be, for example, a mobile phone, a tablet computer, a personal digital assistant, or other hardware device with various operating systems.

[0041] Among them, when the electronic device is a vehicle, the present application does not limit the vehicle type to which the vehicle belongs. For example, the vehicle type includes but is not limited to: multi-purpose vehicles (Sport Utility Vehicle, abbreviated as SUV), family cars, vans, etc.

[0042] like Figure 1 As shown, the audio processing method may include the following steps S101 to S104:

[0043] Step S101: play the original audio signal through the speaker array of the electronic device, and pick up the ambient audio signal in the environment where the electronic device is located through the microphone array of the electronic device.

[0044] The original audio signal includes, but is not limited to, audio files stored on electronic devices, audio files downloaded by electronic devices, audio files acquired in real time through wireless communication, etc.

[0045] Among them, there is no restriction on the type of audio files. For example, the audio files can be song files (used for karaoke singing, users can choose their favorite song files and sing along with the accompaniment), recitation files (users can practice reading along with the accompaniment), dubbing files (users can dub along with the accompaniment), etc.

[0046] Exemplarily, the original audio signal may be an audio signal selected by a user of the electronic device.

[0047] The original audio signal may include the original singer's vocals, or may not include the original singer's vocals, that is, the original audio signal may be pure music or accompaniment, and this embodiment of the present application does not limit this.

[0048] In an embodiment of the present application, the original audio signal can be played through the speaker array of the electronic device, and the ambient audio signal in the environment in which the electronic device is located can be picked up through the microphone array of the electronic device; wherein the ambient audio signal includes the human voice of the user of the electronic device. Taking the electronic device as a vehicle as an example, the ambient audio signal may include the human voice of objects in the vehicle (such as the driver and passengers).

[0049] Step S102: estimate the sound source direction of the ambient audio signal to obtain at least one target sound source direction.

[0050] In an embodiment of the present application, a direction estimation (DOA) algorithm (including but not limited to a generalized cross-correlation (GCC) algorithm, a steering response power-phase transform (SRP-PHAT) algorithm, etc.) can be used to estimate the sound source direction of the ambient audio signal to obtain at least one target sound source direction.

[0051] Step S103: Based on the directions of the target sound sources, the ambient audio signal is speech enhanced to obtain an enhanced audio signal in the direction of the target sound source.

[0052] In the embodiment of the present application, a beamforming algorithm (including but not limited to a minimum variance distortionless response (MVDR) algorithm, etc.) may be used to perform speech enhancement on the ambient audio signal based on each target sound source direction to obtain an enhanced audio signal in each target sound source direction. That is, the beamforming algorithm is used to enhance the audio signal in each target sound source direction while suppressing noise and interference signals in other directions to obtain an enhanced audio signal in each target sound source direction.

[0053] Step S104: generating a target audio signal based on the enhanced audio signals and the original audio signal.

[0054] In the embodiment of the present application, audio processing may be performed based on the enhanced audio signals and the original audio signals in the directions of the target sound sources to obtain the target audio signals.

[0055] The audio processing method of the embodiment of the present application can realize the multiplexing of the built-in microphone array of electronic devices (such as vehicles, smart furniture devices, etc.) to realize the entertainment function of Karaoke. Since the built-in microphone array usually has lower latency, it can provide a smoother and more natural listening experience. For example, taking the electronic device as a vehicle as an example, the built-in microphone array does not need to be held by the user, which reduces the risk of distraction of the user during driving and improves driving safety. It can not only enhance the intelligent entertainment of the cockpit and enhance the user's riding experience, but also the user can sing while the vehicle is driving, not limited to traffic jams or parking scenes, thereby expanding the use scenarios of the in-vehicle KTV system and improving the user's experience.

[0056] This embodiment of the application provides another audio processing method. Figure 2 A flowchart of a second audio processing method provided in an embodiment of the present application.

[0057] It should be noted that the audio processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0058] like Figure 2 As shown, the audio processing method may include the following steps S201 to S206:

[0059] Step S201: play the original audio signal through the speaker array of the electronic device.

[0060] For the explanation of step S201, please refer to the relevant description in any embodiment of the present application, which will not be repeated here.

[0061] Step S202: during the playback of the original audio signal, subtitles associated with the original audio signal are synchronously displayed.

[0062] In an embodiment of the present application, subtitles can be extracted from the original audio signal, or subtitles can be obtained from subtitle text associated with the original audio signal, and subtitles adapted to the currently played audio data can be synchronously displayed during the playback of the original audio signal.

[0063] As an example, taking the electronic device as a vehicle, the original audio signal as a song, and the subtitles as lyrics, the lyrics of the original audio signal can be displayed on a display screen or on a vehicle-mounted head-up display (HUD) system, wherein the HUD system can obtain the lyrics from the audio stream or lyrics text stream of the original audio signal, and display or project the lyrics in an appropriate font at an appropriate position on the vehicle's windshield.

[0064] Step S203, picking up an ambient audio signal in the environment where the electronic device is located through the microphone array of the electronic device; wherein the ambient audio signal includes a human voice input in response to subtitles.

[0065] In an embodiment of the present application, an ambient audio signal in the environment in which the electronic device is located is picked up by a microphone array of the electronic device; wherein the ambient audio signal includes a human voice input by a user of the electronic device in response to subtitles displayed in real time.

[0066] Step S204: estimate the sound source direction of the ambient audio signal to obtain at least one target sound source direction.

[0067] Step S205: Based on the directions of the target sound sources, the ambient audio signal is speech enhanced to obtain an enhanced audio signal in the direction of each target sound source.

[0068] Step S206: Generate a target audio signal based on the enhanced audio signals and the original audio signal.

[0069] The explanation of steps S204 to S206 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0070] In any embodiment of the present application, after the target audio signal is generated, the target audio signal may be played through a speaker array.

[0071] As an example, the following steps A to D may be used to play the target audio signal:

[0072] Step A: Perform power amplification processing on the target audio signal to obtain an amplified audio signal.

[0073] Step B: determining a target playback mode selected by a user of the electronic device from among a plurality of audio playback modes.

[0074] Exemplarily, taking the electronic device as an example of a vehicle, the audio playback modes include but are not limited to: full-vehicle playback, separate sound zone playback, and other modes; among which, the separate sound zone playback modes include but are not limited to: main driver's sound zone playback mode, co-driver's sound zone playback mode, and the like.

[0075] Step C: Determine a target speaker adapted to the target playback mode from the speaker array.

[0076] For example, taking the electronic device as a vehicle as an example, when the target playback mode is the whole-vehicle playback mode, the target speaker may be the whole-vehicle speakers, that is, the target speaker may be all the speakers in the speaker array; and when the target playback mode is the separate sound zone playback mode (such as the main driver's sound zone playback mode), the target speaker may be the headrest speaker on the main driver's seat.

[0077] Step D: Play the amplified audio signal through the target speaker.

[0078] Therefore, based on the playback mode selected by the user of the electronic device, the amplified target audio signal is played, so that the effects of sound zone isolation and separate playback can be achieved, thereby meeting the personalized audio playback needs of different users.

[0079] The audio processing method of the embodiment of the present application takes the electronic device as a vehicle and the original audio signal as a song as an example. During the playback of the original audio signal, subtitles can be delivered through the display screen or HUD system in the car to remind the objects in the car of subtitles (such as lyrics) that are adapted to the current real-time playback progress. This can not only help the objects in the car to sing along better and improve the accuracy and fluency of the singing, but also the real-time synchronized lyrics increase the interactivity and versatility of the system, making the in-car KTV system a richer and more convenient entertainment tool.

[0080] This embodiment of the application provides another audio processing method. Figure 3 A flowchart of a third audio processing method provided in an embodiment of the present application.

[0081] It should be noted that the audio processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0082] like Figure 3As shown, the audio processing method may include the following steps S301 to S307:

[0083] Step S301: play the original audio signal through the speaker array of the electronic device, and pick up the ambient audio signal in the environment where the electronic device is located through the microphone array of the electronic device.

[0084] For the explanation of step S301, please refer to the relevant description in any embodiment of the present application, and will not be repeated here.

[0085] Step S302: determining a target audio mode associated with the original audio signal from a plurality of candidate audio modes.

[0086] Among them, the candidate audio mode is an audio mode pre-provided by the electronic device, and the candidate audio mode includes but is not limited to: candidate singing mode, candidate reading mode, candidate dubbing mode, etc. Taking the electronic device as a vehicle, the candidate audio mode is an example of a candidate singing mode, and the candidate audio mode is but is not limited to: full-car karaoke, exclusive karaoke, specific area karaoke, etc.; among them, the exclusive karaoke mode includes but is not limited to: exclusive karaoke mode for the main driver, exclusive karaoke mode for the co-driver, exclusive karaoke mode for the back row, etc.

[0087] The target audio mode is an audio mode selected by the user of the electronic device from a plurality of candidate audio modes.

[0088] In the embodiment of the present application, a target audio mode associated with the original audio signal manually selected by the user of the electronic device can be determined from a plurality of candidate audio modes.

[0089] Exemplarily, a target audio mode associated with the original audio signal may be determined from a plurality of candidate audio modes according to a selection instruction (referred to as a first selection instruction in this application) triggered by a user of the electronic device.

[0090] There is no limitation on the triggering method of the first selection instruction. For example, the user of the electronic device can trigger the first selection instruction by clicking, speaking, or the like.

[0091] Step S303: determining the number of sound-emitting objects in the environment where the electronic device is located based on the target audio mode.

[0092] Among them, the voice-generating objects include but are not limited to: singers, readers, dubbing artists, etc.

[0093] As an example, taking the electronic device as a vehicle and the target audio mode as a target singing mode, when the target audio mode is a karaoke mode in a specific area (such as the main driver's seat and the co-driver's seat), the number of sound objects in the vehicle's environment may be 2; when the target audio mode is a karaoke mode exclusively for the main driver, the number of sound objects in the vehicle's environment may be 1.

[0094] When the target audio mode is the whole-car karaoke mode, the number of objects in the car can be further detected by relevant sensors, and the number of objects in the car can be used as the number of sound-emitting objects in the vehicle's environment, that is, the number of sound-emitting objects in the vehicle's environment matches or is consistent with the number of objects in the car.

[0095] Step S304: determining the number of sound-emitting objects in the environment where the electronic device is located according to the configuration instruction triggered by the user object of the electronic device.

[0096] There is no limitation on the triggering method of the configuration instruction. For example, the user of the electronic device can trigger the configuration instruction by clicking, inputting, speaking, etc.

[0097] In an embodiment of the present application, the number of sound-emitting objects in the environment of the electronic device can be manually specified, that is, in the present application, the number of sound-emitting objects in the environment of the electronic device can be determined based on the configuration instructions triggered by the user object of the electronic device.

[0098] It should be noted that steps S302 to S303 and step S304 are two parallel implementation methods. In actual application, only one needs to be executed.

[0099] Step S305 : Based on the above quantity, estimate the sound source direction of the ambient audio signal to obtain the target sound source directions of the quantity.

[0100] In the embodiment of the present application, a DOA algorithm may be used to estimate the sound source direction of the ambient audio signal to obtain the number of target sound source directions.

[0101] As an example, the DOA algorithm can be used to estimate the sound source direction of the ambient audio signal, obtain multiple candidate sound source directions, and calculate the output power in the multiple candidate sound source directions. Therefore, in the present application, the number of target sound source directions can be screened out from the multiple candidate sound source directions based on the output power in the multiple candidate sound source directions.

[0102] For example, the number of sound-emitting objects in the environment where the electronic device is located is marked as N, and the candidate sound source directions can be arranged in descending order according to the corresponding output power, and the first N candidate sound source directions are used as the target sound source directions. That is, the N candidate sound source directions with the largest output power can be used as the target sound source directions.

[0103] Step S306: Based on the directions of the target sound sources, the ambient audio signal is speech enhanced to obtain an enhanced audio signal in the direction of each target sound source.

[0104] Step S307: Generate a target audio signal based on the enhanced audio signals and the original audio signal.

[0105] The explanation of steps S306 to S307 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0106] The audio processing method of the embodiment of the present application adopts a DOA algorithm to estimate the sound source direction of the ambient audio signal to obtain the target sound source direction, and adopts a beamforming algorithm to enhance the audio signal in the direction of each target sound source to obtain an enhanced audio signal in the direction of each target sound source. This can suppress noise and interference signals in other directions, thereby improving the quality of the target audio signal finally generated.

[0107] This embodiment of the application provides another audio processing method. Figure 4 A flowchart of the fourth audio processing method provided in an embodiment of the present application.

[0108] It should be noted that the audio processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0109] like Figure 4 As shown, the audio processing method may include the following steps S401 to S407:

[0110] Step S401: play the original audio signal through the speaker array of the electronic device, and pick up the ambient audio signal in the environment where the electronic device is located through the microphone array of the electronic device.

[0111] Step S402: estimate the sound source direction of the ambient audio signal to obtain at least one target sound source direction.

[0112] Step S403: Based on the directions of the target sound sources, the ambient audio signal is speech enhanced to obtain an enhanced audio signal in the direction of each target sound source.

[0113] The explanation of steps S401 to S403 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0114] Step S404: for any enhanced audio signal, perform echo cancellation and noise cancellation processing on any enhanced audio signal to obtain a first audio signal.

[0115] In an embodiment of the present application, for an enhanced audio signal in the direction of any target sound source, first, an echo cancellation (Acoustic Echo Cancellation, referred to as AEC) algorithm (including but not limited to a normalized least mean squares (Normalized Least Mean Squares, referred to as NLMS) algorithm, etc.) can be used to perform echo cancellation on the enhanced audio signal to obtain an intermediate audio signal. Thereafter, a noise cancellation algorithm can be used to perform noise cancellation processing on the intermediate audio signal to obtain a first audio signal.

[0116] Alternatively, a noise elimination algorithm may be first used to perform noise elimination processing on the enhanced audio signal to obtain an intermediate audio signal, and then an echo elimination algorithm may be used to perform echo elimination processing on the intermediate audio signal to obtain the first audio signal.

[0117] Step S405: perform howling suppression processing on the first audio signal to obtain a second audio signal.

[0118] In the embodiment of the present application, a howling suppression algorithm may be used to perform howling suppression processing on the first audio signal to obtain a second audio signal.

[0119] As a possible implementation manner, a frequency shift technology may be used to perform frequency shift processing on the first audio signal to obtain the second audio signal.

[0120] As another possible implementation, a notch technology may be used to detect a target frequency point where howling occurs in the first audio signal, and reduce the gain of the target frequency point in the first audio signal to obtain a second audio signal; wherein the target frequency point includes a frequency point where howling occurs. That is, the frequency point where obvious howling occurs in the first audio signal is processed to reduce the decibel level, thereby achieving the effect of suppressing howling.

[0121] As another possible implementation, an adaptive feedback suppression algorithm (such as Least Mean Squares (LMS), NLMS and other adaptive algorithms) may be used to filter the second collected feedback signal from the first audio signal to obtain the second audio signal.

[0122] It should be understood that the above three howling suppression algorithms are only exemplary descriptions, and the present application is not limited thereto. In actual application, other howling suppression algorithms may also be used to perform howling suppression processing on the first audio signal, and the present application does not impose any limitation thereto.

[0123] Step S406: add the reverberation in the target audio scene to the second audio signal to obtain a third audio signal.

[0124] The target audio scene is an audio scene manually selected by the user of the electronic device, and the target audio scene includes but is not limited to: a concert hall, a recording studio, a KTV room and other audio scenes.

[0125] Exemplarily, the target audio scene may be determined from a plurality of candidate audio scenes according to a selection instruction (referred to as a third selection instruction in this application) triggered by a user of the electronic device.

[0126] There is no limitation on the triggering method of the third selection instruction. For example, the user of the electronic device can trigger the third selection instruction by clicking, speaking, or the like.

[0127] In the embodiment of the present application, in order to improve the quality of the audio finally generated, the reverberation in the target audio scene required by the user can be added to the second audio signal to obtain a third audio signal.

[0128] In any embodiment of the present application, the following steps E to G may be used to add reverberation in the target audio scene to the second audio signal:

[0129] Step E: Determine a target audio scene selected by a user of the electronic device from among the multiple audio scenes.

[0130] That is, the target audio scene is manually selected from multiple audio scenes.

[0131] Step F: Obtain or query an impulse response that is adapted to the reverberation in the target audio scene.

[0132] It should be noted that the reverberation in different acoustic environments is different. For example, there is a difference in the reverberation time in a concert hall and a recording studio. In order to achieve the effects of different audio scenes, it is necessary to generate reverberation in different audio scenes. For example, the impulse response of each audio scene can be measured, or the impulse responses of different audio scenes can be generated by simulation, and a matching relationship between different audio scenes and impulse responses can be established. Therefore, in this application, the above matching relationship can be queried according to the target audio scene to obtain an impulse response adapted to the reverberation in the target audio scene.

[0133] Step G: Convolve the impulse response with the second audio signal to obtain a third audio signal with reverberation added.

[0134] As an example, the second audio signal is marked as x AN_T (t), the impulse response is rir(t), and the third audio signal is y(t), then:

[0135] y(t)=x AN_T (t)*rir(t);

[0136] Step S407: Generate a target audio signal according to each third audio signal and the original audio signal.

[0137] As a possible implementation manner, each third audio signal and the original audio signal may be synthesized to obtain a target audio signal.

[0138] As another possible implementation, in order to further improve the quality of the target audio signal, the human voice in the third audio signal may be compensated for by sound correction, and the third audio signal after sound correction and compensation may be synthesized with the original audio signal to obtain the target audio signal.

[0139] As an example, the spectral features of the first sound object (such as the original singer) in the original audio signal (referred to as the first spectral features in this application) can be extracted, and the second spectral features of the second sound object in each third audio signal can be extracted respectively. Then, based on the first spectral features, the second spectral features in each third audio signal can be adjusted to obtain each fourth audio signal. Finally, each fourth audio signal can be synthesized with the original audio signal to obtain the target audio signal.

[0140] The audio processing method of the embodiment of the present application processes the audio signal based on beamforming technology, echo cancellation technology, noise suppression technology, howling suppression technology and reverberation synthesis technology, which can further improve the quality of the target audio signal finally generated.

[0141] This embodiment of the application provides another audio processing method. Figure 5 A flowchart of the fifth audio processing method provided in an embodiment of the present application.

[0142] It should be noted that the audio processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0143] like Figure 5 As shown, the audio processing method may include the following steps S501 to S507:

[0144] Step S501, in response to a second selection instruction triggered by a user of the electronic device, an original audio signal to be played is selected from at least one candidate audio signal, and sound source separation is performed on the original audio signal to obtain and display multiple instrument sound sources.

[0145] The candidate audio signals include, but are not limited to: audio files stored on electronic devices, audio files downloaded by electronic devices, audio files acquired in real time through wireless communication and the like, and the like.

[0146] Among them, musical instruments include but are not limited to: drums, cello, violin, saxophone and other instruments.

[0147] Among them, the instrument sound source can also be called a music element, including but not limited to: piano sound, guitar sound, drum sound and other music elements.

[0148] In an embodiment of the present application, the original audio signal to be played can be manually selected. In the present application, in response to a second selection instruction triggered by a user of the electronic device, the original audio signal to be played can be selected from at least one candidate audio signal, and based on artificial intelligence technology, the original audio signal can be separated from the sound source to obtain multiple musical instrument sound sources. Exemplarily, a large model can be called to separate the sound source of the original audio signal to be played to obtain multiple musical instrument sound sources output by the large model.

[0149] There is no limitation on the triggering method of the second selection instruction. For example, the user of the electronic device can trigger the second selection instruction by clicking, speaking, or the like.

[0150] In the embodiment of the present application, multiple instrument sound sources can also be displayed in a visual manner.

[0151] Step S502: In response to the instrument position adjustment operation, the sound field position of at least one instrument sound source among the plurality of instrument sound sources is adjusted.

[0152] The musical instrument position adjustment operation is triggered by the user of the electronic device and is used to adjust the sound field position of the musical instrument sound source.

[0153] As an example, a user of the electronic device can adjust the sound field position of each musical instrument sound source by dragging and dropping on a visual user interface (UI) to achieve customized sound field reconstruction.

[0154] As another example, a plurality of different sound field reconstruction styles may be provided, wherein each sound field reconstruction style is configured with sound field positions of different musical instrument sound sources. In the present application, the user of the electronic device may select a desired target sound field reconstruction style from the plurality of sound field reconstruction styles, and adjust the sound field positions of each musical instrument sound source based on the sound field positions of different musical instrument sound sources configured with the target sound field reconstruction style.

[0155] For example, the multiple instrument sound sources separated from the original audio signal can be Figure 6 As shown in FIG. 1 , the sound field positions of the various musical instrument sound sources adjusted by the user can be shown as Figure 7 (a) or Figure 7 (b) as shown.

[0156] Step S503: Play the instrument sound source after the sound field position is adjusted through the speaker array of the electronic device.

[0157] As an example, the instrument sound source with the sound field position adjusted may be played through all players in the speaker array.

[0158] As another example, the target playback mode selected by the user of the electronic device can be determined from a variety of audio playback modes, and a target speaker adapted to the target playback mode can be determined from the speaker array. Thus, in the present application, the instrument sound source with the sound field position adjusted can be played through the target speaker.

[0159] Step S504: pick up ambient audio signals in the environment where the electronic device is located through the microphone array of the electronic device.

[0160] Step S505: estimate the sound source direction of the ambient audio signal to obtain at least one target sound source direction.

[0161] Step S506: Based on the directions of the target sound sources, the ambient audio signal is speech enhanced to obtain an enhanced audio signal in the direction of each target sound source.

[0162] The explanation of steps S504 to S506 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0163] Step S507: generating a target audio signal based on each enhanced audio signal and the instrument sound source after the sound field position is adjusted.

[0164] It should be noted that the implementation principle of step S507 is similar to that of step S104, and will not be elaborated here.

[0165] The audio processing method of the embodiment of the present application can filter out the original singer's voice in the original audio signal by performing sound source separation on the original audio signal to obtain and display multiple instrument sound sources, so that the target audio signal finally generated only contains the human voice of the sound-emitting object in the environment where the electronic device is located, thereby meeting the personalized entertainment needs of the user. By adjusting the sound field position of at least one instrument sound source among the multiple instrument sound sources in response to the instrument position adjustment operation, and playing the instrument sound source after the sound field position adjustment through the speaker array, the personalized audio playback needs of different users can be met.

[0166] In any embodiment of the present application, the electronic device is a vehicle, the audio signal is a song, and the audio mode is a singing mode. The present application provides a complete in-car Karaoke system without a handheld microphone (hereinafter referred to as a non-microphone Karaoke system) for the vehicle, which can realize the entertainment function of Karaoke by reusing the in-car call microphone, and can also be subtitled through the display screen or the HUD system in the car to remind passengers of the lyrics. The audio signal of the passenger singing is processed by the microphone array built into the vehicle for echo elimination, noise reduction, and reverberation. The processed audio signal is mixed with the original audio signal played (or called background sound), transmitted to the power amplifier module to amplify the volume, and then played through the speaker array. Among them, the original audio signal can be edited twice by an algorithm, such as separating the sound source of the original copyrighted music song, removing the human voice, or choosing to mix and repair the sound with the original singer, and can separate the sound of different instruments such as drums, violins, saxophones, etc., that is, separate multi-track signals, and users can rearrange the spatial distribution of the instrument sound source on the car interface to achieve different effects.

[0167] As an example, taking the original audio signal as a song selected by a user from a microphone-free karaoke system, the audio processing flow can be as follows: Figure 8 As shown, it mainly includes the following steps:

[0168] Step S81, the user can start the microphone-free karaoke system by clicking on the interface chart or using the voice assistant, and then select the song to be played and the karaoke mode. For example, the microphone-free karaoke system can provide two karaoke modes, namely, the whole car karaoke mode and the specific area karaoke mode (or solo mode). For the whole car karaoke mode, the user can select it by directly clicking the "whole car mode" button, and for the solo mode, the user can select it by directly clicking on the corresponding seat chart (for example Fig. 9 Assuming the user clicks on the A1 position, it is the driver's solo mode.

[0169] As an example, the karaoke mode can be Fig. 9 Select from the selection interface shown.

[0170] Step S82, displaying the lyrics of the song on a display screen or in a HUD system, wherein the HUD module can obtain the lyrics from an audio stream or a lyrics text stream, and display the lyrics in an appropriate font at an appropriate position on the windshield.

[0171] Step S83, receiving the singing voice of the passenger or the driver through the microphone array in the cabin originally used for calls or voice assistants, and obtaining a signal s(t).

[0172] Step S84, performs beam forming according to the karaoke mode selected in step S81. If the selected karaoke mode is the whole-car karaoke mode, a whole-car beam is used, that is, a beam is formed at each position to pick up the sound at each sound source position and suppress environmental noise. If the selected karaoke mode is the solo mode, the beam is directed to the solo position, while suppressing environmental noise and interfering human voices from other seats to obtain a signal s(t).

[0173] Step S85, performing echo cancellation on the signal s(t) output by the beamforming in step S84, that is, eliminating the background music sound in the audio signal output by the beamforming, and obtaining a signal x A (t).

[0174] Step S86: Use a noise elimination algorithm to eliminate x A (t) so that the clean human voice is played in the speaker array without playing the environmental noise, and x AN (t).

[0175] Step S87: Since the distance between the speaker array and the microphone array in the cockpit is relatively close, there is a risk of howling. Therefore, it is necessary to suppress the howling of the audio signal after noise reduction to obtain x AN_T (t). Exemplarily, the following technologies can be used to suppress howling in the audio signal after noise reduction: 1. Frequency shifting technology: by adjusting the frequency of the input audio signal, the output audio signal will not be superimposed with the frequency of the original audio signal when it enters the system again, thereby suppressing howling; 2. Notch technology: by reducing the gain of a specific frequency point, the conditions for howling are destroyed, that is, the frequency point where obvious howling appears in the audio signal is processed to reduce the decibel, so as to achieve the effect of suppressing howling; 3. Adaptive feedback suppression algorithm: using adaptive algorithms such as LMS and NLMS to filter out the feedback signal collected twice from the audio signal.

[0176] Step S88, the reverberation in different acoustic environments is different. For example, the reverberation time in a concert hall and a recording studio is different. In order to achieve the effects of different audio scenes (such as singing scenes), different karaoke scenes can be selected. For example, to restore the karaoke scene in a KTV room, it is necessary to generate the reverberation in the KTV room. The impulse response of the KTV room can be measured or simulated. The impulse response rir(t) of the KTV room is then compared with x AN_T (t) is convolved to achieve a better-sounding voice and the convolved signal y(t) is obtained.

[0177] Step S89: Since some passengers are tone-deaf, some passengers want to modify their voices to make them sound better. In this application, the audio signal after adding reverberation can be beautified. The spectrum characteristics of the human voice in the original singer are extracted, and the spectrum characteristics of the passenger's voice are supplemented according to the spectrum characteristics, so that the spectrum of the passenger's voice is fuller.

[0178] As another example, the original audio signal is taken as the music material selected by the user from the microphone-free karaoke system. The audio processing flow can also be as follows: Fig.10 As shown, it mainly includes the following steps:

[0179] Step S11, in the karaoke mode, background music needs to be rendered to achieve a better karaoke effect. For example, the microphone-free karaoke system installed in the cockpit has obtained the song editing copyright, and the original song is used as the music material to be processed, or the original pure music or background music (excluding the original singing voice) with editing copyright is used as the music material to be processed.

[0180] In step S12, the user can use the music material as background music, or perform secondary editing on the background music with editing copyright. That is, the background music is separated from the sound source, for example, the performance audio of the symphony drums, cello, violin, saxophone and other instruments is separated to form multiple tracks, and then the separated instruments are displayed on the UI interface, such as Figure 6 shown.

[0181] Step S13, after successfully separating the instrument sound source, you can perform any of the following: 1. The user can drag the instrument to a different position in the UI interface to achieve customized sound field reconstruction, such as Figure 7 (a) and 7(b); 2. The microphone-free karaoke system can provide several different sound field reconstruction styles for users to choose from.

[0182] Step S14, mixing the singing voice of the passenger or the driver with the background music obtained in step S13 to form a mixed signal of the singing voice and the background music;

[0183] Step S15, in order to make the loudness of the played song loud enough, the mixed signal can be power amplified by the power amplifier module.

[0184] Step S16, select the speakers for playing audio according to the playback mode selected by the user (such as the whole-vehicle playback mode and the individual sound zone playback mode), wherein the whole-vehicle playback mode uses the whole-vehicle speakers for audio playback, and the individual sound zone playback mode uses the headrest speakers on the seats for audio playback, and beamforming is performed on the headrest speakers, thereby achieving the effect of sound zone isolation and individual playback.

[0185] Step S17, using the speaker corresponding to step S16 to play audio.

[0186] In summary, the microphone-free karaoke system can enhance the intelligent entertainment of the cockpit and improve the user's riding experience. Moreover, the microphone-free karaoke system can be applied to common vehicle models such as multi-purpose vehicles, family cars, and vans, which can enhance the applicability of this method.

[0187] In order to implement the above embodiment, the embodiment of the present application also proposes an audio processing device.

[0188] Fig.11 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application.

[0189] like Fig.11 As shown, the audio processing device 1100 may include: a playback and pickup module 1110 , an estimation module 1120 , an enhancement module 1130 , and a generation module 1140 .

[0190] The playing and picking up module 1110 is used to play the original audio signal through the speaker array of the electronic device, and pick up the ambient audio signal in the environment where the electronic device is located through the microphone array of the electronic device;

[0191] An estimation module 1120, configured to estimate a sound source direction of an ambient audio signal to obtain at least one target sound source direction;

[0192] The enhancement module 1130 is used to perform speech enhancement on the ambient audio signal based on the directions of each target sound source to obtain an enhanced audio signal in the direction of each target sound source;

[0193] The generating module 1140 is configured to generate a target audio signal based on the enhanced audio signals and the original audio signal.

[0194] Furthermore, in one implementation of the embodiment of the present application, the playback and pickup module 1110 is specifically used to: play the original audio signal through the speaker array of the electronic device; while playing the original audio signal, synchronously display subtitles associated with the original audio signal; and pick up the ambient audio signal in the environment in which the electronic device is located through the microphone array; wherein the ambient audio signal includes the human voice input in response to the subtitles.

[0195] In one implementation of an embodiment of the present application, the estimation module 1120 is specifically used to: determine a target audio mode associated with the original audio signal from multiple candidate audio modes; determine the number of sound-emitting objects in the environment in which the electronic device is located based on the target audio mode; or determine the number of sound-emitting objects in the environment in which the electronic device is located according to a configuration instruction triggered by a user object of the electronic device; and estimate the sound source direction of the ambient audio signal based on the quantity to obtain a target sound source direction of the quantity.

[0196] In one implementation of an embodiment of the present application, the estimation module 1120 is specifically used to: estimate the sound source direction of the ambient audio signal to obtain multiple candidate sound source directions; calculate the output power in the multiple candidate sound source directions; and screen out a number of target sound source directions from the multiple candidate sound source directions based on the output power in the multiple candidate sound source directions.

[0197] In one implementation of the embodiment of the present application, the estimation module 1120 is specifically used to determine a target audio mode associated with the original audio signal from a plurality of candidate audio modes according to a first selection instruction triggered by a user object of the electronic device.

[0198] In one implementation of an embodiment of the present application, the generation module 1140 is specifically used to: for any enhanced audio signal, perform echo cancellation and noise cancellation processing on any enhanced audio signal to obtain a first audio signal; perform howling suppression processing on the first audio signal to obtain a second audio signal; add reverberation in the target audio scene to the second audio signal to obtain a third audio signal; and generate a target audio signal based on each third audio signal and the original audio signal.

[0199] In one implementation of an embodiment of the present application, the generation module 1140 is specifically used to: extract a first spectral feature of a first sound object in an original audio signal; based on the first spectral feature, adjust the second spectral feature of a second sound object in each third audio signal to obtain each fourth audio signal; and synthesize each fourth audio signal with the original audio signal to obtain a target audio signal.

[0200] In one implementation of an embodiment of the present application, the generation module 1140 is specifically used to: determine a target audio scene selected by a user of the electronic device from multiple audio scenes; obtain an impulse response adapted to the reverberation in the target audio scene; and convolve the impulse response with the second audio signal to obtain a third audio signal with reverberation added.

[0201] In one implementation of an embodiment of the present application, the generation module 1140 is specifically used to perform any of the following: performing frequency offset processing on the first audio signal to obtain a second audio signal; detecting a target frequency point where howling occurs in the first audio signal, reducing the gain of the target frequency point in the first audio signal, and obtaining a second audio signal; wherein the target frequency point includes a frequency point where howling occurs; filtering a secondary collected feedback signal from the first audio signal to obtain a second audio signal.

[0202] In one implementation of the embodiment of the present application, the playback pickup module 1110 is specifically used to: in response to a second selection instruction triggered by a user of the electronic device, select an original audio signal to be played from at least one candidate audio signal; perform sound source separation on the original audio signal to obtain and display multiple musical instrument sound sources; in response to a musical instrument position adjustment operation, adjust the sound field position of at least one musical instrument sound source among the multiple musical instrument sound sources; and play the musical instrument sound source after the sound field position is adjusted through a speaker array.

[0203] In one implementation of an embodiment of the present application, the playback pickup module 1110 is specifically used to: determine a target playback mode selected by a user of the electronic device from a plurality of audio playback modes; determine a target speaker adapted to the target playback mode from a speaker array; and play the instrument sound source after the sound field position is adjusted through the target speaker.

[0204] In one implementation of the embodiment of the present application, the generation module 1140 is specifically used to generate a target audio signal based on each enhanced audio signal and the instrument sound source after the sound field position is adjusted.

[0205] In an implementation of the embodiment of the present application, the audio processing device 1100 may further include:

[0206] The processing module is used to perform power amplification processing on the target audio signal to obtain an amplified audio signal; determine the target playback mode selected by the user of the electronic device from multiple audio playback modes; determine the target speaker adapted to the target playback mode from the speaker array; and play the amplified audio signal through the target speaker.

[0207] It should be noted that the above explanation of the audio processing method embodiment is also applicable to the audio processing device of this embodiment, and will not be repeated here.

[0208] In the audio processing device of the embodiment of the present application, the built-in microphone array of the electronic device (such as a vehicle, smart furniture device, etc.) can be reused to realize the entertainment function of Karaoke. Since the built-in microphone array usually has a lower delay, it can provide a smoother and more natural listening experience. For example, taking the electronic device as a vehicle as an example, the built-in microphone array does not need to be held by the user, which reduces the risk of distraction of the user during driving and improves driving safety. It can not only enhance the intelligent entertainment of the cockpit and enhance the user's riding experience, but also the user can sing while the vehicle is driving, not limited to traffic jams or parking scenes, thereby expanding the use scenarios of the in-vehicle KTV system and improving the user's experience.

[0209] In order to implement the above embodiments, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the audio processing method as described in any of the above embodiments is implemented.

[0210] In order to implement the above embodiments, the present application also proposes a vehicle, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an audio processing method as described in any of the above embodiments.

[0211] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the audio processing method as described in any of the above embodiments is implemented.

[0212] In order to implement the above embodiments, the present application further proposes a computer program product, on which a computer program is stored. When the computer program is executed by a processor, the audio processing method as described in any of the above embodiments is implemented.

[0213] Fig.12 1 is a block diagram of a vehicle 1200 according to an exemplary embodiment. For example, the vehicle 1200 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. The vehicle 1200 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0214] Reference Fig.12, the vehicle 1200 may include various subsystems, for example, an infotainment system 1210, a perception system 1220, a decision control system 1230, a drive system 1240, and a computing platform 1250. The vehicle 1200 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of the vehicle 1200 may be interconnected by wire or wireless means.

[0215] In some embodiments, the infotainment system 1210 may include a communication system, an entertainment system, and a navigation system, etc.

[0216] The perception system 1220 may include several sensors for sensing information about the environment around the vehicle 1200. For example, the perception system 1220 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU for short), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0217] The decision control system 1230 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0218] The drive system 1240 may include components that provide powered motion for the vehicle 1200. In one embodiment, the drive system 1240 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of an internal combustion engine, an electric motor, and an air compression engine. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0219] Some or all functions of the vehicle 1200 are controlled by a computing platform 1250. The computing platform 1250 may include at least one processor 1251 and a memory 1252, and the processor 1251 may execute instructions 1253 stored in the memory 1252.

[0220] The processor 1251 may be any conventional processor, such as a commercially available central processing unit (CPU). The processor may also include a graphics processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0221] Memory 1252 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0222] In addition to instructions 1253 , memory 1252 may also store data, such as road maps, route information, vehicle location, direction, speed, etc. The data stored in memory 1252 may be used by computing platform 1250 .

[0223] In the embodiment of the present application, the processor 1251 can execute instruction 1253 to complete all or part of the steps of the above-mentioned method embodiment.

[0224] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0225] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0226] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0227] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic devices), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0228] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA for short), a field programmable gate array (FPGA for short), etc.

[0229] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0230] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0231] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An audio processing method, characterized in that: include: Playing the original audio signal through the speaker array of the electronic device, and picking up the ambient audio signal in the environment where the electronic device is located through the microphone array of the electronic device; Performing sound source direction estimation on the ambient audio signal to obtain at least one target sound source direction; Based on the directions of the target sound sources, the ambient audio signal is speech enhanced to obtain enhanced audio signals in the directions of the target sound sources; A target audio signal is generated based on each of the enhanced audio signals and the original audio signal.

2. The method according to claim 1, characterized in that The method of playing the original audio signal through the speaker array of the electronic device and picking up the ambient audio signal in the environment where the electronic device is located through the microphone array of the electronic device includes: Playing the original audio signal through the speaker array of the electronic device; During the playing of the original audio signal, synchronously displaying subtitles associated with the original audio signal; The microphone array is used to pick up an ambient audio signal in the environment where the electronic device is located; wherein the ambient audio signal includes a human voice input in response to the subtitles.

3. The method according to claim 1, characterized in that The step of estimating the sound source direction of the ambient audio signal to obtain at least one target sound source direction includes: Determine a target audio mode associated with the original audio signal from a plurality of candidate audio modes, and determine the number of sound-emitting objects in the environment where the electronic device is located based on the target audio mode; or, Determining the number of sound-emitting objects in the environment where the electronic device is located according to a configuration instruction triggered by a user object of the electronic device; Based on the quantity, the sound source direction of the ambient audio signal is estimated to obtain the quantity of target sound source directions.

4. The method according to claim 3, characterized in that The step of estimating the sound source direction of the ambient audio signal based on the quantity to obtain the target sound source directions of the quantity includes: Estimating the sound source direction of the ambient audio signal to obtain a plurality of candidate sound source directions; Calculate the output power in the directions of the multiple candidate sound sources; According to the output powers in the multiple candidate sound source directions, the number of target sound source directions are screened out from the multiple candidate sound source directions.

5. The method according to claim 3, characterized in that: The step of determining a target audio mode associated with the original audio signal from a plurality of candidate audio modes comprises: According to a first selection instruction triggered by a user object of the electronic device, the target audio mode associated with the original audio signal is determined from the multiple candidate audio modes.

6. The method according to claim 1, characterized in that The generating a target audio signal based on each of the enhanced audio signals and the original audio signal comprises: For any enhanced audio signal, perform echo cancellation and noise cancellation processing on the any enhanced audio signal to obtain a first audio signal; Performing howling suppression processing on the first audio signal to obtain a second audio signal; adding reverberation in the target audio scene to the second audio signal to obtain a third audio signal; The target audio signal is generated according to each of the third audio signals and the original audio signal.

7. The method according to claim 6, characterized in that The step of generating the target audio signal according to each of the third audio signals and the original audio signal comprises: Extracting a first frequency spectrum feature of a first sound object in the original audio signal; Based on the first frequency spectrum feature, adjusting the second frequency spectrum feature of the second sound object in each of the third audio signals respectively to obtain each fourth audio signal; Each of the fourth audio signals is synthesized with the original audio signal to obtain the target audio signal.

8. The method according to claim 6, characterized in that The adding the reverberation in the target audio scene to the second audio signal to obtain a third audio signal includes: Determining the target audio scene selected by a user of the electronic device from among a plurality of audio scenes; Acquire an impulse response adapted to the reverberation in the target audio scene; The impulse response is convolved with the second audio signal to obtain a third audio signal to which the reverberation is added.

9. The method according to claim 6, characterized in that The performing the howling suppression process on the first audio signal to obtain the second audio signal includes any one of the following: Performing frequency shift processing on the first audio signal to obtain the second audio signal; Detecting a target frequency point where howling occurs in the first audio signal, reducing the gain of the target frequency point in the first audio signal, and obtaining the second audio signal; wherein the target frequency point includes a frequency point where howling occurs; The second audio signal is obtained by filtering the second collected feedback signal from the first audio signal.

10. The method according to any one of claims 1 to 9, characterized in that The original audio signal is played through the following steps: In response to a second selection instruction triggered by a user of the electronic device, selecting an original audio signal to be played from at least one candidate audio signal; Performing sound source separation on the original audio signal to obtain and display multiple instrument sound sources; In response to the musical instrument position adjustment operation, adjusting the sound field position of at least one musical instrument sound source among the plurality of musical instrument sound sources; The musical instrument sound source after the sound field position is adjusted is played through the speaker array.

11. The method according to claim 10, characterized in that The step of playing the instrument sound source after the sound field position is adjusted through the speaker array comprises: Determining a target playback mode selected by a user of the electronic device from among a plurality of audio playback modes; Determining a target speaker adapted to the target playback mode from the speaker array; The target loudspeaker plays the instrument sound source after the sound field position is adjusted.

12. The method according to claim 10, characterized in that The generating a target audio signal based on each of the enhanced audio signals and the original audio signal comprises: The target audio signal is generated based on each of the enhanced audio signals and the instrument sound source after the sound field position is adjusted.

13. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Performing power amplification processing on the target audio signal to obtain an amplified audio signal; Determining a target playback mode selected by a user of the electronic device from among a plurality of audio playback modes; Determining a target speaker adapted to the target playback mode from the speaker array; The amplified audio signal is played through the target speaker.

14. An audio processing device, characterized in that: include: A playback and pickup module, used to play the original audio signal through the speaker array of the electronic device, and to pick up the ambient audio signal in the environment where the electronic device is located through the microphone array of the electronic device; An estimation module, configured to estimate the sound source direction of the ambient audio signal to obtain at least one target sound source direction; An enhancement module, configured to perform speech enhancement on the ambient audio signal based on the directions of the target sound sources to obtain enhanced audio signals in the directions of the target sound sources; A generating module is used to generate a target audio signal based on each of the enhanced audio signals and the original audio signal.

15. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Implement the steps of the method according to any one of claims 1 to 13.

16. A vehicle, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 13.

17. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 13 are implemented.

18. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

Images