A vehicle-mounted sound source positioning method, device, equipment, medium and product

Abstract

The present application discloses a method for locating a vehicle-mounted sound source, comprising: an electronic device obtains a multi-channel time domain signal received by a plurality of microphones respectively, wherein the plurality of microphones are deployed at a target position in the vehicle without material shielding, and the target position in the vehicle may be, for example, a ceiling light position, and then Fourier transforms the multi-channel time domain signals respectively to obtain a multi-channel frequency domain signal. The electronic device obtains the signal energy proportion of each sound zone in the signal by inputting the multi-channel frequency domain signal into a prediction model, and the electronic device determines the sound zone to which the vehicle-mounted sound source belongs based on the energy proportion.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a vehicle-mounted sound source localization method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of voice processing technologies such as voice wake-up and voice recognition, various voice applications have emerged. As a typical voice application, voice assistants are widely used in different scenarios. Voice assistants usually support voice wake-up and recognition within a certain range. For example, in-vehicle voice assistants usually support voice wake-up and recognition within the entire vehicle.

[0003] When people in different areas of the car speak at the same time, it can have a serious impact on voice wake-up and recognition. To this end, the wake-up direction can be determined by sound source localization, and then the beamforming algorithm can be used to effectively suppress voices from other directions, thereby improving the wake-up rate and recognition rate and enhancing the user's interactive experience.

[0004] Among them, the sound source positioning can be achieved through a beamforming algorithm based on a microphone array. Specifically, two microphones can be deployed at the position of the ceiling light and the position of the B-pillar in the car to form a microphone array. The signal collected by each array element (i.e. each microphone) in the microphone array is weighted and summed to form a beam, and then the beam is guided by searching for the possible position of the sound source, and finally the position where the beam output power is the largest is the position of the sound source.

[0005] However, the above-mentioned sound source localization method is easily affected by the position of the microphone array and the surrounding materials. For example, the microphone on the B-pillar may be blocked by the material, which may cause the microphone array to fail. This affects the accuracy of sound source localization and further affects the user's interactive experience. Summary of the invention

[0006] The main purpose of the embodiments of the present application is to provide a vehicle-mounted sound source localization method, which can reduce the influence of the microphone array position on the sound source localization and improve the accuracy of the sound source localization. The present application also provides a device, equipment, computer-readable storage medium and computer program product corresponding to the above-mentioned vehicle-mounted sound source localization method.

[0007] In a first aspect, the present application provides a vehicle-mounted sound source localization method, which is applied to an electronic device, and the method includes:

[0008] Acquire multi-channel time domain signals received by multiple microphones deployed at a target position in the vehicle, where the target position is not blocked by any material, and the multi-channel time domain signals correspond one-to-one to the multiple microphones;

[0009] Performing Fourier transform on the multiple time domain signals respectively to obtain multiple frequency domain signals;

[0010] Inputting the multi-channel frequency domain signals into a prediction model to obtain a signal energy ratio of each sound zone in the vehicle, wherein the sound zone represents an area into which the vehicle interior space is divided;

[0011] The sound zone to which the vehicle-mounted sound source belongs is determined according to the signal energy proportions of each sound zone in the vehicle.

[0012] In one possible implementation, the prediction model is obtained by performing model training using a first type of sample data, a second type of sample data, and a third type of sample data, wherein the sound sources in the first type of sample data belong to a sound range with the same phase and different amplitudes, the sound sources in the second type of sample data belong to a sound range with the same amplitude and different phases, and the sound sources in the third type of sample data belong to a sound range with different amplitudes and different phases.

[0013] In a possible implementation manner, the phase information in the first type of sample data is set as a mask, and the amplitude information in the second type of sample data is set as a mask.

[0014] In a possible implementation, the prediction model is obtained by training a plurality of U-type networks.

[0015] In a possible implementation, the target position is a ceiling light position, at which at least two microphones are deployed, and the interior space of the vehicle is divided into four or five sound zones.

[0016] In one possible implementation, the vehicle interior space accommodates n rows of seats, n is greater than or equal to 3, the target position is the middle position between two adjacent rows of seats, at least two microphones are deployed at each middle position, and the vehicle interior space is divided into 2n sound zones or 2n+1 sound zones.

[0017] In a second aspect, the present application provides a vehicle-mounted sound source localization device, characterized in that it includes:

[0018] A communication module, used to obtain multi-channel time domain signals received by multiple microphones deployed at a target position in the vehicle, the target position is not blocked by materials, and the multi-channel time domain signals correspond one-to-one to the multiple microphones;

[0019] A transformation module, used for performing Fourier transformation on the multiple time domain signals respectively to obtain multiple frequency domain signals;

[0020] A prediction module, used to input the multi-channel frequency domain signals into a prediction model to obtain a signal energy ratio of each sound zone in the vehicle, wherein the sound zone represents an area into which the vehicle interior space is divided;

[0021] The positioning module is used to determine the sound zone to which the vehicle-mounted sound source belongs according to the signal energy proportion of each sound zone in the vehicle.

[0022] In one possible implementation, the prediction model is obtained by performing model training using a first type of sample data, a second type of sample data, and a third type of sample data, wherein the sound sources in the first type of sample data belong to a sound range with the same phase and different amplitudes, the sound sources in the second type of sample data belong to a sound range with the same amplitude and different phases, and the sound sources in the third type of sample data belong to a sound range with different amplitudes and different phases.

[0023] In a possible implementation manner, the phase information in the first type of sample data is set as a mask, and the amplitude information in the second type of sample data is set as a mask.

[0024] In a possible implementation, the prediction model is obtained by training a plurality of U-type networks.

[0025] In a possible implementation, the target position is a ceiling light position, at which at least two microphones are deployed, and the interior space of the vehicle is divided into four or five sound zones.

[0026] In one possible implementation, the vehicle interior space accommodates n rows of seats, n is greater than or equal to 3, the target position is the middle position between two adjacent rows of seats, at least two microphones are deployed at each middle position, and the vehicle interior space is divided into 2n sound zones or 2n+1 sound zones.

[0027] In a third aspect, an embodiment of the present application further provides a vehicle-mounted sound source localization device, including: a processor, a memory, and a system bus;

[0028] The processor and the memory are connected via the system bus;

[0029] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned vehicle-mounted sound source localization method.

[0030] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned vehicle-mounted sound source localization method.

[0031] In a fifth aspect, an embodiment of the present application further provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned vehicle-mounted sound source localization method.

[0032] An embodiment of the present application provides a method for locating a vehicle-mounted sound source. The method first obtains multi-channel time domain signals received by multiple microphones deployed at a target position in the vehicle that is not blocked by materials, wherein the multi-channel time domain signals correspond one-to-one to the multiple microphones. The multi-channel time domain signals are then Fourier transformed respectively to obtain corresponding multi-channel frequency domain signals. The obtained multi-channel frequency domain signals are input into a prediction model to obtain the signal energy proportion of each sound zone in the vehicle, thereby determining the sound zone of the vehicle-mounted sound source according to the signal energy proportion in each sound zone.

[0033] On the one hand, the signal energy proportion of the sound source signal in each sound zone in the car is predicted according to the prediction model, and the sound zone of the sound source is determined according to the signal energy proportion, thereby avoiding the influence of the position of the microphone array on the sound source positioning.

[0034] On the other hand, the location where the microphone is deployed is not blocked by any material, so the recognition result can be avoided from being affected by the surrounding materials, thereby improving the accuracy of sound source positioning and enhancing the user's interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 A scene diagram of a vehicle-mounted sound source localization method provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a process flow of a vehicle-mounted sound source localization method provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of the division of in-vehicle sound zones provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of another in-car sound zone division provided by an embodiment of the present application;

[0040] Figure 5 A schematic diagram of a geometric acoustic reflection provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the structure of a prediction model provided in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of the structure of a sound source localization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] Voice interaction refers to human-computer interaction through voice. Different from human-computer interaction based on graphical interface, voice interaction has the advantages of convenient operation, low threshold for use (visually impaired people can also use it), and high interaction efficiency. Therefore, it is widely used in different fields. For example, in the field of leisure and entertainment, smart speakers support users to control the speakers to play music through voice; for another example, in the automotive field, cars support drivers to control the vehicle through voice.

[0044] The key to voice interaction lies in voice wake-up and voice recognition. Voice wake-up means that the device (such as mobile phones, toys, home appliances, etc.) can detect the user's voice (set voice command, i.e. wake-up word) even when it is in sleep or locked state, so that the device in sleep or locked state directly enters the waiting state for commands, starting the first step of voice interaction. Speech recognition (speech recognition) is also called automatic speech recognition (Automatic Speech Recognition, ASR), computer speech recognition (Computer Speech Recognition) or speech to text, which specifically processes human speech into text in written format. Further semantic understanding of the text obtained based on speech recognition can realize voice interaction.

[0045] In a voice interaction scenario, multiple users can speak. For example, in a car scenario, both the driver and passengers can speak, and the in-car voice assistant can support voice wake-up and voice recognition throughout the car. However, when people in different areas of the car speak at the same time, it may have a serious impact on the accuracy of voice wake-up and recognition. For example, the voice of the passenger can interfere with the voice of the driver, resulting in the driver's voice not being correctly recognized by the in-car voice assistant, which in turn affects the driver's control of the vehicle and affects the user experience.

[0046] By locating the direction of the sound source through sound source localization technology and then suppressing voices from other directions through beamforming algorithm, the wake-up rate and recognition rate can be improved, thereby enhancing the user's interactive experience.

[0047] Related technologies usually use microphone arrays for sound source localization. Taking the in-vehicle scenario as an example, two microphones can be deployed at the position of the ceiling light and the position of the B-pillar in the car to form a microphone array. The signals collected by each array element (i.e. each microphone) in the microphone array are weighted and summed to form a beam, and then the beam is guided by searching for the possible position of the sound source. Finally, the position where the beam output power is the largest is the position of the sound source to achieve sound source localization.

[0048] However, this sound source localization method is easily affected by the position of the microphone array and the surrounding materials. For example, the microphone on the B-pillar may be blocked by the material, which may cause the microphone array to fail, affect the accuracy of sound source localization, and further affect the user's interactive experience.

[0049] In view of this, the present application provides a vehicle-mounted sound source localization method, which is applied to electronic devices. Electronic devices refer to devices with data processing capabilities, such as servers or terminals. Among them, terminals include but are not limited to in-vehicle entertainment systems (In-Vehicle Infotainment / On-Board Infotainment), smart phones, tablet computers, laptops, personal digital assistants (personal digital assistant, PDA) or smart wearable devices. In this embodiment, the electronic device is taken as an in-vehicle entertainment system as an example for explanation.

[0050] Specifically, the in-vehicle entertainment system obtains multiple time domain signals received by multiple microphones respectively, wherein the multiple microphones are deployed at a target position in the vehicle without material obstruction, and the target position in the vehicle can be, for example, the position of the ceiling light, and then the multiple time domain signals are Fourier transformed respectively to obtain multiple frequency domain signals. The in-vehicle entertainment system obtains the signal energy ratio of each sound zone in the signal by inputting the multiple frequency domain signals into the prediction model (for the convenience of description as an example, in this embodiment, it is also referred to as a model in some cases), and the in-vehicle entertainment system determines the sound zone to which the vehicle sound source belongs according to the energy ratio. On the one hand, the in-vehicle entertainment system can obtain the signal energy ratio of the sound source signal in each sound zone in the vehicle according to the prediction model, and determine the sound zone described by the sound source according to the signal energy ratio, thereby avoiding the influence of the position of the microphone array on the sound source positioning, and improving the accuracy of positioning. On the other hand, the position where the microphone is deployed is not blocked by materials, so the recognition result can be avoided from being affected by the surrounding materials, thereby improving the accuracy of sound source positioning and improving the user's interactive experience.

[0051] The application scenarios of this method can be as follows Figure 1 As shown, the vehicle includes an in-vehicle entertainment system 100 and a microphone 200, wherein the microphone further includes 200-1 and 200-2. Microphones 200-1 and 200-2 receive two time domain signals respectively, and the in-vehicle entertainment system 100 obtains the two time domain signals from microphones 200-1 and 200-2 respectively, and then performs Fourier transform on the two time domain signals respectively to obtain two corresponding frequency domain signals. Then the in-vehicle entertainment system 100 inputs the two frequency domain signals into the prediction model to obtain the signal energy proportion of each sound zone in the vehicle, thereby determining the sound zone to which the vehicle sound source belongs based on the signal energy proportion of each sound zone in the vehicle, and realizing the positioning of the vehicle sound source.

[0052] In order to make the technical solution of the present disclosure clearer and easier to understand, the vehicle-mounted sound source localization method provided by the embodiment of the present disclosure is introduced below by taking the electronic device as the vehicle-mounted entertainment system 100 as an example. Figure 2 As shown, the method comprises the following steps:

[0053] S202: The in-vehicle entertainment system obtains multi-channel time domain signals received by multiple microphones deployed at target locations in the vehicle.

[0054] The target position refers to the position where the microphone is deployed. Generally, in order for the microphone to receive high-quality sound signals, the target position should not be blocked by any material.

[0055] In some possible implementations, the interior space of the vehicle includes n rows of seats, and the target position for deploying the microphone is the position of the ceiling light between two adjacent rows of seats, wherein at least two microphones are deployed at each target position. The division of the sound zones in the interior space of the vehicle can be performed according to the distribution of the seats in the vehicle. When the interior space of the vehicle includes n rows of seats, the interior space of the vehicle is divided into 2n sound zones or 2n+1 sound zones.

[0056] For example, Figure 3 As shown in (A), the interior space of the vehicle includes two rows of seats, and two microphones 200-1 and 200-2 are deployed. The interior space of the vehicle can be divided into five sound zones according to the seats: S1, S2, S3, S4 and S5. Figure 3 As shown in (B), the interior space of the vehicle includes two rows of seats, and two microphones 200-1 and 200-2 are deployed. The interior space of the vehicle can also be divided into four sound zones according to the seats: S1, S2, S3 and S4.

[0057] For example, Figure 4 As shown, the interior space of the vehicle includes three rows of seats, including two target positions A1 and A2, A1 deploys two microphones 200-3 and 200-4, and A2 deploys two microphones 200-5 and 200-6. In this way, the interior space of the vehicle can be divided into seven sound zones according to the seats: S1, S2, S3, S4, S5, S6 and S7.

[0058] The following is an introduction using an example in which the vehicle interior space includes two rows of seats, two microphones 200 - 1 and 200 - 2 are deployed, and the vehicle interior space includes five sound zones.

[0059] The time domain is used to describe the relationship between physical signals and time. Time domain signals are used to represent the relationship between signal changes over time. In this embodiment, the in-vehicle entertainment system obtains two time domain signals received by two microphones deployed at a target position in the vehicle, and the two time domain signals can be y1 and y2 respectively.

[0060] S204: The in-vehicle entertainment system performs Fourier transform on the multiple time domain signals respectively to obtain multiple frequency domain signals.

[0061] Fourier transform refers to the operation of converting a function representing a wave from the time domain (the relationship between time and amplitude) to the frequency domain (the relationship between frequency and amplitude). The spectrum of a time domain signal after Fourier transform is the representation of the time domain signal in the frequency domain.

[0062] Taking two time domain signals y1 and y2 collected by two microphones as an example, the frequency domain signal after the Fourier transform of the time domain signal y1 is Y1, and the corresponding relationship can be: F(y1) = Y1. The frequency domain signal after the Fourier transform of the time domain signal y1 is Y2, and the corresponding relationship can be: F(y2) = Y2.

[0063] S206: The in-vehicle entertainment system inputs the multi-channel frequency domain signals into the prediction model to obtain the signal energy proportion of each sound zone in the vehicle.

[0064] The prediction model refers to a model that can predict the signal energy proportion of the frequency domain signal in each sound zone, and can be obtained through neural network training. Specifically, the prediction model can be obtained by training the model using the first type of sample data, the second type of sample data, and the third type of sample data.

[0065] The training of the prediction model can be performed by a processing device, wherein the processing device can be a device with data processing capability. The device for training the prediction model and the electronic device for executing the method can be the same device or different devices.

[0066] The sound sources in the first type of sample data belong to sound zones with the same phase and different amplitudes, the sound sources in the second type of sample data belong to sound zones with the same amplitude and different phases, and the sound sources in the third type of sample data belong to sound zones with different amplitudes and different phases. The sample data includes an input frequency domain signal and the signal energy ratio of the frequency domain signal in each sound zone in the car. The input frequency domain signal can be obtained based on the impulse response and the speech signal.

[0067] In this embodiment, the processing device may obtain the impulse response in the vehicle by using the Image method. Image is a method for obtaining the impulse response of a target point based on room geometric acoustics. Figure 5 (A) shows the situation after one reflection, S is the sound source position, D is the receiving point position, S' is the mirror image sound source position behind the wall, and there are two sound signals (L1 and L2) reaching the receiving point. L1 is the path of the direct sound, and L2 is the path of the reflected sound. The path length of L1 can be calculated from the positions of S and D, and the path length of L2 can be calculated from the symmetry of the triangle.

[0068] like Figure 5 (B) shows the situation after two reflections, S′ is the position of the image sound source after the first reflection, S″′ is the position of the image sound source after the second reflection, and L3 is the path of the sound after the second reflection.

[0069] The principle of the processing device obtaining the impulse response in the car through Image is as follows: Assume a rectangular room with length, width and height of Lx, Ly and Lz respectively. Taking a corner of the room as the origin, the position of the sound source is rs = [xs, ys, zs], and the microphone position is r = [x, y, x]. The position of the wall is x = 0, y = 0 and z = 0. The position of the image sound source can be expressed as RP = [xs-x + 2qx, ys-y + 2jy, zs-z + 2kz]. Among them, q, j and k can all take the value of 0 or 1, so RP has a total of 8 permutations and combinations. Representing q, j and k with a set P, we have P = {(q, j, k): q, j, k∈{0,1}}.

[0070] When each dimension of the elements of P is 1, the image of the sound source in that direction will be included in the calculation. Considering that some images will be reflected multiple times, there exists Rm = [2mxLx, 2myLy, 2mzLz]. Among them, mx, my and mz are all integers, and the value range of the set m = (mx, my, mz) is -N to N. The reflection order of the image at the position Rm + Rp can be expressed as: p,m =|2mx+q|+|2my+j|+|2mz+k|. The distance from the image source to the microphone receiving position can be expressed as d=||Rp+Rm||, and the arrival delay of any image source is: Where 0 represents the speed of sound.

[0071] Therefore, the impulse response from the sound source to the microphone receiving position can be expressed as:

[0072]

[0073] Wherein, M = {(mx, my, mz): -N≤mx, my, mz≤N}, which means that all combinations of m are covered, β x1 , β x2 , β y1 , β y2 , β z1 and β z2 is the reflection coefficient of the six walls, and P represents the eight combinations covered. The elements of set m range from -N to N, with a total of (2N+1) 3 There are 8 (2N+1) combinations. 3 Different paths.

[0074] Furthermore, when simulating the discrete version of the impulse response, there is a problem of misalignment of the sampling points. Therefore, the discrete version of the impulse response for this problem is in the form of:

[0075]

[0076] Where fs is the sampling rate and LFP is an ideal low-pass filter with a cutoff frequency of fs / 2. The arrival time will be shifted to the nearest integer value, so the corresponding approximation is:

[0077] LPF{δ(n-τfs)}≈(n-round{τfs}) (3)

[0078] In this embodiment, an example is given with two microphones and five sound zones. The processing device records the impulse responses of the five sound zones received by the first microphone as I s11 ,I s21 ,I s31 ,I s41 and I s51 The impulse responses of the five sound zones received by the second microphone are respectively denoted as I s12 ,I s22 ,I s32 ,I s42 and I s52 , the speech signals of the five tone divisions are recorded as s 1 、s 2 、s 3 、s 4 and 5 The processing device obtains the signals y1 and y2 received by the microphone by convolving the speech signals of each zone with the impulse responses of each zone:

[0079]

[0080]

[0081] Among them, s i Represents the speech signal of each tone, I si1 Indicates the impulse response of each sound zone received by the first microphone, I si2 represents the impulse response of each sound zone received by the second microphone, n1 represents the noise received by the first microphone, and G2 represents the noise received by the first microphone. In this embodiment, i=1, 2, 3, 4, 5.

[0082] The processing device performs short-time Fourier transform on the two microphone signals y1 and y2 obtained by convolution, respectively, to obtain two corresponding complex frequency domain signals Y1 and Y2. The complex signal includes both the amplitude information and the phase information of the signal. The complex number can be expressed in the form of a+bi (a and b are both real numbers) or r (cosθ+isinθ), where the modulus r represents the amplitude of the signal and the angle θ represents the phase of the signal.

[0083] The complex frequency domain signal obtained in the above manner can be used as input training data of the neural network model, and then the output training data corresponding to the complex signal is obtained according to the sound source position. Among them, the output training data of the prediction model is the signal energy proportion of the signal in each sound zone in the car.

[0084] When the two complex frequency domain signals corresponding to the sound source are Y1 and Y2, the energy proportion of the signal in each sound zone is Mi. In this embodiment, i can be 1, 2, 3, 4, 5. The energy proportion of the signal of the sound source in the S1 sound zone in each sound zone is M1, M2, M3, M4 and M5.

[0085] Mi=Si / (S1+S2+S3+S4+S5) (6)

[0086] For example, when the sound source is in the S1 sound zone, the corresponding two complex signals are Y1 and Y2, and the energy proportion of the signal in each sound zone is M1, M2, M3, M4 and M5. Among them, the value of M1 is calculated based on the amplitude of the sound source, and the value is 1, and the values ​​of M2, M3, M4 and M5 are 0.

[0087] When the sound source is in the S2 sound zone, the corresponding two complex signals are Y1 and Y2, and the energy proportion of the signal in each sound zone is M1, M2, M3, M4 and M5. Among them, the value of M2 is calculated according to the amplitude of the sound source, and the value is 1, and the values ​​of M1, M3, M4 and M5 are 0.

[0088] When the sound source is in the S1 and S2 sound zones, the corresponding two complex signals are Y1 and Y2, and the energy proportion of the signal in each sound zone is M1, M2, M3, M4 and M5. Among them, the values ​​of M1 and M2 are calculated according to the amplitude of the sound source, the value of M1 can be a, the value of M2 can be b, where a+b=1, and the values ​​of M3, M4 and M5 are 0.

[0089] When the sound source is in the S1 and S3 sound zones, the corresponding two complex signals are Y1 and Y2, and the energy proportion of the signal in each sound zone is M1, M2, M3, M4 and M5. The values ​​of M1 and M3 are calculated based on the amplitude of the sound source, the value of M1 can be c, the value of M3 can be d, where c+d=1, and the values ​​of M2, M4 and M5 are 0.

[0090] When the sound source is in the sound zone of S1, S2 and S3, the corresponding two complex signals are Y1 and Y2, and the energy proportion of the signal in each sound zone is M1, M2, M3, M4 and M5. Among them, the values ​​of M1, M2 and M3 are calculated according to the amplitude of the sound source, the value of M1 can be e, the value of M2 can be f, and the value of M3 can be g, where e+f+g=1, and the values ​​of M4 and M5 are 0.

[0091] When the sound source is in the sound zone of S1, S2, S3 and S4, the corresponding two complex signals are Y1 and Y2, and the energy proportion of the signal in each sound zone is M1, M2, M3, M4 and M5. Among them, the values ​​of M1, M2, M3 and M4 are calculated according to the amplitude of the sound source, the value of M1 can be h, the value of M2 can be i, the value of M3 can be j, and the value of M4 can be k, wherein h+i+j+k=1, and the value of M5 is 0.

[0092] In this way, the prediction model can obtain the complex frequency domain signal as input data, and the signal energy proportion of each sound zone in the car corresponding to the input data as output data.

[0093] The prediction model can use the Unet network structure. The Unet network is a network with a shape similar to a U, such as Figure 6 As shown in Figure 2. Unet consists of two parts, one for feature extraction and the other for upsampling. Figure 6 As shown, in the feature extraction part, feature extraction is performed by convolution, and then pooling can be performed for each feature map extracted, for example, maximum pooling or average pooling is performed with 2x2. In the upsampling part, each upsampling is performed at the same scale as the number of channels corresponding to the feature extraction part. In this embodiment, the model can adopt a 5-layer two-dimensional convolution structure, in which the number of channels of each layer is 16, 32, 64, 128 and 256 respectively, and the size of the kernel can be (4, 4).

[0094] Since the input data includes the amplitude information and phase information of the signal, the front and rear rows and directions in the vehicle can be distinguished. Specifically, the front and rear rows can be distinguished by the amplitude information, and the directions can be distinguished by the phase information.

[0095] For data from the same row, the amplitude information can be masked to force the model to learn phase information. For data from different rows and the same direction, the phase information can be masked to force the model to learn amplitude information.

[0096] In this embodiment, during the training of the model, the model may be first trained with data from two sound sources and a high signal-to-interference ratio, so that the model gradually learns the representation information of amplitude and phase for the model area. Then, the difficulty of the training data may be gradually increased, such as expanding the number of sound sources and reducing the signal-to-interference ratio.

[0097] In the process of training the recognition of two sound sources, sample data from the same sound range may be trained first, and then sample data of the first type from two sound sources belonging to two sound ranges with the same phase and different amplitudes may be trained (for example, two sound sources are located in different sound ranges in the front row), and then sample data of the second type from two sound sources belonging to two sound ranges with the same amplitude and different phases may be trained (for example, Figure 3 As shown in (B), the sound source is located in the S2 and S4 sound zones), and finally the third type of sample data (e.g. Figure 3 (B) As shown, the sound source is located in the S2 and S3 sound zones).

[0098] In this way, the prediction model obtained through training can obtain the signal energy proportion of each sound zone in the car according to the input multi-channel frequency domain signal. In this embodiment, the energy proportion corresponding to each sound zone can be M1, M2, M3, M4 and M5.

[0099] S208: The in-vehicle entertainment system determines the sound zone to which the in-vehicle sound source belongs according to the signal energy proportions of each sound zone in the vehicle.

[0100] According to the introduction of signal energy in training data in S206, the energy proportion of the frequency domain signal in the sound zone can reflect the possibility of the sound source in the sound zone. In the absence of noise, the corresponding signal energy proportion in the sound zone without the sound source can be 0. Therefore, a threshold value can be set for the signal energy proportion, and when the corresponding signal energy proportion is higher than the threshold value, it indicates that the sound zone includes the sound source.

[0101] In this way, the in-vehicle entertainment system can determine the sound zone to which the sound source belongs based on the signal energy ratio of each sound zone in the car. Furthermore, the number of identifiable sound sources can also be set in the in-vehicle entertainment system. For example, when the number of identifiable sound sources set in the in-vehicle entertainment system is 2, when the signal energy ratios of the corresponding sound zones are higher than the preset threshold, the two sound zones with higher signal energy ratios can be determined as the sound zones where the sound source is located.

[0102] Furthermore, the in-vehicle entertainment system can accurately determine the wake-up direction based on the determined sound zone of the sound source, thereby suppressing voices from other directions to improve the wake-up rate and recognition rate.

[0103] Based on the description of the above content, the present application provides a method for locating a vehicle-mounted sound source. First, a multi-channel time domain signal received by multiple microphones deployed at a target position in the vehicle without material obstruction is obtained through an in-vehicle entertainment system, wherein the multi-channel time domain signal corresponds to multiple microphones one-to-one. Then, the multi-channel time domain signals are Fourier transformed respectively to obtain corresponding multi-channel frequency domain signals. The obtained multi-channel frequency domain signals are input into a prediction model to obtain the signal energy proportion of each sound zone in the vehicle, thereby determining the sound zone described by the vehicle-mounted sound source according to the signal energy proportion in each sound zone.

[0104] In this way, the in-vehicle entertainment system can predict the signal energy ratio of the sound source signal in each sound zone in the car according to the prediction model, and determine the sound zone described by the sound source according to the signal energy ratio, thereby avoiding the influence of the position of the microphone array on the sound source positioning.

[0105] In addition, the location where the microphone is deployed is not blocked by any material, so the recognition result can be prevented from being affected by the surrounding materials, thereby improving the accuracy of sound source positioning and enhancing the user's interactive experience.

[0106] Figure 7 is a schematic diagram of a vehicle-mounted sound source localization device according to an exemplary disclosed embodiment. Figure 7 As shown, the vehicle-mounted sound source localization device 700 includes:

[0107] A communication module 702 is used to obtain multi-channel time domain signals received by multiple microphones deployed at a target position in the vehicle, where the target position is not blocked by any material, and the multi-channel time domain signals correspond one-to-one to the multiple microphones;

[0108] The transformation module 704 is used to perform Fourier transformation on the multiple time domain signals to obtain multiple frequency domain signals;

[0109] A prediction module 706 is used to input the multi-channel frequency domain signals into a prediction model to obtain a signal energy ratio of each sound zone in the vehicle, where the sound zone represents an area into which the vehicle interior space is divided;

[0110] The positioning module 708 is used to determine the sound zone to which the vehicle-mounted sound source belongs according to the signal energy proportion of each sound zone in the vehicle.

[0111] In one possible implementation, the prediction model is obtained by performing model training using a first type of sample data, a second type of sample data, and a third type of sample data, wherein the sound sources in the first type of sample data belong to a sound range with the same phase and different amplitudes, the sound sources in the second type of sample data belong to a sound range with the same amplitude and different phases, and the sound sources in the third type of sample data belong to a sound range with different amplitudes and different phases.

[0112] In a possible implementation manner, the phase information in the first type of sample data is set as a mask, and the amplitude information in the second type of sample data is set as a mask.

[0113] In a possible implementation, the prediction model is obtained by training a plurality of U-type networks.

[0114] In a possible implementation, the target position is a ceiling light position, at which at least two microphones are deployed, and the interior space of the vehicle is divided into four or five sound zones.

[0115] In one possible implementation, the vehicle interior space accommodates n rows of seats, n is greater than or equal to 3, the target position is the middle position between two adjacent rows of seats, at least two microphones are deployed at each middle position, and the vehicle interior space is divided into 2n sound zones or 2n+1 sound zones.

[0116] Furthermore, an embodiment of the present application also provides a vehicle-mounted sound source localization device, comprising: a processor, a memory, and a system bus;

[0117] The processor and the memory are connected via the system bus;

[0118] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned vehicle-mounted sound source localization method.

[0119] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned vehicle-mounted sound source localization method.

[0120] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned vehicle-mounted sound source localization method.

[0121] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0122] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0123] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0124] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vehicle-mounted sound source localization method, characterized in that: include: Acquire multi-channel time domain signals received by multiple microphones deployed at a target position in the vehicle, where the target position is not blocked by any material, and the multi-channel time domain signals correspond one-to-one to the multiple microphones; Performing Fourier transform on the multiple time domain signals respectively to obtain multiple frequency domain signals; Input the multi-channel frequency domain signal to the prediction model to obtain the signal energy proportion of each sound zone in the vehicle, the sound zone represents the area in which the vehicle interior space is divided, and the prediction model is obtained by using the first type of sample data, the second type of sample data and the third type of sample data for model training, the sound source in the first type of sample data belongs to the sound zone with the same phase and different amplitude, the sound source in the second type of sample data belongs to the sound zone with the same amplitude and different phase, and the sound source in the third type of sample data belongs to the sound zone with different amplitude and different phase; The sound zone to which the vehicle-mounted sound source belongs is determined according to the signal energy proportions of each sound zone in the vehicle.

2. The method according to claim 1, characterized in that The phase information in the sample data of the first type is set as a mask, and the amplitude information in the sample data of the second type is set as a mask.

3. The method according to any one of claims 1 or 2, characterized in that: The prediction model is obtained by training a complex U-type network.

4. The method according to any one of claims 1 or 2, characterized in that: The target position is a ceiling light position, at which at least two microphones are deployed, and the interior space of the vehicle is divided into four sound zones or five sound zones.

5. The method according to any one of claims 1 or 2, characterized in that: The in-car space accommodates n rows of seats, where n is greater than or equal to 3, the target position is the middle position between two adjacent rows of seats, at least two microphones are deployed at each middle position, and the in-car space is divided into 2n sound zones or 2n+1 sound zones.

6. A vehicle-mounted sound source localization device, characterized in that: include: A communication module, used to obtain multi-channel time domain signals received by multiple microphones deployed at a target position in the vehicle, the target position is not blocked by materials, and the multi-channel time domain signals correspond one-to-one to the multiple microphones; A transformation module, used for performing Fourier transformation on the multiple time domain signals respectively to obtain multiple frequency domain signals; A prediction module, used for inputting the multi-channel frequency domain signals into a prediction model to obtain a signal energy ratio of each sound zone in the vehicle, wherein the sound zone represents an area in which the vehicle interior space is divided, and the prediction model is obtained by performing model training using first type sample data, second type sample data, and third type sample data, wherein the sound sources in the first type sample data belong to sound zones with the same phase and different amplitudes, the sound sources in the second type sample data belong to sound zones with the same amplitude and different phases, and the sound sources in the third type sample data belong to sound zones with different amplitudes and different phases; The positioning module is used to determine the sound zone to which the vehicle-mounted sound source belongs according to the signal energy proportion of each sound zone in the vehicle.

7. An electronic device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that When the computer program product runs on a computer, the computer executes the vehicle-mounted sound source localization method according to any one of claims 1 to 5.

Images