Sound source positioning method, sound recognition method and related devices

Abstract

The invention discloses a sound source positioning method, a sound recognition method, a sound source positioning device, a sound source recognition device, electronic equipment and a computer readable storage medium, and the sound source positioning method comprises the steps: obtaining quality parameters of a first number of paths of candidate sound signals, the first number of paths of candidate sound signals are respectively collected by a first number of microphones in the microphone array; selecting a second number of paths of target sound signals with quality parameters satisfying a quality parameter relationship from the first number of paths of candidate sound signals; and carrying out sound source localization based on each path of target sound signal to obtain a sound source localization result. The scheme can be flexibly applied to various sound source positioning environments, and the accuracy of a sound source positioning result can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a sound source localization method, a sound recognition method, a sound source localization device, a sound source recognition device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In application scenarios such as speech recognition and machine fault diagnosis, it is necessary to locate the sound source based on a microphone array. A microphone array is a multi-microphone system consisting of a certain number of acoustic sensors (microphones) arranged in a certain pattern. Sound source localization refers to collecting sound signals with the microphones of the microphone array, analyzing and processing the collected sound signals, and obtaining the location of the sound source.

[0003] However, the existing sound source localization solutions cannot be flexibly applied to various sound source localization environments, and the accuracy of the sound source localization results is not high. Summary of the invention

[0004] The present application provides a sound source localization method, a sound recognition method, a sound source localization device, a sound source recognition device, an electronic device and a computer-readable storage medium, which can solve the problem that the existing sound source localization solutions cannot be flexibly applied to various sound source localization environments and the accuracy of the sound source localization results is not high.

[0005] The present application provides a sound source localization method, comprising: obtaining quality parameters of a first number of candidate sound signals, the first number of candidate sound signals being respectively collected by a first number of microphones in a microphone array; selecting a second number of target sound signals whose quality parameters satisfy a quality parameter relationship from the first number of candidate sound signals; and performing sound source localization based on each target sound signal to obtain a sound source localization result.

[0006] The present application provides a sound recognition method, comprising: obtaining quality parameters of a first number of candidate sound signals, the first number of candidate sound signals being respectively collected by a first number of microphones in a microphone array; selecting a second number of target sound signals whose quality parameters satisfy a quality parameter relationship from the first number of candidate sound signals; performing sound source positioning based on each target sound signal to obtain a sound source positioning result; performing sound recognition based on the sound source positioning result and the target sound signal to obtain a sound recognition result.

[0007] The present application provides a sound source localization device, including: a sound signal acquisition module, a selection module and a sound source localization module. The sound signal acquisition module is used to obtain quality parameters of a first number of candidate sound signals, where the first number of candidate sound signals are respectively collected by a first number of microphones in a microphone array; the selection module is used to select a second number of target sound signals whose quality parameters satisfy the quality parameter relationship from the first number of candidate sound signals; and the sound source localization module is used to perform sound source localization based on each target sound signal to obtain a sound source localization result.

[0008] The present application provides a sound recognition device, including a sound signal acquisition module, a selection module, a sound source localization module and a sound source recognition module. The sound signal acquisition module is used to obtain the quality parameters of a first number of candidate sound signals, which are respectively collected by a first number of microphones in a microphone array; the selection module is used to select a second number of target sound signals whose quality parameters satisfy the quality parameter relationship from the first number of candidate sound signals; the sound source localization module is used to perform sound source localization based on each target sound signal to obtain a sound source localization result; the sound source recognition module is used to perform sound recognition based on the sound source localization result and the target sound signal to obtain a sound recognition result.

[0009] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above method.

[0010] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above method is implemented.

[0011] The above scheme obtains quality parameters of a first number of candidate sound signals, which are respectively collected by a first number of microphones in a microphone array; from the first number of candidate sound signals, a second number of target sound signals whose quality parameters satisfy the quality parameter relationship are selected; sound source positioning is performed based on each target sound signal to obtain a sound source positioning result. On the one hand, the first number and / or the second number can be freely configured according to the sound source positioning environment, so the sound source positioning method is not limited to the specific number of microphones and the specific microphone layout of the microphone array in the environment, and can therefore be flexibly applied to various sound source positioning environments. On the other hand, instead of directly applying the first number of candidate sound signals to sound source positioning, the second number of target sound signals whose quality parameters satisfy the quality parameter relationship are used for sound source positioning, and the target sound signal is more conducive to sound source positioning, thereby improving the accuracy of the sound source positioning result.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0014] Figure 1 It is a flowchart of an embodiment of a sound source localization method provided by the present application;

[0015] Figure 2 It is a flowchart of an embodiment of a sound source localization method provided by the present application;

[0016] Figure 3 It is a flow chart of an embodiment of a sound recognition method provided by the present application;

[0017] Figure 4 It is a schematic diagram of the joint training of the sound source localization network and the sound source identification network of the present application;

[0018] Figure 5 It is a schematic diagram of the simultaneous training of the sound source localization network and the sound recognition network of the present application;

[0019] Figure 6 is a flow chart of another embodiment of the sound recognition method provided by the present application;

[0020] Figure 7 It is a structural schematic diagram of an embodiment of a sound source localization device of the present application;

[0021] Figure 8 It is a structural schematic diagram of an embodiment of a sound recognition device of the present application;

[0022] Fig. 9 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0023] Fig.10 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0024] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0025] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0026] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0027] Figure 1 1 is a flow chart of an embodiment of a sound source localization method provided by the present application. Figure 1 As shown, in this embodiment, the sound source localization method may include the following steps:

[0028] S110: Obtain quality parameters of a first number of candidate sound signals.

[0029] The first number of candidate sound signals are respectively collected by a first number of microphones in the microphone array.

[0030] The subject of the sound source localization method embodiment of the present application is a sound source localization device, which can be any electronic device with the ability to localize a sound source. The subject that collects the sound signal and the subject that performs sound source localization based on the sound signal can be the same or different.

[0031] The microphone array includes a first number of microphones arranged in a target sound source localization environment (hereinafter referred to as the target environment) according to a target layout. The target environment can be a mobile space (such as inside a car) or a fixed space (such as a machine plant). For example, microphones are arranged in the front driver's seat, the front passenger seat, the left side of the rear seat, and the right side of the rear seat in the car.

[0032] S120: Selecting a second number of target sound signals whose quality parameters satisfy the quality parameter relationship from the first number of candidate sound signals.

[0033] The quality parameter is used to characterize the quality of the candidate sound signal. It is understandable that factors such as the noise of the target environment, the performance of the microphone, and signal transmission may interfere with the quality of the candidate sound signal. For example, during a vehicle journey, wind noise, tire noise, discussions between passengers in the car, media echo, and other noises in the car may interfere with the quality of the candidate sound signal. The degree of interference of each interference factor on the quality of the candidate sound signal can be measured by at least one indicator.

[0034] The quality parameters may include at least one of distortion, signal-to-noise ratio, correlation, etc., which are used to measure the candidate sound signal. Distortion refers to the change or loss of the candidate sound signal relative to the original sound signal collected. Signal-to-noise ratio refers to the ratio of the target sound emitted by the sound source to the ambient noise in the candidate sound signal. The larger the signal-to-noise ratio, the lower the proportion of noise. Correlation refers to the similarity between two candidate sound signals. Correlation can be time domain correlation, frequency domain correlation, phase correlation, or time-frequency correlation.

[0035] The quality parameter can be obtained by a specific formula or by a pre-trained quality parameter network. In the case of obtaining by a quality parameter network, the candidate sound signal can be input into the quality parameter network to obtain the quality parameter of the candidate sound signal. Different quality parameters are obtained by different or the same quality parameter models.

[0036] A second number of target sound signals is selected from a first number of candidate sound signals based on a quality parameter. The first number can be set according to the adaptability of the target environment. The first number is at least part of the number of microphones included in the microphone array in the target environment. The second number is less than or equal to the first number. The second number can be fixed in the target environment or dynamic in the target environment (the candidate sound signals that meet the quality parameter relationship in the target environment are not fixed).

[0037] The quality parameter satisfies the quality parameter relationship, which means that the quality meets the requirements of sound source localization. In other words, the target sound signal whose quality parameter satisfies the quality parameter relationship is more conducive to sound source localization and can improve the accuracy of the sound source localization result.

[0038] In some embodiments, the quality parameter includes a single quality parameter, and the candidate sound signal whose single quality parameter satisfies the quality parameter relationship can be used as the target sound signal. The following is an example of a single quality parameter:

[0039] In the case where the quality parameter includes a signal-to-noise ratio, a second number of target sound signals whose signal-to-noise ratios satisfy a first magnitude relationship may be selected from each candidate sound signal. The signal-to-noise ratio satisfies the first magnitude relationship, which may be that the signal-to-noise ratio is greater than a preset signal-to-noise ratio threshold, or that the difference with the highest signal-to-noise ratio is less than a preset signal-to-noise ratio difference.

[0040] In the case where the quality parameter includes correlation, a second number of target sound signals whose correlations satisfy a strong-weak relationship can be selected from the candidate sound signals. The correlation satisfies the strong-weak relationship, which can be a correlation that is stronger than a preset correlation threshold. Alternatively, the correlation between any two of them can be the strongest, or the correlation of the candidate sound signal combination can be the highest. The first number of candidate sound signals can be divided into several candidate sound signal combinations, each candidate sound signal combination including a second number of candidate sound signals. The correlation of the candidate sound signal combination can be the mean, variance, weighted sum, etc. of the correlations between any two of the candidate sound signals in the candidate sound signal combination. In some embodiments, the correlations between any two of the candidate sound signals can be obtained; the second number of candidate sound signals with the strongest correlation between any two of them can be determined as the target sound signal.

[0041] In some embodiments, the quality parameters include multiple types, and the intersection of candidate sound signals that satisfy various quality parameter relationships can be used as the target sound signal. Alternatively, priorities can be set according to the importance of various quality parameters, and a third number of candidate sound signals that satisfy a higher priority quality parameter relationship can be selected first, where the third number is greater than or equal to the second number, and then a second number of candidate sound signals that satisfy a lower priority are selected from the third number of candidate sound signals. In the latter case, the lower priority selection step performed on the basis of the higher priority selection result can be set as an optional step. That is, when the second number is fixed and the third number is equal to the second number, there is no need to execute the lower priority selection step. When the second number is dynamic, it is necessary to continue to execute the lower priority selection step. The following is an example of the case of multiple quality parameters:

[0042] In the case where the quality parameters include signal-to-noise ratio and correlation, a second number of target sound signals whose signal-to-noise ratio satisfies the first size relationship and whose correlation satisfies the strength relationship can be selected from each candidate sound signal. Specifically, a third number of candidate sound signals whose signal-to-noise ratio satisfies the first size relationship can be selected from the first number of candidate sound signals; and then a second number of target sound signals whose correlation satisfies the strength relationship can be selected from the third number of candidate sound signals. Alternatively, a third number of candidate sound signals whose signal-to-noise ratio satisfies the first size relationship can be selected from the first number of candidate sound signals, and a fourth number of candidate sound signals whose correlation satisfies the strength relationship can be selected from the first number of candidate sound signals; the intersection of the third number of candidate sound signals and the fourth number of target sound signals is obtained to obtain the second number of target sound signals.

[0043] S130: Perform sound source localization based on each channel of target sound signals to obtain a sound source localization result.

[0044] The sound source localization result can represent the sound source position / sound zone. The sound source position can include the angle and distance of the microphone that collects each target sound signal relative to the sound source.

[0045] It is understandable that the microphone arrays in different sound source localization environments may be different. Different microphone arrays refer to different microphone layouts and / or numbers of microphones. For example, the number and layout of microphones inside vehicles of different car manufacturers are different. If the sound source localization method is only targeted at a specific microphone layout and a specific number of microphones, it cannot be applied to a variety of sound source localization environments. Then it is necessary to provide personalized customized services for various sound source localization environments and develop different sound source localization methods, which is complex and requires a lot of resource investment.

[0046] The above scheme obtains quality parameters of a first number of candidate sound signals, which are respectively collected by a first number of microphones in a microphone array; from the first number of candidate sound signals, a second number of target sound signals whose quality parameters satisfy the quality parameter relationship are selected; sound source positioning is performed based on each target sound signal to obtain a sound source positioning result. On the one hand, the first number and / or the second number can be freely configured according to the sound source positioning environment, so the sound source positioning method is not limited to the specific number of microphones and the specific microphone layout of the microphone array in the environment, and can therefore be flexibly applied to various sound source positioning environments. On the other hand, instead of directly applying the first number of candidate sound signals to sound source positioning, the second number of target sound signals whose quality parameters satisfy the quality parameter relationship are used for sound source positioning, and the target sound signal is more conducive to sound source positioning, thereby improving the accuracy of the sound source positioning result.

[0047] Figure 2 1 is a flow chart of an embodiment of a sound source localization method provided by the present application. Figure 2 As shown, this embodiment is a further extension of the above embodiment. In this embodiment, the target sound signal includes a first target sound signal and a second target sound signal, the quality parameter includes a signal-to-noise ratio, and the sound source localization method may include the following steps:

[0048] S210: Selecting, from among the candidate sound signals, a candidate sound signal whose signal-to-noise ratio satisfies a signal-to-noise ratio condition as a first target sound signal.

[0049] In some embodiments, the signal-to-noise ratio condition is that the signal-to-noise ratio is the highest. In this case, the first target sound signal is the candidate sound signal with the best quality.

[0050] In some embodiments, the signal-to-noise ratio condition is that the signal-to-noise ratio is greater than the median value of the signal-to-noise ratios of the candidate sound signals. The median value may be the median of the signal-to-noise ratios of the candidate sound signals, or the average of the maximum signal-to-noise ratio and the minimum signal-to-noise ratio of the signal-to-noise ratios of the candidate sound signals, or the average of the signal-to-noise ratios of the candidate sound signals. In this case, the first target sound signal is a candidate sound signal that is above the minimum quality standard.

[0051] S220: Selecting, from the remaining candidate sound signals except the first target sound signal, a second target sound signal whose signal-to-noise ratio magnitude relationship with the first target sound signal satisfies a second magnitude relationship.

[0052] When the signal-to-noise ratio condition is the highest signal-to-noise ratio, the second magnitude relationship may be that the difference between the signal-to-noise ratios of the remaining candidate sound signals and the first target sound signal is less than the signal-to-noise ratio difference threshold. That is, the first target sound signal that is most conducive to sound source localization (with the best quality) is first selected, and then the second target sound signal is selected based on the first target sound signal. In this manner, the number of second target sound signals may be 0 or greater than 0.

[0053] When the signal-to-noise ratio condition is that the signal-to-noise ratio is greater than the median value of the signal-to-noise ratios of the candidate sound signals, the second magnitude relationship may be that the signal-to-noise ratios of the remaining candidate sound signals are greater than or equal to the median value of the signal-to-noise ratios. That is, the first target sound signal that meets the minimum quality standard is first selected, and then the second target sound signal is selected based on the first target sound signal.

[0054] The following describes S210-S220 using the target environment of a car as an example:

[0055] Obtain the signal-to-noise ratio of N candidate sound signals; select the candidate sound signal with the highest signal-to-noise ratio as the first target sound signal; select M-1 (1<=M<=N) candidate sound signals whose signal-to-noise ratio difference with the signal-to-noise ratio of the first target sound signal is less than the signal-to-noise ratio difference threshold as the second target sound signal. It can be understood that the difference between the signal-to-noise ratio and the signal-to-noise ratio of the first target sound signal is greater than or equal to the signal-to-noise ratio difference threshold, which means that the quality loss is serious (for example, microphone clipping caused by excessive wind noise in the vehicle during driving). The number of second target sound signals is 0, which means that the microphone array of the in-vehicle environment is in omnidirectional mode without sound zone separation. The number of second target sound signals is greater than 0, which means that the microphone array of the in-vehicle environment is in directional mode with sound zone separation.

[0056] The above method of first selecting the first target sound signal with the lowest standard / best quality and then selecting the second target sound signal based on the first target sound signal is not limited to the implementation on the quality parameter of signal-to-noise ratio, and can also be implemented on other quality parameters.

[0057] The above scheme provides a method for selecting a target sound signal from candidate sound signals. The target sound signal includes a first target sound signal and a second target sound signal. The first target sound signal is selected first, and then the second target sound signal is selected based on the first target sound signal, thereby achieving the selection of the target sound signal.

[0058] In some embodiments, the sound source localization based on each channel of target sound signals in the above S130 may be directly based on each channel of target sound signals.

[0059] In some embodiments, in the above S130, the sound source is localized based on each target sound signal, and the time delay of the remaining target sound signals relative to the target sound signal with the best quality can be obtained, and the sound source is localized based on the time delay and the target sound signal with the best quality. In the case where the aforementioned first target sound signal is the best quality among the target sound signals, the target sound signal with the best quality referred to here can continue to use the first target sound signal, and the remaining target sound signals can continue to use the aforementioned second target sound signal. That is, the time delay of the second target sound signal relative to the first target sound signal is obtained; based on the time delay and the first target sound signal, the sound source localization result is obtained.

[0060] Among them, time delay refers to the time difference collected. In some embodiments, the frequency domain characteristics of the first target sound signal and the second target sound signal can be obtained, and the frequency domain characteristics include the logarithmic power spectrum of the first target sound signal and the second target sound signal, and the phase difference of the second target sound signal relative to the first target sound signal; the time delay is obtained based on the frequency domain characteristics. In some embodiments, the time delay can be calculated based on a specific formula between the frequency domain characteristics and the time delay. In some embodiments, the time delay can be obtained based on the frequency domain characteristics using a trained time delay acquisition network. It can be understood that the time delay is an explicit sound source position feature, and the introduction of the time delay when locating the sound source can improve the accuracy of the sound source positioning.

[0061] For example, there are M target sound signals in total, of which 1 is a first target sound signal and M-1 are second target sound signals.

[0062] Perform Fourier transform on M target sound signals to obtain frequency domain features (M groups of logarithmic power spectra and M-1 groups of phase differences). Among them, the mth target sound signal (the second target sound signal) is y m (t), for y m (t) Perform Fourier transform Y m The logarithmic power spectrum obtained by (t, f) is LPS m (t, f) = log(|Y m (t, f)| 2). Where m<=M, t represents the time frame, and f represents the frequency. The phase difference IPD between the mth target sound signal and the refth target sound signal (the first target sound signal) is m (t, f) = ∠Y m (t, f)-∠Y ref (t, f).

[0063] Based on M groups of logarithmic power spectra and M-1 groups of phase differences, we can obtain M-1 groups of time delays τ m (t, f).

[0064] In some embodiments, the sound source positioning based on each channel of target sound signals in the above S130 can also be implemented based on a controllable beamforming positioning method (beamforming method) with maximum output power, a positioning method based on high-resolution spectrum estimation, and the like.

[0065] Furthermore, the sound source localization method provided in the above embodiment can be applied to application scenarios such as fault detection and sound recognition.

[0066] In the application scenario of machine fault detection, a microphone can be used to obtain sound signals at a certain distance from the machine parts, and then the sound source localization technology can be used as a virtual sensing tool to measure the vibroacoustic information.

[0067] Voice recognition application scenarios may include intelligent voice interaction, voice input, voice commands, voice search, etc. Taking in-car voice interaction as an example, the driver's voice command can be recognized, and personalized responses (such as the car computer broadcasting "Hello, driver") or personalized services (such as opening the driver's window or turning on the driver's seat heating) can be made based on the recognition results.

[0068] Figure 3 1 is a flow chart of an embodiment of a sound recognition method provided by the present application. Figure 3 As shown, the voice recognition method provided in this embodiment includes the following steps:

[0069] S310: Obtain quality parameters of a first number of candidate sound signals.

[0070] The first number of candidate sound signals are respectively collected by a first number of microphones in the microphone array.

[0071] S320: Selecting a second number of target sound signals whose quality parameters satisfy the quality parameter relationship from the first number of candidate sound signals.

[0072] S330: Perform sound source localization based on each channel of target sound signals to obtain a sound source localization result.

[0073] For the detailed description of S310 - S330 in this embodiment, please refer to the previous embodiment and will not be repeated here.

[0074] S340: Performing sound recognition based on the sound source localization result and the target sound signal to obtain a sound recognition result.

[0075] In the above scheme, on the one hand, the method for obtaining the sound source localization result is not limited to the specific number of microphones and the specific layout of the microphone array in the environment, and can be flexibly applied to various sound source localization environments. On the other hand, instead of directly applying the first number of candidate sound signals to the sound source localization, the second number of target sound signals whose quality parameters satisfy the quality parameter relationship are used for the sound source localization. The target sound signal is more conducive to the sound source localization, thereby improving the accuracy of the sound source localization result, and further improving the accuracy of the sound recognition result.

[0076] In some embodiments, at least some steps / sub-steps in the sound signal selection (S110-S120 / S310-S320) are implemented based on a pre-trained sound signal selection network. For example, all steps of the sound signal selection are implemented in the sound signal selection network, the input of the sound signal selection network is the candidate sound signal, and the output is the target sound signal.

[0077] In some embodiments, at least some steps / sub-steps in sound source localization (S130 / S330) are implemented based on a pre-trained sound source localization network. For example, all steps of sound source localization are implemented in the sound source localization network, the input of the sound source localization network is the target sound signal, and the output is the sound source localization result. For another example, the step after obtaining the time delay of the second target sound signal relative to the first target sound signal in the sound source localization is implemented in the sound source localization network, the input of the sound source localization network is the time delay and the first target sound signal, and the output is the sound source localization result. It can be understood that the input includes the time delay, which is an explicit sound source location feature. Compared with the method of only inputting the first target sound signal into the sound source localization network, it can avoid the implicit learning of the sound source location by the sound source localization network and improve the accuracy of sound source localization.

[0078] In some embodiments, the sound source localization network may include a feature extraction layer and a localization layer. The feature extraction layer is used to extract sound source location features based on each target sound signal; the localization layer is used to obtain a sound source localization result based on the sound source location features.

[0079] In some embodiments, at least some steps / sub-steps of sound recognition (S340) are implemented based on a pre-trained sound recognition network. The input of the sound recognition network is the sound source localization result (the final output of the sound source localization network) and the second number of target sound signals, or the input is the sound source position feature (the hidden layer output of the sound source localization network) and the second number of target sound signals, and the output is the sound recognition result. It can be understood that, relative to the sound source localization result, since the sound source position feature can express the sound source position and is closer to the target sound signal, it is beneficial to improve the accuracy of the sound recognition result.

[0080] In some embodiments, the training data used to train the sound signal selection network, the sound source localization network, and the sound source identification network are similar to the sound signals in the application phase, and both contain the target sound emitted by the sound source and the interference sound in the target environment.

[0081] In some embodiments, the sound source localization network and the sound source identification network can be trained simultaneously.

[0082] In some embodiments, the sound source localization network and the sound source identification network can be trained jointly, that is, the sound source localization network is trained first, and after the sound source localization network is trained, it is used to train the sound source identification network, thereby improving the training effect.

[0083] Figure 4 This is a schematic diagram of the joint training of the sound source localization network and the sound source recognition network of this application. Figure 4 As shown, the training process is as follows:

[0084] The training data set is input into the sound signal selection network to obtain a training sound signal selection result. The training data set includes several groups of training sound signals, each group of training sound signals includes N training candidate sound signals. The training sound signal selection result corresponding to each group of training sound signals includes 1 training first target sound signal and M-1 training second target sound signals.

[0085] The sound source localization network is trained using the sound signal selection results to obtain a trained sound source localization network. The input of the sound source localization network during the training process is the training sound signal selection results, the output of the feature extraction layer is the training sound source position feature, and the output of the localization layer is the training sound source localization result. The outputs of the feature extraction layer and the localization layer can both be used for the training of the sound source localization network.

[0086] The sound recognition network is trained using the output of the feature extraction layer (training sound source location features) / the output of the localization layer (training sound source localization results) of the trained sound source localization network and the training sound signal selection results. The output of the sound recognition network during the training process is the training sound recognition result.

[0087] Figure 5Schematic diagram of the simultaneous training of the sound source localization network and the sound recognition network of the present application. Figure 5 As shown in the figure, the sound source localization network and the sound source recognition network are combined into a sound source localization and sound recognition network for training. The training process is as follows:

[0088] The training data set is input into the sound signal selection network to obtain a training sound signal selection result. The training data set includes several groups of training sound signals, each group of training sound signals includes N training candidate sound signals. The training sound signal selection result corresponding to each group of training sound signals includes 1 training first target sound signal and M-1 training second target sound signals.

[0089] The training sound signal selection result is input into the sound source localization and sound recognition network to obtain the training sound source localization result and the training sound recognition result.

[0090] Figure 6 : is a flow chart of another embodiment of the sound recognition method provided by the present application. In this embodiment, the step of performing sound source localization based on each channel of target sound signals to obtain the sound source localization result is implemented by using a sound source localization network; the step of performing sound recognition based on the sound source localization result and the target sound signal to obtain the sound recognition result is implemented by using a sound recognition network. The sound source localization network includes a feature extraction layer and a localization layer. Figure 6 As shown, the voice recognition method provided in this embodiment includes the following steps:

[0091] S410: Extracting sound source position features based on each target sound signal using a feature extraction layer.

[0092] The sound source location feature refers to the feature that can express the location of the sound source in the target environment.

[0093] S420: Obtain a sound source localization result based on the sound source position feature using the localization layer.

[0094] S430: Obtain a sound recognition result based on the sound source position feature and each channel of target sound signals using a sound recognition network; or obtain a sound recognition result based on the sound source positioning result and each channel of target sound signals using a sound recognition network.

[0095] The above S410-S420 are steps of performing sound source localization using a sound source localization network, and S430 is a step of performing sound recognition using a sound recognition network.

[0096] Through the implementation of S410-S430, the sound source localization network is used to realize the sound source localization, and the sound recognition network is further used to perform sound recognition to obtain the sound recognition result. On the one hand, the input of the sound recognition network can be the sound source position feature or the sound source localization result output by the sound source localization network. Compared with the sound source localization result, the sound source position feature can characterize the sound source position while retaining the information of the target sound signal. When used for sound recognition, it can improve the accuracy of the sound recognition result. On the other hand, not only the sound source position feature or the sound source localization result is input into the sound recognition network, but also the target sound signal is input into the sound recognition network, which can make up for the loss of the target sound signal information in the sound source localization process, and further improve the accuracy of the sound recognition result.

[0097] In order to facilitate a better understanding of the present application, the sound source localization method and sound recognition method provided by the present application are explained as an example in the in-vehicle voice interaction scenario as follows.

[0098] In the in-car voice interaction scenario, possible noise sources in the car include wind noise and tire noise generated by driving, discussions between passengers and media echoes. These noises will affect the quality of the sound signals collected by the microphone.

[0099] S1: Obtain N candidate sound signals collected by N microphones in a microphone array respectively.

[0100] S2: Obtain the signal-to-noise ratios of N candidate sound signals.

[0101] S3: taking the candidate sound signal with the highest signal-to-noise ratio as the first target sound signal.

[0102] S4: Determine M-1 candidate sound signals whose difference between the signal-to-noise ratio and the highest signal-to-noise ratio is greater than a preset signal-to-noise ratio difference threshold as second target sound signals.

[0103] S5: extracting frequency domain features of the first target sound signal and the second target sound signal. The frequency domain features include M groups of logarithmic power spectra and M-1 groups of phase differences. The M-1 groups of phase differences include the phase differences of M-1 second target sound signals relative to the first target sound signal.

[0104] S6: Calculate M-1 groups of delays based on the frequency domain features. The M-1 groups of delays include the delays of M-1 second target sound signals relative to the first target sound signal.

[0105] S7: Based on the first target sound signal and M-1 groups of time delays, obtain a sound source position feature / sound source localization result.

[0106] S8: Perform sound recognition based on the sound source position feature / sound source localization result, the first target sound signal and the second target sound signal to obtain a sound recognition result.

[0107] The sound source localization method provided in the above embodiments of the present application can be implemented by a sound source localization device. And, the sound recognition method based on the sound source localization result can be implemented by a sound recognition device. The sound source localization device and the sound recognition device can be the same or different.

[0108] Figure 7 Schematic diagram of the structure of an embodiment of the sound source localization device of the present application. Figure 7 As shown, the sound source localization device 50 includes a sound signal acquisition module 51 , a selection module 52 and a sound source localization module 53 .

[0109] Among them, the sound signal acquisition module 51 can be used to obtain quality parameters of a first number of candidate sound signals, which are respectively collected by a first number of microphones in the microphone array; the selection module 52 can be used to select a second number of target sound signals whose quality parameters satisfy the quality parameter relationship from the first number of candidate sound signals; the sound source localization module 53 can be used to perform sound source localization based on each target sound signal to obtain a sound source localization result.

[0110] For other detailed descriptions of the sound source localization device 50 , please refer to the previous method embodiments, which will not be repeated here.

[0111] Figure 8 Schematic diagram of the structure of an embodiment of the sound recognition device of the present application. Figure 8 As shown, the sound recognition device 60 includes a sound source localization result acquisition module 61 and a sound recognition module 62 .

[0112] The sound source localization result acquisition module 61 can be used to acquire the sound source localization result, and the sound recognition module 62 can be used to perform sound recognition based on the sound source localization result and the target sound signal to obtain the sound recognition result.

[0113] In some embodiments, the sound source localization result acquisition module 61 may acquire the sound source localization result from the aforementioned sound source localization device 50 .

[0114] In some embodiments, the sound source localization result acquisition module 61 can acquire the sound source localization result according to the sound source localization method provided in the above embodiments. In this case, the sound source localization result acquisition module 61 can include a sound signal acquisition module 611, a selection module 612 and a sound source localization module 613.

[0115] Among them, the sound signal acquisition module 611 is used to obtain quality parameters of a first number of candidate sound signals, which are respectively collected by a first number of microphones in the microphone array; the selection module 612 is used to select a second number of target sound signals whose quality parameters satisfy the quality parameter relationship from the first number of candidate sound signals; the sound source localization module 613 is used to perform sound source localization based on each target sound signal to obtain a sound source localization result.

[0116] For other detailed descriptions of the voice recognition device 60 , please refer to the previous method embodiments, which will not be repeated here.

[0117] Fig. 9 Schematic diagram of the structure of an embodiment of the electronic device of the present application. Fig. 9 As shown, the electronic device 70 includes a memory 71 and a processor 72, and the processor 72 is used to execute the program instructions stored in the memory 71 to implement the steps in any of the above method embodiments. In a specific implementation scenario, the electronic device 70 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 70 may also include a carrier device such as a laptop computer and a tablet computer, which is not limited here.

[0118] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps in any of the above method embodiments. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0119] See also Fig.10 , Fig.10 The computer-readable storage medium 80 stores program instructions 801, which implement the steps in any of the above method embodiments when executed by a processor.

[0120] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0121] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. In another image position, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0123] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0124] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A sound source localization method, characterized in that: include: Acquire quality parameters of a first number of candidate sound signals, where the first number of candidate sound signals are respectively collected by the first number of microphones in the microphone array; Selecting, from the first number of candidate sound signals, a second number of target sound signals whose quality parameters satisfy a quality parameter relationship; The sound source is localized based on the target sound signals of each channel to obtain a sound source localization result.

2. The method according to claim 1, characterized in that The selecting, from the first number of candidate sound signals, a second number of target sound signals whose quality parameters satisfy a quality parameter relationship comprises: In the case where the quality parameter includes a signal-to-noise ratio, selecting a second number of target sound signals whose signal-to-noise ratios satisfy a first magnitude relationship from the candidate sound signals; and / or In the case where the quality parameter includes correlation, a second number of target sound signals whose correlations satisfy a strong-weak relationship are selected from the candidate sound signals.

3. The method according to claim 2, characterized in that The target sound signal includes a first target sound signal and a second target sound signal, and selecting a first number of target sound signals whose signal-to-noise ratios satisfy a first magnitude relationship from the candidate sound signals includes: Selecting, from the candidate sound signals of each channel, a candidate sound signal whose signal-to-noise ratio satisfies a signal-to-noise ratio condition as the first target sound signal; From the remaining candidate sound signals except the first target sound signal, the second target sound signal whose signal-to-noise ratio magnitude relationship with the first target sound signal satisfies a second magnitude relationship is selected.

4. The method according to claim 3, characterized in that The signal-to-noise ratio condition is that the signal-to-noise ratio is the highest; and / or The second magnitude relationship is that a difference between the signal-to-noise ratios of the remaining candidate sound signals and the first target sound signal is less than a signal-to-noise ratio difference threshold.

5. The method according to claim 2, characterized in that: The step of selecting a first number of target sound signals whose correlations satisfy a strong-weak relationship from the candidate sound signals comprises: The candidate sound signals of the second number having the strongest correlations between each other are determined as the target sound signals.

6. The method according to claim 1, characterized in that The second number of target sound signals include a first target sound signal and a second target sound signal, and performing sound source localization based on each of the target sound signals to obtain a sound source localization result includes: Acquiring a time delay of the second target sound signal relative to the first target sound signal; The sound source localization result is obtained based on the time delay and the first target sound signal.

7. A sound recognition method, characterized in that: include: Acquire quality parameters of a first number of candidate sound signals, where the first number of candidate sound signals are respectively collected by the first number of microphones in the microphone array; Selecting, from the first number of candidate sound signals, a second number of target sound signals whose quality parameters satisfy a quality parameter relationship; Performing sound source positioning based on the target sound signals of each channel to obtain a sound source positioning result; Sound recognition is performed based on the sound source localization result and the target sound signal to obtain a sound recognition result.

8. The method according to claim 7, characterized in that The step of performing sound source localization based on each channel of the target sound signal to obtain a sound source localization result is implemented by using a sound source localization network; and / or The step of performing sound recognition based on the sound source localization result and the target sound signal to obtain the sound recognition result is achieved by using a sound recognition network.

9. The method according to claim 8, characterized in that The sound source localization network includes a feature extraction layer and a localization layer. The steps of using the sound source localization network to localize the sound source include: Utilizing the feature extraction layer to extract the sound source position features based on each channel of the target sound signal; Obtaining the sound source localization result based on the sound source position feature using the localization layer; The steps of using the voice recognition network to perform voice recognition include: Obtaining the sound recognition result based on the sound source position feature and each channel of the target sound signal using the sound recognition network; or The sound recognition network is used to obtain the sound recognition result based on the sound source localization result and each channel of the target sound signal.

10. A sound source localization device, characterized in that: include: A sound signal acquisition module, used to acquire quality parameters of a first number of candidate sound signals, where the first number of candidate sound signals are respectively collected by the first number of microphones in the microphone array; A selection module, configured to select, from the first number of candidate sound signals, a second number of target sound signals whose quality parameters satisfy a quality parameter relationship; The sound source localization module is used to perform sound source localization based on the target sound signals of each channel to obtain a sound source localization result.

11. A voice recognition device, characterized in that: include: A sound signal acquisition module, used to acquire quality parameters of a first number of candidate sound signals, where the first number of candidate sound signals are respectively collected by the first number of microphones in the microphone array; A selection module, configured to select, from the first number of candidate sound signals, a second number of target sound signals whose quality parameters satisfy a quality parameter relationship; A sound source localization module is used to localize the sound source based on the target sound signals of each channel to obtain a sound source localization result; The sound source recognition module is used to perform sound recognition based on the sound source positioning result and the target sound signal to obtain a sound recognition result.

12. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images