Sound source localization method, device, computer equipment and storage medium

Abstract

The present invention discloses a method for locating a sound source, which includes obtaining an audio acquisition signal, preprocessing the audio acquisition signal, and obtaining an audio signal to be located; obtaining a preset voiceprint recognition model, and performing voiceprint matching on the audio signal to be located through the preset voiceprint recognition model to determine whether the audio signal to be located includes a target audio signal; when the audio signal to be located includes a target audio signal, obtaining at least one target acquisition image; obtaining a preset image recognition model, and performing image matching on the target acquisition image through the preset image recognition model to obtain a target image; one target image is associated with one sound source audio signal; and the sound source audio signal and the target audio signal are matched to obtain a sound source locating result. In this way, the acquisition of the audio signal to be located is achieved by preprocessing the audio acquisition signal. The determination of the target audio signal is achieved through voiceprint matching. The acquisition of the target image is achieved through image matching.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to a sound source localization method, device, computer equipment and storage medium. Background Art

[0002] Voice interaction is widely used in scenarios such as smart conferences or smart customer service. In the application process of voice interaction, it is often necessary to collect the voice signal of the target sound source in a noisy environment. Therefore, it is very important to accurately locate the target sound source.

[0003] In the prior art, the microphone position is often manually adjusted so that the microphone can face the target sound source to collect voice signals. However, the target sound source is constantly changing, and the manual method cannot adjust the microphone to the correct position in time. This results in low accuracy and efficiency in locating the target sound source, thereby reducing the quality of voice signal collection. Summary of the invention

[0004] The present invention provides a sound source localization method, device, computer equipment and storage medium, which solves the problems of low positioning accuracy and low efficiency in the prior art.

[0005] A sound source localization method, comprising:

[0006] Acquire an audio acquisition signal, and preprocess the audio acquisition signal to obtain an audio signal to be located;

[0007] Acquire a preset voiceprint recognition model, input the audio signal to be located into the preset voiceprint recognition model, and perform voiceprint matching on the audio signal to be located through the preset voiceprint recognition model to determine whether the audio signal to be located includes a target audio signal;

[0008] When the audio signal to be located includes the target audio signal, obtaining at least one target acquisition image, wherein the target acquisition image is obtained by photographing an image acquisition device in a target acquisition area; the target acquisition area is generated according to a signal acquisition area of ​​the target audio signal;

[0009] Obtain a preset image recognition model, input the target captured image into the preset image recognition model, perform image matching on the target captured image through the preset image recognition model, and obtain a target image; one target image is associated with one sound source audio signal;

[0010] The sound source audio signal and the target audio signal are matched to obtain a sound source localization result.

[0011] A sound source localization device, comprising:

[0012] An acquisition module is used to acquire an audio acquisition signal and pre-process the audio acquisition signal to obtain an audio signal to be located;

[0013] a voiceprint matching module, configured to obtain a preset voiceprint recognition model, input the audio signal to be located into the preset voiceprint recognition model, and perform voiceprint matching on the audio signal to be located through the preset voiceprint recognition model to determine whether the audio signal to be located includes a target audio signal;

[0014] A collection module, used for acquiring at least one target collection image when the audio signal to be located includes the target audio signal, wherein the target collection image is obtained by photographing in a target collection area by an image collection device; the target collection area is generated according to a signal collection area of ​​the target audio signal;

[0015] An image matching module is used to obtain a preset image recognition model, input the target acquisition image into the preset image recognition model, and perform image matching on the target acquisition image through the preset image recognition model to obtain a target image; one target image is associated with one sound source audio signal;

[0016] The result module is used to perform signal matching on the sound source audio signal and the target audio signal to obtain a sound source localization result.

[0017] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned sound source localization method when executing the computer program.

[0018] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the sound source localization method described above is implemented.

[0019] The sound source localization method, device, electronic device and storage medium provided by the present invention realize the acquisition of the audio signal to be located by preprocessing the audio acquisition signal, thereby improving the signal-to-noise ratio and quality of the audio signal. By performing voiceprint matching through a preset voiceprint recognition model, it can be quickly determined whether the audio signal to be located includes the target audio signal. Then, the target acquisition image captured by the image acquisition device is matched with the preset image acquisition device, and the target sound source is accurately located in combination with the audio signal matching method. The accuracy and efficiency of sound source localization are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0021] Figure 1 is a flow chart of a sound source localization method in one embodiment of the present invention;

[0022] Figure 2 is a flow chart of step S20 of the sound source localization method in one embodiment of the present invention;

[0023] Figure 3 is a flow chart of step S30 of the sound source localization method in one embodiment of the present invention;

[0024] Figure 4 is a principle block diagram of a sound source localization device in one embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In one embodiment, if Figure 1 As shown, a sound source localization method is provided, comprising the following steps S10-S50:

[0028] S10, acquiring an audio acquisition signal, and preprocessing the audio acquisition signal to obtain an audio signal to be located.

[0029] Understandably, the audio collection signal is an audio signal collected by a directional microphone array. The audio signal is a frequency and amplitude variation information carrier of a regular sound wave with voice. The audio signal to be located is an audio signal for which the direction and position need to be determined. Among them, the microphone array can be a cardioid microphone array, a ring microphone array, a linear microphone array, a bidirectional microphone array or an omnidirectional microphone array. The microphones in the microphone array are all multi-channel microphones. The number of microphones in the microphone array can be set according to actual conditions, such as when the spatial range is small, 4 microphones or 6 microphones are set, and when the spatial range is large, 8 microphones or 12 microphones are set. And the four areas of the front, back, left and right of the microphone array are divided into different microphone collection areas. For example, when there are 4 microphones in the microphone array, one microphone corresponds to collecting the audio signal in one microphone collection area. That is, one multi-channel microphone is used to collect audio signals from the microphone collection area in front of the microphone array; one multi-channel microphone is used to collect audio signals from the microphone collection area behind the microphone array; one multi-channel microphone is used to collect audio signals from the microphone collection area to the left of the microphone array; and one multi-channel microphone is used to collect audio signals from the microphone collection area to the right of the microphone array.

[0030] Specifically, the audio signals in the surrounding environment are collected by the microphone array to obtain an audio collection signal. By summing, amplifying and noise reducing the audio collection signals collected by the microphone array in turn, the preprocessing of the audio collection signal can be completed to obtain the audio signal to be located. Assuming that in the application scenario of smart conference, when there are multiple multi-channel microphones, each multi-channel microphone corresponds to a different audio signal collection area. Therefore, the audio collection signal includes the audio signals collected by all multi-channel microphones. Furthermore, there may be multiple different audio signals in the audio signal to be located.

[0031] Exemplarily, in a smart conference scenario, all multi-channel microphones in the microphone array will collect audio signals in the surrounding environment, when a multi-channel microphone collects audio signals in the microphone collection area in front of the microphone array. The multi-channel microphone on the left collects audio signals in the microphone collection area on the left side of the microphone array. The multi-channel microphone on the right collects audio signals in the microphone collection area on the right side of the microphone array. The multi-channel microphone at the back collects audio signals in the microphone collection area behind the microphone array, that is, multiple multi-channel microphones in the microphone array collect audio signals in the surrounding environment at the same time.

[0032] Further, in one embodiment, when the collection direction changes, that is, the original left, right or back becomes the current front, for example, the multi-channel microphone originally on the left becomes the multi-channel microphone currently in front, then the audio signal in the collection area of ​​the microphone currently in front is collected. The multi-channel microphone originally in front becomes the multi-channel microphone currently on the right, then the audio signal in the collection area of ​​the microphone currently on the right is collected. The multi-channel microphone originally on the right becomes the multi-channel microphone currently at the back, then the audio signal in the collection area of ​​the microphone currently at the back is collected. The multi-channel microphone originally at the back becomes the multi-channel microphone currently on the left, then the audio signal in the collection area of ​​the microphone currently on the left is collected.

[0033] S20, obtaining a preset voiceprint recognition model, and inputting the audio signal to be located into the preset voiceprint recognition model, performing voiceprint matching on the audio signal to be located through the preset voiceprint recognition model to determine whether the audio signal to be located includes a target audio signal.

[0034] Understandably, the preset voiceprint recognition model is a voiceprint recognition model that has been trained in advance, such as inputting the voice of a person or animal into the model as training data, and training the model. When the accuracy of model recognition reaches the set threshold (such as when the accuracy is above 95%), the model training is terminated to obtain the voiceprint recognition model. The target audio signal is the audio signal emitted by the target object. Among them, the target object is different in different application scenarios. For example, in a smart conference scenario, it is necessary to collect audio signals of different users, and the target object is the user. In the animal sound collection scenario, the target object is the animal.

[0035] Specifically, after obtaining the audio signal to be located, the preset voiceprint recognition model is retrieved, the audio signal to be located is input into the preset voiceprint recognition model, and the audio signal to be located is matched with the voiceprint through the preset voiceprint recognition model. That is, a number of audio signals are pre-stored in the preset voiceprint recognition model (the audio signal can be determined according to different application scenarios. For example, in a smart conference, the audio signal can be the audio signal of the user participating in the conference). When the target audio signal and the audio signal in the preset voiceprint recognition model fail to match, it is determined that the audio signal to be located does not include the target audio signal. When the target audio signal and the audio signal in the preset voiceprint recognition model successfully match, it is determined that the audio signal to be located includes the target audio signal.

[0036] S30, when the audio signal to be located includes the target audio signal, obtaining at least one target acquisition image, wherein the target acquisition image is obtained by photographing an image acquisition device in a target acquisition area; the target acquisition area is generated according to a signal acquisition area of ​​the target audio signal.

[0037] Understandably, the target acquisition image is an image captured by the image acquisition device in the target acquisition area. The image acquisition device is a device for capturing images, such as a camera or a digital camera. The signal acquisition area is the acquisition area corresponding to the multi-channel microphone that captures the target audio signal. The target acquisition area is the shooting area after the position or field of view of the image acquisition device is adjusted by the signal acquisition area. The target acquisition area can be the same as the signal acquisition area, or it can be within the signal acquisition area (that is, the target acquisition area is smaller than the signal acquisition area).

[0038] Specifically, after determining that the audio signal to be located includes the target audio signal, the microphone collection area of ​​the microphone array corresponding to the target audio signal is obtained, and the microphone collection area is determined as the signal collection area. Then, the shooting area of ​​the image acquisition device can be adjusted by the signal collection area to obtain the target collection area. Thus, the image acquisition device is allowed to shoot in the target collection area to obtain the target collection image. Among them, the frequency of shooting of the image acquisition device is not limited here. For example, the image acquisition device can shoot once every 1s.

[0039] S40, obtaining a preset image recognition model, inputting the target collected image into the preset image recognition model, performing image matching on the target collected image through the preset image recognition model, and obtaining a target image; one target image is associated with one sound source audio signal.

[0040] Understandably, the preset image recognition model is an image recognition model that has been trained in advance, such as inputting images of people or animals into the model as training data, and training the model. When the accuracy of model recognition reaches the set threshold (such as when the accuracy is above 97%), the model training is terminated to obtain the image recognition model. The target image is a target acquisition image that successfully matches the image in the preset image recognition model. One target image is associated with one sound source audio signal. The sound source audio signal is the audio signal emitted by the target object corresponding to the pre-captured target image.

[0041] Specifically, after obtaining the target acquisition image, the preset image recognition model is retrieved, the target acquisition image is input into the preset image recognition model, and all target acquisition images and images in the preset image recognition model are matched in turn. That is, a number of images are pre-stored in the preset image recognition model (the image can be determined according to different application scenarios. For example, in a smart meeting, the image can be a user participating in the meeting). When the target acquisition image and the image in the preset image recognition model are different, a matching failure result is obtained, and the user is prompted that the matching failed. When the target acquisition image and the image in the preset image recognition model are the same, the target acquisition image is determined as the target image.

[0042] S50: performing signal matching on the sound source audio signal and the target audio signal to obtain a sound source localization result.

[0043] Specifically, after obtaining the target image, the sound source audio signal associated with the target image is retrieved, and the sound source audio signal and the target audio signal are matched. When the sound source audio signal and the target audio signal are the same, a sound source localization result indicating a successful match is obtained. When the sound source audio signal and the target audio signal are different, a sound source localization result indicating a failed match is obtained. The sound source localization result indicates whether the sound source audio signal and the target audio signal are successfully matched.

[0044] The present invention achieves the acquisition of the audio signal to be located by preprocessing the audio acquisition signal, thereby improving the signal-to-noise ratio and quality of the audio signal. By performing voiceprint matching through a preset voiceprint recognition model, it can be quickly determined whether the audio signal to be located includes the target audio signal. Then, the target acquisition image captured by the image acquisition device is matched by a preset image acquisition device, and the target sound source is accurately located by combining the audio signal matching method. The accuracy and efficiency of sound source positioning are improved.

[0045] In one embodiment, the step S10, i.e. obtaining an audio acquisition signal, includes:

[0046] (1) Acquiring sound wave signals through a microphone array and converting the sound wave signals into electric signals to obtain analog audio signals.

[0047] Understandably, the sound wave signal is all the sounds collected by the microphone array, such as human voices and scene noise. The analog audio signal is the electrical signal after the sound-to-electric conversion. The electrical signal is a signal with voltage, current or electromagnetic waves as carriers. The multi-channel microphones in the microphone array collect the sound wave signals in the surrounding environment, and the electrical signal obtained after the sound wave signals are converted from sound to electricity is the analog audio signal.

[0048] (2) Performing analog-to-digital conversion on the analog audio signal to obtain the audio acquisition signal.

[0049] Specifically, an analog-to-digital conversion circuit performs analog-to-digital conversion on an analog audio signal to obtain a digital audio signal, and the obtained digital audio signal is determined as an audio acquisition signal. The analog-to-digital conversion is to convert an analog signal into a digital signal.

[0050] The present invention collects sound wave signals through a microphone array, and performs acoustic-to-electrical conversion and analog-to-digital conversion on the sound wave signals, thereby achieving the acquisition of audio collection signals.

[0051] In one embodiment, the step S10, i.e., preprocessing the audio acquisition signal to obtain the audio signal to be located, includes:

[0052] (1) The audio acquisition signal is weighted and summed using a beamforming algorithm to obtain a valid audio signal.

[0053] It can be understood that the effective audio signal is an audio signal obtained by adding multiple sound wave signals. The window function includes rectangular window, Hamming window, Hanning window and Blackman window.

[0054] Specifically, after obtaining the audio acquisition signal, the audio acquisition signal is framed, that is, the audio acquisition signal is divided into segments of fixed duration, such as 25 milliseconds. The divided audio signal is windowed, that is, a window function is added to each segment of fixed duration, so that the discontinuous audio signal after division becomes continuous and exhibits the characteristics of a periodic function. The windowed audio signal is processed by Fourier transform, and the windowing in the time domain is converted into a convolution in the frequency domain. And a delayed summation beamforming algorithm is used to sum the audio acquisition signal of each multi-channel microphone in the frequency domain to obtain a valid audio signal. Among them, the beamforming algorithm is an algorithm for summing multiple sound wave signals. Further, when a directional microphone array is used, the audio acquisition signal can be directly processed by the microphone array to obtain a valid audio signal. When a circular microphone array or a linear microphone array is used, the audio acquisition signal needs to be summed by a beamforming algorithm, that is, a valid audio signal is obtained through the above process.

[0055] (2) The effective audio acquisition signal is amplified by an automatic gain control circuit to obtain an amplified audio signal.

[0056] Understandably, the automatic gain control circuit is a circuit that automatically adjusts the gain of the amplifier through a feedback system, usually a control circuit that automatically adjusts the gain of the amplifier circuit with the signal strength. The feedback system is a circuit that adjusts the input signal according to the output signal. The amplifier circuit is a signal that amplifies the input weak signal to the required amplitude value and is consistent with the change law of the original input signal. Among them, the amplifier circuit includes a linear amplifier circuit and a compression amplifier circuit.

[0057] Specifically, after obtaining a valid audio signal, the valid audio signal is input into an automatic gain control circuit, the valid audio signal is amplified by an amplifier circuit, and the output audio signal is input into a feedback system, and the feedback system detects and processes the output audio signal to determine whether the output audio signal meets the intensity range requirement of the output signal, such as setting the signal intensity range to 60-80dbm. When the output audio signal does not meet the signal intensity range, the gain of the amplifier is adjusted. Among them, when a weak signal is input, the linear amplifier circuit works to ensure the intensity of the output signal; when the input signal reaches a certain intensity, the compression amplifier circuit is started to reduce the intensity of the output signal. The effective audio signal is amplified by the adjusted amplifier gain to obtain an amplified audio signal.

[0058] (3) Performing noise reduction processing on the amplified audio signal based on active noise reduction technology to obtain the audio signal to be located.

[0059] It can be understood that active noise reduction technology is a technology that obtains the opposite phase of noise through electronic circuits and uses signals with opposite phases and the same amplitude to cancel out the noise.

[0060] Specifically, after obtaining the amplified audio signal, the amplified audio signal is subjected to spectrum analysis to obtain the signal spectrum of the amplified audio signal. The phase of the scene noise is processed by an electronic circuit to obtain a signal with a phase opposite to the scene noise and the same amplitude, and the noise spectrum of the signal is obtained by spectrum analysis. The signal with a phase opposite to the noise spectrum and the same amplitude in the signal spectrum is offset to obtain the audio signal to be located. Among them, the preferred active noise reduction technology is ENC noise reduction. The signal spectrum is the spectrum obtained by performing spectrum analysis on the amplified audio signal. Spectral analysis is a technology for decomposing complex signals into simple signals, that is, finding out the information of the audio signal at different frequencies. The noise spectrum is the spectrum of the signal with a phase opposite to the scene noise and the same amplitude.

[0061] The present invention performs summation calculation on audio acquisition signals through a beamforming algorithm, thereby achieving acquisition of effective audio signals. The effective audio signals are amplified by an automatic gain control circuit and the amplified audio signals are denoised by an active noise reduction technology, thereby achieving acquisition of the audio signals to be located and improving the signal-to-noise ratio and quality of the audio signals.

[0062] In one embodiment, if Figure 2 As shown, in the step S20, that is, performing voiceprint matching on the audio signal to be located by using the preset voiceprint recognition model to determine whether the audio signal to be located includes the target audio signal, including:

[0063] S201: Perform voice endpoint detection on the audio signal to be located to determine whether the audio signal to be located includes a sound wave audio signal.

[0064] Understandably, speech endpoint detection is to detect the range of valid signals from a continuous signal stream. Sound wave audio signals are audio signals emitted by humans or animals.

[0065] Specifically, after obtaining the audio signal to be located, voice endpoint detection is performed on the audio signal to be located. The starting point and / or ending point of the sound wave audio signal are accurately located from the audio signal to be located, that is, only the starting point and / or ending point need to be detected in the audio signal to be located to determine that the audio signal to be located includes the sound wave audio signal. The starting point is the starting time when the sound wave audio signal is collected in the audio signal to be located; the ending point is the time when the sound wave audio signal is not collected after the sound is collected in the audio signal to be located.

[0066] Furthermore, in this embodiment, speech endpoint detection is performed by changing the signal energy value. First, the audio signal to be located is segmented according to a fixed duration, such as 30 milliseconds, and each segmentation unit contains the same number of signal sampling points, and then the energy value of the signal in each segmentation unit is calculated. If the energy values ​​of several consecutive segmentation units in the front end of the audio signal to be located are lower than the preset energy value threshold (the preset energy value threshold can be set according to demand), and the energy values ​​of the next several consecutive segmentation units are greater than or equal to the preset energy value threshold, then the place where the signal energy value increases is the starting point of the sound wave audio signal. Similarly, if the energy values ​​of the speech in several consecutive segmentation units are large, and then the energy values ​​of the speech in several segmentation units become smaller, and continue for a certain period of time, it can be considered that the place where the energy value decreases is the end point of the sound wave audio signal.

[0067] S202, when the audio signal to be located includes a sonic audio signal, performing voiceprint feature recognition on the sonic audio signal to obtain a voiceprint feature to be located corresponding to the sonic audio signal; one sonic audio signal corresponds to one voiceprint feature to be located.

[0068] It can be understood that the voiceprint feature to be located is the voiceprint feature carried in the sound wave audio signal. The voiceprint feature to be located can be extracted from the sound wave audio signal by using Mel frequency cepstrum coefficients.

[0069] Specifically, when it is determined that the audio signal to be located includes an acoustic audio signal, the Mel-frequency cepstrum coefficient is used to extract features of the acoustic audio signal. First, the audio signal to be located is preprocessed to obtain a plurality of sampling points. The audio signal to be located is framed with 256 or 512 sampling points to obtain a plurality of frame units. In order to avoid excessive changes in adjacent frame units, there is an overlapping area between two adjacent frame units, and the overlapping area contains 128 or 256 sampling points. Each frame unit is multiplied by a window function so that the left and right ends of each frame unit are continuous, thereby obtaining a continuous time window. Secondly, all windowed frame units are subjected to fast Fourier transform to obtain a spectrum corresponding to each frame unit, that is, a spectrum distributed in different time windows on the time axis is obtained.

[0070] Furthermore, the obtained spectrum is processed by a Mel filter to obtain a Mel spectrum, that is, the linear natural spectrum is converted into a Mel spectrum that reflects the characteristics of human hearing. Then the logarithm of the Mel spectrum is taken to obtain the logarithmic energy of the Mel spectrum, and the logarithmic energy is inversely transformed according to the discrete cosine transform, and the second to thirteenth coefficients after the discrete cosine transform are taken as the Mel frequency cepstrum coefficients, and the Mel frequency cepstrum coefficients are determined as the voiceprint features to be located.

[0071] S203, performing feature matching between the voiceprint feature to be located and the target voiceprint feature in the preset voiceprint recognition model to obtain a voiceprint matching result; one voiceprint feature to be located corresponds to one voiceprint matching result.

[0072] Understandably, the target voiceprint feature is the voiceprint feature of the target object pre-stored in the preset voiceprint recognition model. For example, in the application scenario of smart conference, it is necessary to collect the audio signal of the speaker. Then the target object is the speaker in the smart conference. Therefore, the speaker's audio signal can be collected in advance, and the voiceprint feature of the speaker's audio signal can be extracted to obtain the target voiceprint feature. The voiceprint matching result is the result of matching the voiceprint feature to be located with the target voiceprint feature.

[0073] Specifically, after obtaining the voiceprint features to be located, the voiceprint features to be located are input into the preset voiceprint recognition model and matched with the target voiceprint features in the preset voiceprint recognition model, that is, the similarity between the voiceprint features to be located and the target voiceprint features is calculated. When the similarity between the voiceprint features to be located and the target voiceprint features is greater than or equal to the preset target similarity threshold, a voiceprint matching result indicating a successful match is obtained. When the similarity between the voiceprint features to be located and the target voiceprint features is less than the preset target similarity threshold, a voiceprint matching result indicating a failed match is obtained. Among them, the preset target similarity threshold is a threshold set in advance for judging the similarity between the voiceprint features to be located and the target voiceprint features.

[0074] S204: When any of the voiceprint matching results indicates a successful match, determine that the audio signal to be located includes a target audio signal.

[0075] Specifically, in different application scenarios, the number of target voiceprint features is one or more. Therefore, after feature matching is performed on the voiceprint features to be located and the target voiceprint features to obtain the voiceprint matching results, when any voiceprint matching result indicates a successful match, it indicates that the audio signal to be located contains the audio signal of the target object that needs to be collected in the current application scenario. That is, the audio signal to be located includes the target audio signal. When all voiceprint matching results indicate a failed match, it indicates that the audio signal to be located does not contain the audio signal of the target object that needs to be collected in the current application scenario. That is, the audio signal to be located does not include the target audio signal.

[0076] Furthermore, if the audio signal to be located does not include the target audio signal, that is, the audio signal to be located is entirely scene noise, the audio signal in the surrounding environment is re-collected through the microphone array, and the collected audio signal is pre-processed and voiceprint matched according to the above steps until it is determined that the audio signal to be located includes the target audio signal. The specific implementation process is the same as the above steps S10-S20 and will not be repeated here. Furthermore, when the same microphone array determines that the audio signal to be located does not include the target audio signal for more than two times, the microphone array is replaced, and the audio signal in the surrounding environment is re-collected with a new microphone array.

[0077] The present invention can quickly determine whether the audio signal to be located includes a sound wave audio signal by performing voice endpoint detection on the audio signal to be located. The voiceprint feature to be located of the sound wave audio signal can be extracted by performing voiceprint feature recognition on the sound wave audio signal. By matching the voiceprint feature to be located with the target voiceprint feature and determining the voiceprint matching result indicating a successful match, it is possible to determine whether the audio signal to be located includes the target audio signal.

[0078] In one embodiment, if Figure 3 As shown, in the step S30, that is, when the audio signal to be located includes the target audio signal, obtaining the target acquisition image includes:

[0079] S301: When the audio signal to be located includes a target audio signal, obtain a signal collection area corresponding to the target audio signal.

[0080] Specifically, after obtaining the sound source localization result indicating successful matching, that is, when it is determined that the audio signal to be located includes the target audio signal, the area for collecting the target audio signal is determined. The signal strength of the audio signals in the four microphone collection areas is calculated to obtain the signal strength values ​​corresponding to the respective microphone collection areas. The four signal strength values ​​are compared, and the microphone collection area corresponding to the maximum value of the four signal strength values ​​is determined as the signal collection area.

[0081] S302, acquiring a current acquisition area of ​​the image acquisition device, and adjusting the current acquisition area according to the signal acquisition area to obtain a target acquisition area.

[0082] S303: Capture an image in the target acquisition area by using the image acquisition device to obtain the target acquisition image.

[0083] Specifically, after acquiring the signal acquisition area, the current field of view is photographed by the image acquisition device to obtain the current acquisition area, and the signal acquisition area and the current acquisition area are compared to see if they are in the same direction. When the signal acquisition area and the current acquisition area are in the same direction, the current acquisition area is determined as the target acquisition area. When the signal acquisition area and the current acquisition area are in different directions, the shooting area of ​​the image acquisition device is adjusted by the terminal control device, the shooting area of ​​the image acquisition device is adjusted to the signal acquisition area, and the adjusted shooting area is determined as the target acquisition area. Among them, the current acquisition area is the range of the current field of view of the image acquisition device. The terminal control device is a device that can adjust the position or direction of the image acquisition device.

[0084] Further, after determining the target acquisition area, an image in the target acquisition area is captured by an image acquisition device, and the captured image is determined as a target acquisition image. In different application scenarios, the number of target objects in the target acquisition area may be one or more. Therefore, there may be one or more acquisition objects in one target acquisition image.

[0085] In one embodiment, when the multi-channel microphone on the left, the multi-channel microphone on the right, or the multi-channel microphone at the back of the microphone array collects audio signals in each microphone collection area, the image acquisition device will capture the image in the microphone collection area that collects the audio signal, and present the corresponding picture window in the current picture of the client, that is, present the left, right, or back picture window. When it is necessary to convert the picture, you can click on the left, right, or back picture window on the client or other methods to trigger a jump instruction, jump from the current picture to the picture corresponding to the left, right, or back picture window, that is, the picture corresponding to the left, right, or back picture window becomes the current picture. At the same time, the collection direction of the multi-channel microphone in the microphone array also changes, and the specific change process is the same as the embodiment in the above step S10, which will not be repeated here. Among them, the client includes but is not limited to display screens, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0086] The present invention acquires the signal acquisition area and the current acquisition area, and adjusts the shooting area of ​​the image acquisition device according to the signal acquisition area, thereby acquiring the target acquisition area. The image acquisition device captures the image of the target acquisition area, thereby acquiring the target acquisition image.

[0087] In one embodiment, in step S40, image matching is performed on the target captured image by using the preset image recognition model to obtain the target image, including:

[0088] (1) performing image recognition on the target acquisition image to obtain at least one intercepted acquisition image; the intercepted acquisition image refers to an acquisition image intercepted from the target acquisition image and containing only one acquisition object;

[0089] (2) The captured images containing the same captured object are associated and recorded as a captured image group.

[0090] It can be understood that the intercepted acquisition image is an acquisition picture intercepted from the target acquisition image and contains only one acquisition object. The shooting acquisition group is a set of intercepted acquisition images of the same acquisition object. The acquisition object is the image captured by the image acquisition device.

[0091] Specifically, after obtaining the target acquisition image, the acquisition object in the target acquisition image is identified. The edge of the acquisition object in the target acquisition image is detected by calculating the gray value of the target acquisition image, and the edge of the acquisition object in the target acquisition image is obtained. The acquisition object image in the target acquisition image is segmented according to the detected edge of the acquisition object to obtain at least one intercepted acquisition image. Image matching is performed on all the intercepted acquisition images to obtain intercepted acquisition images containing the same acquisition object. The intercepted acquisition images containing the same acquisition object are associated, and all the intercepted acquisition images containing the same acquisition object are associated in turn, and the intercepted acquisition images containing the same acquisition object after association are recorded as a shooting acquisition group.

[0092] (3) for each of the photographing and collecting groups, determining the image similarity between the target person image and the intercepted and collected images in the photographing and collecting group;

[0093] (4) Recording the captured image corresponding to the image similarity greater than or equal to the preset image similarity threshold as the target image.

[0094] Specifically, after obtaining the shooting and acquisition group, the intercepted acquisition images containing the same acquisition object in the shooting and acquisition group are input into the preset image recognition model, and similarity matching is performed with the target person image in the preset image recognition model. For example, similarity matching is performed between the intercepted acquisition image and the target person image through grayscale matching, and the pixels of the intercepted acquisition image are calculated to obtain the pixel value of the intercepted acquisition image. The pixels of the target person image are calculated to obtain the pixel value of the target person image. The pixel value of the intercepted acquisition image is compared with the pixel value of the target person image to obtain the image similarity between the two images. When the image similarity is greater than or equal to the preset image similarity threshold, the corresponding intercepted acquisition image is recorded as the target image. When the image similarity is less than the preset image similarity threshold, it is determined that the intercepted acquisition image is not the target image. Among them, the preset image similarity threshold is a threshold of similarity between images set in advance. The target person image is a picture of the target object used as training data.

[0095] Furthermore, when it is determined that the intercepted image is not the target image, the intercepted image in the next shooting and acquisition group is matched with the target person image until the similarity between the intercepted image in a certain shooting and acquisition group and the target person image is greater than or equal to the preset image similarity threshold, thereby obtaining the target image. When the intercepted images in all shooting and acquisition groups fail to match, the field of view or position of the image acquisition device is adjusted, and the adjusted target acquisition area is re-photographed to obtain a new target acquisition image. The new target acquisition image is segmented to obtain a new intercepted image, which is then re-matched with the target person image.

[0096] The invention obtains the intercepted captured image by performing image recognition on the target captured image, and realizes the acquisition of the shooting and capturing group by associating and recording the intercepted captured images containing the same captured object. The target image is determined by performing similarity matching between the intercepted captured image and the target person image.

[0097] In one embodiment, in step S50, signal matching is performed on the sound source audio signal and the target audio signal to obtain a sound source localization result, including:

[0098] (1) Performing voiceprint feature recognition on the source audio signal and the target audio signal to obtain a source audio voiceprint feature corresponding to the source audio signal and a target audio voiceprint feature corresponding to the target audio signal.

[0099] Understandably, the sound source audio voiceprint feature is the voiceprint feature of the sound source audio line signal, and the target audio voiceprint feature is the voiceprint feature of the target audio signal.

[0100] Specifically, after obtaining the target image, the sound source audio signal associated with the target image is obtained, and the feature extraction of the sound source audio signal is performed through the Mel frequency cepstrum coefficient. First, the audio signal to be positioned is preprocessed to obtain multiple sampling points. The sound source audio signal is framed with 256 or 512 sampling points to obtain multiple frame units. Among them, in order to avoid excessive changes in adjacent frame units, there is an overlapping area between two adjacent frame units, and this overlapping area contains 128 or 256 sampling points. Each frame unit is multiplied by a window function so that the left and right ends of each frame unit are continuous, thereby obtaining a continuous time window. Secondly, all windowed frame units are subjected to fast Fourier transform to obtain the frequency spectrum corresponding to each frame unit in the frequency domain, that is, the spectrum distributed in different time windows on the time axis is obtained.

[0101] Furthermore, the obtained spectrum is processed by a Mel filter to obtain a Mel spectrum, that is, the linear natural spectrum is converted into a Mel spectrum that reflects the characteristics of human hearing. Then the logarithm of the Mel spectrum is taken to obtain the logarithmic energy of the Mel spectrum, and the logarithmic energy is inversely transformed according to the discrete cosine transform, and the second to thirteenth coefficients after the discrete cosine transform are taken as the Mel frequency cepstrum coefficients, and the Mel frequency cepstrum coefficients are determined as the audio voiceprint features of the sound source. Similarly, the target audio voiceprint features corresponding to the target audio signal are obtained. The specific implementation process is the same as the above steps and will not be repeated here.

[0102] (2) Perform similarity matching on the sound source audio voiceprint features and the target audio voiceprint features to obtain a sound source localization result.

[0103] Specifically, after obtaining the sound source audio voiceprint features and the target audio voiceprint features, it is detected whether the similarity between the sound source audio voiceprint features and the target audio voiceprint features exceeds the preset voiceprint similarity threshold. When the similarity between the sound source audio voiceprint features and the target audio voiceprint features is less than the preset voiceprint similarity threshold, a sound source localization result indicating a failed match is obtained, and a prompt is given for the sound source localization result indicating a failed match. When the similarity between the sound source audio voiceprint features and the target audio voiceprint features is greater than or equal to the preset voiceprint similarity threshold, a sound source localization result indicating a successful match is obtained, and the position of the target object can be determined. Among them, the sound source localization result is the similarity matching result between the sound source audio voiceprint features and the target audio voiceprint features. The preset voiceprint similarity threshold is a threshold set in advance for judging the similarity between the sound source audio voiceprint features and the target audio voiceprint features.

[0104] Furthermore, in one embodiment, the sound source audio voiceprint feature and the target audio voiceprint feature are matched for similarity, and the similarity is 0.9. The preset voiceprint similarity threshold is set to 0.8. The similarity between the sound source audio voiceprint feature and the target audio voiceprint feature is greater than the preset voiceprint similarity threshold, and a successful match result is obtained, and the location of the target object can be determined. Among them, the preset voiceprint similarity threshold can be set according to actual conditions, such as setting the preset voiceprint similarity threshold to 0.6 to avoid the situation where the similarity between the sound source audio voiceprint feature and the target audio voiceprint feature is less than the preset voiceprint similarity threshold due to hoarseness or other factors, resulting in the inability to locate the target object.

[0105] The present invention realizes the acquisition of the source audio voiceprint features and the target audio voiceprint features by performing voiceprint feature recognition on the source audio signal and the target audio signal, and realizes the acquisition of the sound source positioning result by performing similarity matching on the source audio voiceprint features and the target audio voiceprint features.

[0106] Furthermore, if the sound source is updated in the current application scenario (such as in the smart conference scenario, after the first batch of users in the meeting end the meeting, the second batch of other users join the smart conference), or the position of the current target object changes (such as moving from the collection area of ​​one multi-channel microphone to the collection area of ​​another multi-channel microphone), or the current application scenario changes (such as when the smart conference scenario switches to the smart customer service scenario), the new sound source positioning result can be determined according to the method of steps S10 to S50 above. The specific implementation process is the same as the above steps S10-S50, which will not be repeated here.

[0107] In one embodiment, a sound source localization device is provided, and the sound source localization device corresponds one-to-one to the sound source localization method in the above embodiment. Figure 4As shown, the sound source localization device includes an acquisition module 11, a voiceprint matching module 12, a collection module 13, an image matching module 14 and a result module 15. The functional modules are described in detail as follows:

[0108] The acquisition module 11 is used to acquire an audio acquisition signal and pre-process the audio acquisition signal to obtain an audio signal to be located;

[0109] A voiceprint matching module 12 is used to obtain a preset voiceprint recognition model, and input the audio signal to be located into the preset voiceprint recognition model, and perform voiceprint matching on the audio signal to be located through the preset voiceprint recognition model to determine whether the audio signal to be located includes a target audio signal;

[0110] The acquisition module 13 is used to acquire at least one target acquisition image when the audio signal to be located includes the target audio signal, wherein the target acquisition image is obtained by photographing the target acquisition area by an image acquisition device; the target acquisition area is generated according to the signal acquisition area of ​​the target audio signal;

[0111] The image matching module 14 is used to obtain a preset image recognition model, input the target acquisition image into the preset image recognition model, and perform image matching on the target acquisition image through the preset image recognition model to obtain a target image; one target image is associated with one sound source audio signal;

[0112] The result module 15 is used to perform signal matching on the sound source audio signal and the target audio signal to obtain a sound source localization result.

[0113] In one embodiment, the acquisition module 11 includes:

[0114] The acoustic-to-electric conversion unit is used to collect the acoustic wave signal through the microphone array, and perform acoustic-to-electric conversion on the acoustic wave signal to obtain an analog audio signal;

[0115] The analog-to-digital conversion unit is used to perform analog-to-digital conversion on the analog audio signal to obtain the audio acquisition signal.

[0116] In one embodiment, the acquisition module 11 further includes:

[0117] A weighted summing unit, used for performing weighted summing processing on the audio acquisition signal through a beamforming algorithm to obtain a valid audio signal;

[0118] an amplifying unit, used to amplify the effective audio acquisition signal through an automatic gain control circuit to obtain an amplified audio signal;

[0119] The noise reduction unit is used to perform noise reduction processing on the amplified audio signal based on active noise reduction technology to obtain the audio signal to be located.

[0120] In one embodiment, the voiceprint matching module 12 includes:

[0121] A detection unit, configured to perform voice endpoint detection on the audio signal to be located to determine whether the audio signal to be located includes a sound wave audio signal;

[0122] a voiceprint feature recognition unit, configured to, when the audio signal to be located includes a sound wave audio signal, perform voiceprint feature recognition on the sound wave audio signal to obtain a voiceprint feature to be located corresponding to the sound wave audio signal; one sound wave audio signal corresponds to one voiceprint feature to be located;

[0123] A voiceprint feature matching unit, used for matching the voiceprint feature to be located with the target voiceprint feature in the preset voiceprint recognition model to obtain a voiceprint matching result; one voiceprint feature to be located corresponds to one voiceprint matching result;

[0124] A determining unit is configured to determine that the audio signal to be located includes a target audio signal when any of the voiceprint matching results indicates a successful match.

[0125] In one embodiment, the acquisition module 13 includes:

[0126] An area unit, used to obtain a signal collection area corresponding to the target audio signal;

[0127] An adjustment unit, used for acquiring a current acquisition area of ​​the image acquisition device, and adjusting the current acquisition area according to the signal acquisition area to obtain a target acquisition area;

[0128] The acquisition unit is used to acquire images in the target acquisition area through the image acquisition device to obtain the target acquisition image.

[0129] In one embodiment, the image matching module 14 includes:

[0130] An image recognition unit, configured to perform image recognition on the target acquisition image to obtain at least one intercepted acquisition image; the intercepted acquisition image refers to an acquisition image intercepted from the target acquisition image and containing only one acquisition object;

[0131] An associating unit, used for associating and recording the intercepted and collected images containing the same collection object as a shooting and collection group;

[0132] A similarity determination unit, used to determine, for each of the shooting and acquisition groups, the image similarity between the target person image and the intercepted acquisition images in the shooting and acquisition group;

[0133] The recording unit is used to record the captured image corresponding to the picture similarity greater than or equal to the preset image similarity threshold as the target image.

[0134] In one embodiment, the result module 15 includes:

[0135] an identification unit, configured to perform voiceprint feature identification on the source audio signal and the target audio signal to obtain a source audio voiceprint feature corresponding to the source audio signal and a target audio voiceprint feature corresponding to the target audio signal;

[0136] The similarity matching unit is used to perform similarity matching on the sound source audio voiceprint feature and the target audio voiceprint feature to obtain a sound source positioning result.

[0137] For the specific definition of the sound source localization device, please refer to the definition of the sound source localization method above, which will not be repeated here. Each module in the above-mentioned sound source localization device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0138] In one embodiment, a computer device is provided. The computer device may be a client or a server. The internal structure diagram thereof may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the readable storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a sound source localization method is implemented.

[0139] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the sound source localization method in the above embodiment is implemented.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the sound source localization method in the above embodiment is implemented.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0142] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0143] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A sound source localization method, characterized in that: include: Acquire an audio acquisition signal, and preprocess the audio acquisition signal to obtain an audio signal to be located; Acquire a preset voiceprint recognition model, input the audio signal to be located into the preset voiceprint recognition model, and perform voiceprint matching on the audio signal to be located through the preset voiceprint recognition model to determine whether the audio signal to be located includes a target audio signal; When the audio signal to be located includes the target audio signal, obtaining at least one target acquisition image, wherein the target acquisition image is obtained by photographing an image acquisition device in a target acquisition area; the target acquisition area is generated according to a signal acquisition area of ​​the target audio signal; Obtain a preset image recognition model, input the target captured image into the preset image recognition model, perform image matching on the target captured image through the preset image recognition model, and obtain a target image; one target image is associated with one sound source audio signal; The sound source audio signal and the target audio signal are matched to obtain a sound source localization result.

2. The sound source localization method according to claim 1, characterized in that: The step of obtaining the audio acquisition signal comprises: Acquiring sound wave signals through a microphone array and converting the sound wave signals into electric signals to obtain analog audio signals; Perform analog-to-digital conversion on the analog audio signal to obtain the audio acquisition signal.

3. The sound source localization method according to claim 1, characterized in that: The preprocessing of the audio acquisition signal to obtain the audio signal to be located includes: Performing weighted sum processing on the audio acquisition signal through a beamforming algorithm to obtain a valid audio signal; Amplifying the effective audio acquisition signal through an automatic gain control circuit to obtain an amplified audio signal; The amplified audio signal is subjected to noise reduction processing based on active noise reduction technology to obtain the audio signal to be located.

4. The sound source localization method according to claim 1, characterized in that: The performing voiceprint matching on the audio signal to be located by using the preset voiceprint recognition model to determine whether the audio signal to be located includes a target audio signal, includes: Performing voice endpoint detection on the audio signal to be located to determine whether the audio signal to be located includes a sound wave audio signal; When the audio signal to be located includes a sound wave audio signal, voiceprint feature recognition is performed on the sound wave audio signal to obtain a voiceprint feature to be located corresponding to the sound wave audio signal; one sound wave audio signal corresponds to one voiceprint feature to be located; Perform feature matching on the voiceprint feature to be located and the target voiceprint feature in the preset voiceprint recognition model to obtain a voiceprint matching result; one voiceprint feature to be located corresponds to one voiceprint matching result; When any of the voiceprint matching results indicates a successful match, it is determined that the audio signal to be located includes a target audio signal.

5. The sound source localization method according to claim 1, characterized in that: The acquiring of at least one target acquisition image comprises: Acquire a signal collection area corresponding to the target audio signal; Acquire a current acquisition area of ​​the image acquisition device, and adjust the current acquisition area according to the signal acquisition area to obtain a target acquisition area; The image acquisition device is used to acquire an image in the target acquisition area to obtain the target acquisition image.

6. The sound source localization method according to claim 1, characterized in that: One of the target acquisition images includes at least one acquisition object; the preset image recognition model includes at least one target person image; The performing image matching on the target captured image by using the preset image recognition model to obtain the target image includes: Performing image recognition on the target captured image to obtain at least one intercepted captured image; The captured acquisition image refers to an acquisition picture captured from the target acquisition image and containing only one acquisition object; Associating and recording the intercepted and collected images containing the same collection object as a shooting and collection group; For each of the shooting and collecting groups, determining the image similarity between the target person image and the intercepted and collected images in the shooting and collecting group; The captured image corresponding to the picture similarity greater than or equal to the preset image similarity threshold is recorded as the target image.

7. The sound source localization method according to claim 1, characterized in that: The performing signal matching on the sound source audio signal and the target audio signal to obtain a sound source localization result includes: Performing voiceprint feature recognition on the sound source audio signal and the target audio signal to obtain a sound source audio voiceprint feature corresponding to the sound source audio signal and a target audio voiceprint feature corresponding to the target audio signal; The sound source audio voiceprint feature and the target audio voiceprint feature are matched for similarity to obtain a sound source positioning result.

8. A sound source localization device, characterized in that: include: An acquisition module is used to acquire an audio acquisition signal and pre-process the audio acquisition signal to obtain an audio signal to be located; a voiceprint matching module, configured to obtain a preset voiceprint recognition model, input the audio signal to be located into the preset voiceprint recognition model, and perform voiceprint matching on the audio signal to be located through the preset voiceprint recognition model to determine whether the audio signal to be located includes a target audio signal; A collection module, used for acquiring at least one target collection image when the audio signal to be located includes the target audio signal, wherein the target collection image is obtained by photographing in a target collection area by an image collection device; the target collection area is generated according to a signal collection area of ​​the target audio signal; An image matching module is used to obtain a preset image recognition model, input the target acquisition image into the preset image recognition model, and perform image matching on the target acquisition image through the preset image recognition model to obtain a target image; one target image is associated with one sound source audio signal; The result module is used to perform signal matching on the sound source audio signal and the target audio signal to obtain a sound source localization result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the sound source localization method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the sound source localization method according to any one of claims 1 to 7 is implemented.

Images