Sound source positioning method and device based on direction correction, equipment, storage medium

Abstract

The application provides a sound source positioning method and device based on direction correction, equipment and a storage medium, and the method comprises the following steps: a double-microphone array obtains double-channel voice information based on a uniform linear array model, and determines a frequency domain signal obtained by preprocessing the double-channel voice information as a microphone array signal; sound source direction of arrival estimation is performed based on the microphone array signal to obtain a plurality of sound source estimation directions; a portrait direction angle is obtained by portrait positioning and focusing through a camera, and the portrait direction angle is converted to a portrait azimuth angle based on the uniform linear array model; a complete square difference between the portrait azimuth angle and each sound source estimation direction is calculated, and a sound source estimation direction corresponding to the smallest complete square difference is determined as a target sound source direction. According to the technical scheme of the embodiment of the application, after the sound wave estimation direction is obtained, the direction is corrected by portrait positioning, and the accuracy of sound source positioning is improved under the condition that the dimension of the microphone array is limited.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a method, apparatus, device, and storage medium for sound source localization based on direction correction. Background Technology

[0002] With the normalization of working from home and cross-regional communication, video conferencing has become an essential means of remote communication and collaboration. Video conferencing equipment equipped with cameras and microphones is becoming increasingly popular, and the demands for a superior user experience are rising daily. This requires cameras to be capable of functions such as rapid positioning, automatic zoom, and AI-powered automatic imaging, while microphones must have the ability to actively reduce noise and suppress ambient noise. Regarding voice quality improvement, microphone array algorithms are typically used to reduce noise in the voice signal, followed by voice enhancement through accurate localization of the sound source.

[0003] Currently, microphone array sound source localization algorithms can achieve sound source localization. The more dimensions of the microphone array, the higher the accuracy of sound source localization. However, for scenarios such as home office, due to limitations in equipment size and cost, it is not possible to set up too many microphones in video conferencing equipment. Usually, only a dual-microphone array is used. The accuracy of sound source localization achieved by dual-microphone array algorithms is not high, which affects the effect of voice enhancement and the user experience. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a sound source localization method, apparatus, device, and storage medium based on direction correction, which can correct the sound source direction according to the direction of a person's image, thereby improving the accuracy of sound source localization.

[0005] In a first aspect, embodiments of the present invention provide a sound source localization method based on direction correction, applied to a video conferencing device. The video conferencing device includes a camera and a dual-microphone array, the dual-microphone array being pre-configured with a uniform linear array model. The sound source localization method based on direction correction includes:

[0006] The dual-microphone array acquires dual-channel speech information based on the uniform linear array model, and the frequency domain signal obtained by preprocessing the dual-channel speech information is determined as the microphone array signal;

[0007] Multiple estimated directions of sound sources are obtained by estimating the direction of arrival of the sound sources based on the microphone array signals.

[0008] The camera is used to locate and focus the human face to obtain the human face direction angle, and the human face direction angle is converted into the human face orientation angle based on the uniform linear array model.

[0009] Calculate the perfect square difference between the orientation angle of the human figure and the estimated directions of each sound source, and determine the estimated direction of the sound source corresponding to the smallest calculated perfect square difference as the target sound source direction.

[0010] According to some embodiments of the present invention, determining the frequency domain signal obtained by preprocessing the dual-channel voice information as a microphone array signal includes:

[0011] The dual-channel speech information is separated into left and right channels to obtain left and right channel speech information;

[0012] The left and right channel speech information is obtained and converted into frequency domain speech signals. The frequency domain speech signals are then subjected to anti-aliasing downsampling to obtain sampled signals.

[0013] The sampled signal is denoised and then converted to the time domain to obtain a denoised time domain signal.

[0014] After performing dereverberation processing on the denoised time-domain signal, an audio frame sequence is obtained, wherein the audio frame sequence includes multiple audio frames;

[0015] Based on the audio frame sequence, VAD decision is performed to filter out multiple voice frames, and the sequence formed by combining the voice frames is determined as the microphone array signal.

[0016] According to some embodiments of the present invention, the step of converting the sampled signal to the time domain after denoising processing to obtain a denoised time-domain signal includes:

[0017] The sampled signal is sequentially windowed, FFT transformed, and modulus calculated to obtain the original signal power spectrum;

[0018] Based on the original signal power spectrum, quantile noise estimation, speech presence probability estimation, noise update and suppression coefficient calculation are performed sequentially to determine the noise suppression coefficient for each frequency point.

[0019] Multiply the noise suppression coefficient by the original signal power spectrum to obtain the noise reduction power spectrum;

[0020] The noise-reduced power spectrum is transformed by IFFT to obtain the noise-reduced time-domain signal.

[0021] According to some embodiments of the present invention, the step of performing VAD decision filtering based on the audio frame sequence to select multiple audio frames includes:

[0022] After performing an FFT transform on the audio frame sequence, the frame power and prior and posterior SNR of each audio frame are determined.

[0023] The audio frame, the frame power, and the prior and posterior SNR are input into a preset probability model for VAD decision, and the audio frame is determined to be a speech frame or a silent frame based on the decision result.

[0024] According to some embodiments of the present invention, the step of estimating the direction of arrival (DOA) of sound sources based on the microphone array signal to obtain multiple estimated directions of sound sources includes:

[0025] The microphone array signal is divided into multiple sample angles in a preset spatial domain, and the steering vector information of each sample angle is calculated.

[0026] Calculate the matrix estimate of the spatial correlation matrix of the microphone array signal;

[0027] Based on the transpose of the steering vector information and the matrix estimate, the sample power spectrum corresponding to each sample angle is calculated;

[0028] Based on the sorting of the sample power spectra from largest to smallest, the sample angles corresponding to the top-ranked sample power spectra are determined as the sound source estimation directions according to a preset number of directions.

[0029] According to some embodiments of the present invention, the sample power spectrum is obtained by the following formula:

[0030]

[0031]

[0032]

[0033] Where, θ q Let be the q-th sample angle, where q = 1, 2, 3, ..., Q, and Q is the number of sample angles and is a natural number. q ) represents the guidance vector information for the q-th sample angle, a H (θ q 0 is a(θ) q The conjugate transpose of a matrix, [] T This represents the matrix transpose, where e is the natural constant. λ is the wavelength of the dual-channel voice information, and d is the wavelength of a(θ). q The element spacing of the array is n = 1, 2, ..., N, where N is the frame length of the dual-channel speech information, and x(n) is the microphone array signal. H x(n) is the matrix conjugate transpose of x(n); R is the matrix estimate of the microphone array signal. -1 for Find the inverse of the matrix, P(θ) qThe power spectrum of the sample is denoted as .

[0034] According to some embodiments of the present invention, after determining the sound source estimation direction corresponding to the smallest calculated perfect square difference as the target sound source direction, the method further includes:

[0035] Based on the target sound source direction and beamforming method, the dual-channel speech information is enhanced to obtain enhanced speech information;

[0036] The enhanced voice information is determined as the output signal of the dual-channel voice information and transmitted over the network.

[0037] In a second aspect, embodiments of the present invention provide a sound source localization device based on direction correction, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform the sound source localization method based on direction correction as described in the first aspect above.

[0038] Thirdly, embodiments of the present invention provide an electronic device including a direction-correction-based sound source localization device as described in the second aspect above.

[0039] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions for performing the direction-correction-based sound source localization method as described in the first aspect above.

[0040] The direction-correction-based sound source localization method according to embodiments of the present invention has at least the following beneficial effects: the dual-microphone array acquires dual-channel speech information based on the uniform linear array model, and determines the frequency domain signal obtained by preprocessing the dual-channel speech information as the microphone array signal; multiple sound source estimated directions are obtained by estimating the direction of arrival of the sound source based on the microphone array signal; the image is located and focused by the camera to obtain the image direction angle, and the image direction angle is converted into an image azimuth angle based on the uniform linear array model; the perfect square difference between the image azimuth angle and each of the sound source estimated directions is calculated, and the sound source estimated direction corresponding to the smallest calculated perfect square difference is determined as the target sound source direction. According to the technical solution of the embodiments of the present invention, after obtaining the sound wave estimated direction, direction correction can be performed using image localization, improving the accuracy of sound source localization when the microphone array dimension is limited. Attached Figure Description

[0041] Figure 1This is a flowchart of a sound source localization method based on direction correction provided in an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of left and right channel extraction provided in another embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of downsampling provided in another embodiment of the present invention;

[0044] Figure 4 This is a flowchart of noise reduction processing provided in another embodiment of the present invention;

[0045] Figure 5 This is a flowchart of VAD decision provided in another embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the sample power spectrum provided in another embodiment of the present invention;

[0047] Figure 7 This is a structural diagram of a sound source localization device based on direction correction provided in another embodiment of the present invention. Detailed Implementation

[0048] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0049] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0050] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0051] In the description of this invention, unless otherwise explicitly defined, terms such as "setting," "installing," and "connecting" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0052] This invention provides a method, apparatus, device, and storage medium for sound source localization based on direction correction. The method includes: a dual-microphone array acquiring dual-channel speech information based on a uniform linear array model; determining the frequency domain signal obtained by preprocessing the dual-channel speech information as the microphone array signal; estimating the direction of arrival (DOA) of the sound source based on the microphone array signal to obtain multiple estimated sound source directions; using a camera for portrait positioning and focusing to obtain a portrait direction angle; converting the portrait direction angle to a portrait azimuth angle based on the uniform linear array model; calculating the perfect square difference between the portrait azimuth angle and each estimated sound source direction; and determining the estimated sound source direction corresponding to the smallest calculated perfect square difference as the target sound source direction. According to the technical solution of this invention, after obtaining the estimated sound wave direction, direction correction can be performed using portrait positioning, improving the accuracy of sound source localization even when the microphone array dimension is limited.

[0053] The control method of the present invention will be further described below with reference to the accompanying drawings.

[0054] Reference Figure 1 , Figure 1 This is a flowchart of a sound source localization method based on direction correction provided in an embodiment of the present invention. The sound source localization method based on direction correction includes, but is not limited to, steps S11 to S14:

[0055] S11, the dual-microphone array acquires dual-channel speech information based on a uniform linear array model, and the frequency domain signal obtained by preprocessing the dual-channel speech information is determined as the microphone array signal.

[0056] It should be noted that the video conferencing equipment has a dual-microphone array, and the uniform linear array model of the dual-microphone array is configured as the signal receiving model, so that the voice signal acquired by the dual-microphone array is a uniform linear array signal. The uniform linear array is a technology well known to those skilled in the art, and will not be elaborated on here.

[0057] In one embodiment, the preprocessing further includes, but is not limited to, the following steps: separating the left and right channels of the dual-channel speech information to obtain left and right channel speech information; converting the left and right channel speech information into frequency domain speech signals, performing anti-aliasing downsampling on the frequency domain speech signals to obtain sampled signals; performing noise reduction processing on the sampled signals and converting them to the time domain to obtain a noise-reduced time domain signal; performing de-reverberation processing on the noise-reduced time domain signal to obtain an audio frame sequence, wherein the audio frame sequence includes multiple audio frames; performing VAD decision based on the audio frame sequence to select multiple speech frames, and determining the sequence of speech frames as a microphone array signal.

[0058] It should be noted that dual-channel voice information typically includes left and right channels. After acquiring the dual-channel voice information, the underlying AD chip can sample the PCM audio data, which is the PCM audio data of the left and right stereo channels of a sound card, to obtain the dual-channel PCM audio data. The underlying AD chip uses an interleaved mode by default for dual-channel sampling to arrange the recording data. That is, the acquired dual-channel PCM data is arranged according to the interleaved order of the left and right channel data. (See reference...) Figure 2 As shown, L1 to L4 represent the left channel audio data, and R1 to R4 represent the right channel audio data. Channel separation requires extracting and separating all left and right channels into separate left channel data and separate right channel data according to the fixed order in which the left and right channels are arranged alternately.

[0059] It should be noted that because the sampling specifications supported by the underlying ADC hardware vary, there may be situations where the actual sampling rate configured on the machine differs from the sampling rate required by the algorithm. Furthermore, after switching the analog signal to the frequency domain for sampling, it is equivalent to periodically extending the frequency domain at the sampling rate. Therefore, when the actual sampling rate supported by the ADC hardware is greater than the sampling rate required by the algorithm, downsampling is necessary, i.e., resampling is performed by reducing the sampling rate. Figure 3 As shown, Figure 3 The first image shows the base signal, and the second image shows the downsampled signal. Downsampling can easily cause truncation of effective speech frequencies and aliasing, resulting in jagged edges. Therefore, this embodiment uses a downsampling anti-aliasing algorithm to eliminate jagged edges by using a low-pass filter, thereby achieving anti-aliasing. Figure 3 As shown, Figure 3 The third image is a schematic diagram of a low-pass filter. Taking fs as the original sampling rate as an example, the resampling rate after downsampling is fs / 4. The cutoff frequency of the low-pass filter is half of the resampling rate. The effect diagram obtained after filtering based on the above low-pass filter is shown below. Figure 3 As shown in the fourth figure, the signal does not exhibit aliasing or jagged edges, thus achieving signal optimization.

[0060] In one embodiment, reference is made to Figure 4 The noise reduction processing of the sampled signal also includes, but is not limited to, the following steps: windowing, FFT transformation and modulus calculation are performed on the sampled signal in sequence to obtain the original signal power spectrum; based on the original signal power spectrum, quantile noise estimation, speech presence probability estimation, noise update and suppression coefficient calculation are performed in sequence to determine the noise suppression coefficient of each frequency point; the noise suppression coefficient is multiplied by the original signal power spectrum to obtain the noise-reduced power spectrum; the noise-reduced power spectrum is transformed by IFFT to obtain the noise-reduced time domain signal.

[0061] It should be noted that the sampled signal is a noisy signal. After windowing the noisy signal, an FFT transform is performed to convert it to the frequency domain, and the original signal power spectrum is obtained by modulo operation. The original signal power spectrum can characterize the noise situation. Based on this, this embodiment uses the original signal power spectrum to perform quantile noise estimation, speech presence probability estimation, and noise update, thereby assessing the noise level. The above estimation methods are all techniques well known to those skilled in the art, and will not be elaborated further here.

[0062] It should be noted that after determining the noise level, the noise suppression coefficient of each frequency point can be calculated based on the noise level. After multiplying by the original signal power spectrum, noise reduction of each frequency point can be achieved in the frequency domain. After noise reduction in the frequency domain, IFFT is performed to return to the time domain to obtain the denoised time domain signal.

[0063] It is worth noting that due to the multipath effect of speech signals in spatial propagation, reverberation will occur. De-reverberation algorithms are mainly used to filter out the linear delay interference. Those skilled in the art are familiar with how to perform de-reverberation processing, so we will not go into details here.

[0064] Additionally, in one embodiment, reference is made to Figure 5 The VAD decision to filter out multiple speech frames based on the audio frame sequence includes, but is not limited to, the following steps: after performing FFT transformation on the audio frame sequence, determine the frame power and prior and posterior SNR of each audio frame; input the audio frame, frame power, prior and posterior SNR into a preset probability model to perform VAD decision, and determine whether the audio frame is a speech frame or a silent frame according to the decision result.

[0065] It should be noted that the VAD criterion process in this embodiment is based on... Figure 5 As shown, after performing an FFT transform on each audio frame in the audio frame sequence, the frame power of each frame is calculated, and then the prior and posterior SNRs are calculated. The prior and posterior SNRs, including the prior SNR and posterior SNR, are used to characterize the signal-to-noise ratio of the audio frame. The specific calculation of the prior and posterior SNRs is a technique well-known to those skilled in the art and will not be elaborated upon here. After obtaining the power and SNR, they are simultaneously input into a probability model for decision-making. Finally, after smoothing, the criteria results for silent frames and speech frames are obtained. The decided speech frames are used as the microphone array signals for subsequent calculations. This embodiment uses Voice Activity Detection (VAD) to filter out silent frames and speech frames. No algorithm processing is performed on silent frames, which effectively saves computing power and improves the efficiency of sound source localization.

[0066] S12, based on the microphone array signal, the direction of arrival of the sound source is estimated to obtain the estimated direction of multiple sound sources.

[0067] It should be noted that the direction of arrival of the sound source wave is used to characterize the direction that the sound wave of the sound source can reach, and therefore can be used as a characterization of the sound source. In this embodiment, the direction can be an angle, which is convenient for subsequent calculations.

[0068] This embodiment determines multiple sound source estimation directions through sound source direction-of-arrival estimation, specifically including the following steps: dividing the microphone array signal into multiple sample angles in a preset spatial domain, and calculating the steering vector information for each sample angle; calculating the matrix estimate of the spatial correlation matrix of the microphone array signal; calculating the sample power spectrum corresponding to each sample angle based on the transpose of the steering vector information and the matrix estimate; and determining the sample angles corresponding to the top-ranked sample power spectra as sound source estimation directions based on the sorting of the sample power spectra from largest to smallest and according to the preset number of directions.

[0069] It should be noted that the preset spatial domain can be [0 degrees, 360 degrees]. Q sample angles are divided within the preset spatial domain, where Q is a natural number, and the q-th sample angle is θ. q Given q = 1, 2, 3, ... Q, calculate the guidance vector information for each sample angle. The guidance vector information for the q-th sample angle is represented by a(θ). q The specific calculation formula is as follows: in, λ is the wavelength of the dual-channel speech information. Since the speech signal in this embodiment is a uniform linear array, the signal is represented in matrix form, and d is a(θ) q The element spacing of the array is []. T This indicates the matrix transpose.

[0070] It should be noted that, based on the obtained steering vector information, the matrix estimate of the spatial correlation matrix of the microphone array signal x(n) is calculated. The specific formula is as follows: Where x(n) is the microphone array signal, x H x(n) is the matrix conjugate transpose of x(n), where n = 1, 2, ..., N, and N is the frame length of the dual-channel speech information.

[0071] Finally, the sample power spectrum P(θ) at each sample location is calculated. q The specific calculation formula is as follows: Among them, a H (θ q ) is a(θ q The matrix conjugate transpose of ), where e is the natural constant, R -1 for Find the inverse of the matrix.

[0072] It should be noted that after calculating the sample power spectrum, the samples can be sorted according to their numerical values, and the m sample angles with the largest values ​​can be selected as the estimated direction of the sound source.

[0073] For example, when Q is set to 72, such as Figure 6 As shown, if m=4, there are four main peak positions, and the corresponding sample angles are [75 degrees, 90 degrees, 180 degrees, 195 degrees], which serve as the direction for sound source estimation.

[0074] S13: The camera is used to locate and focus the human face to obtain the direction angle of the human face, and the direction angle of the human face is converted into the orientation angle of the human face based on the uniform linear array model.

[0075] It should be noted that the camera in this embodiment can be a camera equipped with AI automatic imaging and fast positioning functions. Integrating a camera with these functions into video conferencing equipment is a technology well-known to those skilled in the art, and no hardware limitations are imposed here. The AI ​​automatic imaging camera captures a person's image, performs rapid positioning and focusing, and provides the direction angle of the person, thus obtaining the image's direction angle. Since the microphone array's signal model is a uniform linear array, after obtaining the image's direction angle, it can be converted into an image azimuth angle, representing the image's direction in a uniform linear array manner. This ensures that the image azimuth angle and the subsequently obtained sound source estimation direction belong to the same dimension of data, providing a basis for data correction.

[0076] S14, calculate the perfect square difference between the orientation angle of the human face and the estimated directions of each sound source, and determine the estimated direction of the sound source corresponding to the smallest perfect square difference as the target sound source direction.

[0077] The technical solution in this embodiment utilizes the human image positioning direction of the camera to assist the microphone array in sound source localization. This enables high-precision sound source localization with limited resources, even with fewer microphone dimensions. This provides a foundation for further optimizing noise reduction effects and enhancing the user's voice interaction experience.

[0078] It should be noted that after obtaining the orientation angle θ of the human figure p Then, calculate the azimuth angle θ of the portrait. p With the direction of sound source estimation Choose the θ with the smallest perfect square difference between them. q As the final target direction. If the camera does not obtain the direction information of the human face within the preset time, the first obtained sound source estimation direction can be directly used as the target sound source direction to avoid the target sound source direction determination time being too long.

[0079] In another embodiment, after performing step S14, the following steps are included, but are not limited to:

[0080] The dual-channel speech information is enhanced based on the target sound source direction and beamforming method to obtain enhanced speech information; the enhanced speech information is then determined as the output signal of the dual-channel speech information for network transmission.

[0081] It should be noted that the direction of the target sound source is θ q For example, the enhanced speech signal y(n) is calculated using beamforming, and the specific formula is as follows: y(n) = BF(θ) q ,x(n)), where beamforming methods include, but are not limited to, minimum variance distortionless response algorithm, generalized sidelobe cancellation algorithm, etc.

[0082] like Figure 7 As shown, Figure 7 This is a structural diagram of a direction-correction-based sound source localization device according to an embodiment of the present invention. The present invention also provides a direction-correction-based sound source localization device, comprising:

[0083] The processor 701 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0084] The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 to execute the direction-correction-based sound source localization method of the embodiments of this application.

[0085] The input / output interface 703 is used to implement information input and output;

[0086] The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0087] Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704);

[0088] The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0089] This application also provides an electronic device, including the direction-correction-based sound source localization device described above.

[0090] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described sound source localization method based on direction correction.

[0091] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] It will be understood by those skilled in the art that all or some of the steps and apparatuses in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0093] The above provides a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A sound source localization method based on direction correction, characterized in that, The method for locating sound sources based on direction correction is applied to video conferencing equipment, which includes a camera and a dual-microphone array. The dual-microphone array has a pre-set uniform linear array model. The dual-microphone array acquires dual-channel speech information based on the uniform linear array model, and the frequency domain signal obtained by preprocessing the dual-channel speech information is determined as the microphone array signal; Based on the microphone array signal, the direction of arrival of the sound source is estimated to obtain multiple estimated directions of the sound source; The camera is used to locate and focus the human face to obtain the human face direction angle, and the human face direction angle is converted into the human face orientation angle based on the uniform linear array model. Calculate the perfect square difference between the human face orientation angle and each of the estimated directions of the sound sources, and determine the estimated direction of the sound source corresponding to the smallest calculated perfect square difference as the target sound source direction; The step of determining the frequency domain signal obtained by preprocessing the dual-channel voice information as a microphone array signal includes: The dual-channel speech information is separated into left and right channels to obtain left and right channel speech information; The left and right channel speech information is obtained and converted into frequency domain speech signals. The frequency domain speech signals are then downsampled to obtain sampled signals. The sampled signal is denoised and then converted to the time domain to obtain a denoised time domain signal; After performing dereverberation processing on the denoised time-domain signal, an audio frame sequence is obtained, wherein the audio frame sequence includes multiple audio frames; Based on the audio frame sequence, VAD decision is performed to filter out multiple voice frames, and the sequence formed by combining the voice frames is determined as the microphone array signal.

2. The sound source localization method based on direction correction according to claim 1, characterized in that, The step of converting the sampled signal to the time domain after noise reduction processing to obtain a noise-reduced time-domain signal includes: The sampled signal is sequentially windowed, FFT transformed, and modulus calculated to obtain the original signal power spectrum; Based on the original signal power spectrum, quantile noise estimation, speech presence probability estimation, noise update and suppression coefficient calculation are performed sequentially to determine the noise suppression coefficient for each frequency point. Multiply the noise suppression coefficient by the original signal power spectrum to obtain the noise reduction power spectrum; The denoised power spectrum is transformed by IFFT to obtain the denoised time-domain signal.

3. The sound source localization method based on direction correction according to claim 1, characterized in that, The step of filtering out multiple speech frames based on the audio frame sequence using VAD decision includes: After performing an FFT transform on the audio frame sequence, the frame power and prior and posterior SNR of each audio frame are determined. The audio frame, the frame power, and the prior and posterior SNR are input into a preset probability model for VAD decision, and the audio frame is determined to be a speech frame or a silent frame based on the decision result.

4. The sound source localization method based on direction correction according to claim 1, characterized in that, The method of estimating the direction of arrival (DOA) of sound sources based on the microphone array signals to obtain multiple estimated directions of sound sources includes: The microphone array signal is divided into multiple sample angles in a preset spatial domain, and the steering vector information of each sample angle is calculated. Calculate the matrix estimate of the spatial correlation matrix of the microphone array signal; Based on the transpose of the steering vector information and the matrix estimate, the sample power spectrum corresponding to each sample angle is calculated; Based on the sorting of the sample power spectra from largest to smallest, the sample angles corresponding to the top-ranked sample power spectra are determined as the sound source estimation directions according to a preset number of directions.

5. The sound source localization method based on direction correction according to claim 4, characterized in that, The sample power spectrum is obtained by the following formula: ； ； ； in, Let q be the q-th sample angle, where q = 1, 2, 3, ..., Q, and Q is the number of sample angles and is a natural number. ) represents the guidance vector information for the q-th sample angle. For a( The conjugate transpose of a matrix. This represents the matrix transpose, where e is the natural constant. , The wavelength of the dual-channel voice information is d, where d is a( The element spacing of the array is n=1,2,…,N, where N is the frame length of the dual-channel speech information. The signal is from the microphone array. for The matrix conjugate transpose; This is a matrix estimate of the microphone array signal. for Find the inverse of the matrix. The power spectrum of the sample.

6. The sound source localization method based on direction correction according to claim 1, characterized in that, After determining the sound source estimation direction corresponding to the smallest calculated perfect square difference as the target sound source direction, the method further includes: Based on the target sound source direction and beamforming method, the dual-channel speech information is enhanced to obtain enhanced speech information; The enhanced voice information is determined as the output signal of the dual-channel voice information and transmitted over the network.

7. A sound source localization device based on direction correction, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; The memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enables the at least one control processor to perform the direction-correction-based sound source localization method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, Includes the sound source localization device based on direction correction as described in claim 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the direction-correction-based sound source localization method as described in any one of claims 1 to 6.

Images