Speaker location method, device and equipment

Abstract

The present application discloses a speaker localization method for a conference device, the device comprising a first microphone array, a second microphone array, a third microphone array and at least one speaker. In addition, a speech processing method is disclosed. Among them, when the local conference device plays the voice of the speaker at the remote venue, the speaker localization method performs speaker localization according to the signal collected by the microphone far from the speaker to avoid being affected by the sound emitted by the speaker; therefore, the accuracy of speaker localization can be effectively improved. When no one is speaking at the remote venue, the speaker localization is performed according to the signals collected by all microphones; therefore, the robustness and accuracy of speaker localization can be effectively improved.

Description

[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on March 22, 2022, with application number 2022102847233 and invention name “Speaker Localization Method, Device and Equipment”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of speech processing technology, and in particular to a speaker positioning method and device, a speech processing method and device, and a conference terminal. Background Art

[0003] Internet technology has brought about changes in people's communication tools, and audio and video conferencing systems based on cloud computing have gradually become popular. Among them, speaker localization in audio and video conferencing environments has always been a hot topic of research.

[0004] A typical speaker localization method is to perform speaker localization processing based on the same microphone information regardless of whether there is someone speaking at the remote venue. However, in the process of implementing the present invention, the inventors found that the existing solutions have at least the following problems: on the one hand, when someone is speaking at the remote venue, speaker localization will be affected by the sound emitted by the loudspeaker, so the accuracy of speaker localization is low; on the other hand, when no one is speaking at the remote venue, the robustness and accuracy of speaker localization need to be improved. Summary of the invention

[0005] The present application provides a speaker positioning method to solve the problem of low speaker positioning accuracy in the prior art. The present application also provides a speaker positioning device, an echo elimination method and device, and a conference terminal.

[0006] The present application provides a speaker positioning method for a conference device, wherein the conference device comprises: a first microphone array, a second microphone array, a third microphone array, and at least one speaker;

[0007] The method comprises:

[0008] Collecting multiple first microphone signals through a first microphone array; collecting multiple second microphone signals through a second microphone array; collecting multiple third microphone signals through a third microphone array; and obtaining a collection reference signal through a speaker;

[0009] Detecting whether the speaker makes a sound according to the reference signal;

[0010] If it is detected that the speaker emits a sound, first positioning information of a horizontal angle is obtained according to the multi-channel third microphone signal; and second positioning information of a vertical angle is obtained according to the multi-channel second microphone signal;

[0011] The first speaker's position information is acquired according to the first positioning information and the second positioning information.

[0012] Optionally, also include:

[0013] If no sound is detected from the speaker, obtaining third positioning information of a horizontal angle according to the multiple first microphone signals and the multiple third microphone signals; obtaining second positioning information of a vertical angle according to the multiple second microphone signals;

[0014] The second speaker position information is acquired according to the third positioning information and the second positioning information.

[0015] Optionally, the conference device further includes: at least one camera;

[0016] The method further comprises:

[0017] Collecting image signals through the at least one camera;

[0018] Acquiring speaker image information according to the image signal;

[0019] The first speaker position information is acquired according to the first positioning information, the second positioning information and the speaker image information.

[0020] Optionally, the second microphone array and the third microphone array are in an orthogonal configuration, and share a microphone at the intersection position.

[0021] Optionally, the first microphone array and the third microphone array include linear arrays in the horizontal direction;

[0022] The second microphone array includes a vertical linear array.

[0023] Optionally, the microphone comprises an omnidirectional microphone.

[0024] Optionally, the microphone collects far-field sound source signals.

[0025] The present application also provides a speech processing method for a conference device, the conference device comprising: a first microphone array, a third microphone array, at least one fourth microphone, and at least one speaker;

[0026] The at least one fourth microphone is proximate to the at least one speaker array;

[0027] The method comprises:

[0028] Collecting multiple first microphone signals through a first microphone array; collecting multiple third microphone signals through a third microphone array; and collecting at least one fourth microphone signal through at least one fourth microphone as an acoustic reference microphone signal; and obtaining a back-collection reference signal through a speaker;

[0029] Detecting whether the speaker makes a sound according to the reference signal;

[0030] If it is detected that the speaker emits sound, an echo cancellation signal is obtained according to the multi-channel third microphone signal, the acoustic reference microphone signal and the echo reference signal.

[0031] Optionally, also include:

[0032] If no sound is detected from the speaker, a sound signal is acquired according to the multiple first microphone signals and the multiple third microphone signals.

[0033] Optionally, the fourth microphone comprises a low-sensitivity microphone.

[0034] The present application also provides a conference device, including:

[0035] a first microphone array, a second microphone array, a third microphone array, and at least one speaker;

[0036] A processor; and a memory for storing a program for implementing the above-mentioned speaker positioning method, the terminal is powered on and runs the program of the method through the processor.

[0037] Optionally, the conference device also includes: at least one camera.

[0038] The present application also provides a conference device, including:

[0039] a first microphone array, a third microphone array, at least one fourth microphone, and at least one speaker;

[0040] Processor; and

[0041] The memory is used to store a program for implementing the above-mentioned voice processing method. The terminal is powered on and runs the program of the method through the processor.

[0042] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the computer-readable storage medium is run on a computer, the computer executes the above-mentioned various methods.

[0043] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned various methods.

[0044] Compared with the prior art, this application has the following advantages:

[0045] The speaker localization method provided in the embodiment of the present application is used for conference equipment, which includes a first microphone array, a second microphone array, a third microphone array and at least one speaker. The method collects multiple first microphone signals through the first microphone array; collects multiple second microphone signals through the second microphone array; collects multiple third microphone signals through the third microphone array; and obtains a back-collection reference signal through the speaker; detects whether the speaker makes a sound according to the back-collection reference signal; if the speaker makes a sound, obtains the first positioning information of the horizontal angle according to the multiple third microphone signals; obtains the second positioning information of the vertical angle according to the multiple second microphone signals; and obtains the first speaker position information according to the first positioning information and the second positioning information. By adopting this processing method, when the local conference equipment plays the voice of the speaker in the remote venue, the speaker in the local venue is located according to the signal collected by the microphone far away from the speaker, avoiding the influence of the sound emitted by the speaker; therefore, the accuracy of speaker localization can be effectively improved. When no one speaks in the remote venue, the speaker in the local venue is located according to the signals collected by all microphones; therefore, the robustness and accuracy of speaker localization can be effectively improved.

[0046] The speech processing method provided in the embodiment of the present application is used for conference equipment, which includes a first microphone array, a third microphone array, at least one fourth microphone and at least one speaker, wherein the at least one fourth microphone is close to the at least one speaker array, and the method collects multiple first microphone signals through the first microphone array; collects multiple third microphone signals through the third microphone array; and collects at least one fourth microphone signal through at least one fourth microphone as an acoustic reference microphone signal; and obtains a back-collection reference signal through the speaker; detects whether the speaker makes a sound according to the back-collection reference signal; if the speaker is detected to make a sound, obtains an echo cancellation signal according to the multiple third microphone signals, the acoustic reference microphone signal and the back-collection reference signal. By adopting this processing method, when the local conference equipment plays the voice of the speaker in the remote venue, the echo cancellation processing is performed according to the signal collected by the microphone far away from the speaker to avoid being affected by the sound emitted by the speaker; therefore, the echo cancellation effect can be effectively improved. When no one is speaking in the remote venue, the signals collected by all microphones are obtained; therefore, the robustness and accuracy of sound collection can be effectively improved. Among them, the electrical reference signal in traditional echo cancellation is used, combined with microphone array beamforming and acoustic reference microphones, to achieve multi-reference echo cancellation. This processing method allows the microphone array to estimate the echo signal, which can not only estimate the linear part of the echo signal during the propagation process, but also estimate the nonlinear part. In this way, the echo signal estimated by the microphone array can be used as a new reference signal for linear adaptive filtering, which can reduce the impact of nonlinearity in the actual product on the echo cancellation system, effectively filter out the nonlinear components in the echo signal, and thus improve the echo cancellation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of an embodiment of a speaker localization method provided by the present application;

[0048] Figure 2 A schematic diagram of the device structure of an embodiment of the speaker localization method provided by the present application;

[0049] Figure 3 A schematic diagram of a specific flow chart of an embodiment of a speaker localization method provided in the present application;

[0050] Figure 4 A flowchart of an embodiment of a speech processing method provided by the present application;

[0051] Figure 5 A schematic diagram of a specific flow chart of an embodiment of the speech processing method provided in the present application;

[0052] Figure 6 A schematic diagram of the sound direction of an embodiment of the speech processing method provided by the present application;

[0053] Figure 7 A beam diagram of an embodiment of a speech processing method provided by the present application;

[0054] Figure 8 The present application provides a specific flowchart of an embodiment of a speech processing method. DETAILED DESCRIPTION

[0055] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0056] In the present application, a speaker localization method and device, an echo cancellation method and device, and a conference terminal are provided. In the following embodiments, various solutions are described in detail one by one.

[0057] First embodiment

[0058] Please see Figure 1 , which is a flow chart of an embodiment of the speaker localization method of the present application. In this embodiment, the method may include the following steps:

[0059] Step S101: collecting multiple first microphone signals through a first microphone array; collecting multiple second microphone signals through a second microphone array; collecting multiple third microphone signals through a third microphone array; and obtaining a sampling reference signal through a speaker.

[0060] The method can be applied to the speaker location scenario of conference terminal equipment in an audio and video conference system. An audio and video conference system is a system device that enables two or more individuals or groups in different places to communicate with each other through transmission lines and conference terminals to achieve instant and interactive communication and to conduct conferences at the same time. Therefore, an audio and video conference system is a typical real-time communication system.

[0061] The conference device includes: a first microphone array, a second microphone array, a third microphone array, and at least one speaker. The microphone array can pick up sound from a remote sound source. The microphone can be an omnidirectional microphone or a directional microphone.

[0062] The reference signal can be a signal source of a loudspeaker, a sound signal from a remote venue, and can include the voice of a remote speaker. The local conference terminal deployed at the local venue receives the sound signal collected by the remote conference terminal deployed at the remote venue through a communication network, and plays it through the loudspeaker of the local conference terminal.

[0063] In one example, the conference device may further include: at least one camera. Figure 2 As shown, it is an audio and video all-in-one machine, which includes a speaker array (at least one speaker), a first microphone array (including N microphones) for horizontal sound source positioning, a second microphone array (including M microphones) for vertical sound source positioning, a third microphone array (including P microphones) for horizontal sound source positioning and a camera array (at least one camera).

[0064] Depend on Figure 2 It can be seen that a lens group module (i.e., camera array) composed of multiple lenses can be placed at the center of the device for video information collection of the audio and video integrated machine. The sound playback system is located on one side of the camera and can be composed of a speaker array composed of multiple omnidirectional speakers. There is a horizontal microphone array module (first microphone array) on the lower side of the speaker system, which is composed of N omnidirectional microphones and can be used for horizontal sound source positioning.

[0065] Depend on Figure 2 It can be seen that on the other side of the camera module, there is a vertical microphone array module (second microphone array) for vertical sound source localization, which consists of M microphones. Orthogonal to this vertical array is a horizontal microphone array module (third microphone array), which shares a microphone with the vertical array and can consist of P omnidirectional microphones.

[0066] visible, Figure 2 The sound pickup system shown is composed of a microphone array system, which is composed of three microphone array modules, each of which can be composed of omnidirectional microphones. Therefore, the entire system is composed of M+N+P-1 normal omnidirectional microphones, a speaker array module, and a multi-lens camera module.

[0067] Step S103: Detect whether the speaker makes a sound according to the sampling reference signal.

[0068] The local conference terminal plays the received sound signal (the acquisition reference signal) from the remote conference site through a loudspeaker. Sometimes someone speaks at the remote conference site, and sometimes no one speaks. Therefore, the acquisition reference signal includes speech signals and non-speech signals. This embodiment detects the blank in the acquisition reference signal through voice activity detection (VAD).

[0069] Step S105: if it is detected that the speaker emits sound, first positioning information of a horizontal angle is obtained according to the multiple third microphone signals; and second positioning information of a vertical angle is obtained according to the multiple second microphone signals.

[0070] Acquiring the first positioning information at the horizontal angle based on the multi-channel third microphone signals collected by the multiple microphones at the horizontal angle is a relatively mature existing technology and will not be described in detail here.

[0071] Acquiring second positioning information at a vertical angle based on multi-channel second microphone signals collected by multiple microphones at vertical angles is a relatively mature prior art technology and will not be described in detail here.

[0072] Step S107: Acquire the first speaker's position information according to the first positioning information and the second positioning information.

[0073] Acquiring the first speaker's position information according to the first positioning information at a horizontal angle and the second positioning information at a vertical angle is a relatively mature prior art and will not be described in detail herein.

[0074] In one example, the method may further include the following steps:

[0075] Step S201: if no sound is detected from the speaker, obtaining third positioning information of a horizontal angle according to the multiple first microphone signals and the multiple third microphone signals; obtaining second positioning information of a vertical angle according to the multiple second microphone signals.

[0076] Step S203: Acquire the second speaker's position information according to the third positioning information and the second positioning information.

[0077] like Figure 3 As shown, in one example, based on Figure 2 The conference device shown in the figure, the method may also include the following steps:

[0078] Step S301: collecting image signals through the at least one camera.

[0079] Step S303: acquiring speaker image information according to the image signal.

[0080] Acquiring speaker image information (such as face information) according to the image signal is a relatively mature prior art and will not be described in detail here.

[0081] Accordingly, step S107 may be implemented in the following manner: acquiring the first speaker's position information according to the first positioning information, the second positioning information and the speaker's image information.

[0082] Accordingly, step S203 may be implemented in the following manner: acquiring the second speaker position information according to the third positioning information, the second positioning information and the speaker image information.

[0083] In one example, after obtaining the first speaker's position information according to the first positioning information and the second positioning information, the method may further include the following steps:

[0084] Step S401: judging whether the speaker is a real person according to the speaker image information;

[0085] Step S403: If the speaker is identified as a person according to the speaker image, the speaker image is enlarged.

[0086] With this processing method, the facial image of the speaker at the local venue can be used as auxiliary information, so that even if the speaker is a mobile phone or other device at the local venue, the device can be accurately located. If the speaker image is identified as a person, not a device, a close-up shot of the speaker can be given.

[0087] As can be seen from the above embodiments, the speaker positioning method provided by the embodiment of the present application is used for conference equipment, which includes a first microphone array, a second microphone array, a third microphone array and at least one speaker. The method collects multiple first microphone signals through the first microphone array; collects multiple second microphone signals through the second microphone array; collects multiple third microphone signals through the third microphone array; and obtains a back-collection reference signal through the speaker; detects whether the speaker makes a sound according to the back-collection reference signal; if the speaker is detected to make a sound, obtains the first positioning information of the horizontal angle according to the multiple third microphone signals; obtains the second positioning information of the vertical angle according to the multiple second microphone signals; obtains the first speaker position information according to the first positioning information and the second positioning information. By adopting this processing method, when the local conference equipment plays the voice of the speaker in the remote venue, the speaker in the local venue is positioned according to the signal collected by the microphone far away from the speaker, avoiding the influence of the sound emitted by the speaker; therefore, the accuracy of speaker positioning can be effectively improved. When no one speaks in the remote venue, the speaker in the local venue is positioned according to the signals collected by all microphones; therefore, the robustness and accuracy of speaker positioning can be effectively improved.

[0088] Second embodiment

[0089] Please see Figure 4 , which is a flow chart of an embodiment of the speech processing method of the present application. In this embodiment, the method may include the following steps:

[0090] Step S401: collecting multiple first microphone signals through a first microphone array; collecting multiple third microphone signals through a third microphone array; and collecting at least one fourth microphone signal through at least one fourth microphone as an acoustic reference microphone signal; and obtaining a sampling reference signal through a speaker.

[0091] The method can be applied to the echo cancellation scenario of conference terminal equipment in an audio and video conference system. An audio and video conference system is a system device that enables two or more individuals or groups in different places to communicate with each other through transmission lines and conference terminals to achieve instant and interactive communication and to conduct conferences at the same time. Therefore, an audio and video conference system is a typical real-time communication system.

[0092] The conference device includes: a first microphone array, a third microphone array, at least one fourth microphone, and at least one speaker, wherein the at least one fourth microphone is close to the at least one speaker array. The microphone array can pick up sound from a remote sound source. The microphone can be an omnidirectional microphone or a directional microphone.

[0093] The reference signal can be a signal source of a loudspeaker, a sound signal from a remote venue, and can include the voice of a remote speaker. The local conference terminal deployed at the local venue receives the sound signal collected by the remote conference terminal deployed at the remote venue through a communication network, and plays it through the loudspeaker of the local conference terminal.

[0094] In one example, the conference device may further include: at least one camera. Figure 2 As shown, it is an audio and video all-in-one machine, which includes a speaker array (at least one speaker), a first microphone array (including N microphones) for horizontal sound source positioning, a second microphone array (including M microphones) for vertical sound source positioning, a third microphone array (including P microphones) for horizontal sound source positioning, at least one fourth microphone and a camera array (at least one camera).

[0095] Depend on Figure 2 It can be seen that the sound playback system is located on one side of the camera. It is a speaker array composed of multiple omnidirectional speakers. A low-sensitivity sound reference microphone can be placed in the center of the speaker array module near a speaker. Due to its low sensitivity, this microphone can record the working status of the speaker without being broken by the speaker.

[0096] Depend on Figure 2It can be seen that on the other side of the camera module, the sound pickup system is composed of a microphone array system, which consists of three microphone array modules, each of which can be composed of omnidirectional microphones. A vertical microphone array module is used for vertical sound source positioning, and the number consists of M. Orthogonal to this vertical array is a horizontal microphone array module, which shares a microphone with the vertical direction and can be composed of P omnidirectional microphones. Therefore, the entire system consists of M+N+P-1 normal omnidirectional microphones, one or more low-sensitivity sound reference microphones, a speaker array module, and a multi-lens camera module.

[0097] Generally speaking, an audio communication device has a microphone array as a sound pickup unit and a speaker as a sound playback unit. In actual applications, such audio communication devices will encounter echo cancellation problems. In this embodiment, the audio and video all-in-one device has a number P of omnidirectional microphones arranged in a linear array as a sound pickup unit. Figure 2 In the audio and video all-in-one machine provided, the speaker sound source will be on one side of the third microphone array, and a microphone will be included next to the speaker. In this embodiment, the microphone is called the acoustic reference microphone. Each microphone unit of the third microphone array will pick up the sound of the near-end sound source and the speaker at the same time. For a communication system, the signal that the device finally receives and transmits is only expected to be the voice of the local sound source. The speaker signal received by the third microphone array is called an acoustic echo, which needs to be eliminated and not transmitted to the far end by the communication system. Effectively eliminating this acoustic echo is crucial for any communication device. In specific implementation, the microphone array uses multiple microphones arranged in linear, annular, spherical and other array forms to collect sound at the same time. In specific implementation, the third microphone array picks up the near-end sound source, which can further suppress the echo to a certain extent. In specific implementation, it can be an omnidirectional microphone or a directional microphone.

[0098] Step S403: Detect whether the speaker makes a sound according to the sampling reference signal.

[0099] The local conference terminal plays the received sound signal (the acquisition reference signal) from the remote conference site through a loudspeaker. Sometimes someone speaks at the remote conference site, and sometimes no one speaks. Therefore, the acquisition reference signal includes speech signals and non-speech signals. This embodiment detects the blank in the acquisition reference signal through voice activity detection (VAD).

[0100] Step S405: if it is detected that the speaker emits sound, an echo cancellation signal is obtained according to the multi-channel third microphone signal, the acoustic reference microphone signal and the echo reference signal.

[0101] The third microphone signal includes a sound source signal and an echo signal.

[0102] like Figure 5 As shown, in this embodiment, the method may further include the following steps:

[0103] Step S501: if no sound is detected from the speaker, a sound signal is acquired according to the multiple first microphone signals and the multiple third microphone signals.

[0104] In this embodiment, step S405 may include the following sub-steps:

[0105] Step S601: enhancing the echo signal and suppressing the sound source signal by a first beamforming algorithm directed to a speaker to obtain a first sound signal, wherein the first sound signal includes a linear echo signal and a nonlinear echo signal.

[0106] The core idea of ​​the method provided in the embodiment of the present application is to use a third microphone array to obtain beamforming directed to the speaker. The beamforming algorithm is a spatial filter algorithm based on the microphone array. The so-called spatial filtering is to set a target direction. Signals within the target direction range will be picked up, while signals outside the target direction range will be suppressed. Therefore, based on the beamforming algorithm, the microphone array can achieve sound pickup within a specific direction range, while sounds outside the specific direction will be suppressed.

[0107] like Figure 6 As shown, the near-end sound source is generally at the side direction (broadside) of the linear microphone array, which can be defined as 90 degrees or 270 degrees, and the speaker sound source is at the end-fire direction (end-fire) of the linear microphone array, which can be defined as 0 degrees. In this embodiment, the audio and video communication integrated machine application device has M omnidirectional microphones arranged in a linear array as a sound pickup unit, and another microphone is used as a sound reference microphone, which is located near the speaker unit. Since the microphone is close to the speaker, if the speaker sound is too loud, the sound pressure received by the sound reference microphone will exceed the upper limit allowed by the normal operation of the general microphone, so a microphone with lower sensitivity can be selected as the sound reference microphone.

[0108] For beamforming directed toward the loudspeaker, the beams will appear corresponding to the above beams, and the 3D beam diagram is as follows: Figure 7 As shown, the sound from the speaker will be picked up, and the sound pickup direction directed toward the speaker will be suppressed by the beamforming algorithm.

[0109] In specific implementation, the design of beamforming directed toward a speaker or beamforming directed toward a speaker may be based on different theories, such as differential beamforming theory, super-directional beamforming theory, etc., but is not limited to any of the above theories.

[0110] In a specific implementation, the method may further include the following step: converting the multi-channel microphone signals into time-frequency domain sound signals.

[0111] For the microphone input of number M, the microphone signal is transformed into the time-frequency domain based on Fourier transform as follows:

[0112]

[0113] in,[] T is the linear algebra transposition operation, ω represents the corresponding frequency domain subband, and n represents the time frame identifier. This microphone array signal mainly consists of the near-end sound source signal and the acoustic echo signal from the speaker The composition is expressed as follows:

[0114]

[0115] The above formula can be further expressed as:

[0116]

[0117] Among them, s(ω,n) represents the signal of the near-end sound source at the sounding point, represents the acoustic propagation function between the near-end sound source and the microphone array; u(ω, n) represents the signal at the speaker’s sound point, Represents the acoustic propagation function between the speaker sound source and the microphone array.

[0118] It should be emphasized that the method provided in this application uses end-fire beamforming to process the z endfire (ω, n) not only includes the linear playback signal in the speaker, but also includes the nonlinear components in the speaker playback process. Therefore, it will be used as a reference signal in the echo cancellation system in the future.

[0119] Step S603: Perform linear adaptive filtering processing according to the first sound signal, the acoustic reference microphone signal, the echo reference signal and the third microphone signal to obtain an echo cancellation signal.

[0120] The third microphone array collects the multi-channel third microphone signals and the first sound signal z endfire (ω, n), the circuit returns the electrical reference signal (e ref ) and the acoustic reference microphone acoustic reference signal (m ref) is subjected to adaptive filtering to obtain an echo cancellation signal. The adaptive filtering algorithm includes, but is not limited to, normalized least mean square (NLMS), iterative least square (RLS) and other algorithms.

[0121] In one example, the method may further include the following steps:

[0122] Step S701: suppressing the echo signal and enhancing the sound source signal by a second beamforming algorithm directed toward the target sound source, so as to obtain a second sound signal.

[0123] The core idea of ​​this implementation method is to use the third microphone array to obtain beamforming directed to the speaker (end side direction) and beamforming directed to the speaker (side direction). Figure 6 As shown, an example of a 3D beam pattern of beamforming pointing to the side direction is shown. It can be seen that the sound of the speaker's voice source will be picked up, and the echo sound will be suppressed.

[0124] The circular microphone array is used to design beamforming in two directions. The pickup direction of one beamforming (first beamforming algorithm) can be set to point to the speaker inside the array, and the pickup direction of the other beamforming (second beamforming algorithm) can be set to point to the speaker outside the circular array. The beamforming directed to the speaker outside the circular array will suppress the signal from the end-fire direction of the array as interference noise, and mainly pick up the near-end sound source; the beamforming directed to the speaker inside the array will suppress the signal of the near-end speaker outside the array as interference noise, and mainly pick up the end-fire speaker sound source.

[0125] The linear microphone array is used to design beamforming in two directions. The pickup direction of one beamforming (second beamforming algorithm) can be set to the side direction, and the pickup direction of the other beamforming (first beamforming algorithm) can be set to the endfire direction. The side beamforming will suppress the signal from the endfire direction of the array as interference noise, and mainly pick up the near-end sound source; the endfire beamforming will suppress the signal from the side direction of the array as interference noise, and mainly pick up the endfire speaker sound source.

[0126] In this embodiment, the first beamforming algorithm (endfire beamforming algorithm) directed toward the loudspeaker calculates a complex weight vector in the frequency domain subband as follows:

[0127]

[0128] Each microphone corresponds to a complex weight, and the complex weights of multiple microphones form a complex weight vector.

[0129] The second beamforming algorithm (side beamforming algorithm) directed toward the speaker calculates a complex weight vector in the frequency domain subband as follows:

[0130]

[0131] In order to better describe the method provided in the embodiments of the present application, Figure 2 As shown, a linear microphone with an equal diameter and spacing consisting of 16 microphones is taken as an example, and the distance between each microphone is 2 cm, which is analyzed and explained as an example.

[0132] Generally speaking, the most important frequency band for speech is 1khz, which is used as an example frequency band for analysis in this embodiment. For the convenience of analysis and explanation, this embodiment stipulates that the end-fire direction is 0 degrees and the side direction is 90 degrees. Due to the shape of the linear array, it is obvious that the 270-degree direction is also the side direction. The beamforming characteristics proposed in this application are described based on the beamforming beam pattern. Generally speaking, the beam pattern describes the response of the beamforming algorithm in all directions in space. For example, 0db means that the response of the beamforming is 1, that is, the signal is picked up without distortion, and -10db means that the beam will suppress the signal by 10db.

[0133] The beamforming method can be based on different theories, such as differential beamforming theory, super-directional beamforming theory, etc., and is not limited to any of the above. This embodiment is based on the differential beamforming algorithm principle as an example, and can be designed in the end-fire direction and the side direction respectively as follows: Figure 4 The corresponding beamforming algorithm. Figure 4 It can be seen that the beamforming in the endfire direction can theoretically pick up the signal in the endfire direction (0 degrees) without distortion, and suppress the signal in the side direction to the greatest extent; the beamforming in the side direction can suppress the signal in the endfire direction to the greatest extent, and pick up the signal in the side direction without distortion.

[0134] Based on the above ideas, under the theoretical assumptions of free field (anechoic field) and acoustic parallel wave transmission, the beamforming in the end-fire direction (first beamforming algorithm) and the transfer function in the near-end sound source direction and the transfer function in the horn direction obtain the following relationship:

[0135]

[0136] At the same time, the beamforming in the side direction (second beamforming algorithm) and the transfer function in the near-end sound source direction and the transfer function in the speaker direction obtain the following relationship:

[0137]

[0138] like Figure 8 As shown, the microphone array input signal is processed by end-fire beamforming based on the following formula and the output is as follows:

[0139]

[0140] This formula indicates that: the first sound signal (z endfire (ω, n)) is the multi-channel microphone signal in the time-frequency domain ) and the first beamforming algorithm (beamforming function). represents the first beamforming algorithm.

[0141] Based on the above analysis, the following relationship can be obtained for the end-fire direction in free field and parallel wave theory:

[0142] z inner (ω, n) = u(ω, n)

[0143] This formula indicates that the first sound signal after suppressing the direction of the speaker's sound source is the signal of the speaker's sounding point.

[0144] However, in practical environments, the free field assumption no longer holds. This relationship no longer holds, but The result will also be a relatively small value, especially in the application scenario of the embodiment. Generally, the speaker is closer to the microphone in space. The closer the distance is, the more signal energy can be preserved after acoustic propagation. Therefore, even if it is not the theoretical assumption of parallel waves and free fields, the following conditions still hold:

[0145]

[0146] This formula can indicate that the loudspeaker direction signal after the first beamforming process is much larger than the signal in the direction of the speaker's sound source.

[0147] Therefore, in a practical environment, for the output of end-fire beamforming, the following relationship can be obtained:

[0148] z endfire (ω,n)≈u(ω,n)

[0149] This formula can represent that the first sound signal is approximately equal to the loudspeaker direction signal.

[0150] The microphone array input signal is processed by side beamforming based on the following formula and the output is as follows:

[0151]

[0152] This formula indicates that the second sound signal after suppressing the speaker direction signal and enhancing the sound source signal is the product of the multi-channel microphone signal in the time-frequency domain and the second beamforming algorithm (second beamforming function). represents the second beamforming algorithm.

[0153] Based on the above analysis, the following relationship can be obtained in free field and parallel wave theory:

[0154] z broadside (ω, n) = s(ω, n)

[0155] This formula indicates that the second sound signal after suppressing the loudspeaker direction signal is the signal of the near-end sound source at the sound emission point.

[0156] In one example, step S701 may include the following sub-steps: 1) determining a suppression coefficient based on an acoustic propagation function between a speaker and a microphone array and weight vectors of each microphone calculated in the frequency domain by the second beamforming algorithm; 2) obtaining a suppressed echo signal based on the suppression coefficient; 3) taking the sum of the suppressed echo signal and the sound source signal as the second sound signal.

[0157] In a practical environment, since the free field assumption no longer holds, This relationship no longer holds true, and the speaker is close to the microphone. After the signal from the speaker's sound point is transmitted to the microphone, the energy is still very large. Therefore, the side beamforming algorithm can only suppress the echo signal of the speaker to a certain extent. Therefore, the method provided in the embodiment of the present application provides the following conditions β<1, further give the following relationship:

[0158] z broadside (ω,n)≈s(ω,n)+β*u(ω,n)

[0159] This formula can represent that the second sound signal after the second beamforming process is approximately equal to the sum of the signal of the near-end sound source at the sound emission point and the loudspeaker directional signal after being suppressed to a certain extent.

[0160] In this embodiment, step S105 may be implemented in the following manner: performing linear adaptive filtering processing according to the first sound signal, the second sound signal, the acoustic reference microphone signal and the echo reference signal to obtain an echo cancellation signal.

[0161] like Figure 8As shown, based on the analysis of the above signal model, using the second sound signal z broadside (ω, n), the first sound signal z endfire (ω, n), the circuit returns the electrical reference signal (e ref ) and the acoustic reference microphone acoustic reference signal (m ref ) is adaptively filtered to obtain the echo cancellation signal output1(ω, n). The following is an example of the normalized minimum mean square error (NLMS) commonly used in the industry. The echo cancellation signal (output1) can be calculated according to the following formula:

[0162]

[0163] in, is an NLMS adaptive filter with a tap length of N. is the current time frame data z mix (ω, n) and the previous N-1 frames of historical time frame data [z mix (ω, n-1), …, z mix (ω, n-N+1)]; NLMS adaptive filter It can be obtained by the following formula:

[0164]

[0165] Wherein, μ is the adaptive filtering step size. The size of this step size is based on the characteristics of the NLMS filter. Generally, when only the speaker is sounding, a fixed parameter is set, such as μ = 0.1. When the speaker and the near-end sound source are sounding at the same time, μ = 0.

[0166] In order to get z mix (ω, n), in Figure 8 In the fusion device, the array reference signal z endfire (ω, n), the circuit returns the electrical reference signal (e ref ) and the acoustic reference microphone acoustic reference signal (m ref ) to obtain the fusion signal z mix (ω, n), the fusion method is as follows (but not limited to this method):

[0167] z mix (ω, n) = α*z endfire (ω,n)+β*m ref (ω,n)+ρ*e ref

[0168] Among them, α, β or ρ can be fixed constants and can be zero to indicate that a certain reference is not used.

[0169] It should be noted that, in specific implementation, the following formula may also be used to determine the echo cancellation signal:

[0170]

[0171] That is, beamforming processing may not be performed on the signal in the direction of the speaker.

[0172] It can be seen from the above embodiments that the speech processing method provided by the embodiment of the present application is used for conference equipment, which includes a first microphone array, a third microphone array, at least one fourth microphone and at least one speaker, wherein the at least one fourth microphone is close to the at least one speaker array, and the method collects multiple first microphone signals through the first microphone array; collects multiple third microphone signals through the third microphone array; and collects at least one fourth microphone signal through at least one fourth microphone as an acoustic reference microphone signal; and obtains a back-collection reference signal through the speaker; detects whether the speaker makes a sound according to the back-collection reference signal; if the speaker is detected to make a sound, obtains an echo cancellation signal according to the multiple third microphone signals, the acoustic reference microphone signal and the back-collection reference signal. By adopting this processing method, when the local conference equipment plays the voice of the speaker in the remote venue, the echo cancellation processing is performed according to the signal collected by the microphone far away from the speaker to avoid being affected by the sound emitted by the speaker; therefore, the echo cancellation effect can be effectively improved. When no one speaks in the remote venue, the signals collected by all microphones are obtained; therefore, the robustness and accuracy of sound collection can be effectively improved. Among them, the electrical reference signal in traditional echo cancellation is used, combined with microphone array beamforming and acoustic reference microphones, to achieve multi-reference echo cancellation. This processing method allows the microphone array to estimate the echo signal, which can not only estimate the linear part of the echo signal during the propagation process, but also estimate the nonlinear part. In this way, the echo signal estimated by the microphone array can be used as a new reference signal for linear adaptive filtering, which can reduce the impact of nonlinearity in the actual product on the echo cancellation system, effectively filter out the nonlinear components in the echo signal, and thus improve the echo cancellation effect.

[0173] Third embodiment

[0174] In the above-mentioned embodiment, a speech processing method is provided, and correspondingly, the present application also provides a speech processing device. The device corresponds to the embodiment of the above-mentioned method. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0175] The present application further provides a speech processing device for a conference device, wherein the conference device comprises: a first microphone array, a third microphone array, at least one fourth microphone, and at least one speaker; the at least one fourth microphone is close to the at least one speaker array.

[0176] The device comprises:

[0177] A signal acquisition unit, configured to acquire multiple first microphone signals through a first microphone array; acquire multiple third microphone signals through a third microphone array; and acquire at least one fourth microphone signal through at least one fourth microphone as an acoustic reference microphone signal; and acquire a back-acquisition reference signal through a speaker;

[0178] A VAD unit, used for detecting whether the speaker makes a sound according to the sampling reference signal;

[0179] The first signal processing unit is used to obtain an echo cancellation signal according to the multi-channel third microphone signal, the acoustic reference microphone signal and the echo reference signal if it is detected that the speaker emits sound.

[0180] Optionally, the device may further include: a second signal processing unit, configured to obtain a sound signal according to the multiple first microphone signals and the multiple third microphone signals if no sound is detected from the speaker.

[0181] Fourth embodiment

[0182] In the above-mentioned embodiment, a speaker location method is provided. Correspondingly, the present application also provides a speaker location device. The device corresponds to the embodiment of the above-mentioned method. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, please refer to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0183] The present application further provides a speaker positioning device for use in conference equipment, wherein the conference equipment includes: a first microphone array, a second microphone array, a third microphone array, and at least one loudspeaker.

[0184] The device comprises:

[0185] A signal acquisition unit, configured to acquire a plurality of first microphone signals through a first microphone array; acquire a plurality of second microphone signals through a second microphone array; acquire a plurality of third microphone signals through a third microphone array; and acquire a collection reference signal through a speaker;

[0186] A VAD unit, used for detecting whether the speaker makes a sound according to the sampling reference signal;

[0187] A first positioning unit is configured to obtain first positioning information of a horizontal angle according to the multiple third microphone signals if a sound is detected from the speaker; and obtain second positioning information of a vertical angle according to the multiple second microphone signals;

[0188] The second positioning unit is used to obtain the position information of the first speaker according to the first positioning information and the second positioning information.

[0189] Optionally, the device may further include:

[0190] A third positioning unit is configured to obtain third positioning information of a horizontal angle according to the multiple first microphone signals and the multiple third microphone signals if no sound is detected from the speaker; and obtain second positioning information of a vertical angle according to the multiple second microphone signals;

[0191] The fourth positioning unit is used to obtain the second speaker's position information according to the third positioning information and the second positioning information.

[0192] Fifth embodiment

[0193] In the above-mentioned embodiment, a speaker location method is provided, and correspondingly, the present application also provides an electronic device. The device corresponds to the embodiment of the above-mentioned method. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0194] The present application further provides an electronic device, comprising: a first microphone array, a second microphone array, a third microphone array and at least one speaker; a processor; and a memory. The memory is used to store a program for implementing the above-mentioned speaker location method, and the terminal is powered on and runs the program of the method through the processor.

[0195] The electronic device may be an audio or video conferencing terminal or a sound pickup device.

[0196] In one example, the conference device may further include: at least one camera.

[0197] Sixth embodiment

[0198] In the above-mentioned embodiment, a speech processing method is provided, and correspondingly, the present application also provides an electronic device. The device corresponds to the embodiment of the above-mentioned method. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0199] The present application further provides an electronic device, comprising: a first microphone array, a third microphone array, at least one fourth microphone, at least one speaker; a processor; and a memory. The memory is used to store a program for implementing the above-mentioned voice processing method, and the terminal is powered on and runs the program of the method through the processor.

[0200] The electronic device may be an audio or video conferencing terminal or a sound pickup device.

[0201] In one example, the conference device may further include: at least one camera.

[0202] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0203] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0204] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0205] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0206] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A speaker localization method for conference equipment, characterized in that: The conference device comprises: a first microphone array, a second microphone array, a third microphone array, and at least one speaker; The method comprises: Collecting multiple first microphone signals through a first microphone array; collecting multiple second microphone signals through a second microphone array; collecting multiple third microphone signals through a third microphone array; and obtaining a collection reference signal through a speaker; Detecting whether the speaker makes a sound according to the reference signal; If it is detected that the speaker emits a sound, first positioning information of a horizontal angle is obtained according to the multi-channel third microphone signal; second positioning information of a vertical angle is obtained according to the multi-channel second microphone signal; and position information of the first speaker is obtained according to the first positioning information and the second positioning information; If no sound is detected from the speaker, the third positioning information of the horizontal angle is obtained according to the multiple first microphone signals and the multiple third microphone signals; the second positioning information of the vertical angle is obtained according to the multiple second microphone signals; and the second speaker position information is obtained according to the third positioning information and the second positioning information.

2. The method according to claim 1, characterized in that The conference device also includes: at least one camera; The method further comprises: Collecting image signals through the at least one camera; Acquiring speaker image information according to the image signal; The first speaker position information is acquired according to the first positioning information, the second positioning information and the speaker image information.

3. The method according to any one of claims 1 to 2, characterized in that: The second microphone array and the third microphone array are in an orthogonal configuration, and share a microphone at the intersection position.

4. The method according to any one of claims 1 to 2, characterized in that: The first microphone array and the third microphone array include linear arrays in the horizontal direction; The second microphone array includes a vertical linear array.

5. The method according to any one of claims 1 to 2, characterized in that: The microphone comprises an omnidirectional microphone.

6. The method according to any one of claims 1 to 2, characterized in that: The microphone collects far-field sound source signals.

7. A speech processing method for conference equipment, characterized in that: The conference device comprises: a first microphone array, a third microphone array, at least one fourth microphone, and at least one loudspeaker; The at least one fourth microphone is proximate to the at least one speaker array; The method comprises: Collecting multiple first microphone signals through a first microphone array; collecting multiple third microphone signals through a third microphone array; and collecting at least one fourth microphone signal through at least one fourth microphone as an acoustic reference microphone signal; and obtaining a back-collection reference signal through a speaker; Detecting whether the speaker makes a sound according to the reference signal; If the speaker is detected to emit sound, an echo cancellation signal is obtained according to the multi-channel third microphone signal, the acoustic reference microphone signal and the echo reference signal; If no sound is detected from the speaker, a sound signal is acquired according to the multiple first microphone signals and the multiple third microphone signals.

8. The method according to claim 7, characterized in that The fourth microphone includes a low-sensitivity microphone.

9. A conference device, characterized in that: a first microphone array, a second microphone array, a third microphone array, and at least one speaker; Processor; and A memory is used to store a program for implementing the method according to any one of claims 1 to 6, and the conference device is powered on and runs the program of the method through the processor.

10. The device according to claim 9, characterized in that The conference device also includes: at least one camera.

11. A conference device, characterized in that: include: a first microphone array, a third microphone array, at least one fourth microphone, and at least one speaker; processor; as well as A memory is used to store a program for implementing the method according to claim 7, and the conference device is powered on and runs the program of the method through the processor.

Images