Multi-microphone cooperative pickup method, device and equipment and storage medium

Abstract

The invention provides a multi-microphone cooperative pickup method, device and equipment and a storage medium, and the method comprises the steps: collecting a first audio signal through a plurality of microphones, carrying out the sound source positioning according to the first audio signal, determining a main microphone and an auxiliary microphone in the plurality of microphones according to a sound source positioning result, and carrying out the sound source positioning according to the main microphone and the auxiliary microphone. Through the main microphone and the auxiliary microphone, the second audio signal is acquired cooperatively in real time, the main microphone and the auxiliary microphone are distinguished, all microphones are prevented from being in an acquisition state at the same time, the influence of background noise is reduced, the pickup effect is improved, and the experience of a speaker is met; in the process of collecting the second audio signal, the signal characteristics of the second audio signal are analyzed in real time, and the main microphone and the auxiliary microphone are adaptively adjusted according to the signal characteristics, so that the pickup effect is ensured.

Description

Technical Field

[0001] The present application relates to the field of audio, and in particular to a multi-microphone collaborative sound pickup method, device, equipment and storage medium. Background Art

[0002] In modern conference room environments, the application of multi-microphone systems is becoming more and more common. However, when multiple microphones work together in the same environment, since each microphone is always in the same collection state, it is more susceptible to background noise, reflected sound, etc., resulting in poor sound pickup, thus affecting the quality of the meeting. Summary of the invention

[0003] The embodiments of the present application provide a multi-microphone collaborative sound pickup method, device, equipment and storage medium to solve at least one problem existing in the related art. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a method for collaborative sound pickup using multiple microphones, including:

[0005] Collecting a first audio signal through a plurality of microphones;

[0006] Performing sound source localization according to the first audio signal, and determining a main microphone and a secondary microphone from the plurality of microphones according to the sound source localization result;

[0007] The second audio signal is collected in real time by the main microphone and the auxiliary microphone;

[0008] In the process of collecting the second audio signal, the signal characteristics of the second audio signal are analyzed in real time, and the main microphone and the auxiliary microphone are adaptively adjusted according to the signal characteristics.

[0009] In one implementation, the performing of sound source localization according to the first audio signal, and determining a main microphone and a secondary microphone from among the plurality of microphones according to the sound source localization result includes:

[0010] Performing sound source localization according to the first audio signal and the time delay estimation algorithm to obtain a sound source localization result;

[0011] According to the sound source localization result, the microphone closest to the sound source is determined as the main microphone, and the microphone closest to the main microphone is determined as the auxiliary sub-microphone.

[0012] In one implementation, the real-time collaborative collection of the second audio signal by the main microphone and the secondary microphone includes:

[0013] collecting a third audio signal through the main microphone, and collecting a fourth audio signal through the secondary microphone;

[0014] Performing quality analysis on the third audio signal to determine a quality analysis result;

[0015] According to the quality analysis result, the third audio signal is subjected to background noise suppression using the fourth audio signal to obtain a second audio signal, or background noise suppression and audio enhancement processing are performed to obtain a second audio signal.

[0016] In one implementation, the adaptively adjusting the main microphone and the secondary microphone according to the signal characteristics includes:

[0017] When the signal quality score represented by the signal feature is lower than the score threshold, determining the first signal quality of the secondary microphone;

[0018] When the first signal quality is greater than or equal to the scoring threshold, the main microphone is used as a new secondary microphone and the secondary microphone is used as a new main microphone; when the first signal quality is less than the scoring threshold, the second signal quality of the microphones within the preset range of the main microphone is determined; when at least one of the second signal qualities is greater than the scoring threshold, the microphone corresponding to the largest second signal quality is used as a new main microphone and the main microphone is used as a new secondary microphone;

[0019] When all the second signal qualities are less than the scoring threshold, the microphone with the highest signal quality between the first signal quality and the second signal quality is used as a new main microphone, and the main microphone is used as a new secondary microphone.

[0020] In one implementation, the adaptively adjusting the main microphone and the secondary microphone according to the signal characteristics includes:

[0021] In the process of collecting the second audio signal, re-localizing the sound source according to the second audio signal to obtain a new sound source localization result, or determining a first voiceprint feature of the first audio signal, and when the first voiceprint feature is different from a second voiceprint feature in the signal feature, re-localizing the sound source according to the second audio signal to obtain a new sound source localization result;

[0022] According to the new sound source localization result, return to the step of determining a main microphone and a secondary microphone from the plurality of microphones according to the sound source localization result.

[0023] In one embodiment, the method further comprises:

[0024] In each area of ​​the deployment space, a fourth audio signal is correspondingly collected;

[0025] Analyze the fourth audio signal respectively to determine the reverberation time, the sound energy attenuation curve, the sound reflection characteristics, the absorption characteristics, the background noise level and the spatial frequency response;

[0026] Determining the acoustic characteristics of each of the areas according to the reverberation time, the sound energy attenuation curve, the sound reflection characteristics, the absorption characteristics, the background noise level, and the spatial frequency response;

[0027] The microphones are deployed in the deployment space according to the acoustic characteristics.

[0028] In one embodiment, the method further comprises:

[0029] Receive user feedback;

[0030] When the user feedback problem is background noise, adjusting the frequency response coefficient of the Wiener filter or increasing the noise threshold to enhance noise suppression of the second audio signal;

[0031] When the user feedback problem is an echo, a delay compensation parameter and an echo suppression gain parameter of an echo cancellation algorithm are adjusted, and the second audio signal is processed by the adjusted echo cancellation algorithm.

[0032] In a second aspect, an embodiment of the present application provides a multi-microphone collaborative sound pickup device, comprising:

[0033] A first acquisition module, used for acquiring a first audio signal through a plurality of microphones;

[0034] a positioning module, configured to perform sound source positioning according to the first audio signal, and determine a main microphone and a secondary microphone from among the plurality of microphones according to the sound source positioning result;

[0035] A second acquisition module, used to collaboratively acquire a second audio signal in real time through the main microphone and the auxiliary microphone;

[0036] The adjustment module is used to analyze the signal characteristics of the second audio signal in real time during the process of collecting the second audio signal, and adaptively adjust the main microphone and the auxiliary microphone according to the signal characteristics.

[0037] In one embodiment, the adjustment module is further used for:

[0038] In each area of ​​the deployment space, a fourth audio signal is correspondingly collected;

[0039] Analyze the fourth audio signal respectively to determine the reverberation time, the sound energy attenuation curve, the sound reflection characteristics, the absorption characteristics, the background noise level and the spatial frequency response;

[0040] Determining the acoustic characteristics of each of the areas according to the reverberation time, the sound energy attenuation curve, the sound reflection characteristics, the absorption characteristics, the background noise level, and the spatial frequency response;

[0041] The microphones are deployed in the deployment space according to the acoustic characteristics.

[0042] In one embodiment, the adjustment module is further used for:

[0043] User feedback issues;

[0044] When the user feedback problem is background noise, adjusting the frequency response coefficient of the Wiener filter or increasing the noise threshold to enhance noise suppression of the second audio signal;

[0045] When the user feedback problem is an echo, a delay compensation parameter and an echo suppression gain parameter of an echo cancellation algorithm are adjusted, and the second audio signal is processed by the adjusted echo cancellation algorithm.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement a method in any one of the above-mentioned embodiments.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, the method in any one of the above-mentioned embodiments is implemented.

[0048] The beneficial effects of the above technical solution include at least:

[0049] A first audio signal is collected by several microphones, and sound source positioning is performed based on the first audio signal. A main microphone and a secondary microphone are determined from the several microphones based on the sound source positioning result. The second audio signal is collected in real time and collaboratively by the main microphone and the secondary microphone, and the main microphone and the secondary microphone are distinguished to avoid all microphones being in the collection state at the same time, which is beneficial to reducing the influence of background noise, improving the sound pickup effect, and satisfying the speaker's experience. In the process of collecting the second audio signal, the signal characteristics of the second audio signal are analyzed in real time, and the main microphone and the secondary microphone are adaptively adjusted according to the signal characteristics to ensure the sound pickup effect.

[0050] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0052] Figure 1 This is a schematic diagram of the steps of a multi-microphone collaborative sound pickup method according to an embodiment of the present application;

[0053] Figure 2 This is a structural block diagram of a multi-microphone cooperative sound pickup device according to an embodiment of the present application;

[0054] Figure 3 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0056] Reference Figure 1 , a flowchart of a multi-microphone collaborative sound pickup method according to an embodiment of the present application is shown, and the multi-microphone collaborative sound pickup method may at least include steps S100-S400:

[0057] S100: Collect a first audio signal using a plurality of microphones.

[0058] S200: localize a sound source according to the first audio signal, and determine a main microphone and a secondary microphone from among a plurality of microphones according to the sound source localization result.

[0059] S300, using the main microphone and the auxiliary microphone to collaboratively collect the second audio signal in real time.

[0060] S400: In the process of collecting the second audio signal, analyze the signal characteristics of the second audio signal in real time, and adaptively adjust the main microphone and the auxiliary microphone according to the signal characteristics.

[0061] The technical solution of the embodiment of the present application collects a first audio signal through several microphones, performs sound source positioning according to the first audio signal, and determines a main microphone and a secondary microphone from several microphones according to the sound source positioning result, and collects a second audio signal in real time and collaboratively through the main microphone and the secondary microphone, distinguishes the main microphone and the secondary microphone, and avoids all microphones being in the collection state at the same time, which is beneficial to reducing the influence of background noise, improving the sound pickup effect, and satisfying the speaker's experience; in the process of collecting the second audio signal, the signal characteristics of the second audio signal are analyzed in real time, and the main microphone and the secondary microphone are adaptively adjusted according to the signal characteristics to ensure the sound pickup effect.

[0062] In one implementation, a scenario in which several microphones are installed in a conference room is used as an example to collect a first audio signal through the several microphones. For example, in an initial state, when a speaker speaks, all microphones are working and collect the corresponding first audio signal.

[0063] In one implementation, step S200 includes steps S210-S220:

[0064] S210: Perform sound source localization according to the first audio signal and a time delay estimation algorithm to obtain a sound source localization result.

[0065] Optionally, the distance between each microphone is fixed and can be measured in advance and recorded in the system. Therefore, after receiving the first audio signal, since the speakers are at different positions and at different distances from different microphones, there will be a difference in sound wave propagation time. Therefore, based on the distance between the microphones and the speed of sound (approximately 343m / s), the system can calculate the sound wave propagation time difference through the time delay estimation algorithm TDOA, thereby locating the sound source and obtaining the sound source localization results, such as the distance and angle between the speaker and each microphone.

[0066] S220: According to the sound source localization result, determine the microphone closest to the sound source as the main microphone, and determine the microphone closest to the main microphone as the auxiliary sub-microphone.

[0067] In the embodiment of the present application, according to the sound source localization result, the microphone closest to the sound source (i.e., the speaker) can be determined as the main microphone, and then the microphone closest to the main microphone can be used as the auxiliary sub-microphone to ensure the sound pickup effect. Among them, the auxiliary sub-microphone can be one or more. The system can reduce or turn off the volume of other microphones except the main microphone and the sub-microphone to reduce power consumption and reduce interference to the speaker. Specifically, when the first audio signal continues, that is, when the speaker keeps speaking, the system keeps the main microphone and other microphones except the sub-microphone turned off or at a low volume. When the first audio signal ends, that is, when the speaker stops speaking, the volume of the remaining microphones is immediately turned on or increased to achieve dynamic adjustment.

[0068] In one implementation, step S300 includes steps S310-S330:

[0069] S310: Collect a third audio signal through the main microphone, and collect a fourth audio signal through the auxiliary microphone.

[0070] Optionally, after the main microphone and the auxiliary microphone are determined, both the main microphone and the auxiliary microphone collect audio signals. For example, when the speaker is speaking, the signal collected by the main microphone is recorded as the third audio signal, and the signal collected by the auxiliary microphone is recorded as the fourth audio signal; the auxiliary microphone is used to eliminate spatial reflections and noise interference for the main microphone.

[0071] S320: Perform quality analysis on the third audio signal to determine a quality analysis result.

[0072] Optionally, the system can analyze parameters such as the signal-to-noise ratio, sound clarity, reverberation time, and spectrum of the third audio signal, and then the numerical value corresponding to each analysis result has a corresponding preset interval and a score corresponding to the preset interval, and then weighted calculation is performed based on the score and weighted parameters corresponding to different parameters to obtain a quality analysis result.

[0073] S330. According to the quality analysis result, use the fourth audio signal and the third audio signal to perform background noise suppression to obtain a second audio signal, or perform background noise suppression and audio enhancement processing to obtain a second audio signal.

[0074] Optionally, when the quality analysis result is characterized as poor, it means that the sound pickup quality may be affected by the large background noise. At this time, the system can use the fourth audio signal to subtract the fourth audio signal from the third audio signal to suppress the background noise to obtain a processed audio signal, i.e., the second audio signal, or further perform audio enhancement processing on the processed audio signal (for example, the system increases the audio gain through its own gain electronic circuit or through software) to obtain the second audio signal.

[0075] In one implementation, in step S400, the signal characteristics of the second audio signal are analyzed in real time, and parameters including but not limited to signal-to-noise ratio, sound clarity, reverberation time, spectrum, etc. may be collected, and then two methods may be used:

[0076] 1. Based on the method of S320, the analysis result of the corresponding parameter can be first determined, and then the quality analysis result can be weighted based on the analysis result and the weight parameter to determine as the signal feature (the signal feature is equivalent to the overall score at this time), and then the signal feature at this time can be compared with the score threshold; because in some cases, even after the third audio signal is processed, the obtained second audio signal may not meet the standard, so it is necessary to further analyze the signal feature of the second audio signal,

[0077] 2. The score threshold corresponding to the analysis result of the second audio signal can be directly set, and then the analysis result is compared with the score threshold to obtain the signal feature. In this case, the signal feature is equivalent to the score including multiple individual indicators.

[0078] Wherein, in step S400, the main microphone and the auxiliary microphone are adaptively adjusted according to the signal characteristics, including steps S410-S430:

[0079] S410: When the signal quality score represented by the signal feature is lower than the score threshold, determine the first signal quality of the secondary microphone.

[0080] Optionally, when the signal characteristic indicates that the signal quality score is lower than a score threshold, for example:

[0081] 1. The overall score of the signal feature is lower than the score threshold, that is, the signal quality score represented by the signal feature is lower than the score threshold. At this time, the first signal quality of the secondary microphone is determined based on the same method 1;

[0082] 2. An individual indicator of the signal feature is lower than the scoring threshold. At this time, the signal feature represents that the signal quality score is lower than the scoring threshold. At this time, the first signal quality of the secondary microphone is determined based on the same method 2.

[0083] S420: When the first signal quality is greater than or equal to the scoring threshold, the main microphone is used as the new secondary microphone and the secondary microphone is used as the new main microphone; when the first signal quality is less than the scoring threshold, the second signal quality of the microphones within the preset range of the main microphone is determined; when at least one second signal quality is greater than the scoring threshold, the microphone corresponding to the largest second signal quality is used as the new main microphone and the main microphone is used as the new secondary microphone.

[0084] Optionally, when the first signal quality is greater than or equal to the scoring threshold, corresponding method 1 is that the overall score of the signal feature is greater than or equal to the scoring threshold, and corresponding method 2 is that each individual indicator is greater than or equal to the scoring threshold, at this time, the main microphone is used as the new secondary microphone and the secondary microphone is used as the new main microphone.

[0085] Optionally, when the first signal quality is less than the scoring threshold, it means that the signal quality of the secondary microphone is still not up to standard, and the second signal quality of the microphone within the preset range of the main microphone is determined (it can also be determined based on method 1 or method 2). When at least one second signal quality is greater than the scoring threshold, corresponding to method 1, the overall score of the second signal quality is greater than or equal to the scoring threshold, and corresponding to method 2, each individual indicator is greater than or equal to the scoring threshold. The microphone corresponding to the largest second signal quality is used as the new main microphone (the corresponding method 1 is the highest overall score, and the corresponding method 2 can further calculate the overall score, and the highest overall score is used as the largest second signal quality), and the original main microphone is used as the new secondary microphone.

[0086] S430: When all the second signal qualities are less than the scoring threshold, use the microphone with the highest signal quality between the first signal quality and the second signal quality as a new main microphone, and use the main microphone as a new secondary microphone.

[0087] Optionally, when all second signal qualities are less than a scoring threshold, the microphone with the highest signal quality among the first signal quality and the second signal quality is used as the new main microphone. For example, when all second signal qualities are less than a scoring threshold, the first signal quality and the second signal quality are determined by calculating the overall score in method 1, and then the microphone with the highest overall score is used as the new main microphone, and the original main microphone is used as the new secondary microphone. It should be noted that if the original main microphone is still determined to be the new main microphone, that is, the main microphone has not changed, then the system can call the AI ​​audio processing model corresponding to the signal-to-noise ratio, sound clarity, reverberation time, and spectrum to process the second audio signal again to obtain the final second audio signal.

[0088] In one embodiment, the signal characteristics of the second audio signal analyzed in real time in step S400 may also include voiceprint characteristics, for example, Mel-frequency cepstral coefficients (MFCC), fundamental frequency, formants, and timbre characteristics may be analyzed to determine the voiceprint characteristics. Optionally, the system may establish a voiceprint model for each speaker based on the extracted sound characteristics, and through a machine learning algorithm (such as GMM-UBM, Gaussian mixture model-universal background model, or deep neural network DNN), the system may be able to distinguish different speakers based on voiceprint characteristics in real time.

[0089] In one implementation, the step S400 adaptively adjusts the main microphone and the auxiliary microphone according to the signal characteristics, and may further include steps S440-S450:

[0090] S440. In the process of collecting the second audio signal, re-localize the sound source according to the second audio signal to obtain a new sound source localization result, or determine the first voiceprint feature of the first audio signal. When the first voiceprint feature is different from the second voiceprint feature in the signal feature, re-localize the sound source according to the second audio signal to obtain a new sound source localization result.

[0091] It should be noted that the second audio signal refers to the audio signal continuously collected after the main microphone is determined. In the meeting, in addition to the current speaker, there may be other speakers. If the speaker is far away from the original speaker, the sound pickup effect of the current main microphone will inevitably not meet the speaker's needs. Therefore, in some embodiments, in the process of collecting the second audio signal, the sound source is re-localized according to the second audio signal to obtain a new sound source localization result, which is equivalent to continuously determining a new sound source localization result in real time.

[0092] In another embodiment, the first voiceprint feature of the first audio signal can be identified through the AI ​​voiceprint recognition model, and then the first voiceprint feature is compared with the second voiceprint feature in the signal feature. If they are the same, the current main microphone is maintained. If they are different, the sound source is re-localized according to the second audio signal to obtain a new sound source localization result. It should be noted that the method of sound source localization is the same as the delay estimation method mentioned above and will not be repeated here.

[0093] S450: According to the new sound source localization result, return to the step of determining a main microphone and a secondary microphone from a plurality of microphones according to the sound source localization result.

[0094] Optionally, after determining a new sound source localization result, for example, after determining the current actual speaker's position, the process returns to step S200 to re-determine the primary microphone and the secondary microphone. It is understandable that if the first voiceprint feature is the same as the second voiceprint feature in the signal feature, or the new sound source localization result is the same as the original sound source localization result, then S450 may not be executed, and the current primary microphone and the current secondary microphone may be maintained to reduce the amount of data processing.

[0095] In one implementation, the multi-microphone collaborative sound pickup method of the embodiment of the present application may further include steps S510-S530:

[0096] S510. In each area of ​​the deployment space, a fourth audio signal is correspondingly collected, and the fourth audio signal is respectively analyzed to determine a reverberation time, a sound energy attenuation curve, a sound reflection characteristic, an absorption characteristic, a background noise level, and a spatial frequency response.

[0097] Optionally, when determining the deployment space where multiple microphones need to be deployed, the areas are divided according to a certain ratio, and then the fourth audio signal is collected correspondingly using audio equipment in each area. The system then analyzes the fourth audio signal separately to determine the reverberation time (RT60, indicating the time for sound to reflect and decay in the area, used to reflect the reverberation effect of the environment), the sound energy attenuation curve (describing the energy attenuation of sound waves when propagating indoors, affecting the clarity and restoration of the sound), sound reflection characteristics (by analyzing the reflection points in the area, understanding the sound reflection characteristics of different areas), absorption characteristics (by analyzing the distribution of sound-absorbing materials in the area, understanding the absorption characteristics of different areas), background noise level (SPL, real-time monitoring of the noise level in the room to provide a reference for the gain adjustment of the microphone) and spatial frequency response (measuring the frequency response of different areas to ensure that the microphone can capture sounds of the full frequency band).

[0098] S520: Determine the acoustic characteristics of each area according to the reverberation time, the sound energy attenuation curve, the sound reflection characteristics, the absorption characteristics, the background noise level, and the spatial frequency response.

[0099] Optionally, an environmental acoustic model of the deployment space is established based on the reverberation time, the sound energy attenuation curve, the sound reflection characteristics, the absorption characteristics, the background noise level, and the spatial frequency response, and the environmental acoustic model reflects the acoustic characteristics of each area.

[0100] S530: Deploy microphones in the deployment space according to the acoustic characteristics.

[0101] Then, based on the acoustic characteristics of the environmental acoustic model, the deployment position of the microphone (such as the placement position and the distance between them) and the audio parameters can be optimized, and each microphone can be deployed in the deployment space to ensure the best sound pickup effect in the conference environment. For example, the microphone can be placed in an area with less reflection or the layout of the microphone array can be adjusted to avoid collecting audio in locations with severe reflections or reverberation to improve signal quality; gain and sensitivity adjustment: reduce the gain in areas with high noise levels or strong reflections, and appropriately increase the gain in quiet areas to balance the sound pickup effect in different areas; frequency response optimization: set a low-pass filter for the microphone close to the wall to reduce the high-frequency interference introduced by reflections; set a high gain for the microphone far away from the sound source to improve the mid-frequency response and ensure speech clarity.

[0102] In one embodiment, when deploying microphones, it is possible to configure the ID, model, distance, audio parameters (such as sensitivity range, frequency response curve), etc. of each microphone and configure them into a parameter preset template. When in use, each microphone can be initialized using a communication protocol.

[0103] In one implementation, the multi-microphone collaborative sound pickup method of the embodiment of the present application may further include steps S610-S630:

[0104] S610: Receive user feedback.

[0105] Optionally, the system provides a selection page for the user to provide feedback or receive voice feedback from the user. For example, the user can provide feedback on background noise, echo, and other issues, without specific limitation.

[0106] S620: When the user feedback problem is background noise, adjust the frequency response coefficient of the Wiener filter or increase the noise threshold to enhance noise suppression of the second audio signal.

[0107] Optionally, when the user feedback problem is background noise, the system can adjust the frequency response coefficient H(n) of the Wiener filter and use the adjusted Wiener filter to filter the second audio signal, for example: S_out(n) = H(n)*S_in(n), S_out(n) is the output signal, S_in(n) is the input signal. Alternatively, the noise threshold of the Wiener filter can be increased to filter out lower noise; or, the adaptability of the Wiener filter can be increased to make it pay more attention to the main speech band (such as 300Hz to 3.4kHz) and suppress the non-speech band.

[0108] S630: When the user feedback problem is an echo, adjust a delay compensation parameter and an echo suppression gain parameter of an echo cancellation algorithm, and process the second audio signal by using the adjusted echo cancellation algorithm.

[0109] Optionally, the system can be configured with several filters or filtering algorithms to process the second audio signal. Therefore, when the user feedback problem is an echo, the system can adjust the delay compensation parameters and echo suppression gain parameters of the echo cancellation algorithm AEC, and then process the second audio signal through the adjusted echo cancellation algorithm to reduce or eliminate the echo.

[0110] In some implementations, an algorithm may also be used for delay compensation: to increase delay detection sensitivity and reduce the lag effect of the echo; and after adjustment, the parameters are continuously updated based on the feedback issues, and the adjustment effects are tested to ensure that the sound quality issues reported by users are improved.

[0111] In some embodiments, when the second audio data contains audio of multiple speakers, the system performs multi-channel signal processing and uses a Kalman filter to fuse the multi-channel signals. The formula is: x_hat_k = A*x_hat_k-1+B*u_k+w_k, x_hat_k is the state estimate at the current time k, A is the state transfer matrix, B is the control input matrix, u_k is the input signal, and w_k is the process noise. The state transfer matrix A: describes the relationship between the previous state and the current state, obtained by modeling the historical audio signal, and is used to predict the next signal state change. The control input matrix B: the influence weight of the control input on the state estimate. The degree of influence of the control input is determined by the test environment and user input. u_k: the input signal received by the system in real time, such as gain control or signal source switching, which is determined by the system according to the feedback loop or user settings. Process noise w_k: represents the noise in the signal processing process. The noise model is dynamically adjusted by measuring the environmental noise level of each channel to suppress interference. Through Kalman filtering, the system fuses the multi-channel signals to obtain a clean target speech signal. Finally, recognition separation can be performed on the clean target speech signal to separate the audio of different speakers.

[0112] In some embodiments, the system provides a visualization page, where users can intuitively understand the sound quality and manually adjust the parameters, such as displaying a real-time spectrum diagram: showing the spectrum of the audio to help users observe the signal strength of each frequency band. If the low-frequency noise is strong, users can adjust the gain parameter, gain low-pass filter, and suppress low-frequency interference; sound quality monitoring chart: providing dynamic curves of key parameters such as real-time SNR and RT60 to display signal fluctuations. If the SNR is found to continue to decrease, users can increase the microphone gain parameter or activate the noise reduction algorithm; parameter adjustment panel: providing manual adjustment options for parameters such as sensitivity, gain, and noise suppression strength. Users can adjust in real time according to actual conditions, and the system will reprocess the audio data under the new parameters, such as reprocessing the second audio signal. Among them, after the user adjusts the parameters on the interface, the system will immediately update the parameters, such as audio parameters or algorithms, and reapply them to the second audio signal. The adjustment effect is observed through visual feedback, making the sound quality optimization process more flexible and efficient.

[0113] Reference Figure 2 , shows a structural block diagram of a multi-microphone collaborative sound pickup device according to an embodiment of the present application, and the device may include:

[0114] A first acquisition module, used for acquiring a first audio signal through a plurality of microphones;

[0115] A positioning module, used to perform sound source positioning according to the first audio signal, and determine a main microphone and a secondary microphone from a plurality of microphones according to the sound source positioning result;

[0116] A second acquisition module is used to collaboratively acquire a second audio signal in real time through the main microphone and the auxiliary microphone;

[0117] The adjustment module is used to analyze the signal characteristics of the second audio signal in real time during the process of collecting the second audio signal, and adaptively adjust the main microphone and the auxiliary microphone according to the signal characteristics.

[0118] In one embodiment, the adjustment module is further configured to:

[0119] In each area of ​​the deployment space, a fourth audio signal is correspondingly collected;

[0120] Analyzing the fourth audio signal respectively to determine the reverberation time, the sound energy attenuation curve, the sound reflection characteristics, the absorption characteristics, the background noise level and the spatial frequency response;

[0121] Determine the acoustic characteristics of each area based on reverberation time, sound energy decay curve, sound reflection characteristics, absorption characteristics, background noise level, and spatial frequency response;

[0122] The microphones are deployed in the deployment space according to the acoustic characteristics.

[0123] In one embodiment, the adjustment module is further configured to:

[0124] User feedback issues;

[0125] When the user feedback problem is background noise, the frequency response coefficient of the Wiener filter is adjusted or the noise threshold is increased to enhance noise suppression of the second audio signal;

[0126] When the user feedback problem is an echo, a delay compensation parameter and an echo suppression gain parameter of the echo cancellation algorithm are adjusted, and the second audio signal is processed by the adjusted echo cancellation algorithm.

[0127] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and will not be repeated here.

[0128] Reference Figure 3 , shows a structural block diagram of an electronic device according to an embodiment of the present application, the electronic device includes: a memory 310 and a processor 320, the memory 310 stores instructions that can be run on the processor 320, and the processor 320 loads and executes the instructions to implement the multi-microphone collaborative sound pickup method in the above embodiment. The number of the memory 310 and the processor 320 can be one or more.

[0129] In one embodiment, the electronic device also includes a communication interface 330 for communicating with external devices and performing data exchange transmission. If the memory 310, the processor 320 and the communication interface 330 are implemented independently, the memory 310, the processor 320 and the communication interface 330 can be interconnected through a bus and complete mutual communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0130] Optionally, in a specific implementation, if the memory 310, the processor 320 and the communication interface 330 are integrated on a chip, the memory 310, the processor 320 and the communication interface 330 can communicate with each other through an internal interface.

[0131] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-microphone collaborative sound pickup method provided in the above embodiment.

[0132] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0133] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0134] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the advanced RISC machines (ARM) architecture.

[0135] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0136] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0137] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the described specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0138] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0139] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0140] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0141] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0142] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-microphone collaborative sound pickup method, characterized in that: include: Collecting a first audio signal through a plurality of microphones; Performing sound source localization according to the first audio signal, and determining a main microphone and a secondary microphone from the plurality of microphones according to the sound source localization result; The second audio signal is collected in real time by the main microphone and the auxiliary microphone; In the process of collecting the second audio signal, the signal characteristics of the second audio signal are analyzed in real time, and the main microphone and the auxiliary microphone are adaptively adjusted according to the signal characteristics.

2. The multi-microphone collaborative sound pickup method according to claim 1, characterized in that: The performing of sound source localization according to the first audio signal, and determining a main microphone and a secondary microphone from the plurality of microphones according to the sound source localization result comprises: Performing sound source localization according to the first audio signal and the time delay estimation algorithm to obtain a sound source localization result; According to the sound source localization result, the microphone closest to the sound source is determined as the main microphone, and the microphone closest to the main microphone is determined as the auxiliary sub-microphone.

3. The multi-microphone collaborative sound pickup method according to claim 2, characterized in that: The real-time collaborative collection of the second audio signal by the main microphone and the auxiliary microphone includes: collecting a third audio signal through the main microphone, and collecting a fourth audio signal through the secondary microphone; Performing quality analysis on the third audio signal to determine a quality analysis result; According to the quality analysis result, the third audio signal is subjected to background noise suppression using the fourth audio signal to obtain a second audio signal, or background noise suppression and audio enhancement processing are performed to obtain a second audio signal.

4. The multi-microphone collaborative sound pickup method according to any one of claims 1 to 3, characterized in that: The adaptively adjusting the main microphone and the auxiliary microphone according to the signal characteristics includes: When the signal quality score represented by the signal feature is lower than the score threshold, determining the first signal quality of the secondary microphone; When the first signal quality is greater than or equal to the scoring threshold, the main microphone is used as a new secondary microphone and the secondary microphone is used as a new main microphone; when the first signal quality is less than the scoring threshold, the second signal quality of the microphones within the preset range of the main microphone is determined; when at least one of the second signal qualities is greater than the scoring threshold, the microphone corresponding to the largest second signal quality is used as a new main microphone and the main microphone is used as a new secondary microphone; When all the second signal qualities are less than the scoring threshold, the microphone with the highest signal quality between the first signal quality and the second signal quality is used as a new main microphone, and the main microphone is used as a new secondary microphone.

5. The multi-microphone collaborative sound pickup method according to any one of claims 1 to 3, characterized in that: The adaptively adjusting the main microphone and the auxiliary microphone according to the signal characteristics includes: In the process of collecting the second audio signal, re-localizing the sound source according to the second audio signal to obtain a new sound source localization result, or determining a first voiceprint feature of the first audio signal, and when the first voiceprint feature is different from a second voiceprint feature in the signal feature, re-localizing the sound source according to the second audio signal to obtain a new sound source localization result; According to the new sound source localization result, return to the step of determining a main microphone and a secondary microphone from the plurality of microphones according to the sound source localization result.

6. The multi-microphone collaborative sound pickup method according to any one of claims 1 to 3, characterized in that: The method further comprises: In each area of ​​the deployment space, a fourth audio signal is correspondingly collected; Analyze the fourth audio signal respectively to determine the reverberation time, the sound energy attenuation curve, the sound reflection characteristics, the absorption characteristics, the background noise level and the spatial frequency response; Determining the acoustic characteristics of each of the areas according to the reverberation time, the sound energy attenuation curve, the sound reflection characteristics, the absorption characteristics, the background noise level, and the spatial frequency response; The microphones are deployed in the deployment space according to the acoustic characteristics.

7. The multi-microphone collaborative sound pickup method according to claim 1, characterized in that: The method further comprises: Receive user feedback; When the user feedback problem is background noise, adjusting the frequency response coefficient of the Wiener filter or increasing the noise threshold to enhance noise suppression of the second audio signal; When the user feedback problem is an echo, a delay compensation parameter and an echo suppression gain parameter of an echo cancellation algorithm are adjusted, and the second audio signal is processed by the adjusted echo cancellation algorithm.

8. A multi-microphone collaborative sound pickup device, characterized in that: include: A first acquisition module, used for acquiring a first audio signal through a plurality of microphones; a positioning module, configured to perform sound source positioning according to the first audio signal, and determine a main microphone and a secondary microphone from among the plurality of microphones according to the sound source positioning result; A second acquisition module, used to collaboratively acquire a second audio signal in real time through the main microphone and the auxiliary microphone; The adjustment module is used to analyze the signal characteristics of the second audio signal in real time during the process of collecting the second audio signal, and adaptively adjust the main microphone and the auxiliary microphone according to the signal characteristics.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images