A microphone calibration method and system for a multi-modal intelligent terminal

By using the frequency domain characteristics of short-time window signal segments and ICA decomposition technology in a multimodal intelligent terminal to extract and adjust the sound source signal, the lack of clarity of the main sound source in the dynamic environment in the prior art is solved. Problem, more efficient sound source separation and beamforming are achieved.

CN118574049BActive Publication Date: 2024-11-08ROPEOK TECHNOLOGY GROUP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411046337.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-08
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

The microphone calibration system of existing multimodal intelligent terminals is difficult to achieve accurate main sound source clarity in dynamic environments, especially when the main and secondary sound sources have low contrast and the main sound sources are frequently switched, it is impossible to effectively suppress subsound source interference.

Method used

By obtaining the mixed sound signals in the microphone array, segmenting using the frequency domain characteristics of the short-term window signal segment, ICA decomposition is performed to extract the main sound source component signal, and the obfuscation coefficient is calculated based on the main sound source probability, and the sound source signal is adaptively adjusted. Intensity to suppress infrasound source interference.

Benefits of technology

The clarity and accuracy of the main sound source components received by the microphone are improved, making the sound beam formation clearer and more accurate in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118574049B_ABST
    Figure CN118574049B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of audio processing technology, and in particular to a microphone calibration method and system for a multimodal intelligent terminal. The method comprises: obtaining a mixed sound signal received by each microphone in a microphone array; segmenting the mixed sound signal received by each microphone to obtain a plurality of sound source intervals corresponding to each microphone; extracting the main sound source component signal of each sound source interval by ICA decomposition; splicing the main sound source component signals of all sound source intervals corresponding to each microphone to obtain the main sound source signal of each microphone; calculating the main sound source probability corresponding to each sound source interval of each microphone; calculating the confusion coefficient of the main and secondary sound sources; and adjusting the sound source signal strength according to the confusion coefficient. Through this method, the secondary sound source signal received at each moment can be adaptively suppressed, so that the main sound source component in the sound beam finally merged is clearer and more accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to a microphone calibration method and system for a multi-modal intelligent terminal. Background Art

[0002] Multimodal smart terminals refer to smart devices with multiple interaction modes (such as touch, voice, vision, etc.). Today, multimodal smart terminals have become an important bridge connecting people and the digital world. By integrating multiple interaction modes such as touch, voice, and vision, they have greatly enriched the user's operating experience. In these interaction modes, microphones play a vital role. They not only support voice recognition and voice commands, but also play a role in video calls, meeting records, environmental sound monitoring and other scenarios.

[0003] In order to support the practicality of smart terminals in complex scenarios, such as remote video conferencing, calls, and video recordings in outdoor environments, more and more terminal devices are equipped with real-time microphone calibration systems. However, environmental noise, sound source objects, and acoustic characteristics may change at any time, such as sudden traffic noise, crowd noise, etc. The current real-time calibration system still has deficiencies in the accuracy of dynamic calibration, especially when the contrast between the primary and secondary sound sources is low and the main sound source switches frequently. The current automatic calibration strategy based on noise level cannot guarantee the clarity of the main sound source.

[0004] Based on this, it is necessary to study a microphone calibration method and system for a multimodal intelligent terminal to adaptively suppress the secondary sound source signal at each moment, so that the main sound source component received by the microphone is clearer and more accurate. Summary of the invention

[0005] In order to solve the above technical problems, the purpose of the present invention is to provide a microphone calibration method and system for a multimodal intelligent terminal, and the technical solutions adopted are as follows:

[0006] One aspect of the present invention provides a microphone calibration method for a multimodal intelligent terminal, the method comprising:

[0007] Acquire a mixed sound signal received by each microphone in the microphone array;

[0008] The mixed sound signal received by each microphone is segmented using the frequency domain characteristics of the short-time window signal segment to obtain multiple sound source intervals corresponding to each microphone;

[0009] Perform ICA decomposition on each sound source interval to extract the main sound source component signal of each sound source interval;

[0010] The main sound source component signals of all sound source intervals corresponding to each microphone are spliced ​​to obtain the main sound source signal of each microphone;

[0011] Calculating the main sound source probability corresponding to each sound source interval in the mixed sound signal received by each microphone according to the main sound source signal;

[0012] Calculate the confusion coefficient of the primary and secondary sound sources at each moment based on the primary sound source probability;

[0013] The intensity of the sound source signal at each moment is adjusted according to the confusion coefficient to achieve microphone calibration.

[0014] In some embodiments, the mixed sound signal received by each microphone is segmented using the frequency domain characteristics of the short-time window signal segment to obtain multiple sound source intervals corresponding to each microphone, including:

[0015] The short-time window length is set according to the sound sampling frequency, and the mixed sound signal received by each microphone is subjected to short-time Fourier transform to obtain the phase spectrum in each short-time window signal segment;

[0016] Merging consecutive short-time windows according to the phase spectrum to obtain one or more target clusters;

[0017] Each of the target clusters is used as a sound source interval corresponding to the microphone.

[0018] In some embodiments, the merging of consecutive short-time windows according to the phase spectrum to obtain one or more target clusters includes:

[0019] According to the number of identical phases between each short-time window and the next adjacent window and the maximum number of phases between the two, the probability of merging each short-time window with the adjacent window is calculated;

[0020] Merging the short-time windows according to the merging probability to obtain a plurality of sub-clusters;

[0021] The multiple sub-clusters are subjected to agglomerative hierarchical clustering according to a preset termination function to obtain one or more target clusters.

[0022] In some embodiments, the preset termination function is obtained based on the following method:

[0023] Calculate the mean of the merging probabilities of all two adjacent short-time windows in each sub-cluster to obtain a first parameter, and obtain a second parameter according to the mean of the first parameters corresponding to all sub-clusters;

[0024] Calculate the non-merging probability of the front-end short-time window and the back-end short-time window in the adjacent sub-clusters, and obtain the third parameter according to the mean of the non-merging probabilities corresponding to all adjacent sub-clusters;

[0025] The second parameter, the third parameter and the penalty item corresponding to the current clustering level are added together to obtain the termination function.

[0026] In some embodiments, performing ICA decomposition on each sound source interval to extract the main sound source component signal of each sound source interval includes:

[0027] Perform ICA signal decomposition on each sound source interval of the mixed sound signal received by each microphone to obtain k types of sound source component signals;

[0028] According to the average amplitudes of the k types of sound source component signals, a type of sound source component signal with the largest average amplitude is screened out as the main sound source component signal of each sound source interval.

[0029] In some embodiments, the step of splicing the main sound source component signals of all sound source intervals corresponding to each microphone to obtain the main sound source signal of each microphone includes:

[0030] The intersection points of the main sound source component signals in all two adjacent sound source intervals are averaged to obtain the main sound source signal of each microphone.

[0031] In some embodiments, calculating the main sound source probability corresponding to each sound source interval in the mixed sound signal received by each microphone according to the main sound source signal includes:

[0032] Obtain the normalized Euclidean norm of the phase error and amplitude error of the target sound source interval of the main sound source signal in the target microphone and the signal segment corresponding to the same time interval in the main sound source signals of other microphones;

[0033] Continuously deriving the Euclidean norm according to the order of distances between the target microphone and other microphones to obtain a first derivative value sequence;

[0034] The signal-to-noise ratios of the target sound source interval of the main sound source signal in the target microphone and the signal segments corresponding to the same time interval in the main sound source signals of other microphones are calculated in sequence, and normalized to obtain a first normalized result;

[0035] Continuously deriving the first normalized result according to the order of distances between the target microphone and other microphones to obtain a second derivative value sequence;

[0036] Based on the Pearson correlation coefficient of the first derivative sequence and the second derivative sequence, a main sound source probability corresponding to a target sound source interval in the mixed sound signal received by the target microphone is obtained.

[0037] In some embodiments, the calculating the confusion coefficient of the primary and secondary sound sources at each moment based on the primary sound source probability includes:

[0038] According to the main sound source probability corresponding to the sound source interval matching the target moment in the mixed sound signals received by all microphones, a microphone with the largest main sound source probability is selected as the main sound source receiving end of the sound source signal at the target moment, and the remaining microphones are used as secondary sound source receiving ends of the sound source signal at the target moment;

[0039] Calculate the variance of the main sound source probability of the sound source signal of the main sound source receiving end at the target time and the main sound source probability of all other secondary sound source receiving ends to obtain a fourth parameter;

[0040] The confusion coefficients of the primary and secondary sound sources at the target moment are obtained based on the fourth parameter.

[0041] In some embodiments, adjusting the intensity of the sound source signal at each moment according to the confusion coefficient to implement microphone calibration includes:

[0042] According to the confusion coefficients of the primary and secondary sound sources corresponding to the target time, a signal strength adjustment coefficient is obtained;

[0043] The amplitude of the sound source signal received by the infrasound source receiving end is adjusted according to the signal strength adjustment coefficient to suppress the infrasound source signal at the target moment.

[0044] Another aspect of the present invention further provides a microphone calibration system for a multimodal intelligent terminal, the system comprising:

[0045] An acquisition module, used to acquire a mixed sound signal received by each microphone in the microphone array;

[0046] A sound source interval determination module is used to segment the mixed sound signal received by each microphone using the frequency domain characteristics of the short-time window signal segment to obtain multiple sound source intervals corresponding to each microphone;

[0047] A main sound source component signal extraction module is used to perform ICA decomposition on each sound source interval to extract the main sound source component signal of each sound source interval;

[0048] A main sound source signal determination module is used to splice the main sound source component signals of all sound source intervals corresponding to each microphone to obtain the main sound source signal of each microphone;

[0049] A main sound source probability calculation module, used to calculate the main sound source probability corresponding to each sound source interval in the mixed sound signal received by each microphone according to the main sound source signal;

[0050] A confusion coefficient calculation module, used to calculate the confusion coefficients of the primary and secondary sound sources at each moment based on the primary sound source probability;

[0051] The calibration module is used to adjust the strength of the sound source signal at each moment according to the confusion coefficient to achieve microphone calibration.

[0052] The present invention has the following beneficial effects:

[0053] In the microphone calibration method and system for a multimodal intelligent terminal provided in some embodiments of the present invention, the mixed sound signal received by each microphone is segmented by utilizing the frequency domain characteristics of the short-time window signal segment to obtain multiple sound source intervals corresponding to each microphone, and then ICA decomposition is performed on each sound source interval separately to extract the main sound source component signal, and the main sound source component signals of all sound source intervals corresponding to each microphone are spliced ​​to obtain the main sound source signal, and then the main sound source probability corresponding to each sound source interval is calculated according to the propagation attenuation characteristics of the main sound source component, and then the main and secondary sound source confusion coefficients at each moment are calculated based on the main sound source probability of each sound source interval, and finally the sound source signal strength at each moment is adaptively adjusted based on the confusion coefficient (mainly suppressing the secondary sound source signal), so that the main sound source component in the final merged sound beam can be clearer and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 An exemplary flow chart of a microphone calibration method for a multi-modal intelligent terminal provided by an embodiment of the present invention;

[0056] Figure 2 An exemplary module diagram of a microphone calibration system for a multi-modal intelligent terminal provided by an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of a short-time window provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the microphone calibration method and system of a multimodal intelligent terminal proposed by the present invention, its specific implementation, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0059] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0060] The specific scheme of a microphone calibration method and system for a multi-modal intelligent terminal provided by the present invention is described in detail below with reference to the accompanying drawings.

[0061] See also Figure 1 , which shows an exemplary flow chart of a microphone calibration method for a multimodal intelligent terminal provided by an embodiment of the present invention.

[0062] S110, obtaining a mixed sound signal received by each microphone in the microphone array. In some embodiments, S110 may be executed by the obtaining module 210 mentioned below.

[0063] Sound generally propagates in the air in the form of waves. The microphone can receive all the sound beams propagating around it and then convert them into digital signals through the A / D module.

[0064] Multimodal intelligent terminal devices can use microphone arrays to receive sound signals, where a microphone array is a sound receiving array composed of multiple individual microphones in a certain geometric structure. Generally, smart speakers or high-end conference systems use flat circular microphone arrays. Due to the large number of array elements and the large spacing between array elements, the signal processing effect is better and the resolution of the received signal is also higher.

[0065] In an embodiment of the present invention, an audio processing system carried by a multimodal intelligent terminal device can be used to obtain a mixed sound signal received by each microphone in a microphone array. The mixed sound signal may include sound signals generated by one or more sound sources.

[0066] S120, using the frequency domain characteristics of the short-time window signal segment to segment the mixed sound signal received by each microphone, to obtain multiple sound source intervals corresponding to each microphone. In some embodiments, S120 may be executed by the sound source interval determination module 220 mentioned below.

[0067] When a multimodal intelligent terminal device is in an outdoor environment or a noisy environment, the sound signals generated by different sound sources will be mixed together and received by the microphone array. In other words, the sound signals received and processed by each microphone may be mixed sound signals that are transmitted by multiple different sound sources and superimposed on each other.

[0068] Since multiple sound sources are mixed asynchronously, and the pauses and sound emission times of the sounds may vary, the mixed sound signal received by each microphone will contain several sound component signals with different phases and multiple segments. Therefore, the mixed sound signal received by each microphone needs to be decomposed first, that is, the sound source separation.

[0069] Specifically, in some embodiments, short-time Fourier transform can be performed on the mixed sound signal received by each microphone to obtain the amplitude spectrum and phase spectrum in all short-time windows of the mixed sound signal received by each microphone. Generally speaking, the microphone sound sampling frequency of a conventional Internet phone is about 16kHz. In some embodiments of the present invention, the length of each short-time window can be set to 400 unit sampling lengths.

[0070] Furthermore, continuous short-time windows may be merged according to the phase spectrum to obtain one or more target clusters, and then each target cluster is used as a sound source interval corresponding to the microphone.

[0071] Specifically, in an embodiment of the present invention, the merging probability of each short-time window and the adjacent window can be calculated based on the number of identical phases between each short-time window and the next adjacent window and the maximum number of phases between the two, and then continuous short-time windows are merged according to the merging probability to obtain multiple sub-clusters. Finally, the multiple sub-clusters obtained after merging are subjected to agglomerative hierarchical clustering according to a preset termination function to obtain one or more target clusters.

[0072] It can be understood that there may be a continuous speech signal of an independent sound source in adjacent short-time windows with the same phase, and the more the number of phase type overlaps, the greater the probability of merging. Figure 3 As shown, A, B, C, and D represent different phase types. Phases of different types and data may exist in each short-time window. When adjacent short-time windows have the same phase, the two adjacent windows may be merged. For example, short-time windows 310, 320, and 330 are three consecutive short-time windows, wherein short-time window 310 includes two types of phases, A and B, short-time window 320 includes two types of phases, A and C, and short-time window 330 includes three types of phases, B, C, and D. Then A can be regarded as the same phase existing in short-time windows 310 and 320, and C can be regarded as the same phase existing in short-time windows 320 and 330.

[0073] In some embodiments, the calculation process of the aforementioned merging probability can be expressed as follows:

[0074]

[0075] Among them, r represents the rth short-term window, Represents the probability of merging the rth short-term window with its next adjacent short-term window (r+1); Represents the number of identical phases between the rth short-time window and its next adjacent short-time window; Represents the maximum number of phases between the rth short-time window and its next adjacent short-time window (i.e., the larger value of the number of phases between the rth short-time window and the r+1th short-time window). The purpose is to Normalized, the larger the value, the closer the phases of adjacent short-time windows are, and the more likely they are in the same continuous sound signal.

[0076] In the embodiment of the present invention, all short-time windows in the mixed sound signal are only determined to be merged with the short-time windows adjacent to the left and right thereof. Exemplarily, in some embodiments, a merging probability threshold (for example, it can be set to 0.5) can be set, and when the merging probability of the r-th short-time window and its next adjacent short-time window (r+1) is greater than the preset merging probability threshold, they are merged, and the result obtained after the merging is used as a sub-cluster.

[0077] In an embodiment of the present invention, a hierarchical clustering algorithm may be used for slicing and merging the time series data of the mixed sound signal. Exemplarily, in some embodiments, agglomerative hierarchical clustering may be performed on the multiple subclusters obtained by the aforementioned merging process, and the clustering is terminated when the preset termination function reaches a maximum value to obtain one or more target clusters, which are one or more sound source intervals in the mixed sound signal received by the microphone.

[0078] It should be noted that agglomerative hierarchical clustering is a hierarchical clustering method that constructs a tree structure (or dendrogram) by connecting similar objects to each other. In this process, the two most similar objects are merged into a cluster, and then in the next layer, the newly formed cluster is merged with other objects or clusters, and so on recursively until all objects are merged into the same cluster. Agglomerative hierarchical clustering adopts a bottom-up strategy to construct a clustering hierarchy. For the embodiment of the present invention, the process first merges each short-time window with continuous adjacent short-time windows as a sub-cluster (in other words, each cluster participating in the merger in the process is a continuous short-time window merged in time sequence), and then merges these sub-clusters into larger and larger clusters until the termination condition of the merger process is met.

[0079] In an embodiment of the present invention, the termination condition of the merging process can be represented by a termination function. Exemplarily, in some embodiments, the termination function can be represented as follows:

[0080]

[0081] Among them, E represents the output value of the termination function; v represents the vth sub-cluster in the clustering process, and all short-time windows of the mixed sound signal are merged into M sub-clusters in time sequence; r represents the rth short-time window, Represents the number of short-term windows of the vth subcluster; Represents the probability of merging the rth short-time window of the vth sub-cluster with its next adjacent short-time window; Represents the average value of the merging probability of any adjacent short-time windows within all sub-clusters. The larger the value, the better the current classification result.

[0082] represents the back-end short-term window of the vth sub-cluster (i.e., the last short-term window in the vth sub-cluster), Represents the front short-term window of the v+1th sub-cluster (i.e., the first short-term window in the v+1th sub-cluster), and the two are adjacent; represents the probability of merging the back-end short-time window of the v-th sub-cluster and the front-end short-time window of the v+1-th sub-cluster. Subtracting the constant 1 from it represents the probability of not merging. Represents the average value of the non-merging probability of the front and back short-term windows of all any two adjacent sub-clusters in time series. The larger the value, the greater the probability of non-merging of adjacent sub-clusters and the greater the difference between clusters.

[0083] It is a penalty term, which represents the normalized value of the number of hierarchical clustering merged into each layer (i.e., the current number of clustering layers) divided by the preset maximum number of layers, and is used to prevent the function from converging prematurely.

[0084] Specifically, in some embodiments of the present invention, the mean of the merging probabilities of all two adjacent short-time windows in each sub-cluster can be calculated to obtain the first parameter ( ), and according to the mean of the first parameter corresponding to all sub-clusters, the second parameter ( At the same time, the probability of non-merging the front-end short-time window and the back-end short-time window in the adjacent sub-clusters is calculated ( ), and the third parameter ( Finally, the penalty item corresponding to the second parameter, the third parameter and the current clustering level ( ) to obtain the termination function.

[0085] In an embodiment of the present invention, the clustering result of a layer corresponding to the maximum output value E of the above-mentioned termination function is the final clustering result of the agglomerative hierarchical clustering. Through the above-mentioned agglomerative hierarchical clustering process, one or more target clusters can be obtained. Since only adjacent windows are merged in the embodiment of the present invention, each target cluster is an independent signal segment in time sequence, that is, each target cluster is an independent sound source interval. For the mixed sound signal of each microphone, one or more independent sound source intervals can be obtained respectively through the above processing.

[0086] S130, performing ICA decomposition on each sound source interval to extract the main sound source component signal of each sound source interval. In some embodiments, S130 may be executed by the main sound source component signal extraction module 230 mentioned below.

[0087] In each independent sound source interval of the mixed sound signal, there may be other surrounding noises, but at least one sound source signal component is continuous. In an embodiment of the present invention, in order to avoid the problem of sound source signal distortion when performing global decomposition on the mixed sound signal, after obtaining one or more sound source intervals corresponding to each microphone through the above steps, ICA (Independent Component Analysis) decomposition can be performed on each sound source interval.

[0088] Specifically, in some embodiments, the number of decompositions of each sound source interval can be set to k. Since the embodiment of the present invention is only for separating the main sound source signal in the mixed sound signal of each microphone, all sounds can be roughly classified into three categories: main sound source, secondary sound source, and environmental sound source, so k=3 is set. In some embodiments, the ICA iterative method can use a gradient descent method. It should be noted that ICA independent component decomposition is one of the commonly used methods for sound source separation problems, which can be regarded as a well-known technology and will not be described in detail in this specification.

[0089] After ICA decomposition, each microphone in the microphone array can obtain k (k=3 in the embodiment of the present invention) types of sound source component signals. In some embodiments of the present invention, according to the average amplitude of the k types of sound source component signals (which can be obtained by the aforementioned amplitude spectrum), a type of sound source component signal with the largest average amplitude can be screened out as the main sound source component signal of each sound source interval.

[0090] S140, concatenating the main sound source component signals of all the sound source intervals corresponding to each microphone to obtain the main sound source signal of each microphone. In some embodiments, S140 may be executed by the main sound source signal determination module 240 mentioned below.

[0091] In some embodiments of the present invention, the main sound source signal determination module 240 can splice the main sound source component signals of all sound source intervals corresponding to each microphone to obtain the main sound source signal of each microphone. Specifically, the main sound source signal determination module 240 can average the intersection points of the main sound source component signals in all two adjacent sound source intervals to obtain the main sound source signal of each microphone. Among them, the intersection points of the main sound source component signals in two adjacent sound source intervals can be understood as the common components of the main sound source component signals in the two adjacent sound source intervals. By averaging the intersection points of the main sound source component signals in all two adjacent sound source intervals, the main sound source component signals in all independent sound source intervals of each microphone can be spliced ​​into a complete main sound source signal.

[0092] S150, calculating the main sound source probability corresponding to each sound source interval in the mixed sound signal received by each microphone according to the main sound source signal. In some embodiments, S150 may be executed by the main sound source probability calculation module 250 mentioned below.

[0093] Through the above steps, the main sound source component signal in each microphone can be obtained. Further, the main sound source probability calculation module 250 can calculate the main sound source probability corresponding to each sound source interval in the mixed sound signal received by each microphone according to the main sound source signal.

[0094] Specifically, in some embodiments, it can be assumed that each microphone may be a direct receiving end of the main sound source, and each microphone takes turns to serve as a target microphone. When the target microphone is the receiving end of the main sound source, the main sound source signal received by other microphones will be relatively weaker than that of the target microphone. Therefore, each target microphone can be sorted in order from near to far according to the distance between the target microphone and the other microphones in the microphone array to obtain a distance sequence corresponding to each target microphone.

[0095] The main sound source probability calculation module 250 can reacquire each sound source interval in the main sound source signal of the target microphone, and according to the distance sequence corresponding to the target microphone, along the order of the distance from the target microphone to other microphones from near to far, sequentially calculate the phase error and amplitude error of each sound source interval of the main sound source signal of the target microphone and the signal segment corresponding to the same time interval in the main sound source signal of other microphones; and then normalize the phase error and amplitude error. It should be noted that in the embodiment of the present invention, the phase value and the phase error value are in complex form, so they are normalized using trigonometric functions.

[0096] Furthermore, the main sound source probability calculation module 250 can calculate the normalized (linear normalization can be used but is not limited to) Euclidean norm of the phase error and amplitude error of each sound source interval of the main sound source signal of the target microphone and the signal segment corresponding to the same time interval in the main sound source signals of other microphones, which is recorded as (in, represents the vth sound source interval of the zth target microphone, and s represents the sth other microphone).

[0097] At the same time, the main sound source probability calculation module 250 can also calculate the signal-to-noise ratio of each sound source interval of the main sound source signal of the target microphone and the signal segment corresponding to the same time interval in the main sound source signals of other microphones according to the distance sequence between the target microphone and other microphones (that is, the distance sequence corresponding to the target microphone), and perform normalization (linear normalization can be used but is not limited to) to obtain a first normalization result, which is recorded as (in, represents the vth sound source interval of the zth target microphone, and s represents the sth other microphone).

[0098] It can be understood that since there is characteristic attenuation and delay between each sound source component signal and the sound source component signals of other microphones, accompanied by a decrease in the signal-to-noise ratio, if the main sound source signal of the target microphone has the aforementioned characteristics relative to the main sound source signals of other microphones in its distance sequence, the more obvious the characteristics are, the greater the probability that the main sound source signal received by the target microphone is the main sound source.

[0099] Based on this, in some embodiments, the Euclidean norms corresponding to the target microphone and all other microphones can be calculated along the distance sequence of the target microphone. Perform continuous derivation to obtain the first derivative sequence, which is recorded as The first derivative sequence represents that the sound source signal presented by the target microphone continuously attenuates as the distance increases compared to other microphones.

[0100] Similarly, in the embodiment of the present invention, the first normalized results corresponding to the target microphone and all other microphones may be calculated along the distance sequence of the target microphone. Perform continuous derivatives (the faster the signal-to-noise ratio decreases, the smaller the derivative value), and obtain the second derivative sequence, which is recorded as The second derivative sequence represents the continuous distortion of the sound source signal presented by the target microphone compared to other microphones as the distance increases.

[0101] When the first derivative sequence and the second derivative sequence have a negative correlation, it means that the propagation relationship between the main sound source and the secondary sound source is more consistent. Therefore, in some embodiments, the main sound source probability of each sound source interval of the mixed sound signal received by each microphone can be calculated as follows:

[0102]

[0103] Where z represents the zth target microphone; represents the probability of the main sound source corresponding to the vth sound source interval of the zth target microphone, represents the aforementioned first derivative sequence, represents the aforementioned second derivative sequence; Represents the Pearson correlation coefficient between the first derivative sequence and the second derivative sequence.

[0104] It should be noted that in the above formula, The closer the value is to -1, the more the sound source signal from the target microphone to other microphones is continuously attenuated, and it is negatively correlated with the continuous distortion of the sound source from the target microphone to other microphones, that is, the more it conforms to the propagation characteristics between the primary and secondary sound sources, the better. The smaller the value between 0 and 1, The larger the value between 0 and 1, the more likely it is that the vth sound source interval of the zth target microphone is the main sound source. The bigger.

[0105] In short, in some embodiments of the present invention, the main sound source probability calculation module 250 can obtain the normalized Euclidean norm of the phase error and amplitude error of the target sound source interval of the main sound source signal in the target microphone and the signal segment corresponding to the same time interval in the main sound source signals of other microphones, and then continuously derive the Euclidean norm according to the distance order of the target microphone and the other microphones to obtain the first derivative sequence . It should be noted that, in an embodiment of the present invention, the aforementioned target microphone can be understood as the object targeted by the current calculation, which can refer to any microphone in the microphone array, and the specific object referred to can be determined based on the current calculation process. The target sound source interval can refer to any sound source interval contained in the main sound source signal in the target microphone. In some embodiments, the main sound source signal in the target microphone may include multiple sound source intervals, each of which can be used as a target sound source interval and the signal segment corresponding to the same time interval in the main sound source signal of other microphones to calculate the phase error and amplitude error, and obtain the corresponding Euclidean norm through normalization processing.

[0106] At the same time, the main sound source probability calculation module 250 can also calculate the signal-to-noise ratio of the target sound source interval of the main sound source signal in the target microphone and the signal segment corresponding to the same time interval in the main sound source signals of other microphones according to the distance sequence between the target microphone and other microphones (that is, the distance sequence corresponding to the target microphone), and perform normalization processing to obtain a first normalized result, and then continuously derive the first normalized result according to the distance sequence between the target microphone and other microphones to obtain a second derivative value sequence. Finally, the main sound source probability calculation module 250 can calculate the probability of the main sound source based on the Pearson correlation coefficient ( ), obtain the main sound source probability corresponding to the target sound source interval in the mixed sound signal received by the target microphone.

[0107] S160, calculating the confusion coefficients of the primary and secondary sound sources at each moment based on the primary sound source probability. In some embodiments, S160 may be executed by the confusion coefficient calculation module 260 mentioned below.

[0108] Through the above steps, the probability of the main sound source corresponding to each sound source interval of each microphone can be calculated. It can be understood that the main sound source is not fixed. As time changes, the direction of the main sound source may change randomly. Therefore, it is necessary to calculate the confusion coefficient of the main sound source and the secondary sound source based on the main sound source probability, so as to correct the signal strength based on this and improve the frequency contrast of the main and secondary sound sources.

[0109] Specifically, since a single sound source must have at least one microphone in the microphone array that receives the strongest sound source signal, and the sound source signals of other microphones are relatively weakened with the distance. Therefore, in an embodiment of the present invention, according to the main sound source probability corresponding to the sound source interval that matches the target moment (i.e., the moment currently calculated) in the mixed sound signal received by all microphones, the microphone with the highest main sound source probability can be selected as the main sound source receiving end of the sound source signal at the target moment, and the remaining microphones can be used as the secondary sound source receiving ends of the sound source signal at the target moment. Furthermore, the confusion coefficient calculation module 260 can calculate the confusion coefficients of the main and secondary sound sources at each moment based on the following formula:

[0110]

[0111] Among them, i represents the i-th target moment, represents the confusion coefficient of the primary and secondary sound sources corresponding to the i-th target moment; q represents the main sound source receiving end at the i-th target moment, f represents the other f-th sound source receiving end (that is, the f-th microphone except the main sound source receiving end), and T represents the total number of all microphones. represents the probability of the main sound source in the sound source interval (for example, the aforementioned v-th sound source interval) to which the sound source signal of the main sound source receiving end q at the i-th target moment belongs, Represents the probability of the main sound source in the sound source interval (for example, the aforementioned v-th sound source interval) to which the sound source signal of the f-th secondary sound source receiving end at the i-th target moment belongs.

[0112] In the above formula, Represents the variance of the main sound source probability of the sound source signal received by the main sound source at the i-th target moment and the main sound source probability in all other secondary sound source receiving ends; the larger the value, the smaller the confusion between the main and secondary sound sources at the current moment, and the clearer the main sound source; conversely, the smaller the value, the greater the confusion between the main and secondary sound sources at the current moment, and the more blurred the main sound source.

[0113] In other words, in the embodiment of the present invention, the confusion coefficient calculation module 260 can calculate the variance of the main sound source probability of the sound source signal of the main sound source receiving end at the target time and the main sound source probability of all other secondary sound source receiving ends to obtain the fourth parameter ( ), and then obtain the confusion coefficients of the primary and secondary sound sources at the target time based on the fourth parameter. Specifically, in some embodiments, the confusion coefficient calculation module 260 may subtract the constant 1 from the fourth parameter, and then use the difference between the constant 1 and the fourth parameter as the confusion coefficients of the primary and secondary sound sources corresponding to the target time.

[0114] S170, adjusting the strength of the sound source signal at each moment according to the confusion coefficient to achieve microphone calibration. In some embodiments, S170 may be performed by the calibration module 270 mentioned below.

[0115] After obtaining the confusion coefficients of the primary and secondary sound sources corresponding to each moment, the calibration module 270 can use the confusion coefficients of the primary and secondary sound sources corresponding to each moment to reduce the signal strength of all secondary sound sources at each moment to improve the clarity of the primary sound source at that moment.

[0116] In some embodiments, the calibration module 270 may obtain a signal strength adjustment coefficient based on the confusion coefficients of the primary and secondary sound sources corresponding to the target time, and then adjust the amplitudes of the sound source signals received by all secondary sound source receiving ends based on the signal strength adjustment coefficient to suppress the secondary sound source signals at the target time.

[0117] The process of adjusting the signal strength of the sound source signal received by the infrasound source receiving end by the calibration module 270 can be expressed as follows:

[0118]

[0119] Among them, i represents the i-th target moment, represents the amplitude of the sound signal at the f-th sound source receiving end after adjustment at the i-th target time, represents the amplitude of the original sound signal at the receiving end of the f-th subsound source at the i-th target time; Represents the confusion coefficient of the primary and secondary sound sources corresponding to the i-th target moment, Indicates the signal strength adjustment factor.

[0120] From the above formula, we can see that when the confusion coefficient When the value is larger, the signal amplitude of the infrasound source receiving end at the current moment will be adjusted smaller, so according to the confusion coefficient The secondary sound source signal at each moment is adaptively suppressed, making the main sound source signal received at each moment clearer.

[0121] After adaptively suppressing the sound source signals received by all the sub-sound source receiving ends through the above steps, a spatial filter can be used to combine all the sound signals received by all the microphones in the microphone array (the sound signals are the signals processed through the above steps) into one sound signal. This process is the beamforming process of the microphone array, which is a well-known technology and will not be described in detail in this specification.

[0122] It should be pointed out that in an embodiment of the present invention, the microphone of the multimodal smart terminal is calibrated by the above method, so that the main sound source signal at each moment can be located, and the secondary sound source signal at each moment can be adaptively suppressed, so that the final merged sound beam is clearer and more accurate, thereby improving the microphone calibration effect of the smart terminal device.

[0123] Figure 2 An exemplary module diagram of a microphone calibration system for a multi-modal intelligent terminal provided by an embodiment of the present invention. Figure 2 In some embodiments, the system may include an acquisition module 210, a sound source interval determination module 220, a main sound source component signal extraction module 230, a main sound source signal determination module 240, a main sound source probability calculation module 250, a confusion coefficient calculation module 260, and a calibration module 270. Among them:

[0124] The acquisition module 210 may be used to acquire a mixed sound signal received by each microphone in the microphone array.

[0125] The sound source interval determination module 220 can be used to segment the mixed sound signal received by each microphone using the frequency domain characteristics of the short-time window signal segment to obtain multiple sound source intervals corresponding to each microphone.

[0126] The main sound source component signal extraction module 230 may be used to perform ICA decomposition on each sound source interval to extract the main sound source component signal of each sound source interval.

[0127] The main sound source signal determination module 240 may be used to concatenate the main sound source component signals of all sound source intervals corresponding to each microphone to obtain the main sound source signal of each microphone.

[0128] The main sound source probability calculation module 250 may be configured to calculate the main sound source probability corresponding to each sound source interval in the mixed sound signal received by each microphone according to the main sound source signal.

[0129] The confusion coefficient calculation module 260 may be configured to calculate the confusion coefficients of the primary and secondary sound sources at each moment based on the primary sound source probability.

[0130] The calibration module 270 may be configured to adjust the strength of the sound source signal at each moment according to the confusion coefficient to implement microphone calibration.

[0131] about Figure 2 For more detailed description of the microphone calibration system of the multi-modal intelligent terminal shown in FIG. Figure 1 The relevant parts will not be elaborated here.

[0132] It should be understood that Figure 2 The microphone calibration system and its modules of the multimodal intelligent terminal shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art can understand that the above-mentioned method and system can be implemented using computer executable instructions and / or included in a processor control code, for example, such as a carrier medium such as a disk, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium such as a disk, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules of this specification can not only be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software (for example, firmware).

[0133] For more details about the above modules, please refer to other places in this manual (for example Figure 1 part and its related description), which will not be repeated here.

[0134] In summary, the beneficial effects that may be brought about by the present invention include but are not limited to:

[0135] In the microphone calibration method and system for a multimodal intelligent terminal provided in some embodiments of the present invention, the mixed sound signal received by each microphone is segmented by utilizing the frequency domain characteristics of the short-time window signal segment to obtain multiple sound source intervals corresponding to each microphone, and then ICA decomposition is performed on each sound source interval separately to extract the main sound source component signal, and the main sound source component signals of all sound source intervals corresponding to each microphone are spliced ​​to obtain the main sound source signal, and then the main sound source probability corresponding to each sound source interval is calculated according to the propagation attenuation characteristics of the main sound source component, and then the main and secondary sound source confusion coefficients at each moment are calculated based on the main sound source probability of each sound source interval, and finally the sound source signal strength at each moment is adaptively adjusted based on the confusion coefficient (mainly suppressing the secondary sound source signal), so that the main sound source component in the final merged sound beam can be clearer and more accurate.

[0136] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or may be any other beneficial effects that may be obtained.

[0137] It should also be noted that the order of the above embodiments of the present invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0138] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A microphone calibration method for a multimodal intelligent terminal, characterized in that: The method comprises: Obtaining a mixed sound signal received by each microphone in the microphone array; The mixed sound signal received by each microphone is segmented using the frequency domain characteristics of the short-time window signal segment to obtain multiple sound source intervals corresponding to each microphone; Perform ICA decomposition on each sound source interval to extract the main sound source component signal of each sound source interval; The main sound source component signals of all sound source intervals corresponding to each microphone are spliced ​​together to obtain the main sound source signal of each microphone; Calculating the main sound source probability corresponding to each sound source interval in the mixed sound signal received by each microphone according to the main sound source signal; Calculate the confusion coefficient of the primary and secondary sound sources at each moment based on the primary sound source probability; Adjusting the intensity of the sound source signal at each moment according to the confusion coefficient to achieve microphone calibration; The calculating, based on the main sound source signal, the main sound source probability corresponding to each sound source interval in the mixed sound signal received by each microphone includes: Obtain the normalized Euclidean norm of the phase error and amplitude error of the target sound source interval of the main sound source signal in the target microphone and the signal segments corresponding to the same time interval in the main sound source signals of other microphones; Continuously deriving the Euclidean norm according to the order of distances between the target microphone and other microphones to obtain a first derivative value sequence; sequentially calculating the signal-to-noise ratios of the target sound source interval of the main sound source signal in the target microphone and the signal segments corresponding to the same time intervals in the main sound source signals of other microphones, and performing normalization processing to obtain a first normalization result; Continuously deriving the first normalized result according to the order of distances between the target microphone and other microphones to obtain a second derivative value sequence; Obtaining a main sound source probability corresponding to a target sound source interval in a mixed sound signal received by the target microphone based on a Pearson correlation coefficient between the first derivative sequence and the second derivative sequence; Calculating the confusion coefficient of the primary and secondary sound sources at each moment based on the primary sound source probability includes: According to the main sound source probability corresponding to the sound source interval matching the target time in the mixed sound signals received by all microphones, the microphone with the highest main sound source probability is selected as the main sound source receiving end of the sound source signal at the target time, and the remaining microphones are used as secondary sound source receiving ends of the sound source signal at the target time; Calculating the variance of the main sound source probability of the sound source signal at the main sound source receiving end at the target time and the main sound source probabilities at all other secondary sound source receiving ends to obtain a fourth parameter; The confusion coefficients of the primary and secondary sound sources at the target moment are obtained based on the fourth parameter.

2. The microphone calibration method for a multimodal intelligent terminal according to claim 1, wherein: The method of segmenting the mixed sound signal received by each microphone using the frequency domain characteristics of the short-time window signal segment to obtain multiple sound source intervals corresponding to each microphone includes: The short-time window length is set according to the sound sampling frequency, and the mixed sound signal received by each microphone is subjected to short-time Fourier transform to obtain the phase spectrum within each short-time window signal segment; Merging consecutive short-time windows according to the phase spectrum to obtain one or more target clusters; Each of the target clusters is regarded as a sound source interval corresponding to the microphone.

3. The microphone calibration method for a multimodal intelligent terminal according to claim 2, wherein: The merging of continuous short-time windows according to the phase spectrum to obtain one or more target clusters includes: Calculate the probability of merging each short-time window with the adjacent window based on the number of identical phases between each short-time window and the next adjacent window and the maximum number of phases between the two; Merging the short-time windows according to the merging probability to obtain multiple sub-clusters; Performing agglomerative hierarchical clustering on the multiple subclusters according to a preset termination function to obtain one or more target clusters.

4. The microphone calibration method for a multimodal intelligent terminal according to claim 3, wherein: The preset termination function is obtained based on the following method: Calculate the mean of the merging probabilities of all two adjacent short-time windows in each sub-cluster to obtain the first parameter, and obtain the second parameter based on the mean of the first parameters corresponding to all sub-clusters; Calculate the non-merging probability of the front-end short-time window and the back-end short-time window in adjacent sub-clusters, and obtain the third parameter based on the average of the non-merging probabilities corresponding to all adjacent sub-clusters; The second parameter, the third parameter and the penalty term corresponding to the current clustering level are added together to obtain the termination function.

5. The microphone calibration method for a multimodal intelligent terminal according to claim 2, wherein: The ICA decomposition is performed on each sound source interval to extract the main sound source component signal of each sound source interval, including: Perform ICA signal decomposition on each sound source interval of the mixed sound signal received by each microphone to obtain k types of sound source component signals; According to the average amplitudes of the k types of sound source component signals, a type of sound source component signal with the largest average amplitude is screened out as the main sound source component signal of each sound source interval.

6. The microphone calibration method for a multimodal intelligent terminal according to claim 1, wherein: The main sound source component signals of all sound source intervals corresponding to each microphone are spliced ​​together to obtain the main sound source signal of each microphone, including: The intersection points of the main sound source component signals in all two adjacent sound source intervals are averaged to obtain the main sound source signal of each microphone.

7. The microphone calibration method for a multimodal intelligent terminal according to claim 1, wherein: The step of adjusting the intensity of the sound source signal at each moment according to the confusion coefficient to achieve microphone calibration includes: According to the confusion coefficients of the primary and secondary sound sources corresponding to the target time, a signal strength adjustment coefficient is obtained; The amplitude of the sound source signal received by the infrasound source receiving end is adjusted according to the signal strength adjustment coefficient to suppress the infrasound source signal at the target moment.

8. A microphone calibration system for a multimodal intelligent terminal, the system being used to implement the microphone calibration method for a multimodal intelligent terminal according to any one of claims 1 to 7, characterized in that: The system comprises: An acquisition module, used to acquire a mixed sound signal received by each microphone in the microphone array; A sound source interval determination module is used to segment the mixed sound signal received by each microphone using the frequency domain characteristics of the short-time window signal segment to obtain multiple sound source intervals corresponding to each microphone; The main sound source component signal extraction module is used to perform ICA decomposition on each sound source interval to extract the main sound source component signal of each sound source interval; A main sound source signal determination module is used to splice the main sound source component signals of all sound source intervals corresponding to each microphone to obtain the main sound source signal of each microphone; A main sound source probability calculation module is used to calculate the main sound source probability corresponding to each sound source interval in the mixed sound signal received by each microphone according to the main sound source signal; A confusion coefficient calculation module, configured to calculate the confusion coefficients of the primary and secondary sound sources at each moment based on the primary sound source probability; The calibration module is used to adjust the sound source signal strength at each moment according to the confusion coefficient to achieve microphone calibration.