A far-field sound pickup voice recognition method and device of a multi-modal intelligent terminal

Abstract

The application relates to the technical field of speech recognition, in particular to a far-field sound pickup speech recognition method and equipment of a multi-modal intelligent terminal, which comprises the following steps: acquiring speech signals of each microphone in a microphone array, equally dividing a time range where the speech signals are located into a plurality of time periods, acquiring an interference degree of far-field environment noise on sound pickup in each time period, merging the time periods to obtain a plurality of updated time periods, acquiring a fusion coefficient of a speech signal segment of each microphone in each updated time period, thereby acquiring a sound pickup attention degree of each microphone, determining a weight of each microphone, weighting and superimposing speech signals of all the microphones to obtain an enhanced speech signal, and then converting the enhanced speech signal into text content. The application improves the speech signal enhancement effect after the speech signals of all the microphones are weighted and superimposed by adaptively adjusting the weight of each microphone in the microphone array, thereby guaranteeing the accuracy of far-field sound pickup speech recognition.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, specifically to a far-field voice recognition method and device for a multimodal smart terminal. Background Technology

[0002] The core of far-field voice recognition in multimodal intelligent terminals is solving the problem of clear voice acquisition and accurate recognition under conditions of "long distance and strong interference," which is a fundamental technology for realizing natural interaction on the terminal. Its technical implementation mainly relies on the synergy of hardware voice pickup solutions and software algorithm optimization. The ultimate goal is to enable the terminal to accurately capture and understand commands, much like the human ear, at distances of 3 to 5 meters or even further. Hardware voice pickup enhances sound pickup capabilities through a multi-microphone array, while the software algorithm is based on beamforming to acquire voice signals from all microphones, weighted and superimposed to create an enhanced voice signal. Finally, automatic speech recognition (ASR) is used to convert the enhanced voice signal into text.

[0003] The existing problem: Beamforming assigns different weights to the speech signals collected by different microphones in a multi-microphone array, thereby amplifying the sound from the target direction and suppressing the sound from the interfering direction. However, due to the complex and random changes in environmental noise, the impact on the sound pickup of different microphones varies at different times. Furthermore, users may move around the terminal, causing the direction of the sound source to change over time. This can easily lead to unreasonable weighting in the multi-microphone array, resulting in poor quality output speech signals and potentially inaccurate speech recognition results. Summary of the Invention

[0004] This invention provides a far-field voice recognition method and device for multimodal smart terminals to solve existing problems.

[0005] The far-field voice recognition method and device for a multimodal smart terminal of the present invention adopts the following technical solution:

[0006] One embodiment of the present invention provides a far-field voice recognition method and device for a multimodal smart terminal, the method comprising the following steps:

[0007] Acquire the speech signal from each microphone in the microphone array;

[0008] The time range of the speech signal is divided into several time periods; based on the positive correlation between the speech signal segments of different microphones in each time period, the degree of interference of far-field environmental noise on the sound pickup in each time period is obtained; based on the similarity of the degree of interference of far-field environmental noise on the sound pickup in different time periods, the time periods are merged to obtain several updated time periods.

[0009] Based on the energy distribution of the speech signal segment of each microphone in the spectrum during each update period, and the energy comparison results between the speech signal segments of different microphones, combined with the degree of interference of far-field environmental noise on the sound pickup during each period, the fusion coefficient of the speech signal segment of each microphone during each update period is obtained; based on the fusion coefficient of the speech signal segment of each microphone during all update periods, and the duration of each update period, the sound pickup attention of each microphone is obtained.

[0010] Based on the pickup attention of each microphone, the weight of each microphone is determined, and the speech signals of all microphones in the microphone array are weighted and superimposed to obtain an enhanced speech signal; the enhanced speech signal is then converted into text content.

[0011] Furthermore, the specific steps for obtaining the degree of interference of far-field environmental noise on sound pickup in each time period are as follows:

[0012] Within each time period, the inversely proportional normalized value of the mean Pearson correlation coefficient between any two microphone speech signal segments is used as the degree of interference of far-field environmental noise on sound pickup within each time period.

[0013] Furthermore, the specific steps involved in obtaining several update time periods are as follows:

[0014] Based on the time sequence, the interference level sequence is constructed by the degree of interference of far-field environmental noise on the sound pickup in all time periods. The DBSCAN clustering algorithm is used to divide the interference level sequence into several sequence segments, and all time periods corresponding to each sequence segment are merged into each updated time period.

[0015] Furthermore, the specific steps for obtaining the fusion coefficients of the speech signal segments of each microphone within each update time period are as follows:

[0016] The speech signal segment from any microphone within any update time period is used as the reference speech signal segment;

[0017] Obtain the energy value and fundamental frequency of the reference speech signal segment at different frequencies;

[0018] Based on the energy values ​​of the reference speech signal segment at different frequencies and the difference between the frequency and the fundamental frequency, the possibility of the reference speech signal segment being free from interference is obtained.

[0019] The reference speech signal segment is decomposed into several speech components;

[0020] The interference-free probability of each speech component is obtained according to the method for obtaining the interference-free probability of the reference speech signal segment;

[0021] The speech component with the highest probability of being free from interference is denoted as the user speech component.

[0022] Obtain the energy value of the user's speech component at different frequencies, and record the sum of the energy values ​​of the user's speech component at all frequencies as the total energy value of the user's speech component;

[0023] Within each update period, the user speech pickup effect of each microphone's speech signal segment is obtained based on the comparison results between the total energy values ​​of the user speech components of all microphone speech signal segments within each update period.

[0024] Based on the user voice pickup effect and interference-free possibility of each microphone's voice signal segment in each update period, and combined with the degree of interference of far-field environmental noise on the pickup in each period, the fusion coefficient of the voice signal segment of each microphone in each update period is obtained.

[0025] Furthermore, the specific steps for obtaining the interference-free possibility of the reference speech signal segment are as follows:

[0026] The sum of the energy values ​​at all frequencies is recorded as the total spectral energy of the reference speech signal segment.

[0027] Using the absolute value of the difference between any two frequencies as the clustering distance, the DBSCAN clustering algorithm is used to cluster all frequencies to obtain several clusters.

[0028] The sum of the energy values ​​at all frequencies in each cluster is obtained as the total energy of each cluster;

[0029] The ratio of the maximum value of the total energy of all clusters to the total spectral energy of the reference speech signal segment is denoted as the energy concentration of the reference speech signal segment.

[0030] Obtain the remainder of each frequency divided by the base frequency, and denote it as the non-integer multiple probability of each frequency;

[0031] The frequency whose normalized value of a non-integer multiple probability is less than or equal to a preset harmonic threshold is denoted as the target frequency.

[0032] The ratio of the number of all target frequencies to the total number of frequencies is denoted as the first ratio. The ratio of the sum of the energy values ​​at all target frequencies to the total spectral energy of the reference speech signal segment is denoted as the second ratio. The mean of the first ratio and the second ratio is denoted as the harmonic structure significance of the reference speech signal segment.

[0033] The average of the energy concentration and harmonic structure significance of the reference speech signal segment is denoted as the interference-free probability of the reference speech signal segment.

[0034] Furthermore, the specific steps for obtaining the user voice pickup effect of the voice signal segment of each microphone within each update time period are as follows:

[0035] Within each update period, the maximum value of the total energy of the user speech components in the speech signal segments of all microphones is obtained and recorded as the baseline energy value. The ratio of the total energy value of the user speech components in the speech signal segments of each microphone to the baseline energy value is recorded as the user speech pickup effect of the speech signal segments of each microphone within each update period.

[0036] Furthermore, the specific steps for obtaining the fusion coefficient of the speech signal segment of each microphone in each update period, based on the user speech pickup effect and the possibility of no interference for each microphone's speech signal segment in each update period, combined with the degree of interference of far-field environmental noise on the pickup in each period, are as follows:

[0037] The average value of the interference of far-field environmental noise on sound pickup in all time periods within each update period is recorded as the interference of far-field environmental noise on sound pickup in each update period.

[0038] The product of the interference level of far-field environmental noise on sound pickup in each update period and the probability of no interference for the speech signal segment of each microphone in each update period is recorded as the first product. The product of the inverse proportional value of the interference level of far-field environmental noise on sound pickup in each update period and the user speech pickup effect of the speech signal segment of each microphone in each update period is recorded as the second product. The sum of the first product and the second product is recorded as the fusion coefficient of the speech signal segment of each microphone in each update period.

[0039] Furthermore, the specific steps for obtaining the pickup attention of each microphone are as follows:

[0040] The ratio of the duration of each update period to the duration of the speech signal is used as the weight of each update period. The fusion coefficients of the speech signal segments of each microphone in all update periods are weighted and summed to obtain the pickup attention of each microphone.

[0041] Furthermore, the specific steps for determining the weight of each microphone are as follows:

[0042] The weight of each microphone is defined as the ratio of its pickup attention to the sum of the pickup attention of all microphones.

[0043] The present invention also proposes a far-field voice recognition device for a multimodal intelligent terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program stored in the memory to implement the steps of the aforementioned far-field voice recognition method for a multimodal intelligent terminal.

[0044] The beneficial effects of the technical solution of the present invention are:

[0045] In this embodiment of the invention, the speech signal of each microphone in the microphone array is acquired, and the time range of the speech signal is divided into several time periods. Based on the positive correlation between the speech signal segments of different microphones in each time period, the interference degree of far-field environmental noise on the sound pickup in each time period is obtained, and the time periods are merged to obtain several updated time periods. Thus, the updated time periods are divided according to the interference degree of far-field environmental noise on the sound pickup, thereby ensuring that noise suppression is prioritized when there is high noise interference, and user speech enhancement is prioritized when there is low noise interference, thus ensuring the speech signal enhancement effect. Based on the energy distribution of the speech signal segment of each microphone in the spectrum in each updated time period, and the energy comparison results between the speech signal segments of different microphones, the sound pickup attention of each microphone is obtained, which is used to determine the weight of each microphone. Thus, by analyzing the sound pickup effect of each microphone in different updated time periods, the sound pickup attention of each microphone is determined, thereby assigning greater weight to microphones with better sound pickup effects to ensure the speech signal enhancement effect. Finally, the speech signals of all microphones in the microphone array are weighted and superimposed to obtain the enhanced speech signal, which is then converted into text content. Thus, this invention improves the speech signal enhancement effect after weighted superposition of speech signals from all microphones by adaptively adjusting the weight of each microphone in the microphone array, thereby ensuring the accuracy of far-field voice recognition. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the steps of a far-field voice recognition method for a multimodal smart terminal according to the present invention.

[0048] Figure 2 This is a schematic diagram of a 6-microphone array configuration;

[0049] Figure 3 A schematic diagram showing how a 6-microphone array forms 6 beams. Detailed Implementation

[0050] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a far-field voice recognition method and device for a multimodal intelligent terminal according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0052] The following description, in conjunction with the accompanying drawings, details the specific solution of the far-field voice recognition method and device for a multimodal intelligent terminal provided by this invention.

[0053] Please see Figure 1 The diagram illustrates a flowchart of a far-field voice recognition method for a multimodal smart terminal according to an embodiment of the present invention. The method includes the following steps:

[0054] Step S001: Acquire the voice signal from each microphone in the microphone array.

[0055] It should be noted that multimodal intelligent terminals typically include devices such as smart speakers, smart home control systems, in-vehicle voice systems, and conferencing systems. In this embodiment, the multimodal intelligent terminal uses a 6-microphone array, which is arranged in a circular layout. The 6 microphones are evenly distributed around the circumference and are equidistant from each other. A schematic diagram of the 6-microphone array configuration is shown below. Figure 2 As shown. The 6-microphone array forms 6 beams, each beam corresponding to... The recording range, a diagram showing the 6-microphone array forming 6 beams, as shown below. Figure 3 As shown, Figure 3 The middle section is a vertically downward-facing 6-microphone array Direction, then obtain in counter-clockwise order. direction, direction, direction, direction, The directions correspond to the beam 0 of the first microphone (MIC1), the beam 1 of the second microphone (MIC2), the beam 2 of the third microphone (MIC3), the beam 3 of the fourth microphone (MIC4), the beam 4 of the fifth microphone (MIC5), and the beam 5 of the sixth microphone (MIC6), respectively.

[0056] Therefore, the voice signals of distant users are collected through the microphone array of the multimodal smart terminal, acquiring the voice signals simultaneously collected by each microphone in the array. The horizontal axis represents time, and the vertical axis represents signal amplitude.

[0057] Step S002: Divide the time range of the speech signal into several time periods; based on the positive correlation between the speech signal segments of different microphones in each time period, obtain the degree of interference of far-field environmental noise on the sound pickup in each time period; based on the similarity of the degree of interference of far-field environmental noise on the sound pickup in different time periods, merge the time periods to obtain several updated time periods.

[0058] It should be noted that environmental noise interference occurs randomly. Therefore, the impact of interference on the microphone array varies at different times. When the background noise in the far-field environment (such as air conditioner noise, traffic noise, etc.) is low, the focus is on amplifying the sound in the user's direction to ensure clear and distinct voice signals. However, when the background noise in the far-field environment is high, the user's voice signal may be overwhelmed by the noise. In this case, the focus is on suppressing the sound in the direction of interference to reduce environmental noise interference. Therefore, in this embodiment, the time range of the voice signal is first divided into multiple time periods, and the magnitude of environmental noise interference in each time period is analyzed.

[0059] The preset number of equal divisions N is 30, which is used as an example for description. In other implementations, other values ​​can be set according to the duration of the collected voice signal.

[0060] Divide the time range of the speech signal from each microphone in the microphone array into N equal time periods.

[0061] Acquire the audio signal segment from the audio signal of each microphone within each time period.

[0062] Within each time period, obtain the Pearson correlation coefficient between any two microphone speech signal segments, and calculate the mean of the Pearson correlation coefficients between any two microphone speech signal segments in the microphone array. The inversely proportional normalized value is denoted as the degree of interference of far-field environmental noise on sound pickup in each time period.

[0063] It should be noted that the Pearson correlation coefficient is a well-known technique, and the specific method will not be described here. The Pearson correlation coefficient ranges from -1 to 1, therefore, in this embodiment, it is used as... As The inverse proportional normalized value. The closer the Pearson correlation coefficient is to 1, the stronger the positive linear correlation. Since different microphones in a microphone array are at different distances from the user, the attenuation of the speech signal with distance is an amplitude change caused by physical characteristics. However, the "core temporal structure" of the signal remains unchanged at different distances. A positive linear correlation measures the "synchronicity of the changing trends" of two speech signals. Theoretically, in the absence of far-field environmental noise interference, the "fluctuation pattern of the time-domain waveform" of the same speech signal at different distances is completely consistent, only the amplitude differs. Therefore, a significant positive correlation will be observed when calculating the Pearson correlation coefficient. However, the presence of far-field environmental noise interference will disrupt this positive correlation. For example, in... Figure 3 If the user is in In terms of direction, the noise interference source is in In terms of direction, beam 0 acquires the user's voice signal with the highest relative strength and the acquired noise interference voice signal with the lowest relative strength, while beam 3 acquires the user's voice signal with the lowest relative strength and the acquired noise interference voice signal with the highest relative strength. At this point, the noise interference voice signal with the lowest relative strength on beam 0 is superimposed on the user's voice signal with the highest relative strength, resulting in a relatively small change to the time-domain waveform of the user's voice signal on beam 0. Conversely, the noise interference voice signal with the highest relative strength on beam 3 is superimposed on the user's voice signal with the lowest relative strength, resulting in a relatively large change to the time-domain waveform of the user's voice signal on beam 3. This reduces the positive correlation between the voice signals acquired by beams 0 and 3. Therefore, the more noise interference sources there are in different directions within a given time period, and the greater the interference level of these sources, the more likely the problem will be. The smaller.

[0064] Following the chronological order, the interference level sequence is constructed based on the degree of interference of far-field environmental noise on sound pickup in all time periods. The DBSCAN clustering algorithm is used to divide the interference level sequence into several sequence segments. All time periods corresponding to each sequence segment are merged into each update time period. The mean value of the degree of interference of far-field environmental noise on sound pickup in all time periods within each update time period is recorded as the degree of interference of far-field environmental noise on sound pickup in each update time period.

[0065] Among them, the DBSCAN clustering algorithm (Density-Based Spatial Clustering of Applications with Noise) is a well-known technique, and its specific method will not be described here. The interference of far-field environmental noise on sound pickup is similar across all time periods within each segment of the sequence.

[0066] Step S003: Based on the energy distribution of the speech signal segment of each microphone in the spectrum during each update period, and the energy comparison results between the speech signal segments of different microphones, combined with the degree of interference of far-field environmental noise on the sound pickup during each period, obtain the fusion coefficient of the speech signal segment of each microphone during each update period; based on the fusion coefficient of the speech signal segment of each microphone during all update periods, and the duration of each update period, obtain the sound pickup attention of each microphone.

[0067] It should be noted that due to the location distribution of far-field environmental noise sources and the influence of noise interference intensity at different locations, the noise interference intensity experienced by different microphones in the microphone array at the same time will vary. Therefore, further analysis of the interference intensity experienced by the speech signal segment of each microphone in each update period is required. It is known that user speech has a distinct harmonic structure and energy concentration region in its spectrum, while the generation mechanism of environmental noise (such as wind noise, traffic noise, etc.) is irregular, and its spectral characteristics are completely opposite to those of speech. Therefore, superimposed environmental noise will disrupt this characteristic. Thus, in this embodiment, the interference intensity experienced by the speech signal segment of each microphone in each update period is determined by analyzing this spectral characteristic of the speech signal segment of each microphone in each update period.

[0068] Obtain the speech signal segment from the speech signal of any microphone within any update time period, and denote it as the reference speech signal segment.

[0069] Short-time Fourier transform is used to obtain the spectrum of the reference speech signal segment, which includes energy values ​​at several different frequencies. Then, cepstral method is used to obtain the fundamental frequency in the spectrum of the reference speech signal segment.

[0070] Among them, the short-time Fourier transform and the cepstral method are well-known techniques, and the specific methods will not be introduced here.

[0071] In the spectrum of the reference speech signal segment, the sum of the energy values ​​at all frequencies is obtained and denoted as the total spectral energy of the reference speech signal segment. .

[0072] In the spectrum of the reference speech signal segment, the absolute value of the difference between any two frequencies is used as the clustering distance. Then, the DBSCAN clustering algorithm is used to cluster all frequencies to obtain several clusters.

[0073] In this cluster, all frequencies are similar in magnitude, meaning each cluster represents a frequency range.

[0074] The sum of the energy values ​​at all frequencies in each cluster is obtained as the total energy of each cluster.

[0075] Get the maximum value of the total energy of all clusters. The maximum value among the total energy of all clusters Total spectral energy of the reference speech signal segment ratio , denoted as the energy concentration of the reference speech signal segment.

[0076] It should be noted that each cluster represents a frequency range, therefore The closer a value is to 1, the more concentrated the energy of the reference speech signal segment is within a single frequency range.

[0077] The preset harmonic threshold is 0.5, and this will be used as an example for explanation.

[0078] In the spectrum of the reference speech signal segment, the remainder of each frequency divided by the fundamental frequency is obtained and denoted as the non-integer multiple probability of each frequency. Frequency whose normalized non-integer multiple probabilities are less than or equal to a preset harmonic threshold is denoted as the target frequency. All target frequencies are then counted. With all frequency quantities ratio , denoted as the first ratio, is the sum of energy values ​​at all target frequencies. Total spectral energy of the reference speech signal segment ratio The first ratio is denoted as the second ratio, and the average of the first and second ratios is denoted as the harmonic structure significance of the reference speech signal segment.

[0079] In this embodiment, the min-max normalization method is used to normalize the probabilities of non-integer multiples of all frequencies. The min-max normalization method is a well-known technique, and its specific method will not be described here.

[0080] It should be noted that user speech exhibits a distinct harmonic structure in its spectrum, and the spectral energy of the speech signal is primarily concentrated at the fundamental frequency and its integer multiples, a characteristic determined by the physical properties of human vocal cord vibration. However, the harmonics of actual speech are not strictly locked to integer multiples of the fundamental frequency, but are allowed to deviate to some extent. Therefore, the closer a frequency's remainder when divided by the fundamental frequency is to 0, the more likely it is to be the target frequency. The more target frequencies there are, and the greater the energy value at those target frequencies, the more pronounced the harmonic structure of the reference speech signal segment.

[0081] The average of the energy concentration and the harmonic structure of the reference speech signal segment is denoted as the interference-free probability of the reference speech signal segment.

[0082] It should be noted that the greater the energy concentration and harmonic structure of the reference speech signal segment, the less interference environmental noise causes to the user's speech, and the greater the possibility of no interference.

[0083] Using the above method, the possibility of interference-free audio signal segments from any microphone within any update time period can be obtained.

[0084] It should be noted that: further, it is necessary to analyze the distance from the user to each microphone in the microphone array within each update period to determine the user's voice signal with the highest relative intensity collected by the microphone in the user's direction within each update period. Since the energy of the voice signal attenuates with distance, but the superimposed noise interference can cause a large error when directly analyzing the voice signal, this embodiment decomposes the voice analysis to distinguish and analyze the user's voice components in order to reduce noise interference.

[0085] Principal component analysis is used to decompose the reference speech signal segment into several speech components and obtain the eigenvalue of each speech component. The larger the eigenvalue, the more information the speech component contains from the reference speech signal segment.

[0086] Principal component analysis is a well-known technique, and its specific methods will not be described here.

[0087] The interference-free probability of each speech component is obtained by following the method for obtaining the interference-free probability of the reference speech signal segment.

[0088] The speech component with the highest probability of being interference-free is denoted as the user speech component.

[0089] It should be noted that if multiple speech components with the highest probability of being interference-free exist, the speech component with the largest eigenvalue is selected as the user speech component. The speech component with the highest probability of being interference-free indicates that it contains more user speech information and is less affected by environmental noise. Other speech components, on the other hand, contain more environmental noise interference and less user speech information. Therefore, these other speech components cannot be directly removed to suppress noise, as this would result in the loss of some user speech information.

[0090] Then, use the short-time Fourier transform to obtain the energy value of the user's speech component at different frequencies, and record the sum of the energy values ​​of the user's speech component at all frequencies as the total energy value of the user's speech component.

[0091] Using the above method, the total energy value of the user voice component of the voice signal segment of any microphone within any update time period can be obtained.

[0092] Within each update period, the maximum value of the total energy of the user speech components in the speech signal segments of all microphones is obtained and recorded as the baseline energy value. The ratio of the total energy value of the user speech components in the speech signal segments of each microphone to the baseline energy value is recorded as the user speech pickup effect of the speech signal segments of each microphone within each update period.

[0093] It should be noted that: within the same update period, the greater the total energy value of the user speech component in the microphone's speech signal segment, the closer the microphone is to the user, meaning it can pick up the user's speech better.

[0094] The product of the interference level of far-field environmental noise on sound pickup in each update period and the probability of no interference for the speech signal segment of each microphone in each update period is recorded as the first product. The product of the inverse proportional value of the interference level of far-field environmental noise on sound pickup in each update period and the user speech pickup effect of the speech signal segment of each microphone in each update period is recorded as the second product. The sum of the first product and the second product is recorded as the fusion coefficient of the speech signal segment of each microphone in each update period.

[0095] Specifically, the first Within the [number] update period, the [number]th ... Fusion coefficient of speech signal segments from each microphone The calculation formula is:

[0096]

[0097] in, For the first The degree of interference of far-field environmental noise on sound pickup during each update period. For the first Within the [number] update period, the [number]th ... The possibility of interference-free speech signal segments from each microphone. For the first Within the [number] update period, the [number]th ... The user's voice pickup effect of each microphone's voice signal segment.

[0098] It should be noted that: when When it is larger, it indicates that the first Noise suppression is a key focus during each update period, and The smaller the value, the better. Within the [number] update period, the [number]th ... The greater the noise interference to the speech signal segment of each microphone, the smaller the fusion coefficient needs to be assigned, so that the weight of the fused speech signals from multiple microphones is smaller, thus suppressing noise. When When it is smaller, it indicates the first During each update period, the focus should be on enhancing user voice functionality, and The larger it is, the more likely it is to be the first Within the [number] update period, the [number]th ... The better the user's voice pickup effect of each microphone's voice signal segment, the more fusion coefficients need to be assigned, so that the weight of the subsequent multi-microphone voice signal fusion is greater, thereby enhancing the user's voice. Therefore, and As respectively and The weight, The smaller the value, the smaller the fusion coefficient is required. The larger the value, the larger the required fusion coefficient. Therefore... The larger it is, the more likely it is to be the first Within the [number] update period, the [number]th ... The clearer the user's voice is in the audio signal segment of each microphone, and the less interference from environmental noise.

[0099] The ratio of the duration of each update period to the duration of the speech signal of each microphone in the microphone array is used as the weight of each update period. The fusion coefficients of the speech signal segments of each microphone in all update periods are weighted and summed to obtain the pickup attention of each microphone.

[0100] Specifically, the first Microphone pickup focus The calculation formula is:

[0101]

[0102] in, The duration of the speech signal. For the first The duration of each update period, This represents the number of update periods.

[0103] It should be noted that for each microphone's voice signal, when it is consistently in the user's direction and ambient noise interference is minimal, the microphone's pickup focus should be maximized. Therefore, using... We use weights to obtain the pickup attention of each microphone.

[0104] Step S004: Determine the weight of each microphone based on the pickup attention of each microphone, and perform weighted superposition of the speech signals from all microphones in the microphone array to obtain an enhanced speech signal; convert the enhanced speech signal into text content.

[0105] In a microphone array, the pickup attention of each microphone is the sum of the pickup attention of all microphones. The ratio of the values ​​is denoted as the weight of each microphone. Based on beamforming, the speech signals from all microphones are weighted and superimposed to obtain the enhanced speech signal.

[0106] It should be noted that beamforming is a well-known technology, and the specific method will not be described here. In this embodiment, by assigning greater weight to the speech signal from the microphone located in the user's direction where environmental noise interference is relatively small, the user's speech is enhanced, environmental noise interference is suppressed, and the clarity of the user's speech in the enhanced speech signal is ensured, thereby improving the accuracy of subsequent speech recognition.

[0107] Based on Automatic Speech Recognition (ASR) technology, the enhanced speech signal is converted into text content to complete speech recognition.

[0108] It should be noted that in this embodiment, the model architecture used for speech recognition (ASR) technology is the Transformer model (such as Whisper (OpenAI open source model) or Wav2Vec 2.0). It captures long temporal dependencies of speech through a self-attention mechanism and supports the recognition of multilingual and long speech (such as meeting minutes). This is a well-known technology, and the specific method will not be described here.

[0109] The present invention also provides a far-field voice recognition device for a multimodal smart terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program stored in the memory to implement the steps of the aforementioned far-field voice recognition method for a multimodal smart terminal.

[0110] This invention is now complete.

[0111] In summary, in this embodiment of the invention, the speech signal from each microphone in the microphone array is acquired, and the time range of the speech signal is divided into several time periods. Based on the positive correlation between the speech signal segments of different microphones within each time period, the interference level of far-field environmental noise on sound pickup within each time period is obtained, which is then used to merge the time periods to obtain several updated time periods. Based on the energy distribution of the speech signal segment of each microphone in the spectrum within each updated time period, and the energy comparison results between the speech signal segments of different microphones, the pickup attention of each microphone is obtained to determine the weight of each microphone. The speech signals of all microphones in the microphone array are then weighted and superimposed to obtain an enhanced speech signal, which is then converted into text content. This invention improves the speech signal enhancement effect after weighting and superimposing the speech signals of all microphones by adaptively adjusting the weight of each microphone in the microphone array, thereby ensuring the accuracy of far-field speech recognition.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for far-field voice recognition of a multi-modal intelligent terminal, characterized in that, The method comprises the following steps: Obtaining voice signals of each microphone in the microphone array; Equally dividing a time range in which the voice signals are located into a plurality of time periods; obtaining an interference degree of far-field environmental noise on sound pickup in each time period according to positive correlations between voice signal segments of different microphones in each time period; merging the time periods according to similarities of the interference degrees of the far-field environmental noise on the sound pickup in different time periods to obtain a plurality of updated time periods; Obtaining a fusion coefficient of a voice signal segment of each microphone in each updated time period according to energy distributions of the voice signal segments of each microphone in each updated time period on a frequency spectrum, energy contrast results between the voice signal segments of different microphones, and the interference degree of the far-field environmental noise on the sound pickup in each time period; obtaining a sound pickup attention degree of each microphone according to the fusion coefficients of the voice signal segments of each microphone in all the updated time periods and time lengths of the updated time periods; Determining a weight of each microphone according to a size of the sound pickup attention degree of each microphone; weighting and superimposing voice signals of all the microphones in the microphone array to obtain an enhanced voice signal; and converting the enhanced voice signal into text content. The method for obtaining the fusion coefficient of the voice signal segment of each microphone in each updated time period comprises the following specific steps: Taking a voice signal segment of any microphone in any updated time period as a reference voice signal segment; Obtaining energy values of the reference voice signal segment at different frequencies and a fundamental frequency; Obtaining an interference-free possibility of the reference voice signal segment according to the energy values of the reference voice signal segment at different frequencies and differences between the frequencies and the fundamental frequency; Decomposing the reference voice signal segment into a plurality of voice components; Obtaining an interference-free possibility of each voice component according to the obtaining manner of the interference-free possibility of the reference voice signal segment; Taking a voice component with the largest interference-free possibility as a user voice component; Obtaining energy values of the user voice component at different frequencies; and taking a sum of the energy values of the user voice component at all frequencies as a total energy value of the user voice component. In each updated time period, obtaining a user voice sound pickup effect of a voice signal segment of each microphone in each updated time period according to contrast results between the total energy values of the user voice components of the voice signal segments of all the microphones; Taking an average value of the interference degrees of the far-field environmental noise on the sound pickup in all the time periods in each updated time period as the interference degree of the far-field environmental noise on the sound pickup in each updated time period; Obtaining a first product of the interference degree of the far-field environmental noise on the sound pickup in each updated time period and an interference-free possibility of the voice signal segment of each microphone in each updated time period; obtaining a second product of a reciprocal value of the interference degree of the far-field environmental noise on the sound pickup in each updated time period and the user voice sound pickup effect of the voice signal segment of each microphone in each updated time period; and taking a sum of the first product and the second product as the fusion coefficient of the voice signal segment of each microphone in each updated time period.

2. The method of far-field voice recognition of multi-modal intelligent terminal according to claim 1, characterized in that, The method for obtaining the interference degree of the far-field environmental noise on the sound pickup in each time period comprises the following specific steps: The reciprocal normalized value of the average of the Pearson correlation coefficients between the speech signal segments of any two microphones in each time period is recorded as the interference degree of the far-field environmental noise on the sound pickup in each time period. 3.The multi-modal intelligent terminal far-field audio pickup voice recognition method of claim 1, wherein, The specific steps of obtaining the several update time periods include the following: In time sequence, the interference degrees of the far-field environmental noise on the sound pickup in all time periods form an interference degree sequence, the DBSCAN clustering algorithm is used to divide the interference degree sequence into several sequence segments, and all time periods corresponding to each sequence segment are combined into each update time period.

4. The method of far-field audio signal acquisition and speech recognition of a multi-modal intelligent terminal according to claim 1, characterized in that, The specific steps of obtaining the disturbance-free possibility of the reference speech signal segment include the following: The sum of the energy values at all frequencies is obtained, which is recorded as the total spectral energy of the reference speech signal segment; The absolute value of the difference between any two frequencies is used as the clustering distance, the DBSCAN clustering algorithm is used for clustering operation on all frequencies, and several clustering clusters are obtained; The sum of the energy values at all frequencies in each clustering cluster is obtained as the total energy of each clustering cluster; The ratio of the maximum value of the total energy of all clustering clusters to the total spectral energy of the reference speech signal segment is recorded as the energy concentration of the reference speech signal segment; The remainder of each frequency divided by the fundamental frequency is recorded as the non-integer multiple probability of each frequency; The frequencies whose normalized values of the non-integer multiple probability are less than or equal to the preset harmonic threshold are recorded as the target frequencies; The ratio of the number of all target frequencies to the number of all frequencies is recorded as the first ratio, the ratio of the sum of the energy values at all target frequencies to the total spectral energy of the reference speech signal segment is recorded as the second ratio, and the average of the first ratio and the second ratio is recorded as the harmonic structure obviousness of the reference speech signal segment. The average of the energy concentration and the harmonic structure obviousness of the reference speech signal segment is recorded as the disturbance-free possibility of the reference speech signal segment. 5.The multi-modal intelligent terminal far-field audio pickup voice recognition method of claim 1, wherein, The specific steps of obtaining the user speech sound pickup effect of the speech signal segment of each microphone in each update time period include the following: In each update time period, the maximum value of the total energy values of the user speech components of the speech signal segments of all microphones is obtained, which is recorded as the reference energy value, and the ratio of the total energy value of the user speech component of the speech signal segment of each microphone to the reference energy value is recorded as the user speech sound pickup effect of the speech signal segment of each microphone in each update time period.

6. The method of far-field audio pickup and speech recognition of a multi-modal intelligent terminal according to claim 1, characterized in that, The specific steps of obtaining the sound pickup attention degree of each microphone include the following: The ratio of the length of each update time period to the length of the speech signal is used as the weight of each update time period, the fusion coefficients of the speech signal segments of each microphone in all update time periods are weighted and summed to obtain the sound pickup attention degree of each microphone.

7. The method of far-field audio pickup and speech recognition of a multi-modal intelligent terminal according to claim 1, characterized in that, The specific steps of determining the weight of each microphone include the following: The ratio of the sound pickup attention degree of each microphone to the sum of the sound pickup attention degrees of all microphones is recorded as the weight of each microphone.

8. A far-field voice recognition device of a multi-modal intelligent terminal, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The computer program is executed by the processor to realize the steps of the far-field sound pickup speech recognition method of the multi-modal intelligent terminal according to any one of claims 1-7.

Images