An artificial intelligence-based speech recognition method and system

Abstract

The application provides a speech recognition method and system based on artificial intelligence, and relates to the technical field of speech recognition.The application collects speech signals in real time, extracts speech signal features through mel-frequency cepstrum coefficients after denoising the speech signals, and forms speech feature vectors by combining mel-cepstrum coefficients, first-order differences and second-order differences of the mel-cepstrum coefficients.Meanwhile, lip movement images are collected as visual signals, image feature vectors are generated by calculating the LBP values of the pixels in the images after the images are subjected to grayscale processing, the weights of the speech feature vectors and the image feature vectors are dynamically adjusted through a cross-modal attention mechanism, a fusion weight matrix is generated, different fusion weight matrices correspond to different speech instructions, the speech original signals and the visual original images are used as a training set, the speech instructions corresponding to the fusion weight matrix are used as labels to train a deep learning network model, and finally the real-time collected data is input into the trained model for real-time speech recognition.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, specifically to a speech recognition method and system based on artificial intelligence. Background Technology

[0002] Currently, speech recognition technology is widely used in various fields, such as intelligent assistants, automatic translation, and accessibility technology. However, traditional speech recognition systems mainly rely on audio signals, making them susceptible to environmental noise, leading to decreased recognition accuracy and robustness. Furthermore, single audio input performs particularly poorly in complex noisy or multi-person conversation scenarios. Multimodal speech recognition systems can effectively address this problem by simultaneously utilizing audio and visual information for speech recognition, thereby improving recognition performance in noisy environments. The primary challenge lies in effectively integrating these two types of information and adjusting their contribution to the recognition process.

[0003] In the prior art, CN116580706B discloses an artificial intelligence-based speech recognition method. This method involves collecting user-inputted speech audio information, converting the speech audio information into an audio spectrogram, obtaining multiple audio frames from the audio spectrogram, extracting feature information from each audio frame, associating the feature information of multiple audio frames to obtain the data to be recognized, inputting the data to be recognized into a trained speech recognition model, determining the speech content corresponding to the speech audio information, verifying the speech content, obtaining the speech recognition result, and outputting it.

[0004] The main problem with the above methods is that they rely entirely on speech audio information, so the recognition accuracy will be affected in complex noise environments. Different recording devices and recording conditions will lead to differences in audio quality, which will produce certain errors when generating spectrograms. Furthermore, the feature extraction process from audio spectrograms is complex and requires a lot of computing resources and time, which may not meet the needs of real-time speech recognition applications.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a speech recognition method and system based on artificial intelligence to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An artificial intelligence-based speech recognition method, comprising the following steps:

[0009] Step 1: Acquire the original speech signal, perform noise reduction processing on the original speech signal, decompose the speech signal into multiple sub-bands of different frequencies through wavelet transform, each sub-band corresponds to the information of the signal at different time scales, perform threshold processing on the high-frequency sub-bands to reduce noise components, and recombine the processed sub-bands to obtain the denoised speech signal.

[0010] Step 2: Pre-emphasize the denoised speech signal using a high-pass filter, then divide the pre-emphasized speech signal into short frames of 20-40ms, with each frame overlapping by 50% of its length. Window each frame signal, and convert the windowed frame signal from the time domain to the frequency domain using a fast Fourier transform. Generate Mel-frequency cepstral coefficients from the frequency domain signal using a Mel filter, and synthesize the Mel-frequency cepstral coefficients and their first and second differences into a speech feature vector.

[0011] Step 3: Collect the speaker's lip movement information in real time as the original visual image, perform grayscale processing on the original image, and generate the first recognition image;

[0012] Step 4: Normalize the pixels of the first recognition image. For each pixel in the first recognition image, select the 8 neighboring pixels around the pixel, compare the gray values ​​of the center pixel with those of its neighbors, generate a binary number, convert the binary number to a decimal number to generate the LBP value of the center pixel, and calculate the LBP values ​​of all pixels in the first recognition image to form a new image, called the second recognition image.

[0013] Step 5: Divide the second recognition image into non-overlapping units of 8×8 pixels. Units that do not meet the size requirements are directly cropped and discarded. Calculate the distribution of LBP values ​​of all pixels in each unit to generate a histogram of a unit. Piece together the histograms of all units in the order of the units to generate an image feature vector, which serves as the final feature representation of the visual signal.

[0014] Step 6: Convert the speech feature vector and image feature vector into matrix representations, and dynamically adjust the weights of the speech features and image features through a cross-modal attention mechanism to generate a fusion weight matrix;

[0015] Step 7: Construct a deep learning network model. Use the original speech signal and the original visual image as the training set, and use the speech instructions corresponding to the fusion weight matrix as labels. Input these into the deep learning network model for training.

[0016] Step 8: Input the real-time acquired speech signal and visual image into the trained deep learning network model to obtain the current fusion weight matrix, and determine the current speech command based on the fusion weight matrix.

[0017] Furthermore, the principle underlying the determination of the high-frequency sub-band range is as follows:

[0018]

[0019] in, Indicates the frequency of the high-frequency sub-band. Indicates the sampling rate of the original signal. This indicates the corresponding layer number in the wavelet transform.

[0020] Furthermore, the formula used to decompose the speech signal into multiple sub-bands of different frequencies through wavelet transform is as follows:

[0021]

[0022]

[0023]

[0024] in, Represents the Haar wavelet basis functions. Represents a time variable. Represents the wavelet coefficients of the high-frequency subband. Represents the original signal. Represents the wavelet basis functions The result of scaling and translation It is the scaling parameter. It is the translation scale parameter;

[0025] A noise threshold is set, and sub-band coefficients are generated based on the noise threshold and wavelet coefficients. The formula used is as follows:

[0026]

[0027] in, This represents the subband coefficients after processing. Indicates the noise threshold;

[0028] The processed subband coefficients are then subjected to inverse wavelet transform to generate the denoised signal, based on the following formula:

[0029]

[0030] in, This represents the signal after noise reduction. Indicates the sub-band coefficient. Represents the wavelet basis functions The result of scaling and translation It is the scaling parameter. It is the translation scale parameter.

[0031] Furthermore, the principle underlying the generation of Mel frequency cepstral coefficients is as follows:

[0032] First, the denoised signal is pre-emphasized using the following formula:

[0033]

[0034] in, This indicates the time of the pre-emphasized signal. The corresponding value, This indicates the time of the denoised signal. The corresponding value, This indicates the time of the denoised signal. The corresponding value, Indicates the pre-emphasis coefficient. Represents a time variable;

[0035] After the pre-emphasized signal is divided into short time frames, windowing is applied to each frame using the following formula:

[0036]

[0037]

[0038] in, This represents the result after windowing the segmented frame signal. This represents the segmented frame signal. Represents a time variable. This represents the Hamming window function. Indicates a time index. Indicates frame length;

[0039] The formula used to convert the windowed frame signal from a time-domain signal to a frequency-domain signal is:

[0040]

[0041] Among them, Z Represents frequency domain signals, Represents a time-domain signal. Indicates frame length, Represents the imaginary unit. This represents the frequency index, with values ​​ranging from [0, N-1]. This represents a time index, with a value range of [0, N-1].

[0042] The formula for the energy output of each filter is as follows:

[0043]

[0044] in, Indicates the first The value of a Mel filter, Indicates frequency index, Indicates the first The frequency of the Mel filter;

[0045] The logarithm of the energy output of each filter is taken to generate the logarithmic energy. Then, a discrete cosine transform is performed on the logarithmic energy to generate the Mel-frequency cepstral coefficients. The formula used is as follows:

[0046]

[0047]

[0048] Among them, among them, Indicates the first The logarithmic energy of each filter, Represents frequency domain signals, Indicates the first The value of a Mel filter, Indicates frequency index, Indicates frame length, Indicates the first Mel frequency cepstral coefficients, Indicates the number of Mel filters. Indicates the first The logarithmic energy of a Mel filter Indicates the index of the Mel filter. Indicates the index of the Mel frequency cepstral coefficients;

[0049] The formulas used to calculate the first and second differences of the Mel frequency cepstral coefficients are as follows:

[0050]

[0051]

[0052] in, Indicates the first First-order difference of Mel frequency cepstral coefficients Indicates the first The second difference of the cepstral coefficients of the Mel frequency Indicates the frame offset. The size of the difference window is determined by the size of the difference window. The specific value is determined based on the actual application scenario;

[0053] The principle underlying the synthesis of Mel frequency cepstral coefficients, first-order differences, and second-order differences into a single eigenvector is as follows:

[0054]

[0055] in, This represents the final speech feature vector. Indicates the first Mel frequency cepstral coefficients, This represents the first-order difference of the cepstral coefficient at the th Mel frequency. Indicates the first The second difference of the cepstral coefficients of the Mel frequency This indicates the number of Mel frequency cepstral coefficients.

[0056] Furthermore, the original image is converted to grayscale, and the formula used to generate the first recognition image is as follows:

[0057]

[0058] in, Represents the grayscale value of a pixel. This represents the red channel value of a pixel in the original image. This represents the green channel value of a pixel in the original image. This represents the blue channel value of a pixel in the original image.

[0059] Furthermore, the principle behind comparing the grayscale values ​​of the center pixel with those of its neighboring pixels to generate a binary number is as follows:

[0060] For the center pixel with 8 neighboring pixels, it can be represented as:

[0061]

[0062] The formula used to determine the value of a binary number is:

[0063]

[0064] in, This represents the grayscale value of the center pixel. Represents the grayscale value of neighboring pixels. It is a comparison formula that judges based on the size relationship between the center pixel and its neighboring pixels;

[0065] The formula used to convert a binary number into the LBP value of a pixel is:

[0066]

[0067] in, The LBP value represents the center pixel. Indicates the number of neighboring pixels. This represents a comparison formula that judges based on the size relationship between the center pixel and its neighboring pixels.

[0068] Furthermore, the principle underlying the generation of a unit's histogram is as follows:

[0069] The distribution of LBP values ​​of all pixels within a unit is statistically analyzed. The LBP values ​​are divided into 256 intervals according to the range [0,255]. Each interval corresponds to one LBP value. A histogram vector of length 256 is generated, where each element represents the number of LBP values ​​of the pixels in the corresponding interval.

[0070] The formula used to generate an eigenvector by connecting the histograms of all units in sequence is:

[0071]

[0072] in, This represents the final image feature vector. Indicates the first The histogram of the nth unit The number of LBP values ​​in each interval Indicates the number of units.

[0073] Furthermore, the principle underlying the generation of the fusion weight matrix is ​​as follows:

[0074] The speech feature vector can be viewed as a 1×3L matrix, which can be represented as:

[0075]

[0076] in, Represents the speech feature matrix, Indicates the first Mel frequency cepstral coefficients, This represents the first-order difference of the cepstral coefficient at the th Mel frequency. Indicates the first The second difference of the cepstral coefficients of the Mel frequency This indicates the number of Mel frequency cepstral coefficients;

[0077] The image feature vector can be viewed as a 1×256E matrix, which can be represented as:

[0078]

[0079] in, Represents the image feature matrix. Indicates the first The histogram of the nth unit The number of LBP values ​​in each interval Indicates the number of units;

[0080] The speech feature matrix and the image feature matrix have the same number of elements, and their similarity is generated using the following formula:

[0081]

[0082] in, This represents the similarity between the elements of the corresponding speech feature matrix and image feature matrix. The first part of the speech feature matrix represents the... Column elements, The first element of the image feature matrix represents the... Column elements;

[0083] Arrange all the calculated similarity values ​​in order to form a matrix. The matrix That is, the correlation matrix between the speech feature matrix and the image feature matrix;

[0084] The attention weights are calculated based on the relevance matrix using the following formula:

[0085]

[0086]

[0087] in, Attention weights representing speech features Attention weights representing image features Represents the correlation matrix. Represents the transpose of the correlation matrix;

[0088] The calculated weights are used for feature fusion to generate a fusion weight matrix, based on the following formula:

[0089]

[0090] in, Represents the fusion weight matrix. Attention weights representing speech features Represents the speech feature matrix, Attention weights representing image features This represents the image feature matrix.

[0091] The present invention also provides an artificial intelligence-based speech recognition system, the system being used to implement the above-mentioned artificial intelligence-based speech recognition method, comprising:

[0092] The voice acquisition module is used to acquire the raw voice signal, perform noise reduction processing on the raw voice signal, decompose the voice signal into multiple sub-bands of different frequencies through wavelet transform, each sub-band corresponds to the information of the signal at different time scales, perform threshold processing on the high-frequency sub-bands to reduce noise components, and recombine the processed sub-bands to obtain the denoised voice signal.

[0093] The speech feature extraction module is used to pre-emphasize the denoised speech signal through a high-pass filter, divide the pre-emphasized speech signal into short frames of 20~40ms, overlap the frames with 50% of the frame length, window the signal of each frame, and transform the windowed frame signal from the time domain signal to the frequency domain signal through a fast Fourier transform. The frequency domain signal is then passed through a Mel filter to generate Mel cepstral coefficients, and the Mel cepstral coefficients and their first-order and second-order differences are grouped together to form a speech feature vector.

[0094] The image acquisition module is used to acquire the speaker's lip movement information in real time as the raw visual image, perform grayscale processing on the raw image, and generate the first recognition image;

[0095] The image processing module is used to normalize the pixels of the first recognition image. For each pixel in the first recognition image, it selects eight neighboring pixels around the pixel, compares the gray values ​​of the center pixel with those of its neighboring pixels, generates a binary number, converts the binary number into a decimal number to generate the LBP value of the center pixel, and calculates the LBP values ​​of all pixels in the first recognition image to form a new image, called the second recognition image.

[0096] The image feature extraction module is used to divide the second recognition image into non-overlapping units of 8×8 pixels. Units that do not meet the size requirements are directly cropped and discarded. The LBP value distribution of all pixels in each unit is statistically analyzed to generate a histogram of a unit. The histograms of all units are concatenated end to end in the order of the units to generate an image feature vector, which serves as the final feature representation of the visual signal.

[0097] The feature fusion module is used to convert speech feature vectors and image feature vectors into matrix representations, and dynamically adjust the weights of speech features and image features through a cross-modal attention mechanism to generate a fusion weight matrix.

[0098] The model building module is used to build a deep learning network model. It uses the original speech signal and the original visual image as the training set, and the speech command corresponding to the fusion weight matrix as the label, and inputs it into the deep learning network model for training.

[0099] The speech recognition module inputs real-time collected speech signals and visual images into the trained deep learning network model to obtain the current fusion weight matrix, and determines the current speech command based on the fusion weight matrix.

[0100] Compared with the prior art, the beneficial effects of the present invention are:

[0101] This invention, based on a multimodal speech recognition method, comprehensively considers both speech and visual signal features, and employs different processing methods for different signal features: Wavelet transform and thresholding techniques are used to denoise the speech signal, improving speech recognition accuracy in high-noise environments. The denoised speech features are then converted into Mel frequencies to generate Mel frequency cepstral coefficients, extracting key speech features that are closer to human perception. After grayscale processing of the lip movement information image, the LBP value of each pixel is calculated to generate the image's feature vector. Combining the speech and image feature vectors, a cross-modal attention mechanism is used to adjust their attention weights, thereby generating a fusion weight matrix that comprehensively reflects both speech and image features. Different fusion weight matrices correspond to different speech and image features, thus representing different speech commands. Using the original speech signal and original visual image as the training set, and the speech commands corresponding to the fusion weight matrix as labels, a deep learning network model is trained. Inputting real-time acquired speech signals and visual images into the trained model generates different speech commands corresponding to different fusion weight matrices. This invention can adaptively optimize the weight ratio of speech features and image features in the fusion weight matrix under different environments to adjust the feature fusion strategy, thereby providing more accurate and stable recognition results. Attached Figure Description

[0102] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0103] Figure 2 This is a schematic diagram of the system modules in an embodiment of the present invention. Detailed Implementation

[0104] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0105] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0106] Example:

[0107] Please see Figure 1 The present invention provides a technical solution:

[0108] An artificial intelligence-based speech recognition method, comprising the following steps:

[0109] Step 1: Acquire the original speech signal, perform noise reduction processing on the original speech signal, decompose the speech signal into multiple sub-bands of different frequencies through wavelet transform, each sub-band corresponds to the information of the signal at different time scales, perform threshold processing on the high-frequency sub-bands to reduce noise components, and recombine the processed sub-bands to obtain the denoised speech signal.

[0110] In this embodiment, the principle upon which the high-frequency sub-band range is determined is as follows:

[0111]

[0112] in, Indicates the frequency of the high-frequency sub-band. Indicates the sampling rate of the original signal. This indicates the corresponding layer number in the wavelet transform.

[0113] The formula used for wavelet transform is:

[0114]

[0115]

[0116]

[0117] in, Represents the Haar wavelet basis functions. Represents a time variable. Represents the wavelet coefficients of the high-frequency subband. Represents the original signal. Represents the wavelet basis functions The result of scaling and translation It is the scaling parameter. It is the translation scale parameter;

[0118] Wavelet transform decomposes a speech signal into a series of basis functions called wavelets. These basis functions are generated from a single prototype waveform through translation and scaling with different scale and position parameters, simultaneously providing information about the speech signal in both time and scale. Wavelet coefficients represent the characteristics of the signal at different scales and time positions; their numerical values ​​reflect the amplitude of the signal at the corresponding wavelet scale and time position, and their signs reflect the positive or negative polarity of the energy. The scale parameter... The larger the value, the lower the corresponding frequency; the coarser the signal represented by the wavelet coefficients, the smaller the scale parameter. The smaller the value, the higher the corresponding frequency, and the more refined the signal represented by the wavelet coefficients; translation parameter The larger the value, the later the corresponding time position. The wavelet coefficients represent the characteristics of the signal in the later stages, and the shift parameter... The smaller the value, the earlier the corresponding time position; the wavelet coefficient represents the characteristics of the signal in the early stage.

[0119] A noise threshold is set, and sub-band coefficients are generated based on the noise threshold and wavelet coefficients. The formula used is as follows:

[0120]

[0121] in, This represents the subband coefficients after processing. Indicates the noise threshold;

[0122] Subband coefficients are the result of thresholding wavelet coefficients and reflect the form of the signal after denoising. Hard thresholding is used in the thresholding process, where wavelet coefficients not greater than the threshold are set to zero, while wavelet coefficients greater than the threshold are kept unchanged, which can better preserve the original characteristics of the signal.

[0123] The processed subband coefficients are then subjected to inverse wavelet transform to generate the denoised signal, based on the following formula:

[0124]

[0125] in, This represents the signal after noise reduction. Indicates the sub-band coefficient. Represents the wavelet basis functions The result of scaling and translation It is the scaling parameter. It is the translation scale parameter;

[0126] The purpose of inverse wavelet transform is to reconstruct the original signal from the denoised wavelet coefficients. By merging wavelet coefficients of different scales, it reflects the process of reconstructing the original signal from multi-level information from details to overview. The accuracy of the result of inverse wavelet transform is proportional to the number of wavelet coefficients retained and the accuracy of the retained wavelet coefficients.

[0127] Step 2: Pre-emphasize the denoised speech signal using a high-pass filter, then divide the pre-emphasized speech signal into short frames of 20-40ms, with each frame overlapping by 50% of its length. Window each frame signal, and convert the windowed frame signal from the time domain to the frequency domain using a fast Fourier transform. Generate Mel-frequency cepstral coefficients from the frequency domain signal using a Mel filter, and synthesize the Mel-frequency cepstral coefficients and their first and second differences into a speech feature vector.

[0128] In this embodiment, the principle underlying the generation of Mel frequency cepstral coefficients is as follows:

[0129] First, the denoised signal is pre-emphasized using the following formula:

[0130]

[0131] in, This indicates the time of the pre-emphasized signal. The corresponding value, This indicates the time of the denoised signal. The corresponding value, This indicates the time of the denoised signal. The corresponding value, Indicates the pre-emphasis coefficient. Represents a time variable;

[0132] The signal after pre-emphasis reflects the enhancement effect on the original signal. Common pre-emphasis coefficient values ​​are 0.90 to 0.97. The larger the pre-emphasis coefficient, the more obvious the high-frequency enhancement effect. The smaller the pre-emphasis coefficient, the weaker the high-frequency enhancement effect. The signal after pre-emphasis is proportional to the pre-emphasis coefficient and proportional to the strength of the original signal.

[0133] After the pre-emphasized signal is divided into short time frames, windowing is applied to each frame using the following formula:

[0134]

[0135]

[0136] in, This represents the result after windowing the segmented frame signal. This represents the segmented frame signal. Represents a time variable. This represents the Hamming window function. Indicates a time index. Indicates frame length;

[0137] Windowing a signal reduces the discontinuity between two segments, making the spectral representation of the signal clearer and more focused on the actual frequency components of the signal. The purpose of choosing the Hamming window as the window function is to effectively reduce spectral leakage and provide a better balance between time-domain and frequency-domain performance, reducing sidelobe amplitude while maintaining a reasonable main lobe width.

[0138] The formula used to convert the windowed frame signal from a time-domain signal to a frequency-domain signal is:

[0139]

[0140] Among them, Z Represents frequency domain signals, Represents a time-domain signal. Indicates frame length, Represents the imaginary unit. This represents the frequency index, with values ​​ranging from [0, N-1]. This represents a time index, with a value range of [0, N-1].

[0141] The Fourier transform is used to convert the time-domain signal into a frequency-domain signal, reflecting the distribution of the time-domain signal at various frequency indices. An increase means that the time-domain signal is multiplied by a faster oscillating waveform, which directly affects the amplitude and phase of the corresponding frequency component in the frequency domain representation. The frequency component is inversely proportional to the frequency index and directly proportional to the input signal sample.

[0142] The formula for the energy output of each filter is as follows:

[0143]

[0144] in, Indicates the first The value of a Mel filter, Indicates frequency index, Indicates the first The frequency of the Mel filter;

[0145] Taking the logarithm of the energy output of each filter, the formula for generating the logarithmic energy is as follows:

[0146]

[0147] in, Indicates the first The logarithmic energy of each filter, Represents frequency domain signals, Indicates the first The value of a Mel filter, Indicates frequency index, Indicates frame length;

[0148] Logarithmic energy reflects the human ear's perception of sound intensity. The magnitude of logarithmic energy is proportional to the frequency domain components and the response value of the Mel filter.

[0149] The discrete cosine transform of the logarithmic energy is used to generate the Mel frequency cepstral coefficients, based on the following formula:

[0150]

[0151] in, Indicates the first Mel frequency cepstral coefficients, Indicates the number of Mel filters. Indicates the first The logarithmic energy of a Mel filter Indicates the index of the Mel filter. Indicates the index of the Mel frequency cepstral coefficients;

[0152] Mel frequency cepstral coefficients reflect the spectral characteristics of speech signals that match human auditory perception. They reflect the nonlinear perception of frequency by human hearing through a filter bank at the Mel scale. In the low-frequency region, the frequency resolution at the Mel scale is high, while in the high-frequency region, the resolution is low. Mel frequency cepstral coefficients are proportional to the logarithmic energy of the Mel filter and to the number of Mel filters.

[0153] The formula used to calculate the first difference of the Mel frequency cepstral coefficients is as follows:

[0154]

[0155] in, Indicates the first First-order difference of Mel frequency cepstral coefficients Indicates the frame offset. The size of the difference window is determined by the size of the difference window. The value of the difference window size is determined by the specific application scenario;

[0156] The formula used to calculate the second difference of the Mel frequency cepstral coefficients is as follows:

[0157]

[0158] in, Indicates the first The second difference of the cepstral coefficients of the Mel frequency Indicates the frame offset. The size of the difference window is determined by the size of the difference window. The specific values ​​were determined through experiments;

[0159] The principle underlying the synthesis of Mel frequency cepstral coefficients, first-order differences, and second-order differences into a single eigenvector is as follows:

[0160]

[0161] in, This represents the final speech feature vector. Indicates the first Mel frequency cepstral coefficients, This represents the first-order difference of the cepstral coefficient at the th Mel frequency. Indicates the first The second difference of the cepstral coefficients of the Mel frequency This indicates the number of Mel frequency cepstral coefficients.

[0162] The first-order difference of the Mel frequency represents the velocity characteristic, that is, the rate of change of the Mel frequency cepstral coefficients over time, reflecting the dynamic changes of the speech signal. The second-order difference represents the acceleration characteristic, that is, the rate of change of the first-order difference of the Mel frequency cepstral coefficients over time, which can capture more subtle changes in the speech signal. Combining the Mel frequency cepstral coefficients, the first-order difference of the Mel frequency cepstral coefficients, and the second-order difference of the Mel frequency cepstral coefficients to generate a feature vector can capture the rapidly changing dynamic characteristics in the speech signal in a timely manner, thereby improving the accuracy and robustness of speech recognition.

[0163] Step 3: Collect the speaker's lip movement information in real time as the original visual image, perform grayscale processing on the original image, and generate the first recognition image;

[0164] In this embodiment, the formula used to generate the first recognition image by converting the original image to grayscale is as follows:

[0165]

[0166] in, Represents the grayscale value of a pixel. This represents the red channel value of a pixel in the original image. This represents the green channel value of a pixel in the original image. This represents the blue channel value of a pixel in the original image;

[0167] The grayscale conversion formula reflects that green has the greatest weight in influencing grayscale, followed by red, and then blue. This weighting method is based on the optical perception principle of color images, ensuring that the conversion result is as visually consistent as possible with the brightness level of the original image. The purpose of grayscale conversion is to remove color information and retain only brightness information, thus simplifying the complexity of image processing.

[0168] Step 4: Normalize the pixels of the first recognition image. For each pixel in the first recognition image, select the 8 neighboring pixels around the pixel, compare the gray values ​​of the center pixel with those of its neighbors, generate a binary number, convert the binary number to a decimal number to generate the LBP value of the center pixel, and calculate the LBP values ​​of all pixels in the first recognition image to form a new image, called the second recognition image.

[0169] In this embodiment, the principle underlying the generation of a binary number by comparing the grayscale values ​​of the center pixel with those of its neighboring pixels is as follows:

[0170] For the center pixel with 8 neighboring pixels, it can be represented as:

[0171]

[0172] The formula used to determine the value of a binary number is:

[0173]

[0174] in, This represents the grayscale value of the center pixel. Represents the grayscale value of neighboring pixels. It is a comparison formula that judges based on the size relationship between the center pixel and its neighboring pixels;

[0175] The formula used to convert a binary number into the LBP value of a pixel is:

[0176]

[0177] in, The LBP value represents the center pixel. Indicates the number of neighboring pixels. This represents a comparison formula that judges based on the size relationship between the center pixel and its neighboring pixels.

[0178] In LBP calculation, when the gray value of a neighboring pixel... Not less than the gray value of the center pixel When the value is 1, it is the gray value of the neighboring pixels. Gray values ​​smaller than the center pixel When the value is 0, the eight neighboring pixels around the center pixel are evaluated sequentially, starting from the top left neighboring pixel, and a binary sequence is generated. Finally, the binary sequence is converted into decimal, corresponding to the LBP value of the center pixel. The LBP value of a pixel reflects the change of gray values ​​in the neighborhood. If the gray values ​​of the neighboring pixels are generally greater than or equal to those of the center pixel, the LBP value is higher, indicating stronger texture features of the image. If the gray values ​​of the neighboring pixels are generally less than those of the center pixel, the LBP value is lower, indicating weaker texture features of the image. The more edge pixels with gray values ​​greater than those of the center pixel, the higher the LBP value.

[0179] Step 5: Divide the second recognition image into non-overlapping units of 8×8 pixels. Units that do not meet the size requirements are directly cropped and discarded. Calculate the distribution of LBP values ​​of all pixels in each unit to generate a histogram of a unit. Piece together the histograms of all units in the order of the units to generate an image feature vector, which serves as the final feature representation of the visual signal.

[0180] In this embodiment, the principle underlying the generation of a cell's histogram is as follows:

[0181] The distribution of LBP values ​​of all pixels within a unit is statistically analyzed. The LBP values ​​are divided into 256 intervals according to the range [0,255]. Each interval corresponds to one LBP value. A histogram vector of length 256 is generated, where each element represents the number of LBP values ​​of the pixels in the corresponding interval.

[0182] The formula used to generate an eigenvector by connecting the histograms of all units in sequence is:

[0183]

[0184] in, This represents the final image feature vector. Indicates the first The histogram of the nth unit The number of LBP values ​​in each interval Indicates the number of units.

[0185] The second recognition image is divided into several non-overlapping units, each unit being 8×8 in size. Units that do not meet the size requirement are discarded. For each unit, the LBP values ​​of all pixels within it are counted, and the number of LBP values ​​in the range [0, 255] is recorded. After processing all units, all units are connected to generate a feature vector.

[0186] Step 6: Convert the speech feature vector and image feature vector into matrix representations, and dynamically adjust the weights of the speech features and image features through a cross-modal attention mechanism to generate a fusion weight matrix;

[0187] The principle underlying the generation of the fusion weight matrix is ​​as follows:

[0188] The speech feature vector can be viewed as a 1×3L matrix, which can be represented as:

[0189]

[0190] in, Represents the speech feature matrix, Indicates the first Mel frequency cepstral coefficients, This represents the first-order difference of the cepstral coefficient at the th Mel frequency. Indicates the first The second difference of the cepstral coefficients of the Mel frequency This indicates the number of Mel frequency cepstral coefficients;

[0191] The image feature vector can be viewed as a 1×256E matrix, which can be represented as:

[0192]

[0193] in, Represents the image feature matrix. Indicates the first The histogram of the nth unit The number of LBP values ​​in each interval Indicates the number of units;

[0194] The speech feature matrix and the image feature matrix have the same number of elements, and their similarity is generated using the following formula:

[0195]

[0196] in, This represents the similarity between the elements of the corresponding speech feature matrix and image feature matrix. The first part of the speech feature matrix represents the... Column elements, The first element of the image feature matrix represents the... Column elements;

[0197] Arrange all the calculated similarity values ​​in order to form a matrix. The matrix That is, the correlation matrix between the speech feature matrix and the image feature matrix;

[0198] The attention weights are calculated based on the relevance matrix using the following formula:

[0199]

[0200]

[0201] in, Attention weights representing speech features Attention weights representing image features Represents the correlation matrix. Represents the transpose of the correlation matrix;

[0202] The purpose of calculating attention weights using the softmax function is to convert the original relevance scores into probability values ​​ranging from 0 to 1, and to ensure that the sum of all attention weights is 1. This ensures that the model allocates weights reasonably among different modalities, avoiding situations where a certain modality feature is too prominent or ignored.

[0203] The calculated weights are used for feature fusion to generate a fusion weight matrix, based on the following formula:

[0204]

[0205] in, Represents the fusion weight matrix. Attention weights representing speech features Represents the speech feature matrix, Attention weights representing image features Represents the image feature matrix;

[0206] The fusion weight matrix is ​​a feature representation that combines speech features and image features to generate a comprehensive feature representation with richer information. It reflects the correlation between features of different modalities. Each element in the fusion weight matrix represents the correlation between a specific speech feature and a specific image feature. The higher the weight value, the higher the similarity between the two.

[0207] Step 7: Construct a deep learning network model. Use the original speech signal and the original visual image as the training set, and use the speech instructions corresponding to the fusion weight matrix as labels. Input these into the deep learning network model for training.

[0208] In this embodiment, the differences in features extracted from the original speech signal and the original visual image, as well as the differences in their attention weights, will affect the generation result of the fusion weight matrix. The different fusion weight matrices generated correspond to different speech commands based on their unique speech feature vectors and image feature vectors.

[0209] Step 8: Input the real-time acquired speech signal and visual image into the trained deep learning network model, obtain the current fusion weight matrix, and determine the current speech command based on the fusion weight matrix;

[0210] Please see Figure 2 The present invention also provides an artificial intelligence-based speech recognition system, the system being used to implement the above-mentioned artificial intelligence-based speech recognition method, comprising:

[0211] The voice acquisition module is used to acquire the raw voice signal, perform noise reduction processing on the raw voice signal, decompose the voice signal into multiple sub-bands of different frequencies through wavelet transform, each sub-band corresponds to the information of the signal at different time scales, perform threshold processing on the high-frequency sub-bands to reduce noise components, and recombine the processed sub-bands to obtain the denoised voice signal.

[0212] The speech feature extraction module is used to pre-emphasize the denoised speech signal through a high-pass filter, divide the pre-emphasized speech signal into short frames of 20~40ms, overlap the frames with 50% of the frame length, window the signal of each frame, and transform the windowed frame signal from the time domain signal to the frequency domain signal through a fast Fourier transform. The frequency domain signal is then passed through a Mel filter to generate Mel cepstral coefficients, and the Mel cepstral coefficients and their first-order and second-order differences are grouped together to form a speech feature vector.

[0213] The image acquisition module is used to acquire the speaker's lip movement information in real time as the raw visual image, perform grayscale processing on the raw image, and generate the first recognition image;

[0214] The image processing module is used to normalize the pixels of the first recognition image. For each pixel in the first recognition image, it selects eight neighboring pixels around the pixel, compares the gray values ​​of the center pixel with those of its neighboring pixels, generates a binary number, converts the binary number into a decimal number to generate the LBP value of the center pixel, and calculates the LBP values ​​of all pixels in the first recognition image to form a new image, called the second recognition image.

[0215] The image feature extraction module is used to divide the second recognition image into non-overlapping units of 8×8 pixels. Units that do not meet the size requirements are directly cropped and discarded. The LBP value distribution of all pixels in each unit is statistically analyzed to generate a histogram of a unit. The histograms of all units are concatenated end to end in the order of the units to generate an image feature vector, which serves as the final feature representation of the visual signal.

[0216] The feature fusion module is used to convert speech feature vectors and image feature vectors into matrix representations, and dynamically adjust the weights of speech features and image features through a cross-modal attention mechanism to generate a fusion weight matrix.

[0217] The model building module is used to build a deep learning network model. It uses the original speech signal and the original visual image as the training set, and the speech command corresponding to the fusion weight matrix as the label, and inputs it into the deep learning network model for training.

[0218] The speech recognition module inputs real-time collected speech signals and visual images into the trained deep learning network model to obtain the current fusion weight matrix, and determines the current speech command based on the fusion weight matrix.

[0219] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0220] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0221] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0222] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A speech recognition method based on artificial intelligence, characterized in that, The specific steps include: Step 1: Acquire the original speech signal, perform noise reduction processing on the original speech signal, decompose the speech signal into multiple sub-bands of different frequencies through wavelet transform, each sub-band corresponds to the information of the signal at different time scales, perform threshold processing on the high-frequency sub-bands to reduce noise components, and recombine the processed sub-bands to obtain the denoised speech signal. Step 2: Pre-emphasize the denoised speech signal using a high-pass filter, then divide the pre-emphasized speech signal into short frames of 20-40ms, with each frame overlapping by 50% of its length. Window each frame signal, and convert the windowed frame signal from the time domain to the frequency domain using a fast Fourier transform. Generate Mel-frequency cepstral coefficients from the frequency domain signal using a Mel filter, and synthesize the Mel-frequency cepstral coefficients and their first and second differences into a speech feature vector. Step 3: Collect the speaker's lip movement information in real time as the original visual image, perform grayscale processing on the original image, and generate the first recognition image; Step 4: Normalize the pixels of the first recognition image. For each pixel in the first recognition image, select the 8 neighboring pixels around the pixel, compare the gray values ​​of the center pixel with those of its neighbors, generate a binary number, convert the binary number to a decimal number to generate the LBP value of the center pixel, and calculate the LBP values ​​of all pixels in the first recognition image to form a new image, called the second recognition image. Step 5: Divide the second recognition image into non-overlapping units of 8×8 pixels. Units that do not meet the size requirements are directly cropped and discarded. Calculate the distribution of LBP values ​​of all pixels in each unit to generate a histogram of a unit. Piece together the histograms of all units in the order of the units to generate an image feature vector, which serves as the final feature representation of the visual signal. Step 6: Convert the speech feature vector and image feature vector into matrix representations, and dynamically adjust the weights of the speech features and image features through a cross-modal attention mechanism to generate a fusion weight matrix; Step 7: Construct a deep learning network. Use the original speech signal and the original visual image as the training set, and use the speech instructions corresponding to the fusion weight matrix as labels. Input these into the deep learning network for training. Step 8: Input the real-time acquired speech signal and visual image into the trained deep learning network, obtain the current fusion weight matrix, and determine the current speech command based on the fusion weight matrix; The principle underlying the generation of a cell's histogram is as follows: The distribution of LBP values ​​of all pixels within a unit is statistically analyzed. The LBP values ​​are divided into 256 intervals according to the range [0,255]. Each interval corresponds to one LBP value. A histogram vector of length 256 is generated, where each element represents the number of LBP values ​​of the pixels in the corresponding interval. The formula used to generate an eigenvector by connecting the histograms of all units in sequence is: ； in, This represents the final image feature vector. Indicates the first The histogram of the nth unit The number of LBP values ​​in each interval Indicates the number of units; The principle underlying the generation of the fusion weight matrix is ​​as follows: The speech feature vector is considered as a 1×3 L The matrix can be represented as: ； in, Represents the speech feature matrix, Indicates the first Mel frequency cepstral coefficients, This represents the first-order difference of the cepstral coefficient at the th Mel frequency. Indicates the first The second difference of the cepstral coefficients of the Mel frequency This indicates the number of Mel frequency cepstral coefficients; The image feature vector is considered as a 1×256 E The matrix can be represented as: ； in, Represents the image feature matrix. Indicates the first The histogram of the nth unit The number of LBP values ​​in each interval Indicates the number of units; The speech feature matrix and the image feature matrix have the same number of elements, and their similarity is generated using the following formula: ； in, This represents the similarity between the elements of the corresponding speech feature matrix and image feature matrix. The first part of the speech feature matrix represents the... Column elements, The first element of the image feature matrix represents the... Column elements; Arrange all the calculated similarity values ​​in order to form a matrix. The matrix That is, the correlation matrix between the speech feature matrix and the image feature matrix; The attention weights are calculated based on the relevance matrix using the following formula: ； ； in, Attention weights representing speech features Attention weights representing image features Represents the correlation matrix. Represents the transpose of the correlation matrix; The calculated weights are used for feature fusion to generate a fusion weight matrix, based on the following formula: ； in, Represents the fusion weight matrix. Attention weights representing speech features Represents the speech feature matrix, Attention weights representing image features This represents the image feature matrix.

2. The speech recognition method based on artificial intelligence according to claim 1, characterized in that: The principle underlying the determination of the high-frequency sub-band range in step 1 is as follows: ； in, Indicates the frequency of the high-frequency sub-band. Indicates the sampling rate of the original signal. This indicates the corresponding layer number in the wavelet transform.

3. The speech recognition method based on artificial intelligence according to claim 1, characterized in that: The formula used in step 1 to decompose the speech signal into multiple sub-bands of different frequencies using wavelet transform is as follows: ； ； ； in, Represents the Haar wavelet basis functions. Represents a time variable. Represents the wavelet coefficients of the high-frequency subband. Represents the original signal. Represents the wavelet basis functions The result of scaling and translation It is the scaling parameter. It is the translation scale parameter; A noise threshold is set, and sub-band coefficients are generated based on the noise threshold and wavelet coefficients. The formula used is as follows: ； in, This represents the subband coefficients after processing. Indicates the noise threshold; The processed subband coefficients are then subjected to inverse wavelet transform to generate the denoised signal, based on the following formula: ； in, This represents the signal after noise reduction. Indicates the sub-band coefficient. Represents the wavelet basis functions The result of scaling and translation It is the scaling parameter. It is the translation scale parameter.

4. The speech recognition method based on artificial intelligence according to claim 1, characterized in that: The principle underlying the generation of Mel frequency cepstral coefficients in step 2 is as follows: First, the denoised signal is pre-emphasized using the following formula: ； in, This indicates the time of the pre-emphasized signal. The corresponding value, This indicates the time of the denoised signal. The corresponding value, This indicates the time of the denoised signal. The corresponding value, Indicates the pre-emphasis coefficient. Represents a time variable; After the pre-emphasized signal is divided into short time frames, windowing is applied to each frame using the following formula: ； ； in, This represents the result after windowing the segmented frame signal. This represents the segmented frame signal. Represents a time variable. This represents the Hamming window function. Indicates a time index. Indicates frame length; The formula used to convert the windowed frame signal from a time-domain signal to a frequency-domain signal is: ； Among them, Z Represents frequency domain signals, Represents a time-domain signal. Indicates frame length, Represents the imaginary unit. This represents the frequency index, with a value range of [0, ...]. N -1], This represents a time index, with a value range of [0, ...]. N -1]; The formula for the energy output of each filter is as follows: ； in, Indicates the first The value of a Mel filter, Indicates frequency index, Indicates the first The frequency of the Mel filter; The logarithm of the energy output of each filter is taken to generate the logarithmic energy. Then, a discrete cosine transform is performed on the logarithmic energy to generate the Mel-frequency cepstral coefficients. The formula used is as follows: ； ； Among them, among them, Indicates the first The logarithmic energy of each filter, Represents frequency domain signals, Indicates the first The value of a Mel filter, Indicates frequency index, Indicates frame length, Indicates the first Mel frequency cepstral coefficients, Indicates the number of Mel filters. Indicates the first The logarithmic energy of a Mel filter Indicates the index of the Mel filter. Indicates the index of the Mel frequency cepstral coefficients; The formulas used to calculate the first and second differences of the Mel frequency cepstral coefficients are as follows: ； ； in, Indicates the first First-order difference of Mel frequency cepstral coefficients Indicates the first The second difference of the cepstral coefficients of the Mel frequency Indicates the frame offset. The size of the difference window is determined by the size of the difference window. The specific value is determined based on the actual application scenario; The principle underlying the synthesis of Mel frequency cepstral coefficients, first-order differences, and second-order differences into a single eigenvector is as follows: ； in, This represents the final speech feature vector. Indicates the first Mel frequency cepstral coefficients, This represents the first-order difference of the cepstral coefficient at the th Mel frequency. Indicates the first The second difference of the cepstral coefficients of the Mel frequency This indicates the number of Mel frequency cepstral coefficients.

5. The speech recognition method based on artificial intelligence according to claim 1, characterized in that: The formula used in step 3 to convert the original image to grayscale and generate the first recognition image is as follows: ； in, Represents the grayscale value of a pixel. This represents the red channel value of a pixel in the original image. This represents the green channel value of a pixel in the original image. This represents the blue channel value of a pixel in the original image.

6. The speech recognition method based on artificial intelligence according to claim 1, characterized in that: The principle behind comparing the grayscale values ​​of the center pixel with those of its neighboring pixels in step 4 to generate a binary number is as follows: For the center pixel with 8 neighboring pixels, it can be represented as: ； The formula used to determine the value of a binary number is: ； in, This represents the grayscale value of the center pixel. Represents the grayscale value of neighboring pixels. It is a comparison formula that judges based on the size relationship between the center pixel and its neighboring pixels; The formula used to convert a binary number into the LBP value of a pixel is: ； in, The LBP value represents the center pixel. Indicates the number of neighboring pixels. This represents a comparison formula that judges based on the size relationship between the center pixel and its neighboring pixels.

7. A speech recognition system based on artificial intelligence, characterized in that: The system is used to implement the artificial intelligence-based speech recognition method according to any one of claims 1-6, comprising: The voice acquisition module is used to acquire the raw voice signal, perform noise reduction processing on the raw voice signal, decompose the voice signal into multiple sub-bands of different frequencies through wavelet transform, each sub-band corresponds to the information of the signal at different time scales, perform threshold processing on the high-frequency sub-bands to reduce noise components, and recombine the processed sub-bands to obtain the denoised voice signal. The speech feature extraction module is used to pre-emphasize the denoised speech signal through a high-pass filter, divide the pre-emphasized speech signal into short frames of 20~40ms, overlap the frames with 50% of the frame length, window the signal of each frame, and transform the windowed frame signal from the time domain signal to the frequency domain signal through a fast Fourier transform. The frequency domain signal is then passed through a Mel filter to generate Mel cepstral coefficients, and the Mel cepstral coefficients and their first-order and second-order differences are grouped together to form a speech feature vector. The image acquisition module is used to acquire the speaker's lip movement information in real time as the raw visual image, perform grayscale processing on the raw image, and generate the first recognition image; The image processing module is used to normalize the pixels of the first recognition image. For each pixel in the first recognition image, it selects eight neighboring pixels around the pixel, compares the gray values ​​of the center pixel with those of its neighboring pixels, generates a binary number, converts the binary number into a decimal number to generate the LBP value of the center pixel, and calculates the LBP values ​​of all pixels in the first recognition image to form a new image, called the second recognition image. The image feature extraction module is used to divide the second recognition image into non-overlapping units of 8×8 pixels. Units that do not meet the size requirements are directly cropped and discarded. The LBP value distribution of all pixels in each unit is statistically analyzed to generate a histogram of a unit. The histograms of all units are concatenated end to end in the order of the units to generate an image feature vector, which serves as the final feature representation of the visual signal. The feature fusion module is used to convert speech feature vectors and image feature vectors into matrix representations, and dynamically adjust the weights of speech features and image features through a cross-modal attention mechanism to generate a fusion weight matrix. The model building module is used to build a deep learning network model. It uses the original speech signal and the original visual image as the training set, and the speech command corresponding to the fusion weight matrix as the label, and inputs it into the deep learning network model for training. The speech recognition module inputs real-time collected speech signals and visual images into the trained deep learning network model to obtain the current fusion weight matrix, and determines the current speech command based on the fusion weight matrix.

Images