AI voice recognition method and tablet computer

Abstract

The invention belongs to the field of speech recognition, and provides an AI speech recognition method and a tablet personal computer, and the method comprises the steps: extracting the time domain information of an input one-dimensional audio signal, and converting the time domain information into a two-dimensional Mel spectrogram according to the Mel scale; the two-dimensional Mel spectrogram is input into a convolutional neural network model, the convolutional neural network model obtains a three-dimensional feature map by using binarization weight and through sliding window operation and convolution operation during feature extraction, a remodeling layer remodels the three-dimensional feature map into two-dimensional features, a full connection layer classifies the two-dimensional features, and the two-dimensional features are classified into a three-dimensional feature map; mapping to a probability space of a preset Chinese pinyin factor tag, and converting an output value of the full connection layer into probability distribution of each phoneme by the Softmax layer; inputting the output probability distribution of each phoneme into a connection time sequence classification decoder, and converting the optimal phoneme sequence into a Chinese text; according to the invention, the problems of high calculation complexity and high hardware resource demand in traditional speech recognition can be solved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to an AI speech recognition method and a tablet computer. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technology, speech recognition has been widely used as an important part of human-computer interaction. In the field of speech recognition, traditional methods rely on the joint optimization of acoustic models, language models and decoders, while end-to-end models have gradually become mainstream in recent years. These methods extract speech features through convolutional neural networks (CNNs), recurrent neural networks (RNNs) or self-attention mechanisms, directly mapping speech signals to text sequences, significantly improving the accuracy and adaptability of the system.

[0003] Convolutional neural networks have been widely used in the feature extraction stage of speech recognition due to their powerful feature extraction capabilities. However, traditional high-precision convolutional neural networks rely on floating-point weights for calculations, resulting in excessive computational complexity and storage requirements, especially on resource-constrained embedded devices or mobile devices (such as mobile phones and tablets), which are difficult to meet the requirements of real-time performance and low power consumption. Summary of the invention

[0004] In view of the above technical problems, the present invention provides an AI speech recognition method and a tablet computer to solve the problems of high computational complexity and high hardware resource requirements in traditional speech recognition, and to provide an effective solution for realizing real-time Chinese speech recognition in a low-power hardware environment.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present invention, an AI speech recognition method is disclosed, the method comprising: Extract the time domain information of the input one-dimensional audio signal, perform spectrum analysis using short-time Fourier transform, and convert it into a two-dimensional Mel spectrogram according to the Mel scale; Input the generated two-dimensional Mel spectrum map into a convolutional neural network model to obtain the probability distribution of the phoneme corresponding to the one-dimensional audio signal. The convolutional neural network model includes multiple convolutional layers, activation layers, pooling layers, reshaping layers, fully connected layers, and Softmax layers. The multiple convolutional layers extract local features of the two-dimensional Mel spectrum map, and perform layer-by-layer processing to deepen the feature representation. The size of each convolution kernel is 3*3. Binarization weights and sliding window operations are used when extracting features. Convolution operations obtain three-dimensional feature maps. The activation layer performs nonlinear transformation on the three-dimensional feature map to improve feature expression capabilities. The pooling layer reduces the number of time frames and frame segments of the three-dimensional feature map by downsampling to compress the spatial representation of features. The binarization weight is -1 or +1. The reshaping layer reshapes the three-dimensional feature map into two-dimensional features. The fully connected layer classifies the two-dimensional features and maps them to the probability space of preset Chinese pinyin factor labels. The Softmax layer converts the output value of the fully connected layer into the probability distribution of each phoneme. The probability distribution of each output phoneme is input into the connection time classification decoder, an optimal phoneme sequence is generated based on a dynamic programming algorithm, and the optimal phoneme sequence is converted into Chinese text based on a mapping rule from pinyin to Chinese characters.

[0007] Further, when converting the extracted one-dimensional audio signal into the two-dimensional Mel-spectrogram, comprising; The time domain information of the one-dimensional audio signal is divided into a number of time frames of fixed length, and short-time Fourier transform is applied to each time frame to obtain frequency domain information. The power density of the spectrum is calculated, and then the frequency domain information is weighted using a Mel filter bank according to the perception characteristics of the human ear to different frequencies. The high-dimensional frequency domain information is compressed and mapped to the Mel scale to generate the two-dimensional Mel spectrum graph.

[0008] Furthermore, when the convolution layer operates the two-dimensional Mel-spectrogram to obtain the three-dimensional feature map, the following formula is calculated: ; is the two-dimensional Mel-spectrogram; is the bias term for each channel; is the weight parameter of the convolution kernel, which is binarized to -1 or +1, expressed as: ; in, is the full precision weight, is the binarization weight; exist middle, is the ordinate of the pixel in the three-dimensional feature map, is the horizontal coordinate of the pixel in the three-dimensional feature map, is the first channel, corresponding to The operation result of the convolution kernel; exist middle, is the position of the convolution kernel on the ordinate, is the position of the convolution kernel on the horizontal axis, is the channel number of the input feature map, corresponding to the convolution kernel processing Input channels, is the number of the convolution kernel, which determines the output channel currently being processed. and The value range is 1-3.

[0009] Furthermore, the two-dimensional Mel-spectrogram includes the number of time frames, the number of Mel-frequency bands, and the number of feature channels.

[0010] Furthermore, the convolutional neural network model is divided into full-precision training and binarization training during training, wherein: In full-precision training, the weights of the convolutional neural network model are optimized using a standard deep learning training method to obtain the full-precision weights; In the binarization training, the convolutional neural network model with the full-precision weights is used as a reference, the full-precision weights are binarized into the binarized weights, the binarized weights are used in forward propagation and back propagation, and the full-precision weights are used in parameter updating.

[0011] Furthermore, when the Softmax layer calculates the distribution probability, the calculation formula is as follows: ; is the output of the fully connected layer; is the probability distribution, The pre-set Chinese pinyin phoneme labels.

[0012] Furthermore, when converting the probability distribution of each phoneme into the corresponding Chinese text, it includes: After the probability distribution of each output phoneme is input to the connection temporal classification decoder, the connection temporal classification decoder uses a dynamic programming algorithm to calculate the probabilities of all possible alignment paths from the input phoneme sequence to the target text, and generates the optimal phoneme sequence by maximizing the path probability. The loss function of the connection temporal classification decoder is: ; represents possible alignment paths, is the target sequence The set of all legal paths of is the time step, is the time step Upper path probability; The optimal phoneme sequence is extracted from the probability distribution of the output of the connection timing classification decoder through greedy decoding or search algorithm, and the optimal phoneme sequence is matched to the Chinese text output based on the mapping rule from pinyin to Chinese characters.

[0013] According to another aspect of the present invention, an AI speech recognition tablet computer is disclosed, the tablet computer includes an AI speech recognition chip, and the AI ​​speech recognition chip includes: a storage module, the storage module is used to store intermediate feature maps, weights and activation values; a convolution calculation module, the convolution calculation module is used to complete a binary convolution operation; a logic control module, the logic control module is used to implement dynamic configuration to adjust the convolution kernel size and the number of channels; a processing module, the processing module is used to execute the method as described above.

[0014] The technical solution disclosed in this disclosure has the following beneficial effects: The present invention significantly reduces the computational complexity and greatly reduces the power consumption and volume of the hardware by binarizing the weights of the convolutional layer into +1 and −1, and converting the traditional floating-point multiplication operation into a simple logical operation. By combining Mel-spectrogram feature extraction, binarized convolutional neural network and connected temporal classification decoder, the problem of inconsistent input and output sequence lengths in speech recognition is solved, and the classification accuracy of the model for Chinese pinyin phonemes is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of an AI speech recognition method in an embodiment of this specification; Figure 2 This is a structural block diagram of an AI voice recognition tablet computer in an embodiment of this specification. DETAILED DESCRIPTION

[0016] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0017] In addition, the accompanying drawings are only schematic illustrations of the present disclosure. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0018] In one embodiment, if Figure 1 As shown, this specification provides an AI speech recognition method, and the execution subject of the method can be a computer, a server, a mobile phone, a tablet computer, etc. The method can specifically include steps S101 to S103: In step S101, the time domain information of the input one-dimensional audio signal is extracted, and the spectrum is analyzed using short-time Fourier transform, and converted into a two-dimensional Mel-spectrogram according to the Mel scale.

[0019] During the conversion, the time domain information of the one-dimensional audio signal is divided into several time frames of fixed length, and a short-time Fourier transform is applied to each time frame to obtain frequency domain information. The power density of the spectrum is calculated, and the frequency domain information is weighted using a Mel filter bank according to the human ear's perception characteristics of different frequencies. The high-dimensional frequency domain information is compressed and mapped to the Mel scale to generate the two-dimensional Mel spectrum graph.

[0020] Among them, the two-dimensional Mel spectrum graph is expressed as , where H is the number of time frames, W is the number of Mel frequency bands, and N is the number of feature channels. and The padding is for the height and length of the two-dimensional Mel spectrum graph. The feature points on the boundary will be ignored by the convolution kernel without padding. Through the padding operation, the boundary features can be involved in the convolution calculation to improve the integrity of feature extraction. When the convolution kernel slides to the boundary, there is not enough input data for calculation. Through padding, enough data can be provided to the convolution kernel to maintain the size of the output feature map.

[0021] In step S102, the generated two-dimensional Mel-spectrogram is input into a convolutional neural network model to obtain the probability distribution of the phonemes corresponding to the one-dimensional audio signal. The convolutional neural network model includes multiple convolutional layers, activation layers, pooling layers, reshaping layers, fully connected layers, and Softmax layers. The multiple convolutional layers extract local features of the two-dimensional Mel-spectrogram and perform layer-by-layer processing to deepen the feature representation. The size of each convolution kernel is 3*3. Binarization weights and sliding window operations are used when extracting features. The convolution operation obtains a three-dimensional feature map. The activation layer performs a nonlinear transformation on the three-dimensional feature map to improve the feature expression capability. The pooling layer reduces the number of time frames and frame segments of the three-dimensional feature map by downsampling to compress the spatial representation of the feature. The binarization weight is -1 or +1. The reshaping layer reshapes the three-dimensional feature map into a two-dimensional feature. The fully connected layer classifies the two-dimensional feature and maps it to the probability space of the preset Chinese pinyin factor label. The Softmax layer converts the output value of the fully connected layer into the probability distribution of each phoneme.

[0022] Among them, the generated two-dimensional Mel spectrum Input into the convolutional neural network model, the convolution layer extracts features from the local area of ​​the two-dimensional Mel spectrum map, and processes layer by layer to deepen the feature representation. The size of each convolution kernel is fixed to 3*3. Through the sliding window operation, the output feature map is calculated according to the following formula: ; is the two-dimensional Mel-spectrogram; is the bias term for each channel; is the weight parameter of the convolution kernel. To reduce the computational complexity, all convolution layer weights are 1 byte, that is, they are binarized to -1 or +1, expressed as: ; in, is the full precision weight, is the binarization weight; exist middle, is the ordinate of the pixel in the three-dimensional feature map, is the horizontal coordinate of the pixel in the three-dimensional feature map, is the first channel, corresponding to The operation result of the convolution kernel; exist middle, is the position of the convolution kernel on the ordinate, is the position of the convolution kernel on the horizontal axis, is the channel number of the input feature map, corresponding to the convolution kernel processing Input channels, is the number of the convolution kernel, which determines the output channel currently being processed. and The value range is 1-3.

[0023] Taking a speech sentence with a sampling rate of 16kHz as an example, it is converted into a two-dimensional Mel spectrum feature map (size H=800, W=64, N=1). The size of each convolution kernel is 3*3, and the binary weight (+1 or -1) is used for calculation. The output feature map is further extracted through activation functions (such as ReLU) and pooling operations to reduce the data size and obtain a more compact three-dimensional feature representation, that is, a three-dimensional feature map (shape is 100,8,256), which is reshaped into a two-dimensional representation (shape is 100,2048) through the reshaping layer. The two-dimensional feature is mapped to the probability space of m Chinese pinyin phoneme labels through the fully connected layer to obtain a probability distribution. In order to reduce computing resources, the weight of the fully connected layer can be quantized to 8-bit fixed-point numbers, and then the distribution probability is calculated through the Softmax layer. The calculation formula is as follows: ; is the output of the fully connected layer; is the probability distribution, The pre-set Chinese pinyin phoneme labels.

[0024] Finally, the model converts the input two-dimensional Mel-spectrogram into the probability distribution of phonemes, providing a basis for subsequent pinyin decoding and text conversion.

[0025] In step S103, the probability distribution of each phoneme output is input into the connection time classification decoder, an optimal phoneme sequence is generated based on a dynamic programming algorithm, and the optimal phoneme sequence is converted into Chinese text based on a mapping rule from pinyin to Chinese characters.

[0026] Among them, when converting the probability distribution of each phoneme into the corresponding Chinese text, it includes: after the output probability distribution of each phoneme is input into the connection time series classification decoder, the connection time series classification decoder uses a dynamic programming algorithm to calculate the probability of all possible alignment paths from the input phoneme sequence to the target text, and generates the optimal phoneme sequence by maximizing the path probability. The loss function of the connection time series classification decoder is: ; represents possible alignment paths, is the target sequence The set of all legal paths of is the time step, is the time step Upper path probability; The optimal phoneme sequence is extracted from the probability distribution of the output of the connection timing classification decoder through greedy decoding or search algorithm, and the optimal phoneme sequence is matched to the Chinese text output based on the mapping rule from pinyin to Chinese characters.

[0027] In one embodiment, the convolutional neural network model is trained by full-precision training and binarization training, wherein: In full-precision training, the weights of the convolutional neural network model are optimized using a standard deep learning training method to obtain the full-precision weights; In the binarization training, the convolutional neural network model with the full-precision weights is used as a reference, the full-precision weights are binarized into the binarized weights, the binarized weights are used in forward propagation and back propagation, and the full-precision weights are used in parameter updating.

[0028] Among them, the full precision weight Binarization is the binarization weight As mentioned above, the binarization operation simplifies the computation and storage requirements. After binarization, the binarized weights are used in the forward propagation. Convolution operations are performed to calculate the output value and loss function of the model. Binarized weights greatly accelerate convolution operations, and floating-point operations are replaced by simple logical operations (such as XNOR and addition). In back propagation, gradient calculation and propagation are still based on binary weights. , ensuring that the calculation process is consistent with the binary model. In parameter updates, the full-precision weights are still used Update model parameters to avoid the negative impact of binary weights on model optimization. Update full-precision weights Binarize again to generate new binary weights , used for the next iteration. Full-precision training provides high-quality initialization parameters, ensuring that the binarization model can still achieve good performance under low computational complexity conditions. Binarization training reduces the impact of binarization on the model's learning ability by introducing a parameter update mechanism for full-precision weights, thereby achieving a balance between efficient computing and high prediction accuracy.

[0029] Based on the same idea, Figure 2 As shown, an exemplary embodiment of the present disclosure also provides an AI speech recognition tablet computer, wherein the tablet computer 200 includes an AI speech recognition chip 210, and the AI ​​speech recognition chip 210 includes: a storage module 211, wherein the storage module 211 is used to store intermediate feature maps, weights and activation values; a convolution calculation module 212, wherein the convolution calculation module 212 is used to complete a binary convolution operation; a logic control module 213, wherein the logic control module 213 is used to implement dynamic configuration to adjust the convolution kernel size and the number of channels; and a processing module, wherein the processing module 214 is used to execute the method as described above.

[0030] In the tablet computer provided in this embodiment, by binarizing the convolutional layer weights into +1 and −1, the traditional floating-point multiplication operation is converted into a simple logical operation, which significantly reduces the computational complexity and greatly reduces the hardware power consumption and volume; by combining Mel spectrum feature extraction, binarized convolutional neural network and connection time series classification decoder, the problem of inconsistent input and output sequence lengths in speech recognition is solved, and the classification accuracy of the model for Chinese pinyin phonemes is improved.

[0031] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary implementation of the present disclosure.

[0032] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0033] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. An AI speech recognition method, characterized in that: The method comprises: Extract the time domain information of the input one-dimensional audio signal, perform spectrum analysis using short-time Fourier transform, and convert it into a two-dimensional Mel spectrogram according to the Mel scale; Input the generated two-dimensional Mel spectrum map into a convolutional neural network model to obtain the probability distribution of the phoneme corresponding to the one-dimensional audio signal. The convolutional neural network model includes multiple convolutional layers, activation layers, pooling layers, reshaping layers, fully connected layers, and Softmax layers. The multiple convolutional layers extract local features of the two-dimensional Mel spectrum map, and perform layer-by-layer processing to deepen the feature representation. The size of each convolution kernel is 3*3. Binarization weights and sliding window operations are used when extracting features. Convolution operations obtain three-dimensional feature maps. The activation layer performs nonlinear transformation on the three-dimensional feature map to improve feature expression capabilities. The pooling layer reduces the number of time frames and frame segments of the three-dimensional feature map by downsampling to compress the spatial representation of features. The binarization weight is -1 or +1. The reshaping layer reshapes the three-dimensional feature map into two-dimensional features. The fully connected layer classifies the two-dimensional features and maps them to the probability space of preset Chinese pinyin factor labels. The Softmax layer converts the output value of the fully connected layer into the probability distribution of each phoneme. The probability distribution of each output phoneme is input into the connection time classification decoder, an optimal phoneme sequence is generated based on a dynamic programming algorithm, and the optimal phoneme sequence is converted into Chinese text based on a mapping rule from pinyin to Chinese characters.

2. The AI ​​speech recognition method according to claim 1, characterized in that: When converting the extracted one-dimensional audio signal into the two-dimensional Mel-spectrogram, comprising: The time domain information of the one-dimensional audio signal is divided into a number of time frames of fixed length, and short-time Fourier transform is applied to each time frame to obtain frequency domain information. The power density of the spectrum is calculated, and then the frequency domain information is weighted using a Mel filter bank according to the perception characteristics of the human ear to different frequencies. The high-dimensional frequency domain information is compressed and mapped to the Mel scale to generate the two-dimensional Mel spectrum graph.

3. The AI ​​speech recognition method according to claim 1, characterized in that: When the convolution layer operates the two-dimensional Mel-spectrogram to obtain the three-dimensional feature map, the following formula is calculated: ； is the two-dimensional Mel-spectrogram; is the bias term for each channel; is the weight parameter of the convolution kernel, which is binarized to -1 or +1, expressed as: ； in, is the full precision weight, is the binarization weight; exist middle, is the ordinate of the pixel in the three-dimensional feature map, is the horizontal coordinate of the pixel in the three-dimensional feature map, is the first channel, corresponding to The operation result of the convolution kernel; exist middle, is the position of the convolution kernel on the ordinate, is the position of the convolution kernel on the horizontal axis, is the channel number of the input feature map, corresponding to the convolution kernel processing Input channels, is the number of the convolution kernel, which determines the output channel currently being processed. and The value range is 1-3.

4. An AI speech recognition method according to any one of claims 2 or 3, characterized in that: The two-dimensional Mel spectrum graph includes the number of time frames, the number of Mel frequency bands, and the number of feature channels.

5. The AI ​​speech recognition method according to claim 3, characterized in that: The convolutional neural network model is trained in two ways: full-precision training and binarization training. In full-precision training, the weights of the convolutional neural network model are optimized using a standard deep learning training method to obtain the full-precision weights; In the binarization training, the convolutional neural network model with the full-precision weights is used as a reference, the full-precision weights are binarized into the binarized weights, the binarized weights are used in forward propagation and back propagation, and the full-precision weights are used in parameter updating.

6. The AI ​​speech recognition method according to claim 1, characterized in that: When the Softmax layer calculates the distribution probability, the calculation formula is as follows: ; is the output of the fully connected layer; is the probability distribution, The pre-set Chinese pinyin phoneme labels.

7. The AI ​​speech recognition method according to claim 1, characterized in that: When converting the probability distribution of each phoneme into the corresponding Chinese text, it includes: After the probability distribution of each output phoneme is input to the connection temporal classification decoder, the connection temporal classification decoder uses a dynamic programming algorithm to calculate the probabilities of all possible alignment paths from the input phoneme sequence to the target text, and generates the optimal phoneme sequence by maximizing the path probability. The loss function of the connection temporal classification decoder is: ； represents possible alignment paths, is the target sequence The set of all legal paths of is the time step, is the time step Upper path The probability of The optimal phoneme sequence is extracted from the probability distribution of the output of the connection timing classification decoder through greedy decoding or search algorithm, and the optimal phoneme sequence is matched to the Chinese text output based on the mapping rule from pinyin to Chinese characters.

8. An AI voice recognition tablet computer, characterized in that: The tablet computer includes an AI voice recognition chip, and the AI ​​voice recognition chip includes: A storage module, wherein the storage module is used to store intermediate feature maps, weights and activation values; A convolution calculation module, wherein the convolution calculation module is used to complete a binarization convolution operation; A logic control module, which is used to implement dynamic configuration to adjust the convolution kernel size and the number of channels; A processing module, wherein the processing module is used to execute the method according to any one of claims 1 to 6.

Images