Mobile phone interaction method based on improved Transform
By improving the structure of the Transformer model and combining relative position encoding and convolutional neural networks, the real-time and accuracy issues in mobile phone voice interaction were resolved, thereby improving the performance and user experience of the voice interaction system.
Patent Information
- Application Number
- CN202410959471.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-26
AI Technical Summary
Existing Transformer models have limitations in handling mobile phone voice interaction, including insufficient support for scenarios with high real-time requirements and the need to improve speech recognition accuracy.
An improved Transformer model is constructed by introducing an input embedding layer, a pre-convolutional layer, a multi-head self-attention mechanism layer, a feedforward network layer, a cross-attention mechanism layer, a gating mechanism layer, and an output layer, combined with relative position encoding and a convolutional neural network to improve network performance.
The model's context awareness and local feature extraction capabilities have been enhanced, improving the accuracy and response speed of the voice interaction system and providing a more natural and smooth user experience.
Smart Images

Figure CN119028339A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a mobile phone interaction method based on an improved Transformer. Background Art
[0002] With the popularity of smartphones, the ways of interaction between users and mobile phones have gradually diversified, including touch screen, voice recognition, gesture recognition, etc. Among them, voice interaction has become one of the important ways of interaction for users due to its convenience and speed. Existing smartphone voice interaction systems mainly rely on traditional RNN, LSTM and other models, which are inefficient and ineffective when processing long sequences. In recent years, the Transformer model has received widespread attention due to its excellent performance in natural language processing tasks.
[0003] The Transformer model is a deep learning architecture that has revolutionized the field of natural language processing (NLP) since its introduction in 2017. The model was proposed by Vaswani et al. and has become one of the most influential models in the NLP community.
[0004] In general, traditional sequential models, such as recurrent neural networks (RNNs), have limitations in capturing long-range dependencies and implementing parallel computations. To address these issues, the Transformer model introduces a self-attention mechanism that allows the model to weigh the importance of different positions in the input sequence when generating outputs.
[0005] The Transformer model can better handle long-distance dependencies through the advantages of self-attention mechanism and parallel computing, improving the training and reasoning efficiency of the model. It has achieved significant performance improvements in multiple NLP tasks such as machine translation, text summarization, and question-answering systems.
[0006] However, the traditional Transformer model still has some shortcomings when processing mobile phone voice interaction, such as insufficient support for scenarios with high real-time requirements and the need to improve the accuracy of voice recognition. Therefore, it is necessary to improve the existing Transformer model to enhance its application effect in mobile phone interaction. Summary of the invention
[0007] The purpose of the present invention is to provide a mobile phone interaction method based on an improved Transformer, which improves the performance of a voice interaction system and user experience by constructing an improved Transformer model.
[0008] The technical solution of the present invention is:
[0009] The mobile phone interaction method based on the improved Transformer includes the following six steps:
[0010] Step 1: Dataset collection: Dataset collection is the basis for Transformer model training. In the field of mobile phone voice interaction, different types of voice datasets need to be collected.
[0011] Step 2: Data preprocessing: Preprocess the collected data set, including noise elimination, speech segmentation, feature extraction, and data set partitioning, to obtain the processed training set time domain signal and test set time domain signal.
[0012] Step 3: Model construction: Construct an improved Transformer architecture, the improved Transformer model includes an input embedding layer, a pre-convolutional layer after the input embedding layer, a multi-head self-attention mechanism layer after the pre-convolutional layer feedforward network layer, a feedforward network layer after the multi-head self-attention mechanism layer, a cross-attention mechanism layer after the feedforward network layer, a gating mechanism layer between the feedforward network layer and the cross-attention mechanism layer, and an output layer after the cross-attention mechanism layer. Introduce relative position encoding and convolutional neural network (CNN) in the Transformer architecture to improve network performance.
[0013] Step 4: Model training: Use the training data to train the improved Transformer model. During the training process, it is necessary to monitor the performance of the model on the training set and the validation set to prevent overfitting.
[0014] Step 5: Model evaluation: Use the test data to evaluate the model performance and calculate the interaction accuracy.
[0015] Step 6: Apply the model: Apply the trained improved Transformer model to real-time voice interaction tasks, convert the model's response into natural language, and feed it back to the user in the form of text or voice, including speech synthesis (TTS) and text processing.
[0016] A further technical solution of the present invention is: the data set types included in the step 1 data set collection include:
[0017] General speech recognition datasets, containing user voice input and its corresponding text transcription, used to train speech recognition models. Example datasets: LibriSpeech: Contains thousands of hours of English audiobook recordings and their transcriptions. MozillaCommon Voice: A large amount of multilingual voice data contributed by volunteers around the world.
[0018] Command and control datasets contain user voice commands and their corresponding text instructions, which are used to train voice assistants or control systems. Example datasets: Google Speech Commands: Contains tens of thousands of simple voice commands, such as "yes", "no", "stop", etc.
[0019] Dialogue datasets, including multi-round conversation recordings and transcriptions between users and voice assistants, are used to train dialogue systems. Example datasets: DSTC (Dialog State Tracking Challenge): Contains multi-round conversation data for dialogue state tracking.
[0020] The emotional speech dataset contains speech data annotated with emotions and is used to train emotion recognition models.
[0021] Example datasets: IEMOCAP: Contains a variety of speech data annotated with emotions. RAVDESS: Contains emotional speech and singing data annotated with emotions.
[0022] Multilingual datasets, containing speech data and their transcriptions in multiple languages, for training multilingual speech recognition models. Example datasets: Multilingual LibriSpeech (MLS): Contains audiobook recordings and their transcriptions in multiple languages. Google Multilingual Speech: Contains multilingual speech data.
[0023] A further technical solution of the present invention is: the data pre-processing in step 2 mainly includes noise elimination, speech segmentation, feature extraction, data set division, etc.
[0024] Noise elimination: Use techniques such as spectral subtraction to reduce the impact of background noise on speech signals and improve the signal-to-noise ratio.
[0025] Speech segmentation: Divide the continuous speech signal into several short-time windows for subsequent feature extraction and processing.
[0026] Feature extraction: Use methods such as Mel-frequency cepstral coefficients (MFCC) to extract the feature vector of the speech signal as input for subsequent models.
[0027] Dataset partitioning: The dataset is divided into a training set and a test set. The commonly used partitioning ratio is 80% training set and 20% test set. The training set is used for model training, and the test set is used for model evaluation.
[0028] A further technical solution of the present invention is: the model construction in step three mainly includes the following parts: input embedding layer: convert the speech signal into a feature vector (such as MFCC or Mel spectrum), perform linear transformation on the feature vector, and map it to a higher dimensional representation space. Use positional encoding to add position information to maintain the order of the time series. The number of layers is set to 1, the input size feature vector dimension is 512, and the output size feature vector dimension is 512.
[0029] Pre-convolutional layer: A convolutional neural network (CNN) is added after the input embedding layer to extract local features. The convolutional layer can capture dependencies in a short period of time and improve the ability to extract local features.
[0030] Multi-head self-attention mechanism layer: Use the multi-head self-attention mechanism to calculate the self-attention score of each position in the input sequence. Introduce relative positional encoding to replace the traditional absolute position encoding, so that the model can better capture relative position information.
[0031] Feed-forward network layer: Add two fully connected layers and ReLU activation function after each attention head. Use residual connection and layer normalization to stabilize the training process.
[0032] Cross-attention mechanism layer: A cross-attention layer is introduced to enable the representation of the current sequence to pay attention to the user's historical input or other auxiliary information. This helps improve the model's performance in context perception and multi-round dialogue.
[0033] Gating mechanism layer: Introduce a gating mechanism between the self-attention and feedforward networks, such as Gated Linear Units (GLU), to dynamically adjust the information flow and enhance the model's ability to selectively process important information.
[0034] Output layer: Use a linear layer to map the final feature vector to the target output space, such as vocabulary or other classification labels.
[0035] A further technical solution of the present invention is: the model training in step 4 includes two parts, namely, defining hyperparameters and training the model.
[0036] Define hyperparameters: Select the hyperparameters for training, such as the number of training rounds (epochs), batch size (batch_size), number of attention heads, number of Transformer encoder layers, etc.
[0037] Model training: Use the fit method to train the model. During the training process, the model automatically adjusts the weights to minimize the loss function. At the same time, the performance of the model is monitored through the validation set to prevent overfitting.
[0038] The optimizer of the improved Transformer network established is the Stochastic Gradient Descent (SGD) optimization model;
[0039] A further technical solution of the present invention is: the model evaluation in step 5 uses the evaluate method to evaluate the performance of the model on the test set, and the evaluation result includes the loss value and the accuracy rate. The interactive accuracy on the test set is printed to evaluate the performance of the model.
[0040] A further technical solution of the present invention is: the step six application model applies the trained improved Transformer model to real-time voice interaction tasks, converts the model's response into natural language based on the user's input, and feeds back to the user in the form of text or voice, including speech synthesis (TTS) and text processing.
[0041] Effect of the invention: The improved Transformer network structure described above includes input embedding, pre-convolution, multi-head self-attention, feedforward network, cross-attention, gating mechanism and output layer. The context-awareness ability is enhanced. This enhanced context-awareness ability enables the model to perform better when processing multi-round conversations, and can accurately capture user intentions and semantic changes, thereby providing a more natural and smooth interactive experience. The local feature extraction capability is improved. The pre-convolution layer is added after the input embedding layer to effectively extract the local features of the speech signal. By introducing a gating mechanism (such as a gated linear unit) between the self-attention and feedforward networks, the model has the ability to dynamically adjust the information flow. This dynamic control mechanism allows the model to automatically adjust the processing and transmission of important information according to the characteristics of the input data, thereby improving the selective processing capability of key information, reducing the interference of irrelevant or redundant information on the model, and further improving the accuracy and response speed of the interactive system.
[0042] In summary, the present invention significantly improves the performance and user experience of the voice interaction system through a series of structural optimizations and innovative designs. Its enhanced context awareness, local feature extraction capabilities, and dynamic information flow control make this method particularly effective in processing complex voice interaction tasks, and has broad application prospects and huge commercial value. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 This is a flow chart of the mobile phone interaction method based on the improved Transformer. DETAILED DESCRIPTION
[0044] The technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described examples are only part of the examples of the invention, not all of the examples. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Specific implementation method one:
[0046] The technical solution of the present invention is: the mobile phone interaction method based on the improved Transformer, the specific process is as follows Figure 1 As shown, it includes the following six steps:
[0047] S1: Collect different types of speech datasets;
[0048] S2: Preprocessing the speech data set to obtain a processed training set time domain signal and a test set time domain signal; the preprocessing includes noise elimination, speech segmentation, feature extraction and data set division;
[0049] S3: construct an improved Transformer model, wherein the improved Transformer model includes an input embedding layer, a pre-convolutional layer after the input embedding layer, a multi-head self-attention mechanism layer after the pre-convolutional layer feed-forward network layer, a feed-forward network layer after the multi-head self-attention mechanism layer, a cross-attention mechanism layer after the feed-forward network layer, a gating mechanism layer between the feed-forward network layer and the cross-attention mechanism layer, and an output layer after the cross-attention mechanism layer;
[0050] S4: training the improved Transformer model on the training set time domain signal of step 2;
[0051] S5: using the test set time domain signal of step 2 to evaluate the performance of the improved Transformer model after the training convergence of step 3, and calculating the accuracy of recognition of different speech segments;
[0052] S6: Apply the improved Transformer model evaluated in step 5 to the real-time voice interaction task on a mobile phone.
[0053] Specifically,
[0054] S1 is specifically: Dataset collection: Dataset collection is the basis for training the Transformer series models. In the field of mobile phone voice interaction, different types of voice datasets need to be collected.
[0055] S2 is specifically: Data preprocessing: preprocessing the collected data set, including noise elimination, speech segmentation, feature extraction, etc., to obtain a processed time domain signal.
[0056] S3 is specifically: Model construction: Build an improved Transformer architecture, introduce relative position encoding and convolutional neural network (CNN) into the Transformer architecture to improve network performance.
[0057] S4 is as follows: Model training: Use the processed speech training data to train the improved Transformer model. During the training process, it is necessary to monitor the performance of the model on the training set and the validation set to prevent overfitting.
[0058] S5 is specifically: Model evaluation: Use test data to evaluate the model performance after training convergence and calculate the accuracy of recognition of different speech segments.
[0059] S6 is specifically: Application model: Apply the trained improved Transformer model to real-time voice interaction tasks on mobile phones.
[0060] A further technical solution of the present invention is: the types of data sets included in the S1 data set collection are:
[0061] General speech recognition datasets, containing user voice input and its corresponding text transcription, used to train speech recognition models. Example datasets: LibriSpeech: Contains thousands of hours of English audiobook recordings and their transcriptions. MozillaCommon Voice: A large amount of multilingual voice data contributed by volunteers around the world.
[0062] Command and control datasets contain user voice commands and their corresponding text instructions, which are used to train voice assistants or control systems. Example datasets: Google Speech Commands: Contains tens of thousands of simple voice commands, such as "yes", "no", "stop", etc.
[0063] Dialogue datasets, including multi-round conversation recordings and transcriptions between users and voice assistants, are used to train dialogue systems. Example datasets: DSTC (Dialog State Tracking Challenge): Contains multi-round conversation data for dialogue state tracking.
[0064] The emotional speech dataset contains speech data annotated with emotions and is used to train emotion recognition models.
[0065] Example datasets: IEMOCAP: Contains a variety of speech data annotated with emotions. RAVDESS: Contains emotional speech and singing data annotated with emotions.
[0066] Multilingual datasets, containing speech data and their transcriptions in multiple languages, for training multilingual speech recognition models. Example datasets: Multilingual LibriSpeech (MLS): Contains audiobook recordings and their transcriptions in multiple languages. Google Multilingual Speech: Contains multilingual speech data.
[0067] A further technical solution of the present invention is: the S2 pre-processing data mainly includes noise elimination, speech segmentation, feature extraction, data set division, etc.
[0068] Noise elimination: Use techniques such as spectral subtraction to reduce the impact of background noise on speech signals and improve the signal-to-noise ratio.
[0069] Speech segmentation: Divide the continuous speech signal into several short-time windows for subsequent feature extraction and processing.
[0070] Feature extraction: Use methods such as Mel-frequency cepstral coefficients (MFCC) to extract the feature vector of the speech signal as input for subsequent models.
[0071] Dataset partitioning: The dataset is divided into a training set and a test set. The commonly used partitioning ratio is 80% training set and 20% test set. The training set is used for model training, and the test set is used for model evaluation.
[0072] A further technical solution of the present invention is: the construction of the S3 model mainly includes the following parts:
[0073] Input embedding layer: convert the speech signal into a feature vector (such as MFCC or Mel spectrogram), perform a linear transformation on the feature vector, and map it to a higher-dimensional representation space. Use positional encoding to add position information to maintain the order of the time series. The number of layers is set to 1, the input size feature vector dimension is 512, and the output size feature vector dimension is 512.
[0074] The positional encoding embedding formula is as follows:
[0075]
[0076] Among them, pos represents the position, i represents the dimension, and d model Represents the number of data features.
[0077] Pre-convolutional layer: A convolutional neural network (CNN) is added after the input embedding layer to extract local features. The convolutional layer can capture dependencies in a short period of time and improve the ability to extract local features. The number of layers is set to 1, the input size feature vector dimension is 512*T, and the output size feature vector dimension is 512*T, where T is the number of time steps.
[0078] Multi-head self-attention mechanism layer: Use the multi-head self-attention mechanism to calculate the self-attention score of each position in the input sequence. Introduce relative positional encoding to replace the traditional absolute position encoding, so that the model can better capture relative position information. The number of attention heads is set to 8 based on the number of features of the input speech data. Each attention head calculates attention once. The input size is 512*T and the output size is 512*T.
[0079] The calculation formulas for self-attention and multi-head self-attention are as follows:
[0080]
[0081] MultiHead(Q,K,V)=Concat(head1,…,head h )W O
[0082] head i =Attention(QW i Q ,KW i K ,VW i V )
[0083] in, represents the dimension of the Key matrix, Q, K, and V are the Quary matrix, Key matrix, and Value matrix after the input data is linearly transformed. and is the weight matrix.
[0084] Feedforward network layer: Add two fully connected layers and ReLU activation function after each attention head. Use residual connection and layer normalization to stabilize the training process. The number of layers is set to 2, the input size is 512*T, and the output size is 512*T.
[0085] The calculation formulas for residual connection and layer normalization are as follows:
[0086] sub_layer_output=LayerNorm(x+SubLayer(x))
[0087] The calculation formula of the feedforward network is as follows:
[0088] FFN(x)=max(0,xW1+b1)W2+b2
[0089] Among them, W1 and b1 are the weight and bias of the first fully connected layer, and W2 and b2 are the weight and bias of the second fully connected layer.
[0090] Cross-attention mechanism layer: A cross-attention layer is introduced to enable the representation of the current sequence to pay attention to the user's historical input or other auxiliary information. It helps to improve the model's performance in context perception and multi-round dialogue. The number of layers is set to 1, the input size is 512*T, and the output size is 512*T.
[0091] Gating mechanism layer: Introduce a gating mechanism between the self-attention and feedforward networks, such as Gated Linear Units (GLU), to dynamically adjust the information flow. Enhance the model's ability to selectively process important information. The number of layers is set to 1, the input size is 512*T, and the output size is 512*T.
[0092] Output layer: Use a linear layer to map the final feature vector to the target output space, such as a vocabulary or other classification labels. The number of layers is set to 1, the input size is 512*T, and the output size depends on the task, such as the size of the vocabulary or the number of classification labels.
[0093] The convolution calculation formula is as follows:
[0094]
[0095] Among them, W l conv and are the convolution kernel and bias of the lth layer respectively, and σ is the activation function.
[0096] A further technical solution of the present invention is: the model training in step 4 includes two parts, namely, defining hyperparameters and training the model.
[0097] Define hyperparameters: Select the hyperparameters for training, such as the number of training rounds (epochs), batch size (batch_size), number of attention heads, number of Transformer encoder layers, etc. The commonly used hyperparameter settings are 50 rounds of training with 32 samples per batch.
[0098] Model training: Use the fit method to train the model. During the training process, the model automatically adjusts the weights to minimize the loss function. At the same time, the performance of the model is monitored through the validation set to prevent overfitting.
[0099] The loss function is defined as follows:
[0100]
[0101] Among them, y i is the true label, is the predicted probability.
[0102] The optimizer of the improved Transformer network established is the Stochastic Gradient Descent (SGD) optimization model;
[0103] A further technical solution of the present invention is: the S5 model evaluation uses the evaluate method to evaluate the performance of the model on the test set, and the evaluation result includes the loss value and the accuracy rate. The interactive accuracy on the test set is printed to evaluate the performance of the model.
[0104] A further technical solution of the present invention is: the S6 application model applies the trained improved Transformer model to real-time voice interaction tasks, converts the model's response into natural language based on the user's input, and feeds back to the user in the form of text or voice, including speech synthesis (TTS) and text processing.
[0105] Effect of the invention: The improved Transformer network structure described above includes input embedding, pre-convolution, multi-head self-attention, feedforward network, cross-attention, gating mechanism and output layer. The context-awareness ability is enhanced. This enhanced context-awareness ability enables the model to perform better when processing multi-round conversations, and can accurately capture user intentions and semantic changes, thereby providing a more natural and smooth interactive experience. The local feature extraction capability is improved. The pre-convolution layer is added after the input embedding layer to effectively extract the local features of the speech signal. By introducing a gating mechanism (such as a gated linear unit) between the self-attention and feedforward networks, the model has the ability to dynamically adjust the information flow. This dynamic control mechanism allows the model to automatically adjust the processing and transmission of important information according to the characteristics of the input data, thereby improving the selective processing capability of key information, reducing the interference of irrelevant or redundant information on the model, and further improving the accuracy and response speed of the interactive system.
[0106] In summary, the present invention significantly improves the performance and user experience of the voice interaction system through a series of structural optimizations and innovative designs. Its enhanced context awareness, local feature extraction capabilities, and dynamic information flow control make this method particularly effective in processing complex voice interaction tasks, and has broad application prospects and huge commercial value.
Claims
1. A mobile phone interaction method based on an improved Transformer model, characterized in that: The following steps are involved: Step 1: Collect different types of speech datasets; Step 2: preprocessing the speech data set to obtain a processed training set time domain signal and a test set time domain signal; the preprocessing includes noise elimination, speech segmentation, feature extraction and data set division; Step 3: construct an improved Transformer model, wherein the improved Transformer model includes an input embedding layer, a pre-convolutional layer after the input embedding layer, a multi-head self-attention mechanism layer after the pre-convolutional layer feedforward network layer, a feedforward network layer after the multi-head self-attention mechanism layer, a cross-attention mechanism layer after the feedforward network layer, a gating mechanism layer between the feedforward network layer and the cross-attention mechanism layer, and an output layer after the cross-attention mechanism layer; Step 4: training the improved Transformer model on the training set time domain signal of step 2; Step 5: using the test set time domain signal of step 2 to evaluate the performance of the improved Transformer model after the training convergence of step 3, and calculating the accuracy of recognition of different speech segments; Step 6: Apply the improved Transformer model evaluated in step 5 to the real-time voice interaction task on a mobile phone.
2. The mobile phone interaction method based on the improved Transformer model according to claim 1, characterized in that: The speech data set includes at least one of a general speech recognition data set, a command and control data set, a dialogue data set, an emotional speech data set, an example data set, and a multi-language data set.
3. The mobile phone interaction method based on the improved Transformer model according to claim 1, characterized in that: The input layer is used to convert the signal into a feature vector, perform a linear transformation on the feature vector, and map it to a high-dimensional representation space; and the input layer uses position encoding to add position information to maintain the order of the time series; the pre-convolution layer is used to add a layer of convolutional neural network after the input embedding layer to extract local features; The multi-head attention mechanism layer is used to calculate the self-attention score of each position in the input sequence using a multi-head self-attention mechanism; The feedforward network layer is used to add two fully connected layers and a ReLU activation function after each of the multi-head attention mechanism layers; the cross attention mechanism layer is used to enable the representation of the current sequence to pay attention to the user's historical input or other auxiliary information; The gating mechanism layer gates the linear unit to dynamically adjust the information flow; the output layer is used to map the final feature vector to the target output space using the linear layer.
4. The mobile phone interaction method based on the improved Transformer model according to claim 1, characterized in that: The model training in step 4 includes defining hyperparameters and training models, wherein the defined hyperparameters are hyperparameters selected for training, and the hyperparameters include at least one of the number of training rounds, batch size, number of attention heads, and number of Transformer encoder layers; and the training model uses a fit method for model training.
5. The mobile phone interaction method based on the improved Transformer model according to claim 1, characterized in that: The optimizer of the improved Transformer model is a stochastic gradient descent optimization model.
6. The mobile phone interaction method based on the improved Transformer model according to claim 1, characterized in that: The step 5 evaluates the performance of the model on the test set using the evaluate method, and the evaluation results include loss value and accuracy.
7. The mobile phone interaction method based on the improved Transformer model according to claim 1, characterized in that: The step six applies the model to apply the trained improved Transformer model to the real-time voice interaction task, converts the model's response into natural language based on the user's input, and feeds back to the user in the form of text or voice, including speech synthesis (TTS) and text processing.
Citation Information
Cited By
Large language model voice vulnerability test method and device based on Monte Carlo tree search algorithm, and medium
CN119830298A
