Speech recognition method and electronic equipment

Abstract

The invention provides a voice recognition method and electronic equipment, relates to the technical field of voice processing, and can solve the problems of high decoding time complexity, low decoding rate, high power consumption and the like during voice recognition. The method comprises the following steps: processing a tth voice frame in T voice frames by using a voice recognition model to obtain a text prediction value corresponding to the tth voice frame; the text prediction value comprises blank probability and duration, and is used for indicating the probability that the character corresponding to the tth voice frame is a blank character; the duration is used for at least the tth voice frame; under the condition that the blank probability is greater than or equal to a blank probability threshold, processing the (t + n) th voice frame in the T voice frames by using the voice recognition model to obtain a text prediction value corresponding to the (t + n) th voice frame; wherein n is equal to the duration, and n is greater than 1.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech recognition method and electronic device. Background Technology

[0002] Automatic speech recognition (ASR) is a crucial human-computer interaction technology in the field of edge artificial intelligence (AI). It is widely used in various electronic devices (such as mobile phones, tablets, and desktop computers) to improve the efficiency of human-computer interaction. To achieve on-device speech recognition, one can either perform speech recognition in the cloud and then transmit it back to the device, or perform speech data recognition on the device itself. Cloud-based recognition is currently the mainstream solution, leveraging the powerful computing capabilities of cloud servers to achieve good speech recognition results. However, cloud-based recognition solutions suffer from high data latency, high data risk, and high costs. Therefore, implementing speech recognition on the device itself has become a new trend.

[0003] A transducer is an end-to-end neural network architecture. Speech recognition models based on this architecture can be applied to the edge of electronic devices after quantization, giving the devices independent speech recognition capabilities, thus offering broad application prospects. However, in practical applications, it has been found that transducer-based speech recognition models suffer from high decoding time complexity, slow decoding speed, and high power consumption when recognizing speech. Summary of the Invention

[0004] This application provides a speech recognition method and electronic device to solve the problems of high decoding time complexity, slow decoding rate and high power consumption in speech recognition models based on converter architecture.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, a speech recognition method is provided, which uses a speech recognition model to process speech data, the speech data including T speech frames, where T is a positive integer; the speech recognition model includes a joint network, the joint network including at least one fully connected layer, the at least one fully connected layer including a first fully connected layer and a second fully connected layer.

[0007] The method includes: processing the t-th speech frame in T-frame speech frames using a speech recognition model to obtain the text prediction value corresponding to the t-th speech frame; wherein, the text prediction value includes a blank probability and a duration, the blank probability being obtained after passing through a first fully connected layer, used to indicate the probability that the character corresponding to the t-th speech frame is a blank character; the duration being obtained after passing through a second fully connected layer, used to indicate the duration of at least the t-th speech frame; when the blank probability is greater than or equal to a blank probability threshold, and when the duration is greater than 1, processing the (t+n)-th speech frame in T-frame speech frames using a speech recognition model to obtain the text prediction value corresponding to the (t+n)-th speech frame; where n is the duration.

[0008] It should be noted that in the relevant technologies, when recognizing the corresponding character in a certain audio frame, the non-blank probability is output by the joint network, and the corresponding character result is obtained after character search and decoding based on the non-blank probability.

[0009] The scheme of this application, when recognizing the t-th speech frame, outputs the blank probability through the first fully connected layer of the joint network and the non-blank probability through the second fully connected layer of the joint network. Based on this, if the blank probability is greater than or equal to the blank probability threshold, it indicates that the character corresponding to the speech frame is likely a blank character, therefore the non-blank probability can be omitted, reducing power consumption by decreasing the calculation of the non-blank probability. Furthermore, when the duration is greater than 1, it indicates that the duration of the character corresponding to the speech frame is relatively long, for example, the character is present in n consecutive frames after the speech frame. Therefore, the number of frames corresponding to the duration can be skipped, making the actual number of loops for recognizing the speech data less than T, thereby reducing the decoding time complexity and increasing the decoding rate.

[0010] In one possible implementation of the first aspect, at least one fully connected layer further includes a third fully connected layer; wherein, if the blank probability is less than the blank probability threshold, it indicates that the character corresponding to the t-th speech frame is not a blank character, and therefore the non-blank probability can be output by the third fully connected layer of the joint network to recognize the character corresponding to the t-th speech frame, thereby improving the reliability of speech recognition.

[0011] In one possible implementation of the first aspect, the method further includes: when the duration is equal to 1, it indicates that the duration of the character corresponding to the speech frame is short. Therefore, the speech recognition model can be used to process the (t+1)th speech frame in the T-frame speech frame to obtain the text prediction value corresponding to the (t+1)th speech frame. That is, the next speech frame after the t-frame speech frame is processed to improve the reliability of speech recognition.

[0012] In one possible implementation of the first aspect, the speech recognition model further includes an encoder and a prediction network; wherein, processing the t-th speech frame in the T-frame speech frames using the speech recognition model to obtain the text prediction value corresponding to the t-th speech frame includes: inputting the t-th speech frame into the encoder to obtain an acoustic feature vector; inputting the decoded text sequence corresponding to the t-th speech frame into the prediction network to obtain a text feature vector; and inputting the acoustic feature vector and the text feature vector into the joint network to obtain the text prediction value corresponding to the t-th speech frame.

[0013] In one possible implementation of the first aspect, at least one fully connected layer further includes a fourth fully connected layer and a fifth fully connected layer; wherein, inputting acoustic feature vectors and text feature vectors into the joint network to obtain the text prediction value corresponding to the t-th speech frame includes: linearly processing the acoustic features using the fourth fully connected layer, and linearly processing the text feature vectors using the fifth fully connected layer; wherein, the linear processing is used to make the vector dimensions of the acoustic feature vectors and the text feature vectors the same.

[0014] In one possible implementation of the first aspect, the joint network further includes a fusion function; the method further includes: using the fusion function to perform vector accumulation processing on acoustic feature vectors and text feature vectors with the same vector dimension to obtain a joint vector.

[0015] In one possible implementation of the first aspect, the joint network further includes a first activation function and a second activation function; the method further includes: after the joint vector passes through a first fully connected layer and the first activation function, a blank probability is obtained; after the joint vector passes through a second fully connected layer and the second activation function, a duration probability is obtained; and the duration probability is passed through a preset function to obtain the duration.

[0016] In one possible implementation of the first aspect, the joint network further includes a third activation function; the method further includes: after the joint network passes through a third fully connected layer and a third activation function, non-blank probabilities are obtained.

[0017] In a second aspect, an electronic device is provided, which has the functions described in any one of the first aspects above. These functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions.

[0018] Thirdly, an electronic device is provided, comprising: a memory and one or more processors; the memory stores computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method described in any one of the first aspects.

[0019] Fourthly, a chip system is provided for use in an electronic device, the chip system comprising: at least one processor and an interface for receiving instructions and transmitting them to the at least one processor; the at least one processor executes instructions to cause the electronic device to perform the method described in any one of the first aspects.

[0020] Fifthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the method described in any one of the first aspects.

[0021] In a sixth aspect, a computer program product containing instructions is provided, which, when run on a computer, enables the computer to perform the method described in any one of the first aspects above.

[0022] The technical effects of any of the implementation methods in the second to sixth aspects mentioned above can be referred to the technical effects of different implementation methods in the first aspect, and will not be elaborated here. Attached Figure Description

[0023] Figure 1 A schematic diagram of a basic structure based on a converter is provided in an embodiment of this application;

[0024] Figure 2 A schematic diagram illustrating the principle of a speech recognition process provided in an embodiment of this application;

[0025] Figure 3 A schematic diagram of a structure based on a decomposable transformer is provided in an embodiment of this application;

[0026] Figure 4 A schematic diagram of a TD converter based on an embodiment of this application is provided;

[0027] Figure 5 A schematic diagram of a decomposable TD converter provided in this application embodiment;

[0028] Figure 6 A schematic diagram illustrating the principle of a speech recognition method provided in an embodiment of this application;

[0029] Figure 7 A schematic diagram illustrating the training process of a speech recognition model provided in an embodiment of this application;

[0030] Figure 8 A schematic diagram illustrating the training process of another speech recognition model provided in this application embodiment;

[0031] Figure 9 A schematic diagram illustrating the training process of another speech recognition model provided in this application embodiment;

[0032] Figure 10 A flowchart illustrating a speech recognition method provided in an embodiment of this application;

[0033] Figure 11 A schematic diagram of an interface for a speech recognition application scenario provided in this application embodiment;

[0034] Figure 12 A schematic diagram of an interface for another application scenario of speech recognition provided in this application embodiment;

[0035] Figure 13 A schematic diagram of an interface for another application scenario of speech recognition provided in this application embodiment;

[0036] Figure 14 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0037] Figure 15 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0038] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0039] To facilitate understanding of the embodiments of this application, a brief description of the embodiments of this application is provided below:

[0040] Figure 1 This is a basic structural diagram of the transformer. The transformer consists of three core sub-networks: an encoder, a predictor network, and a joint network. The functions of each sub-network are as follows:

[0041] Encoder: Used to extract acoustic features from speech data to obtain corresponding acoustic feature vectors.

[0042] Prediction network: Used to extract features from each character in a decoded text sequence (hereinafter referred to as decoded text sequence) of speech data, resulting in a text feature vector composed of the character features of each character. Here, a text sequence refers to a sequence consisting of one or more characters.

[0043] A joint network is used to combine acoustic feature vectors and text feature vectors to predict the next character in a decoded text sequence. The methods of combining these feature vectors include, but are not limited to, vector accumulation.

[0044] Optional, Figure 1 The transformer shown can be a standard transformer (or standard transducer). In the speech recognition model based on the standard transformer, the vocabulary set during training includes one or more characters (such as Chinese characters, English words, etc.) and a whitespace character to represent blank output. The whitespace character and one or more characters are recorded together in a vocabulary file, and the position of each character in the vocabulary file is fixed. For example, assuming 5000 Chinese characters and one whitespace character are set as characters in this embodiment, the standard transformer will record these 5001 characters in the vocabulary file, and the position of each character in the vocabulary file will be fixed. The whitespace character is usually located at the beginning of the vocabulary file.

[0045] For example, a thesaurus file can be recorded in the form shown in Table 1 below.

[0046] Table 1

[0047] 1 2 3 4 …… 1000 1001 …… 5000 5001 <blk> < / blk> I You You …… You He She His

[0048] It should be noted that Table 1 is an example assuming a dictionary file containing multiple characters. The correspondence between characters and positions in the dictionary file shown in Table 1 is merely an example and does not constitute a limitation on this application. In Table 1, a whitespace character can be represented as "blk".

[0049] In this embodiment, whitespace characters and non-whitespace characters (i.e., one or more of the aforementioned characters, such as Chinese characters, English single characters, etc.) can be collectively referred to as tokens. In natural language processing (NLP) technology, a token refers to the smallest unit or basic element of text processing, which can be a word, a phrase, a punctuation mark, or a character. In NLP technology, text is decomposed into a series of tokens so that the model can understand and process it. For ease of understanding, the following will use characters (including whitespace characters and non-whitespace characters) as an example for illustration, and will not be repeated here.

[0050] When performing speech recognition using a pre-trained speech recognition model based on a standard transformer, the joint network combines the acoustic feature vector and the text feature vector, and then outputs a text prediction value to indicate the prediction result for the next character. For example... Figure 1As shown, the prediction result can include a character probability distribution, which includes whitespace characters and their corresponding probabilities, and non-whitespace characters and their corresponding probabilities. The non-whitespace character and its corresponding probability distribution can include only one non-whitespace character and its corresponding probability, or it can include multiple non-whitespace characters and their corresponding probabilities. Based on Figure 1 The speech recognition process shown can be understood as follows: the joint network of the standard transformer includes an output, namely the character probability distribution described above.

[0051] For example, such as Figure 2 As shown, the speech recognition process is briefly explained using a speech recognition model based on a standard transducer. First, the speech data is processed by the acoustic feature extraction module to obtain acoustic features. The encoder then analyzes and extracts these acoustic features to obtain acoustic feature vectors. Second, the decoded text sequence is input into the prediction network for processing (when using the prediction network for the first time, preset characters or strings can be input as the initial decoded text sequence) to obtain text feature vectors. The joint network then combines the acoustic and text feature vectors to obtain the subsequent character probability distribution and determine multiple decoded text sequences. Finally, the decoding module determines the characters and outputs the final text sequence (i.e., the speech recognition result).

[0052] In this application, the specific acoustic features and acoustic feature extraction methods used are not limited, and can be set by technical personnel according to actual needs. For example, acoustic features can be linear prediction cepstral coefficients (LPCCs), speech spectrum features, or melfrequency cepstral coefficients (MFCCs). Similarly, this application does not limit the specific implementation of the acoustic feature extraction module, and can be set by technical personnel according to the acoustic features used and the extraction requirements. For example, acoustic models based on recurrent neural networks (Long Short-Term Memory, LSTM), transformers, or hybrid network structures (conformers) can be used as the acoustic feature extraction module in this application. Alternatively, acoustic feature extraction functions can be designed and encapsulated as the acoustic feature extraction module in this application.

[0053] It should be noted that, Figure 2This is merely the basic architecture of a speech recognition model based on a standard transducer. Without departing from the technical specifications of the model architecture described in the above embodiments, those skilled in the art can modify it according to actual needs. Figure 2 Modifications, refinements, or additions can be made. For example, in some optional embodiments, structures other than recurrent networks can be added to the encoder and prediction networks according to actual needs to refine the network structure and meet actual data processing requirements. In other optional embodiments, a speech data preprocessing module can be added before the acoustic feature extraction module to perform preprocessing such as noise reduction on the speech data, thereby improving the quality of the speech data. Alternatively, a post-processing module can be added after the decoding module to perform post-processing on the model output results, thereby improving the readability of the final speech recognition results.

[0054] Optional, Figure 1 The transformer shown can also be a decomposable transformer (or decomposable transducer). A decomposable transformer is an extension of a standard transformer. The difference from a standard transformer is that, during training, the speech recognition model based on a decomposable transformer uses a vocabulary containing one or more characters and a whitespace character. The whitespace character and the one or more characters are recorded in two vocabulary files. For example, assuming 5000 Chinese characters and one whitespace character are used in this embodiment, the decomposable transformer will record these 5000 Chinese characters in vocabulary file 1, fixing the text of each character in vocabulary file 1. Furthermore, the decomposable transformer will record one whitespace character in vocabulary file 2.

[0055] For example, vocabulary file 1 and vocabulary file 2 can be recorded in the form shown in Table 2 below.

[0056] Table 2

[0057]

[0058] It should be noted that Table 2 uses a dictionary file 1 containing multiple characters as an example. The correspondence between characters and positions in the dictionary file shown in Table 2 is merely an example and does not constitute a limitation on this application. Of course, there can be other correspondences between characters and positions, which will not be listed here.

[0059] When performing speech recognition using a pre-trained speech recognition model based on a decomposable transformer, the joint network combines the acoustic feature vector and the text feature vector, and then outputs a text prediction value to indicate the prediction result for the next character. For example... Figure 3As shown, the prediction result can include blank probabilities (i.e., blank characters and their corresponding probabilities) and non-blank probabilities (i.e., non-blank characters and their corresponding probabilities). In other words, the joint network of the decomposable transformer includes two outputs: blank probabilities and non-blank probabilities. The non-blank probability can contain only one character and its corresponding probability, or it can contain multiple characters and output their respective probabilities; in this case, the output is the probability matrix (also called the non-blank probability matrix).

[0060] For a brief explanation of the speech data recognition process in speech recognition models employing decomposable transformers, please refer to the above. Figure 2 As shown, this will not be elaborated further here. It is understandable that, in the process of speech recognition, the difference between the speech recognition model of the decomposable transformer and the speech recognition model of the standard transformer lies in the fact that the joint network of the decomposable transformer outputs two classes: one class outputs blank probabilities, and the other class outputs non-blank probabilities.

[0061] Optional, Figure 1 The transformer shown can also be a Token-Duration-Transducer (TD transformer, or TD Transducer). The TD transformer is also an extension of the standard transformer. The difference from the standard transformer is that, during training, the speech recognition model based on the TD transformer sets one or more durations in addition to the vocabulary. The vocabulary set during training includes one or more characters, as well as a whitespace character. The whitespace character and the one or more characters are recorded in a vocabulary file, and the position of each character in the vocabulary file is fixed. The one or more durations set during training represent the duration of each character (or each pronunciation unit) corresponding to a speech frame. It is understood that a speech frame may correspond to one character or multiple characters. For ease of understanding, this embodiment uses a speech frame corresponding to one character as an example; therefore, the duration is the duration corresponding to the character. This is explained uniformly here and will not be repeated below.

[0062] When performing speech recognition using a pre-trained TD-SCDMA-based speech recognition model, the joint network combines the acoustic feature vector and the text feature vector, and then outputs a text prediction value to indicate the prediction result for the next character. For example... Figure 4 As shown, the prediction result can include the character probability distribution and the duration probability. That is, the joint network of the TD converter includes two outputs: the character probability distribution and the duration probability.

[0063] For a brief explanation of the speech data recognition process using a TD converter speech recognition model, please refer to the above. Figure 2 As shown, this will not be elaborated further here. It is understandable that, in the process of speech recognition, the difference between the speech recognition model of the TD converter and the speech recognition model of the standard converter lies in the fact that the joint network of the TD converter outputs two types of data: one type is the character probability distribution, and the other type is the duration probability distribution.

[0064] Optional, Figure 1 The transformer shown can also be a decomposable TD transformer (or decomposable TDTransducer). The decomposable TD transformer combines the features of a decomposable transformer and a TD transformer. Unlike the decomposable transformer and TD transformer mentioned above, the speech recognition model based on the decomposable TD transformer, during training, includes setting one or more durations in addition to setting the vocabulary. The vocabulary set during training includes one or more characters, as well as a whitespace character; the whitespace character and the one or more characters are recorded in two vocabulary files. The one or more durations set during training represent the duration of each character corresponding to a speech frame.

[0065] When performing speech recognition using a pre-trained speech recognition model based on a decomposable TD transformer, the joint network combines the acoustic feature vector and the text feature vector, and then outputs a text prediction value to indicate the prediction result for the next character. For example... Figure 5 As shown, the prediction result includes the blank probability (i.e., the blank character and its corresponding probability), the non-blank probability (i.e., the non-blank character and its corresponding probability), and the duration probability. In other words, the joint network of the decomposable TD converter includes three outputs: the blank probability, the non-blank probability, and the duration probability. The non-blank probability can contain only one character and its corresponding probability, or it can contain multiple characters and output their respective probabilities; in this case, the output is the probability matrix (also called the non-blank probability matrix).

[0066] For a brief explanation of the speech data recognition process in the speech recognition model using a decomposable TD converter, please refer to the above. Figure 2 As shown, this will not be elaborated further here. It is understandable that, in the process of speech recognition, the speech recognition model of the decomposable TD transformer differs from the aforementioned decomposable transformer and TD transformer in that: the joint network of the decomposable TD transformer outputs three types of data: one type of output is the blank probability, one type of output is the non-blank probability, and the other type of output is the duration probability.

[0067] In this embodiment, a speech recognition model based on a decomposable TD converter (hereinafter referred to as the speech recognition model for ease of explanation; however, other naming methods may be used in practical applications) is used for speech recognition. In the speech recognition model, the joint network can output the blank probability, the non-blank probability, and the duration probability, respectively. Based on this, during speech recognition, the joint network can first output the blank probability and the duration probability; the speech recognition model can skip speech frames containing less information during decoding based on the duration probability; and the speech recognition model can determine whether to stop outputting the non-blank probability based on the blank probability, thereby accelerating decoding while reducing the computational load and power consumption of speech recognition.

[0068] The following describes in detail the technical solution of the speech recognition model based on a decomposable TD converter. The speech recognition method provided in this application can be applied to mobile terminal electronic devices such as mobile phones, laptops, embedded devices, tablets, and wearable devices. In this case, these mobile terminals are the executing entities of the speech recognition method provided in this application. It can also be applied to non-mobile terminal electronic devices such as servers and desktop computers. In this case, these non-mobile terminals are the executing entities of the speech recognition method provided in this application. This application does not impose any restrictions on the specific type of electronic device; it can be determined according to the actual application scenario.

[0069] To illustrate the technical solution of this application, specific embodiments are described below. The embodiments of this application will be described sequentially according to the architecture, training process, speech recognition process, and specific application scenarios of the speech recognition model based on a decomposable TD converter. Details are as follows:

[0070] Part One describes the speech recognition model in the embodiments of this application.

[0071] The speech recognition model is based on a decomposable TD converter, therefore the overall architecture of the model can be referenced. Figure 1 . Figure 1 In the speech recognition model architecture shown, in practical applications, the encoder can be based on structures such as LSTM, gated recurrent units (GRU), or transformers, conformers, squeezeformers, zipformers, etc., without limitation. The prediction network can be based on stateful structures such as LSTM and GRU recurrent neural networks, or on stateless structures such as convolutional neural networks (CNN), without limitation.

[0072] Among them, Figure 1 Based on this, the speech recognition model mainly involves structural changes to the joint network. Therefore, the structure of the joint network of the speech recognition model is explained below:

[0073] For federated networks: Reference Figure 6 The federated network in this embodiment consists of N fully connected layers. In this embodiment, the number N of fully connected layers is not strictly limited; when N is greater than 1, it can be set by those skilled in the art according to actual needs. Optionally, such as... Figure 6 As shown, the N fully connected layers include fully connected layer 1 (which can be represented as in_encoder) and fully connected layer 2 (which can be represented as in_decoder). Fully connected layer 1 is used to unify the dimensions of the acoustic feature vectors, and fully connected layer 2 is used to unify the dimensions of the text feature vectors, ensuring that the acoustic feature vectors and text feature vectors have the same data dimensions.

[0074] Optional, such as Figure 6 As shown, the joint network also includes a fusion function, such as the hyperbolic tangent function (which can be represented as tanh), used to accumulate (e.g., weighted processing) the acoustic feature vector and text feature vector of the same dimension to obtain the accumulated joint vector. After obtaining the joint vector, the joint network uses several fully connected layers to process the joint vector to obtain the predicted probability value of the next character. Optionally, such as... Figure 6 As shown, the N fully connected layers also include fully connected layer 3 (which can be represented as out_blk), fully connected layer 4 (which can be represented as out_ilm), and fully connected layer 5 (which can be represented as out_dur) to determine the text prediction values. For example, fully connected layer 3 is used to predict the blank probability, fully connected layer 4 is used to predict the non-blank probability, and fully connected layer 5 is used to predict the duration probability. Optionally, fully connected layer 3 is also connected to a first activation function (which can be represented as sigmoid) to map the blank probability value predicted by fully connected layer 3 to the range of 0 to 1. Fully connected layer 4 is also connected to a second activation function (which can be represented as softmax) to map the non-blank probability value predicted by fully connected layer 4 to the range of 0 to 1. Fully connected layer 5 is also connected to a third activation function (which can be represented as softmax) to map the duration probability predicted by fully connected layer 3 to the range of 0 to 1.

[0075] The above Figure 6 The illustrated embodiment describes the internal architecture of a sub-network (joint network) in a speech recognition model. For descriptions of the internal architecture of other sub-networks (such as encoders, prediction networks, etc.), please refer to descriptions in related technologies; they will not be repeated here. Optionally, such as... Figure 6As shown, the speech recognition model may also include a decoding module (also called a search module). The decoding module is connected to the joint network and is used to search for and determine characters from the output of the joint network. For example, after the joint network outputs multiple characters, the decoding module selects some characters as possible next characters. The speech recognition model then selects one character from these, resulting in a new decoded text sequence composed of the decoded text sequence and the selected character. This new decoded text sequence can be used as the speech recognition result or as new input data for the prediction network. The specific implementation method of character search is not limited here and can be set by technicians according to actual needs. For example, in some optional embodiments, beam search, greedy search, or exhaustive search can be used to implement character search.

[0076] Part Two: Based on the above description of the speech recognition model architecture, the training process of the speech recognition model in this application embodiment is described as follows:

[0077] First, it should be noted that, in the embodiments of this application, the training methods for the speech recognition model may include at least: not pre-training the encoder and the prediction network separately, but jointly training the entire speech recognition model.

[0078] In practical applications, technicians can set the training methods according to their needs. These can be the methods mentioned above, or other training methods set by the technicians; no further restrictions are imposed here.

[0079] In addition to determining the specific training method, it is also necessary to determine the specific training method for the speech recognition model. This application does not impose excessive limitations on the specific training method for the speech recognition model, which can be set by those skilled in the art according to actual needs. For example, in some optional embodiments, considering that the basic architecture of the speech recognition model is similar to that of the standard transformer, the main difference in speech recognition lies in the different text prediction values ​​output by the joint network. Therefore, theoretically, the training method for the speech recognition model based on the standard transformer can be used to train the speech recognition model of this application embodiment. In other optional embodiments, reference can be made to... Figure 7 Based on the above method 2, the speech recognition model is trained through the following steps S101-S105.

[0080] S101. Initialize the weights in the speech recognition model.

[0081] S102. The speech data is propagated forward through the acoustic feature extraction module and the encoder, while the decoded text sequence is propagated forward through the prediction network (the initial decoded text sequence of the prediction network is a preset character or string), and then the joint network propagates the input data of the encoder and the input data of the prediction network forward.

[0082] The decoded text sequence can be a portion of the actual character content corresponding to the speech data.

[0083] S103. Calculate the total error between the speech recognition result output by the speech recognition model and the real character content corresponding to the speech data (e.g., the loss value can be calculated using a loss function and used as the total error).

[0084] S104. If the total error continues to decrease, the weights are updated using the backpropagation algorithm. That is, the total error is propagated back into the speech recognition model to calculate the error of each network layer, and the weights in the speech recognition model are updated according to the errors of the network layers, and then S102 is executed again.

[0085] S105. If the total error no longer changes, then the training ends and the trained speech recognition model is obtained.

[0086] Based on steps S101-S105 above, this embodiment does not impose excessive limitations on the details of speech recognition model training, which can be set by technicians according to actual needs. In this embodiment, the encoder and prediction network are jointly trained within the speech recognition model, rather than pre-trained. Since the encoder and prediction network converge quickly, the joint training time is also short when the error meets the expected value requirement. Therefore, the overall model training complexity and difficulty can be reduced while maintaining good training speed and effect.

[0087] As an optional specific training method for the speech recognition model in this application, method 2 is selected to jointly train the entire model in this embodiment. (See reference...) Figure 8 This is a schematic diagram of the training process based on a partial speech recognition model structure in an embodiment of this application. The training-related explanations are as follows:

[0088] First, it should be understood that the embodiments of this application use a loss function to calculate the loss value between the predicted text value and the real text label, as a quantitative value for evaluating the training effect of the model. The specific loss function used is not limited here and can be set by technical personnel according to actual needs; for example, the decomposable TD transform loss function (Token-Duration-Transducer Loss) can be used.

[0089] Furthermore, in this embodiment, no decoding module is set up during speech recognition model training. Therefore, the loss function essentially measures whether the text prediction value output by the joint network reaches the required accuracy. If it does, it indicates that the training of the encoder, prediction network, and joint network has met the requirements. Based on this, a corresponding decoding module can be added when actually using the speech recognition model. It should be noted that the text prediction value output by the joint network in this embodiment is essentially the prediction result for the next character. This prediction result may include blank probability, non-blank probability, duration probability, etc.

[0090] 1. Preparation of training-related data.

[0091] (1) Sample data:

[0092] This application provides several training audio data sets (i.e., sample data), each with corresponding text annotations. The text annotations are the actual text sequences of the audio content of the sample data. Each character in the text annotation is separated by delimiters. These characters can be Chinese characters, English words, or words from other languages, or they can be symbols with special functions, such as control symbols used to assist training. For example, assuming the actual text sequence of the audio content of sample data A is "Please play music," the corresponding text annotation would be "Please_play_music_".

[0093] In this embodiment, the text label is the actual text annotation corresponding to the sample data, which is used to compare with the text prediction value output by the joint network to calculate the corresponding loss value. For example, in the above embodiment, the text label corresponding to sample data A is "Please play music".

[0094] In this embodiment, to achieve a high training effect for the speech recognition model, a threshold is set for the cumulative duration of all sample data, for example, 1000 hours (the threshold can be modified according to needs). However, the duration of each individual sample data is not overly limited. For example, it can be set between 1 and 20 seconds. Therefore, the specific number of sample data needs to be determined based on the cumulative duration of each sample data and the threshold. If the cumulative value does not reach the threshold, more sample data needs to be added. Since a larger number of samples theoretically results in better training, but also takes longer, once the cumulative value reaches the threshold, technicians can determine whether to continue adding samples based on actual needs. Furthermore, this embodiment does not overly limit the sampling rate, data bit depth, and file format of the sample data, which can be set by technicians according to actual needs. For example, the sampling rate can be set to 16kHz, the data bit depth to 16 bits, and the file format to WAV.

[0095] (2) Preparation of encoder input data:

[0096] Since the encoder takes audio frame data from sample data as input, therefore, during the... Figure 8 Before the training operation shown, the sample data needs to be segmented into frames to obtain multiple speech frames. Segmentation is used to utilize the short-term stationarity of speech data. This embodiment does not impose excessive restrictions on the segmentation rules, which can be set by those skilled in the art. For example, the frame length of each speech frame can be set to 25 milliseconds, and the segmentation step size can be set to 10 milliseconds. In this case, the sample data can be divided into multiple speech frames. For example, assuming a sample data duration of 1 second, segmentation with a frame length of 25 milliseconds and a step size of 10 milliseconds can yield a total of 100 speech frames.

[0097] (3) Preparation of input data for the prediction network (i.e., the decoded text sequence):

[0098] In this embodiment, the output of the decoding module can be used as the input data for the prediction network. Alternatively, a portion of the real text sequence can be used as the input data for the prediction network (in this case, "decoded" in the decoded text sequence essentially means that the character content is known). For example, assuming the real text sequence corresponding to the speech content of sample data A is "Please play music", the text sequence "Please play sound" can be used as the input data for the prediction network, and the last character is predicted. Based on this principle, in preparing the input data for the prediction network, this embodiment removes the last character from the real text sequence corresponding to each sample data to obtain the decoded text sequence used as the input data for the prediction network. Similarly, this input data will also be annotated with text, such as "Please play sound" being annotated as "Please_play_sound".

[0099] After the above preparation operations for the prediction network input, each sample data will correspond to a decoded text sequence and a corresponding text label.

[0100] (4) Dictionary file:

[0101] In this embodiment, whitespace characters and non-whitespace characters are recorded in two dictionary files. For example, whitespace characters are recorded in dictionary file 1, and non-whitespace characters are recorded in dictionary file 2. Assuming that 5000 Chinese characters are set as non-whitespace characters in this embodiment, these 5000 Chinese characters are recorded in the dictionary text, and the position of each character in the dictionary file is fixed.

[0102] (5) Duration file:

[0103] The duration file includes one or more durations, each representing the duration of each character corresponding to a speech frame. Optionally, the duration can also be in frames; for example, a duration of 2 means that the duration of each character corresponding to the current speech frame is two speech frames.

[0104] In this embodiment, the duration can be set in units of frames. For example, all possible frame numbers can be recorded in a single duration file. For instance, the duration file can be represented as [0, 1, 2, 3, 4].

[0105] 2. The training process of the speech recognition model.

[0106] At the start of training, the weights within the speech recognition model are initialized. After preparing all the necessary data for model training, the speech recognition model is trained using the following steps. It should be understood that, assuming the sample data contains T speech frames, the speech recognition model is trained by iterating through each speech frame from 1 to T. For example, as shown... Figure 9 As shown, the training process specifically includes:

[0107] Step 1: Input the t-th audio frame into the encoder and input the decoded text sequence into the prediction network.

[0108] Where t is a positive integer, t∈[1,T]. For example, assuming there are 8 sample data, after the above frame-segmentation process, each sample data corresponds to 1000 speech frames, then T can be 8000, that is, t∈[1,8000].

[0109] Step 2: The encoder extracts features from the t-th speech frame to obtain the acoustic feature vector.

[0110] Step 3: The prediction network extracts features from the decoded text sequence to obtain text feature vectors.

[0111] Step 4: The joint network performs vector accumulation on the acoustic feature vector and the text feature vector, and calculates the text prediction value for the next character.

[0112] Step 5: Calculate the loss value between the predicted text value and the text label using the loss function, and update the weights of the speech recognition model based on the loss value.

[0113] Optionally, in this embodiment of the application, in addition to using the TDT loss value, the connectionist temporal classification (CTC) loss value is also used when calculating the loss value, so as to improve the accuracy of speech recognition while accelerating convergence.

[0114] For example, the loss value can be: L = L TDT +L CTC Where L represents the loss value, L TDT L represents the TDT loss value. CTC This represents the CTC loss value.

[0115] In some optional embodiments of this application, when calculating L... TDT Previously, embodiments of this application could also use the predicted blank probabilities to normalize the non-blank probabilities, mapping the predicted probability values ​​of the non-blank probabilities to a range of 0 to 1. For example, the non-blank probabilities can be normalized using the following expression:

[0116] P T (v|t, u)=(1-P) T (φ|t,u))P T (v|t, u); where P T (v|t, u) represents the non-blank probability, P T (φ|t,u) represents the blank probability.

[0117] It should be noted that the speech recognition model obtained after training is theoretically ready for use in speech recognition. However, in practical applications, to ensure good speech recognition performance, the weights within the model often have high precision. For example, the data type of the weights can generally be set to 64-bit double-precision floating-point numbers or 32-bit single-precision floating-point numbers. The resulting speech recognition model is often large in size and requires significant computational resources during runtime. Therefore, it places high demands on the storage space and computational resources of electronic devices, resulting in lower practicality. To make the speech recognition model more suitable for electronic devices, it can be quantized (in practical applications, if the electronic device has ample storage space and computational resources, quantization can be omitted, and the model can be stored and run on the electronic device). This involves reducing the precision of the weights within the speech recognition model to decrease its size and computational resource consumption. For example, the data type of the weights can be set to 8-bit integers (bytes). Therefore, as an optional embodiment of this application, after training the speech recognition model, the method further includes: quantizing the speech recognition model and storing the quantized speech recognition model on the electronic device. The specific quantification scheme can be set by technical personnel according to the actual situation, and no further restrictions are imposed here.

[0118] Part Three: Based on the completion of speech recognition model training, this application provides a speech recognition method, and the speech recognition process is described as follows:

[0119] First, it should be noted that although quantization changes the size and computational load of the speech recognition model, the process by which the speech recognition model recognizes speech data remains unchanged before and after quantization. Therefore, the speech recognition model referred to in the following description of the speech recognition process can be either the speech recognition model before or after quantization.

[0120] refer to Figure 10 This is a schematic diagram of the speech recognition process provided by the speech recognition model in the embodiments of this application, including:

[0121] Step a: The electronic device acquires voice data and inputs the voice data into the encoder.

[0122] The source of the voice data can be locally stored on the electronic device, acquired via a network, sent by other devices, or collected by the microphone of the electronic device. The specific source needs to be determined based on the actual application scenario, or refer to Part IV of this application's embodiments for details on specific application scenarios of the voice recognition model; no further limitations are imposed here.

[0123] Optionally, before inputting the data to the encoder, the electronic device may perform frame segmentation on the speech data, resulting in multiple corresponding speech frames. The framing rules are not strictly defined here and can be set by technicians according to requirements. For example, the frame length of a speech frame can be set to 25 milliseconds, and the framing step size can be set to 10 milliseconds.

[0124] Step b: The encoder extracts features from the speech data to obtain acoustic feature vectors, and then inputs the acoustic feature vectors into the joint network.

[0125] Understandably, since the voice data input to the encoder is divided into multiple voice frames (such as T-frame voice frames), the electronic device can cyclically traverse each voice frame from frame 1 to T and input each voice frame to the encoder in turn.

[0126] Step c: Input the decoded text sequence into the prediction network for feature extraction to obtain the text feature vector, and then input the text feature vector into the joint network.

[0127] Considering that there are no decoded characters when speech data is initially recognized, this embodiment pre-sets a character or string as the initial "decoded text sequence" input to the prediction network. That is, when processing the current speech data, the first input to the prediction network can be a preset character or string. This preset character or string is generally not retained in the speech recognition result. This embodiment does not impose excessive limitations on the specific content of the preset character or string, which can be set by those skilled in the art. For example, it can be set to an empty character, represented as "blank".

[0128] In the process of recognizing speech data, if there are already some decoded characters and the recognition of the speech data is not yet complete. At this time, the "decoded text sequence" input is the character content that has been decoded from the current speech data. For example, assume that part of the recognized speech data is "Please_play_sound". At this time, "Please_play_sound" will be used as the decoded text sequence and input into the prediction network to continue predicting and recognizing subsequent characters.

[0129] After receiving the acoustic feature vector and the text feature vector, the joint network will combine the two vectors in a way such as vector accumulation, and then obtain a joint vector. After obtaining the joint vector, it will be further processed using a fully connected layer, etc., to recognize the next character based on the decoded text sequence. In the embodiments of this application, the process of character recognition is essentially to recognize characters for each speech frame in the speech data.

[0130] Assume that the current speech data contains T speech frames. The joint network loops through each speech frame from the 1st frame to the Tth frame and performs character recognition on each speech frame. Taking the joint network's character recognition of the tth speech frame as an example, the execution process of the joint network for character recognition is introduced, which specifically includes:

[0131] Step d: The joint network inputs the acoustic feature vector into the first fully connected layer and performs linear processing on the acoustic feature vector.

[0132] Step e: The joint network inputs the text feature vector into the second fully connected layer and performs linear processing on the text feature vector.

[0133] In the embodiments of this application, the acoustic feature vector can be the acoustic feature vector obtained by the encoder after feature extraction of the tth speech frame. The text feature vector can be the acoustic feature vector obtained by the prediction network after feature extraction of the tth speech frame. Exemplarily, the acoustic feature vector can be expressed as: The text feature vector a can be expressed as: It should be noted that the representation methods of the acoustic feature vector and the text feature vector a here are only examples and do not constitute a limitation to this application.

[0134] Understandably, fully connected layer 1 and fully connected layer 2 are used to perform linear processing (such as dimensional data duplication) on the acoustic feature vector and text feature vector, making the data dimensions of the acoustic feature vector and text feature vector the same. For example, assume that the dimensions of the acoustic feature vector are [P, Z, C1], i.e., a three-dimensional feature vector; and the dimensions of the text feature vector are [P, 1, W, C2], i.e., a four-dimensional feature vector. First, fully connected layer 1 expands the dimensions of the acoustic feature vector to make it a four-dimensional feature vector, for example, the expanded acoustic feature vector has dimensions of [P, Z, 1, C1].

[0135] Next, fully connected layer 1 copies the third-dimensional data of the acoustic feature vector with dimensions [P, Z, 1, C1] to the third-dimensional data of the text feature vector, resulting in an acoustic feature vector with dimensions [P, Z, W, C1]. Similarly, fully connected layer 2 copies the second-dimensional data of the text feature vector a with dimensions [P, 1, W, C2] to the second-dimensional data of the acoustic feature vector a, resulting in a text feature vector with dimensions [P, Z, W, C2]. Thus, through the linear processing of the acoustic and text feature vectors by fully connected layers 1 and 2, the first three dimensions of the acoustic and text feature vectors are completely identical.

[0136] Step f: The joint network combines the acoustic feature vector and the text feature vector to obtain the joint vector.

[0137] Since the acoustic feature vector and the text feature vector are completely identical in their first three dimensions after the fully connected layer 1 and the fully connected layer 2 perform linear processing, the joint network can achieve the joint processing of the acoustic feature vector and the text feature vector.

[0138] For example, the joint network uses a fusion function to perform nonlinear processing on the acoustic feature vector and the text feature vector, such as summing the acoustic feature vector and the text feature vector to obtain a fused vector. For example, the fusion function can be a hyperbolic tangent function (such as the tanh function), or it can be any other function that can achieve vector summation, without limitation.

[0139] For example, the joint network utilizes a fusion function to combine acoustic feature vectors with the same dimensionality. and text feature vectors After performing vector accumulation, the resulting joint vector can be represented as: Of course, joint vectors can also be represented in other ways, without limitation.

[0140] After obtaining the joint vector, several fully connected layers are used to process the joint vector to obtain the predicted probability value of the next character. In the embodiment of the present application, a blank character is recorded in the vocabulary file 1. When processing in the fully connected layer 3, for each speech frame, the predicted probability value of this blank character is calculated. At this time, a blank character and the corresponding predicted probability value can be obtained.

[0141] Optionally, the embodiment of the present application also sets a blank probability threshold to indicate whether the t-th speech frame contains valid information. When the predicted probability value corresponding to the blank character is greater than or equal to the blank probability threshold, it means that the t-th speech frame does not contain valid information (that is, does not contain valid characters). Therefore, the non-blank probability does not need to be predicted, and by saving the calculation of the non-blank probability, the calculation amount and power consumption can be reduced. When the predicted probability value corresponding to the blank character is less than the blank probability threshold, it means that the t-th speech frame contains valid information (that is, contains valid characters). Therefore, the non-blank probability can continue to be predicted.

[0142] In the case where the predicted probability value corresponding to the blank character is less than the blank probability threshold, assuming that there are X non-blank characters recorded in the vocabulary file 2, when processing in the fully connected layer 4, for each speech frame, the predicted probability values of each of the X non-blank characters are calculated. At this time, for each speech frame, a probability matrix table composed of X non-blank characters and the corresponding predicted probability values can be obtained. For example, assume that the decoded text sequence of the sample data A is "Please_play_", and the corresponding text label is "Please_play_sound", and assume that the generated probability matrix table is the following Table 3.

[0143] Table 3

[0144] Sound 0.75 Movie 0.15 Go 0.05 …… …… Ah 0.05

[0145] Among them, the first column in Table 3 is all the non-blank characters recorded in the vocabulary file 2, and the second column is the predicted probability values corresponding to each non-blank character for the sample data A.

[0146] Correspondingly, assuming that there are M durations recorded in the duration file, when processing in the fully connected layer 5, for the speech frame, the predicted probability values of each of the M durations are calculated. At this time, a probability matrix table composed of M durations and the corresponding predicted probability values can be obtained. Assume that the M durations recorded in the duration file are [0, 1, 2, 3, 4], and assume that the generated probability matrix table is the following Table 4.

[0147] Table 4

[0148] 0 0.05 1 0.15 2 0.75 3 0.02 4 0.03

[0149] Among them, the first column in Table 4 is the duration of all records in the duration file, and the second column is the predicted probability value corresponding to the duration of the t-th speech frame. Based on this, the duration of the t-th speech frame can be predicted according to the predicted probability value of the duration. For example, referring to Table 4 above, the duration of the t-th speech frame is 2, that is, the duration of the t-th speech frame is two speech frames.

[0150] In the embodiment of the present application, the reason for predicting the duration of the speech frame is that the duration of the speech frame can represent the duration of the speech frame. The longer the duration, the less valid information (or no valid information) is contained in the consecutive multiple speech frames after this speech frame. For example, the subsequent speech frames are empty speech frames. For example, for the t-th speech frame, assuming that the character obtained by the character recognition result is "sound", and the duration of the t-th speech frame is 2, then it can be explained that the consecutive two frames (such as the t + 1-th frame and the t + 2-th frame) after the t-th speech frame contain less valid information, such as empty speech frames. Therefore, the character recognition of the speech frames of the t + 1-th frame and the t + 2-th frame can be skipped, thereby accelerating the recognition of speech data.

[0151] Or, the duration of the speech frame represents the duration of the speech frame. The longer the duration, the more the character results of the consecutive multiple speech frames after this speech frame are the same as the character result corresponding to this speech frame. For example, for the t-th speech frame, assuming that the character obtained by the character recognition result is "sound" and the duration of the t-th speech frame is 2, then it can be explained that the characters corresponding to the consecutive two frames (such as the t + 1-th frame and the t + 2-th frame) after the t-th speech frame are the same as the character corresponding to the t-th speech frame. Therefore, the character recognition of the speech frames of the t + 1-th frame and the t + 2-th frame can be skipped, thereby accelerating the recognition of speech data.

[0152] Optionally, when the predicted probability value corresponding to the blank character is less than the blank probability threshold, after obtaining the probability matrix table, the text prediction value with the largest predicted probability value therein can be used as the text prediction value corresponding to the currently decoded text sequence and classified and output. For example, based on Table 3, "Please_play_sound" will be used as the text prediction value of "Please_play" and subsequent prediction operations will be performed.

[0153] Exemplarily, the process by which the joint network obtains the predicted probability value of the next character based on the joint vector includes:

[0154] Step g: The joint network obtains the blank probability based on the joint vector.

[0155] For example, the joint network inputs the joint vector into the fully connected layer 3. After the fully connected layer 3 performs linear processing on the joint vector, it passes it through a first activation function (such as the sigmoid function) to obtain the blank probability. The first activation function is used to map the blank probability obtained after the linear transformation of the fully connected layer 3 to the range of 0 to 1.

[0156] Optionally, a blank probability threshold is pre-set in this embodiment. The blank probability threshold is used to determine whether the character corresponding to each audio frame is an empty character. For example, the blank probability threshold can be represented as a threshold, or it can be represented in other ways, without limitation. The specific value of the blank probability threshold is not limited in this embodiment and can be set by those skilled in the art according to actual needs. It is understood that the higher the value of the blank probability threshold, the more accurate the result of determining whether the character corresponding to each audio frame is an empty character. Therefore, the blank probability threshold can be set to 0.9, 0.95, etc., without limitation.

[0157] It is understood that in this embodiment, when the blank probability is greater than or equal to the blank probability threshold, it indicates that the character corresponding to the t-th speech frame is an empty character. In some optional embodiments, if the recognition result indicates that the character corresponding to the t-th speech frame is an empty character, the next speech frame will be analyzed until a non-empty character is obtained, and then the non-blank probability will be calculated. For example, step i will be executed after step g.

[0158] Thus, in this embodiment of the application, by first calculating the blank probability, if the character corresponding to the t-th frame of the speech frame indicated by the blank probability is an empty character, the non-blank probability does not need to be calculated. That is, the non-blank probability is not calculated by iterating through each frame, thereby saving the calculation of the non-blank probability and achieving the purpose of reducing the amount of computation and power consumption.

[0159] In some alternative embodiments, if the recognition result indicates that the character corresponding to the t-th audio frame is an empty character, the duration of the t-th audio frame will be calculated to represent the duration of the t-th audio frame. For example, step h below will be executed.

[0160] Step h: The joint network obtains the duration probability based on the joint vector.

[0161] The duration probability is used to indicate the duration of the t-th audio frame.

[0162] For example, the joint network inputs the joint vector into the fully connected layer 5. After the fully connected layer 5 performs linear processing on the joint vector, it passes through the third activation function (softmax) to obtain the duration probability. The third activation function is used to map the duration probability obtained after the linear transformation of the fully connected layer 5 to the range of 0 to 1.

[0163] It is understandable that when a duration file contains one or more durations, after linearly processing the joint vector using a fully connected layer 5, a probability matrix of probability values ​​for one or more durations can be obtained. For example, assuming the obtained duration probabilities (i.e., the probability matrix) are as shown in Table 4 above, it can be seen from Table 4 that the probability value corresponding to a duration of 2 is the highest (0.75), indicating that the duration of the t-th speech frame is two speech frames. That is to say, the two speech frames following the t-th speech frame (i.e., the (t+1)-th and (t+2)-th frames contain less effective information (or no effective information at all), for example, the (t+1)-th and (t+2)-th frames are empty speech frames. Here, empty speech frames could be speech data collected when the user is not speaking, empty speech frames resulting from zero-padding, or speech data collected when the user is speaking and includes prolonged sounds, etc., without limitation.

[0164] Alternatively, the characters corresponding to the two audio frames following the t-th audio frame (i.e., the t+1-th and t+2-th frames) are the same as the characters corresponding to the t-th audio frame.

[0165] In this embodiment, the joint network can determine the duration of each speech frame based on the duration probability, such as calculating the number of frames corresponding to the duration of each speech frame. For example, after obtaining the duration probability, the joint network can calculate the corresponding number of frames according to the following expression:

[0166] Where duration represents the number of frames corresponding to the duration, and argmax represents the function. This represents the probability of duration.

[0167] In this embodiment of the application, after the joint network calculates the number of frames corresponding to the duration of each audio frame, since the number of frames corresponding to the duration indicates that the audio frame following the audio frame is an empty audio frame, or the character corresponding to the audio frame following the audio frame is the same as the character corresponding to the audio frame, the electronic device can skip the audio frames following the audio frame and not perform character recognition on the audio frames following the audio frame, thereby accelerating the recognition of audio data.

[0168] Optionally, assume that the number of frames corresponding to the duration of the t-th speech frame is 2. Then, the electronic device can skip the (t + 1)-th speech frame and the (t + 2)-th speech frame, and does not perform character recognition on the (t + 1)-th speech frame and the (t + 2)-th speech frame. Instead, it starts character recognition from the (t + 3)-th speech frame. That is, when duration > 1, for the T speech frames, when the electronic device traverses each speech frame in the T speech frames in a loop to recognize the character corresponding to each speech frame, the actual number of loops will be less than T. Therefore, the complexity of speech recognition can be reduced, and accelerated recognition of speech data can be achieved.

[0169] Exemplarily, when duration > 1, the electronic device can update the decoded text sequence and continue to execute the above steps a - step h to continue recognizing the character corresponding to the (t + n)-th speech frame. Here, n is the number of frames corresponding to the duration of the t-th speech frame, and n is a positive integer greater than 1.

[0170] For example, assume that when recognizing the character corresponding to the t-th speech frame, the decoded text sequence input to the prediction network is "Please_play_sound", and the two speech frames following the t-th speech frame are empty speech frames. Then, the updated decoded text sequence can be "Please_play_sound_blk_blk". Or, assume that when recognizing the character corresponding to the t-th speech frame, the decoded text sequence input to the prediction network is "Please_play_sound", and the characters corresponding to the two speech frames following the t-th speech frame are the same as the character corresponding to the t-th speech frame. Then, the updated decoded text sequence can be "Please_play_sound_sound_sound".

[0171] It should be noted that in the embodiments of the present application, the order of execution of step g and step h is not limited and depends on the specific implementation. Exemplarily, the electronic device can first execute step g and then execute step h; or, the electronic device can also first execute step h and then execute step g; or, the electronic device can execute step g and step h simultaneously, which is not limited.

[0172] In some alternative embodiments of the present application, after the electronic device finishes performing the above-mentioned step g, that is, obtaining the blank probability, if the blank probability is less than the blank probability threshold, the electronic device continues to perform step i. Alternatively, after the electronic device finishes performing the above-mentioned step g, that is, obtaining the blank probability, if the blank probability is less than the blank probability threshold, the electronic device continues to perform step h, that is, obtaining the duration probability. If the value of the duration corresponding to the duration probability is less than or equal to 1, the electronic device continues to perform step i. Alternatively, the electronic device first performs the above-mentioned step h. After obtaining the duration probability, the electronic device calculates the duration of the t-th speech frame based on the duration probability. The value corresponding to this duration is used to represent the number of speech frames. On this basis, if the value corresponding to the duration is greater than 1, the electronic device continues to perform step i. After recognizing the character corresponding to the t-th speech frame, the electronic device skips the number of frames represented by the value corresponding to the duration and starts from the (t + n)-th speech frame to re-perform the above-mentioned steps a-step h. Here, n refers to the number of frames represented by the value corresponding to the duration.

[0173] Step i: The joint network obtains the non-blank probability based on the joint vector.

[0174] Exemplarily, the joint network inputs the joint vector into the fully connected layer 5. After the fully connected layer 5 performs linear processing on the joint vector, through the third activation function (such as the softmax function), the non-blank probability is obtained. Here, the third activation function is used to map the non-blank probability obtained after linear transformation of the fully connected layer 5 to the range of 0 to 1.

[0175] Exemplarily, in the case where the non-blank symbol includes one or more characters, the joint network calculates the probability matrix corresponding to the next character through the fully connected layer 5 and the third activation function. Suppose that during the processing of speech data, the updated value of the decoded text sequence is "please_play". At this time, the joint network will output the probability matrix of the character next to "please_play".

[0176] Suppose the obtained probability matrix is the same as Table 3. At this time, the decoding module searches according to the probability and determines the character "sound" with the highest probability among them, and places this character after the decoded text sequence "please_play". Then, the corresponding speech recognition result obtained is "please_play_sound".

[0177] Part Four: Describe the scenarios where the speech recognition model is used.

[0178] First, according to the requirements for the real-time performance of speech recognition in the embodiments of the present application, the application scenarios are divided into two categories:

[0179] The first category of scenarios involves applications with high real-time requirements, such as voice assistants, AI-powered calls, voice input methods, dictation systems, and real-time translation programs. In this category, real-time audio stream acquisition and recognition are required, enabling the system to recognize the user's speech as they speak. Furthermore, the duration of the user's speech is often unpredictable, necessitating continuous audio recognition.

[0180] The second type of scenario: Applications with lower real-time requirements, namely offline speech recognition, such as local speech data recognition. In this second type of scenario, the recognition of each piece of speech data is a relatively independent operation, and a complete piece of speech data can often be obtained at once, with relatively low real-time requirements.

[0181] The speech recognition method provided in this application can be applied to the needs of any of the above-mentioned scenarios. Taking a voice assistant as an example, for instance, a user can activate the voice assistant through a corresponding function switch. For example, a user can activate the voice assistant by setting a function switch in the application; or, a user can activate the voice assistant by inputting a wake word (such as "Hello, YOYO"); or, a user can also activate the voice assistant by pressing and holding the power button, etc., without limitation. Figure 11 As shown in (a) above, taking a mobile phone as an example, the phone displays the voice assistant's startup interface, which includes a microphone icon 20 and a prompt message (such as "Welcome~"). In actual applications, other content and quantities of prompt messages can be used, or the prompt message can be omitted.

[0182] After the voice assistant is activated, the phone can initiate voice recognition and use the microphone to capture the user's voice data, while simultaneously loading the voice recognition model into memory. Upon capturing voice data, the processor runs the voice recognition model, performing voice recognition using the process described in Part Three above, and simultaneously displaying the voice recognition result on the screen. Assuming the voice recognition result is "Please play music," then... Figure 11 As shown in (b), the phone will respond based on the voice recognition result and display the response on the screen. For example, the response to "Please play music" would be "Music has started playing".

[0183] Taking AI-powered call scenarios as an example, such as... Figure 12 As shown, the phone displays the call interface. After the AI ​​call function is activated, the phone can use the microphone to pick up the voice data played by the speaker (i.e., pick up the voice data from the other end), and simultaneously load the voice recognition model into memory. When voice data is picked up, the processor runs the voice recognition model, performs voice recognition using the voice recognition process described in Part 3 above, and simultaneously displays the voice recognition result on the screen. Figure 12As shown, the voice recognition result displayed on the phone is "Hello, I am xxx". Correspondingly, the phone can also use its microphone to pick up the user's voice data (i.e., pick up the voice data on its own end), and after performing voice recognition using a voice recognition model, simultaneously display the voice recognition result on the screen. For example, the recognition result might be "Sorry, I'm not available right now".

[0184] For the second type of scenario, the difference from the first is that in the second type, the voice data processed each time can be acquired completely at once. Furthermore, the voice data in this type of scenario does not need to be collected in real time; it is generally read from the existing voice data stored on the phone. After acquiring the voice data to be recognized, the voice recognition method in the third part is then used to recognize the voice data.

[0185] Let's take voice-based social networking software as an example, such as... Figure 13 As shown in (a) above, the chat interface of the social software includes voice data 21 and voice data 22. (In actual applications, the voice data included in chat decoding may differ from the embodiments of this application; this is merely an example). When a user needs to recognize a specific voice data, they can long-press the corresponding voice data. For example, if a user needs to recognize voice data 21, they can refer to... Figure 13 As shown in (b), after performing voice recognition, the phone can simultaneously display the voice recognition result in the chat decoding, such as the voice recognition result being "Want to go cycling this weekend?". Correspondingly, if the user wants to recognize other voice data, they can long-press the corresponding voice data, and the phone will then perform the above operation to recognize the new voice data.

[0186] The speech recognition method provided in this application can be applied to electronic devices that support speech recognition, such as smartphones, personal digital assistants (PDAs), laptops, tablets, handheld computers, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, PDAs, and augmented reality (AR) / virtual reality (VR) devices. This application does not impose any special limitations on the specific form of the electronic device.

[0187] refer to Figure 14 This is a hardware structure diagram of an electronic device provided in an embodiment of this application. Figure 14As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a positioning module 181, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0188] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0189] Processor 110 may include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0190] The charging management module 140 receives charging input from a charger, which can be either a wireless or wired charger. The power management module 141 connects to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0191] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0192] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include one or more filters, switches, power amplifiers, low-noise amplifiers (LANs), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the demodulated signal and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functions of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0193] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices within one or more communication processing modules. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0194] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0195] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-LED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0196] Electronic device 100 can perform shooting functions through an ISP, camera 193, video codec, GPU, display 194, and application processor. The ISP processes data fed back from the camera 193. For example, when taking a picture, the shutter is opened, light is transmitted through the lens to the camera's photosensitive element, the light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into a visible image. The ISP can also perform algorithmic optimization of image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature. In some embodiments, the ISP can be integrated into the camera 193.

[0197] Camera 193 is used to capture images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0198] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0199] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing by running the instructions stored in internal memory 121. For example, processor 110 can display different content on display screen 194 in response to an operation to unfold display screen 194 by executing instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phone book, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0200] Electronic device 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor, such as music playback and recording.

[0201] Buttons 190 include a power button and volume buttons. Buttons 190 can be mechanical buttons or touch buttons. Electronic device 100 can receive button inputs and generate key signal inputs related to user settings and function control of electronic device 100. Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with electronic device 100. Electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1.

[0202] In this application embodiment, in conjunction with the above Figure 14The hardware structure of the electronic device 100 shown is exemplified by the electronic device 100 including a microphone 170C, a processor 110, a memory (such as internal memory 121), and a display device (such as a display screen 194). For example, the microphone 170C is used to collect user voice data, the internal memory 121 is used to load a speech recognition model, and the processor 110 is used to perform calculations on the speech recognition model, process the voice data, and thus obtain the corresponding speech recognition result.

[0203] It should be noted that the contents described in the various embodiments of this application can be used to explain the technical solutions in other embodiments of this application. The technical features described in each embodiment can also be applied in other embodiments, and new solutions can be formed by combining the technical features in other embodiments. This application only provides an exemplary list of several embodiments for illustration and does not mean that this application is limited thereto.

[0204] This application provides an electronic device that may include a memory, one or more processors, and a display screen. The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs the various functions or steps performed by the mobile phone in the above embodiment. The structure of the electronic device can be referred to the above. Figure 14 The structure of the electronic device 100 shown.

[0205] This application also provides a chip system for use in electronic devices. For example... Figure 15 As shown, the chip system 1100 includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 can be one of the embodiments described above. Figure 14 The processor 110 is shown. Based on this, the interface circuit 1102 can be, for example, an interface circuit between the processor 110 and external memory; or an interface circuit between the processor 110 and internal memory.

[0206] The processor 1101 and interface circuit 1102 described above can be interconnected via a line. For example, interface circuit 1102 can be used to receive signals from other devices (e.g., the memory of electronic device 100). As another example, interface circuit 1102 can be used to send signals to other devices (e.g., processor 1101). Exemplarily, interface circuit 1102 can read instructions stored in memory and send those instructions to processor 1101. When the instructions are executed by processor 1101, the electronic device can perform the various functions or steps performed by the mobile phone in the above embodiments. Of course, the chip system may also include other discrete components, and this application embodiment does not specifically limit this.

[0207] This application also provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiments.

[0208] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the mobile phone in the above method embodiments.

[0209] It should be noted that the terms "first" and "second," etc., in the embodiments and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0210] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0211] It should be understood that in this application, "at least one (item)" means one or more. "More than one" means two or more. "At least two (items)" means two or three or more. "And / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple. Both "...when" and "if" indicate that a corresponding action will be taken under certain objective circumstances. They are not time limits, nor do they require a judgment action to be taken when the action is taken, nor do they imply any other limitations.

[0212] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0213] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules according to the system, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0214] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0215] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0216] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional unit.

[0217] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0218] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A voice recognition method, characterized by, The method utilizes a speech recognition model to process speech data, the speech data containing T frames of speech frames, T being a positive integer; the speech recognition model comprising a joint network, the joint network comprising at least one fully connected layer, the at least one fully connected layer comprising a first fully connected layer and a second fully connected layer; the method comprising: processing, by the speech recognition model, a t-th frame of speech frames in the T frames of speech frames to obtain a text prediction value corresponding to the t-th frame of speech frames; wherein the text prediction value comprises a blank probability and a duration; the blank probability is obtained after the first fully connected layer and is used to indicate a probability that a character corresponding to the t-th frame of speech frames is a blank symbol; the duration is obtained after the second fully connected layer and is used to indicate a duration of the t-th frame of speech frames; in a case where the blank probability is greater than or equal to a blank probability threshold, when the duration is greater than 1, processing, by the speech recognition model, a t+n-th frame of speech frames in the T frames of speech frames to obtain a text prediction value corresponding to the t+n-th frame of speech frames; n being the duration.

2. The method of claim 1, wherein, The at least one fully connected layer further comprises a third fully connected layer; in a case where the blank probability is less than a blank probability threshold, the text prediction value corresponding to the t-th frame of speech frames further comprises a non-blank probability; the non-blank probability is obtained after the third fully connected layer and is used to indicate a probability that a character corresponding to the t-th frame of speech frames is a non-blank symbol.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: when the duration is equal to 1, processing, by the speech recognition model, a t+1-th frame of speech frames in the T frames of speech frames to obtain a text prediction value corresponding to the t+1-th frame of speech frames.

4. The method according to any one of claims 1-3, characterized in that, The speech recognition model further comprises an encoder and a prediction network; wherein the processing, by the speech recognition model, of the t-th frame of speech frames in the T frames of speech frames to obtain the text prediction value corresponding to the t-th frame of speech frames comprises: inputting the t-th frame of speech frames into the encoder to obtain an acoustic feature vector; inputting a decoded text sequence corresponding to the t-th frame of speech frames into the prediction network to obtain a text feature vector; inputting the acoustic feature vector and the text feature vector into the joint network to obtain the text prediction value corresponding to the t-th frame of speech frames.

5. The method of claim 4, wherein, The at least one fully connected layer further comprises a fourth fully connected layer and a fifth fully connected layer; wherein the inputting of the acoustic feature vector and the text feature vector into the joint network to obtain the text prediction value corresponding to the t-th frame of speech frames comprises: linearly processing the acoustic feature by the fourth fully connected layer and linearly processing the text feature vector by the fifth fully connected layer; wherein the linear processing is used to make the acoustic feature vector and the text feature vector have the same vector dimension.

6. The method of claim 5, wherein, The joint network further comprises a fusion function; the method further comprises: performing vector accumulation processing on the acoustic feature vector and the text feature vector having the same vector dimension by the fusion function to obtain a joint vector.

7. The method of claim 6, wherein, The joint network further comprises a first activation function and a second activation function; the method further comprises: The joint vector passes through the first full connection layer and the first activation function to obtain the blank probability; The joint vector passes through the second full connection layer and the second activation function to obtain the duration probability, and the duration probability passes through the preset function to obtain the duration.

8. The method according to claim 6 or 7, characterized in that, The joint network further comprises a third activation function; and the method further comprises: The joint network passes through the third full connection layer and the third activation function to obtain the non-blank probability.

9. An electronic device, comprising: Comprise: a memory and one or more processors; The memory stores computer program code comprising computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method as claimed in any one of claims 1-9.

10. A chip system, characterized by Applied to an electronic device, the chip system comprises: at least one processor and an interface; The interface is used to receive instructions and transmit to the at least one processor; the at least one processor runs the instructions to make the electronic device execute the method as claimed in any one of claims 1-9.

11. A computer readable storage medium, characterized in that, Comprise: computer instructions; When the computer instructions run on the electronic device, the electronic device executes the method as claimed in any one of claims 1-9.

12. A computer program product, characterised in that, When the computer program product runs on the computer, the computer executes the method as claimed in any one of claims 1-9.

Images