Model processing method, device, electronic device and readable storage medium

Abstract

The embodiments of the present invention provide a model processing method, device, electronic device and readable storage medium. In the method, the frame features of the audio to be recognized are used as the input of the target acoustic model. For any frame feature, the attention module in the target acoustic model is used to calculate the first attention weight matrix based on the frame feature and the adjacent frame features within the first range of the frame feature. The range of the adjacent frame features used by the target acoustic model during the training process is a second range, and the second range is smaller than the first range. Based on the first attention weight matrix, the output of the target acoustic model is generated to determine the text corresponding to the audio to be recognized. To a certain extent, the processing time of the acoustic model during the training process can be shortened, thereby improving the processing efficiency. At the same time, since more adjacent frame features are used, the accuracy of audio recognition can be improved to a certain extent.

Description

Technical Field

[0001] The present invention belongs to the field of network technology, and in particular relates to a model processing method, device, electronic equipment and readable storage medium. Background Art

[0002] At present, with the continuous development of network technology, speech recognition is more and more widely used. When performing speech recognition, it is often necessary to combine acoustic models to realize speech recognition.

[0003] In the related art, when using acoustic models for speech recognition, there will be problems such as low processing efficiency and long time consumption. Summary of the invention

[0004] The present invention provides a model processing method, device, electronic device and readable storage medium to solve the problem of low efficiency in problem discovery.

[0005] In a first aspect, the present invention provides a model processing method, the method comprising:

[0006] Using the frame features of the audio to be recognized as the input of the target acoustic model;

[0007] For any frame feature, using the attention module in the target acoustic model, a first attention weight matrix is ​​calculated based on the frame feature and adjacent frame features within a first range of the frame feature; the range of adjacent frame features used by the target acoustic model during training is a second range, and the second range is smaller than the first range;

[0008] Based on the first attention weight matrix, the output of the target acoustic model is generated to determine the text corresponding to the audio to be recognized.

[0009] In a second aspect, the present invention provides a model processing device, the device comprising:

[0010] An input module, used for taking the frame features of the audio to be recognized as input of the target acoustic model;

[0011] A first calculation module is used to calculate a first attention weight matrix for any frame feature, using an attention module in the target acoustic model, based on the frame feature and adjacent frame features within a first range of the frame feature; the range of adjacent frame features used by the target acoustic model during training is a second range, and the second range is smaller than the first range;

[0012] A generation module is used to generate the output of the target acoustic model based on the first attention weight matrix to determine the text corresponding to the audio to be recognized.

[0013] In a third aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0014] In a fourth aspect, the present invention provides a readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the above method.

[0015] In an embodiment of the present invention, the frame features of the audio to be recognized are used as the input of the target acoustic model. For any frame feature, the attention module in the target acoustic model is used to calculate the first attention weight matrix based on the frame feature and the adjacent frame features within the first range of the frame feature. The range of the adjacent frame features used by the target acoustic model during the training process is the second range, and the second range is smaller than the first range; based on the first attention weight matrix, the output of the target acoustic model is generated to determine the text corresponding to the audio to be recognized. Compared with the method of directly using all other frame features in the whole sentence audio, in the present invention, since only the adjacent frame features within the second range are used during the training process, that is, only part of the adjacent frame features are used, the processing time of the acoustic model during the training process can be shortened to a certain extent, thereby improving the processing efficiency. At the same time, when processing the speech to be recognized, the first attention weight matrix is ​​calculated in combination with the adjacent frame features within the larger first range to realize speech recognition. Since more adjacent frame features are used, the accuracy of audio recognition can be improved to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a flow chart of steps of a model processing method provided by an embodiment of the present invention;

[0018] Figure 2 is a global schematic diagram showing an embodiment of the present invention;

[0019] Figure 3 is a weight diagram of an initial acoustic model provided by an embodiment of the present disclosure;

[0020] Figure 4 It is a partial schematic diagram provided by an embodiment of the present invention;

[0021] Figure 5is a schematic diagram of a model structure shown in an embodiment of the present invention;

[0022] Figure 6 is a schematic diagram of a reading method shown in an embodiment of the present invention;

[0023] Figure 7 is a schematic diagram of another reading method provided by an embodiment of the present invention;

[0024] Figure 8 is a schematic diagram of an accuracy rate provided by an embodiment of the present invention;

[0025] Fig. 9 is a schematic diagram of a real-time rate provided by an embodiment of the present invention;

[0026] Fig.10 is a structural diagram of a model processing device provided by an embodiment of the present invention;

[0027] Fig.11 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Figure 1 is a flow chart of steps of a model processing method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0030] Step 101: Using the frame features of the audio to be recognized as input of the target acoustic model.

[0031] The model processing method provided in the embodiment of the present invention can be applied to electronic devices. The audio to be recognized in the embodiment of the present invention can be audio that needs to be recognized as text. For example, the audio to be recognized can be the voice input by the user in the network platform. The target acoustic model can be a pre-trained acoustic model. Specifically, the audio to be recognized can be divided into multiple segments according to a preset length to obtain multiple audio frames. For example, assuming that the preset length is 10 milliseconds (ms), then an audio segment of every 10ms in the audio to be recognized can be used as an audio frame. Then, feature extraction can be performed on each audio frame to obtain frame features. Specifically, voice endpoint detection (Voice Activity Detection, VAD) can be performed first, and then feature extraction can be implemented based on preset operations such as Fourier transform. Finally, the frame features can be input into the target acoustic model for processing by the target acoustic model.

[0032] Step 102: For any frame feature, using the attention module in the target acoustic model, calculate a first attention weight matrix based on the frame feature and adjacent frame features within a first range of the frame feature; the range of adjacent frame features used by the target acoustic model during training is a second range, and the second range is smaller than the first range.

[0033] In an embodiment of the present invention, because audio information often has contextual relevance, an attention module is set in the target acoustic model, and the first attention weight matrix is ​​calculated according to the frame features and the adjacent frame features near the frame features through the attention model, so that the acoustic model can pay attention to the contextual relevance, thereby ensuring the recognition effect of the acoustic model to a certain extent. Among them, the adjacent frame features may include frame features directly adjacent to the frame features and frame features indirectly adjacent to the frame features. Further, the first range and the second range may be pre-set according to actual conditions. For example, the adjacent frame features within the first range may be partial / global frame features in the same row as the frame features to be calculated. Since the adjacent frame features within the second range used by the target acoustic model during the training process, that is, the partial frame features near the frame features are combined, it can ensure that the model learns the contextual relevance during the training process while avoiding the problem of taking too long. Further, in the process of using the trained target acoustic model for audio recognition, the first range, that is, the adjacent frame features within a larger range, are used, so that more sufficient contextual information can be provided for the actual recognition process to a certain extent, thereby ensuring the actual recognition effect.

[0034] Step 103: Based on the first attention weight matrix, generate the output of the target acoustic model to determine the text corresponding to the audio to be recognized.

[0035] In an embodiment of the present invention, the first attention weight matrix can be used as the input of the module after the attention module in the target acoustic model, and the output of the target acoustic model can be obtained after being processed by the subsequent modules. Among them, the target acoustic model can be used to construct a probability mapping relationship between the input speech and the output acoustic unit, that is, it can be used to output the pronunciation probability of each frame of audio according to the frame features of the input audio frame (the probability that each frame feature belongs to a certain category, and one category represents a pronunciation). Further, after obtaining the output of the target acoustic model, the decoder can be combined with the output of the target acoustic model and the scores of the preset language model on different phrase combinations for screening, so as to obtain the final recognition result of the speech to be recognized, that is, the text corresponding to the audio to be recognized.

[0036] The model processing method provided by the embodiment of the present invention uses the frame features of the audio to be recognized as the input of the target acoustic model. For any frame feature, the attention module in the target acoustic model is used to calculate the first attention weight matrix based on the frame feature and the adjacent frame features within the first range of the frame feature. The range of the adjacent frame features used by the target acoustic model during the training process is the second range, and the second range is smaller than the first range; based on the first attention weight matrix, the output of the target acoustic model is generated to determine the text corresponding to the audio to be recognized. Compared with the method of directly using all other frame features in the whole sentence audio, the present invention uses only part of the adjacent frame features. Therefore, the processing time of the acoustic model during the training process can be shortened to a certain extent, thereby improving the processing efficiency. At the same time, when processing the speech to be recognized, the first attention weight matrix is ​​calculated in combination with the adjacent frame features within the larger first range to realize speech recognition. Since more adjacent frame features are used, the accuracy of audio recognition can be improved to a certain extent.

[0037] Optionally, in an implementation of the embodiment of the present invention, the target acoustic model may be trained in the following manner:

[0038] Step S21, obtaining a trained initial acoustic model; the range of adjacent frame features used by the attention module in the initial acoustic model during the training process is a third range, and the third range is larger than the second range.

[0039] In an embodiment of the present invention, the initial acoustic model may be a pre-trained acoustic model, and the third range may be set according to actual needs. For example, the third range may cover all frame features input to the initial acoustic model, that is, the initial acoustic model is trained in the training process by combining adjacent frame features within the entire sentence range of the audio input to the initial acoustic model. For example, all other frame features that are in the same row as the frame feature, that is, adjacent to the left and right, may be used as adjacent frame features, and the initial acoustic model may be trained using global frame features. For example, Figure 2is a global schematic diagram showing an embodiment of the present invention, such as Figure 2 As shown, the acoustic model processes data frame by frame during training. The boxes covered by slashes can represent the frame features of each calculation. The other frame features (other boxes) in the same row as the frame features are the adjacent frame features within the third range of the frame features. It can be seen that this method is equivalent to using the scope of the entire sentence to determine the attention weight matrix. Furthermore, an open source acoustic model trained using global frame features can be obtained as the initial acoustic model, thereby obtaining a trained initial acoustic model. Alternatively, the initial acoustic model can be obtained based on autonomous training of global frame features, which is not limited in the embodiments of the present invention.

[0040] Step S22: using the second range as the range of adjacent frame features in the adjustment phase, and adjusting and training the initial acoustic model to obtain the target acoustic model.

[0041] In an embodiment of the present invention, the adjustment stage may be a stage of fine tuning the initial acoustic model. The operation of adjusting and training the initial acoustic model may essentially be equivalent to fine tuning the initial acoustic model. It should be noted that the execution subject of the operation of training to obtain the target acoustic model may be the same as the execution subject of recognizing the audio to be recognized, that is, both are the aforementioned electronic devices. Alternatively, they may be different, that is, the execution subject of the operation of training to obtain the target acoustic model may be other devices, for example, a server, and the embodiment of the present invention does not limit this.

[0042] In an embodiment of the present invention, a trained acoustic model obtained by training using all adjacent frame features during the training process is obtained as an initial acoustic model. Then, based on the initial acoustic model, the second range is used as the range of adjacent frame features in the adjustment stage, and the initial acoustic model is adjusted and trained to obtain a target acoustic model, wherein the second range is smaller than the third range. In this way, since the initial acoustic model has been trained with adjacent frame features within a larger range. Therefore, based on the initial acoustic model, by adjusting to obtain the target acoustic model, the amount of calculation in the adjustment process can be reduced to a certain extent while ensuring the processing effect of the target acoustic model finally obtained, thereby reducing the acquisition cost of the target acoustic model. At the same time, on the basis of the trained initial acoustic model, further training is performed based on the second range, so that the training process is more sufficient, thereby improving the training effect.

[0043] Furthermore, since the trained initial acoustic model using global frame features often has local correlation, in the embodiment of the present invention, an acoustic model trained using global frame features is obtained as the initial acoustic model, and then the initial model can be adjusted in combination with local adjacent frame features, which can ensure the accuracy of the target acoustic model obtained by the final adjustment to a certain extent. Figure 3 is a weight diagram of an initial acoustic model provided in an embodiment of the present disclosure, Figure 3 The initial acoustic model is used to process a randomly selected audio data with 208 frames, and the attention weights learned at the 104th frame are visualized to facilitate viewing the relationship learned by the initial acoustic model on the speech data. Figure 3 As shown, Figure 3 Represents the different components (heads) of the attention module in the 6th layer of the network: "layer6_head0", "layer6_head1", "layer6_head2", "layer6_head3" and "layer6_head_ave", and the relationship between the weights learned in this sentence. As can be seen from the figure, the different heads in the attention module have learned significantly different focus points, and the main relevant parts are around the 104th frame, that is, unlike most text processing tasks, the speech information in the acoustic model has the characteristics of local correlation. Further, the following Table 1 shows the accuracy of the method combining the global frame feature decoding (represented by "Global" in Table 1) and the method combining the local frame feature decoding within the second range (represented by "Local(50)" in Table 1) on speech of different lengths:

[0044]

[0045] Table 1

[0046] It can be seen from Table 1 above that when the decoding method combined with the local frame feature decoding within the second range is used for direct decoding, the acoustic module suffers from performance loss (decreased accuracy), but the accuracy of the acoustic module does not deteriorate significantly. It can be seen that this decoding method can retain most of the learned information in the acoustic model, which is consistent with the local correlation obtained above. Therefore, the acoustic model obtained by training with global frame features is obtained as the initial acoustic model. Subsequently, the initial model can be further adjusted and optimized in combination with the local adjacent frame features, which can ensure the accuracy of the target acoustic model obtained by the final adjustment and ensure the processing performance to a certain extent.

[0047] Optionally, the operation of using the second range as the range of adjacent frame features in the adjustment stage and adjusting and training the initial acoustic model to obtain the target acoustic model may specifically include:

[0048] Step S31: use the sample frame features of the training audio as the input of the initial acoustic model, and start from the specified processing stage corresponding to the initial acoustic model, calculate the second attention weight matrix based on the sample frame features and the adjacent sample frame features within a second range of the sample frame features.

[0049] In an embodiment of the present invention, the training audio may be selected based on actual needs. For example, the training audio may include historical audio input by users in different scenarios in the network platform. The training audio may include multiple pieces of audio. The method for obtaining the sample frame features of each training audio may refer to the aforementioned implementation method for obtaining the frame features of the audio to be identified. The embodiment of the present invention does not limit this. Furthermore, the specified processing stage may be a stage involving the calculation of the attention weight matrix. The attention weight matrix may be used to characterize the frame correlation between frame features. The specified processing stage may be pre-set according to actual needs. For example, training starts from the specified processing stage, which may cover the entire training stage corresponding to the initial acoustic model, or may only cover part of the training stage. Furthermore, when calculating the attention weight matrix, it may be calculated based on the attention module in the initial acoustic model according to a preset calculation formula. Among them, the preset calculation formula may be:

[0050]

[0051] Furthermore, when obtaining the sample frame features and the adjacent sample frame features within the second range of the sample frame features for calculation, the adjacent sample frame features within the second range can be selected by windowing. For example, a sliding window can be used to limit the visible range of a certain frame feature for other frames during calculation, so that it can better focus on calculating the relationship between the frame feature and the features of the nearby frames. For example, Figure 4 is a partial schematic diagram provided by an embodiment of the present invention, such as Figure 4 As shown in the figure, the boxes covered by diagonal lines can represent the frame features of each calculation, and the boxes covered by meshes represent the adjacent frame features in the second range used to calculate the frame features. Since the adjacent frame features involved in each calculation are local partial frame features, the amount of calculation of the Attention operation can be reduced to a certain extent, and thus it can be ensured that no matter how long the sentence is, the acoustic model can have a linear complexity, thereby shortening the time required to process long sentences and reducing the time complexity. For example, the complexity can be reduced from O(N 2 ) is reduced to O(N).

[0052] Step S32: Based on the second attention weight matrix, adjust the parameters of the initial acoustic model to obtain the target acoustic model.

[0053] In this step, the output of the initial acoustic model can be obtained based on the second attention weight matrix. Then, based on the output and the labels corresponding to the sample frame features of the training audio, the parameters of the initial acoustic model are adjusted until the initial acoustic module meets the preset stop condition, thereby obtaining the target acoustic model. Among them, the labels corresponding to the training audio can be pre-acquired. Specifically, the training data of the acoustic model can be audio information, that is, training audio, and the annotation data can be text information at the character level. In order to perform neural network training at the frame level, the training audio can be divided into audio frames in the link of obtaining the sample frame features to obtain sample audio frames. Then, the frame features of the sample audio frames are extracted to obtain sample frame features. Further, unlike conventional supervised tasks, the number of characters in the annotation data is often much smaller than the number of sample frame features in the training data, and the relationship between frame features and characters cannot be determined in advance. Therefore, the HMM-GMM model can be trained to determine how many types of states these frame features can be divided into and the alignment results of each frame feature at the state level (that is, pronunciation probability) according to the frame features and the annotation data, thereby obtaining the label (label) of the sample frame feature.

[0054] Furthermore, the loss value can be calculated based on the output and label of the initial acoustic model, and then the model parameters can be adjusted based on the loss value to optimize the acoustic model. Through multiple rounds of cyclic adjustment, the acoustic model can gradually converge in the fine-tuning stage. Finally, the fine-tuning can be stopped when the preset stop condition is reached. Among them, the preset stop condition can be set according to actual needs. For example, the preset stop condition can be that the number of parameter adjustments reaches a preset number of thresholds, or the current loss value is less than a preset threshold, and so on.

[0055] In an embodiment of the present invention, the sample frame features of the training audio are used as the input of the initial acoustic model, and starting from the designated processing stage corresponding to the initial acoustic model, the second attention weight matrix is ​​calculated based on the sample frame features and the adjacent sample frame features within a second range of the sample frame features; based on the second attention weight matrix, the parameters of the initial acoustic model are adjusted to obtain the target acoustic model. In this way, based on the initial acoustic model trained using a large range of adjacent frame features, the initial acoustic model is further strengthened with adjacent frame features within a smaller range during the fine-tuning stage, so that the acoustic model can better focus on the ability to calculate the relationship between the frame features and the frame features nearby. At the same time, by further adjusting the parameters of the initial acoustic model in the future, the acoustic model can be further optimized, thereby obtaining a target acoustic model with better processing effect.

[0056] In actual application scenarios, the training process of the acoustic model may have multiple stages. For example, the training process of the hybrid acoustic model often has multiple stages. Optionally, the initial acoustic model in the embodiment of the present invention can be obtained by training in the first training stage and the second training stage, and the second training stage is located after the first training stage. Correspondingly, the aforementioned specified processing stage can be the second training stage. Among them, the first training stage and the second training stage are both stages that can fully realize the training of the acoustic model, and the first training stage and the second training stage may both involve the calculation operation of the attention weight matrix. In an embodiment of the present invention, the first training stage is performed first, and then the second training stage is performed to obtain the initial acoustic model, so that the accuracy of the initial acoustic model can be ensured, thereby ensuring the processing performance of the initial acoustic model. For example, the first training stage can be "LF-MMI" and the second training stage can be "SMBR". Among them, the adjacent frame features in the global range are used in the SMBR and LF-MMI training process, that is, Global LF-MMI and Global SMBR are performed. Accordingly, in an embodiment of the present invention, the SMBR model using Global Attention can be used as a basis, and fine-tuning can be started directly with SMBR as the designated processing stage. That is, Global LF-MMI training can be performed first, and then Global SMBR training can be further performed based on the training results. Next, fine-tuning can be started from the SMBR stage, that is, local SMBR is further performed on the results of Global SMBR training to obtain the target acoustic model. In the local SMBR training process, adjacent frame features within a local range are used.

[0057] Of course, in actual application scenarios, the first training stage can also be used as the designated processing stage, that is, fine-tuning can be performed starting from the LF-MMI stage, so that local LF-MMI and local SMBR can be performed in the fine-tuning stage. Alternatively, the initial acoustic model can also be obtained by training in only a single training stage, and the embodiments of the present invention are not limited to this. For example, the initial acoustic model can be an acoustic model obtained only through Global LF-MMI, and accordingly, fine-tuning can be performed starting from the LF-MMI stage, that is, during the fine-tuning process, local LF-MMI and local SMBR operations can be used to train and obtain the target acoustic model.

[0058] In the embodiment of the present invention, fine-tuning training is started from the second half of the training stage involving the initial acoustic model, which can reduce the implementation cost while optimizing the initial acoustic model through fine-tuning. At the same time, by selecting the acoustic model obtained through two training stages as the initial acoustic model, it can be ensured to a certain extent that the selected initial acoustic model has better processing performance, thereby providing a model foundation with better processing performance for the fine-tuning stage, thereby improving the model performance of the target acoustic model finally obtained.

[0059] Optionally, in an implementation of the present invention, the initial acoustic model and the specified processing stage may be selected by the following steps:

[0060] Step S41, obtaining a trained candidate acoustic model; the range of adjacent frame features used by the attention module in the candidate acoustic model during the training process is the third range, and different candidate acoustic models go through different training stages.

[0061] For example, the candidate acoustic models may include the acoustic models trained by Global LF-MMI and Global SMBR, and the acoustic models trained by Global LF-MMI only, wherein each candidate acoustic model is trained in combination with global adjacent frame features during the training process.

[0062] Step S42: For any alternative acoustic model, starting from each training stage of the alternative acoustic model, a third attention weight matrix is ​​calculated based on the frame features of the test sample and the adjacent sample frame features within the second range of the frame features to obtain the optimized alternative acoustic model.

[0063] In this step, the implementation method of calculating the third attention weight matrix can refer to the implementation method of the aforementioned related steps, which will not be repeated here. Furthermore, after calculating the third attention weight matrix, the output of the alternative acoustic model can be further determined based on the third attention weight matrix, and then the parameters of the alternative acoustic model are adjusted based on the output of the alternative acoustic model and the labels corresponding to the frame features of the test sample, so as to achieve optimization. Finally, after the preset stop condition is met, the optimized alternative acoustic model is obtained. Among them, the specific implementation method of these steps can refer to the implementation method of the aforementioned related steps, which will not be repeated here.

[0064] Step S43: Select an initial acoustic model from the candidate acoustic models based on the performance parameters of the optimized candidate acoustic models, and determine a designated processing stage corresponding to the initial acoustic model.

[0065] In this step, the performance parameter may be the accuracy of the alternative acoustic model. For example, the alternative acoustic models after optimization may be used to process audio sets in different scenarios to determine the accuracy of the alternative acoustic model. Further, the initial acoustic model may be selected based on the performance parameter. For example, the alternative acoustic model with the lowest training cost may be selected from the alternative acoustic models whose accuracy meets the preset requirements. For example, the obtained performance parameters may be shown in Table 2 below:

[0066]

[0067] Table 2

[0068] Among them, Global base model represents an alternative acoustic model, "SMBR" in Global base model represents an acoustic model trained with Global LF-MMI and Global SMBR, and "LF-MMI" in Global base model represents an acoustic model trained only with Global LF-MMI. Fine tune stage represents the initial training stage, "LF-MMI" in Fine tune stage represents starting from the LF-MMI stage and optimizing through local LF-MMI and local SMBR, and "SMBR" in Fine tune stage represents starting from the SMBR stage and optimizing through local SMBR. call center, clean, noise, and privacy represent audio sets in different scenarios, respectively. It can be seen that the accuracy of these methods is relatively high. Therefore, considering the training cost, the acoustic model trained with Global LF-MMI and Global SMBR can be used as the initial acoustic model, and the SMBR stage can be used as the designated processing stage.

[0069] It should be noted that the execution subject in the above steps S41 to S43 may be the same as the execution subject in the application link of the target acoustic model and the execution subject of the training target acoustic model, or may be different, and the embodiment of the present invention is not limited to this.

[0070] In an embodiment of the present invention, a trained alternative acoustic model is obtained, and the range of adjacent frame features used by the attention module in the alternative acoustic model during the training process is a third range, and different alternative acoustic models go through different training stages. For any alternative acoustic model, starting from each training stage of the alternative acoustic model, the third attention weight matrix is ​​calculated based on the frame features of the test sample and the adjacent sample frame features within the second range of the frame features to obtain the optimized alternative acoustic model. Based on the performance parameters of each optimized alternative acoustic model, an initial acoustic model is selected from the alternative acoustic model, and the designated processing stage corresponding to the initial acoustic model is determined. In this way, by selecting and determining the performance parameters, selecting the initial acoustic model and the designated processing stage based on the performance parameters, the performance of the target acoustic model obtained by the final training can be ensured to a certain extent.

[0071] Optionally, before using the frame features of the audio to be recognized as the input of the target acoustic model, the following steps can also be performed: Step S51, deploying the target acoustic model to a preset online scoring library. Among them, the audio to be recognized can be the audio sent by the user in the online application link, and the online scoring library can provide online support for speech recognition for various projects of the network platform, thereby realizing various functions of the network platform and realizing joint optimization of engineering and algorithm models. For example, in a segmented speech recognition scenario, functions such as privacy number transcription and customer service transcription can be realized by deploying an online scoring library. Among them, the segmented scenario is a scenario where you can wait until a sentence is finished before recognition, and the information of a whole sentence can be used in the recognition process.

[0072] Accordingly, the above-mentioned operation of generating the output of the target acoustic model based on the first attention weight matrix to determine the text corresponding to the audio to be recognized may specifically include:

[0073] Step S61: Obtain the output of the target acoustic model based on the first attention weight matrix through the online scoring library.

[0074] For example, the first attention weight matrix can be used as the input of the module after the attention module in the target acoustic model, and the output of the target acoustic model can be obtained after being processed by the subsequent modules. For example, the acoustic model is an Encoder of Transformer, where Transformer is an end-to-end structure that uses Attention to model sequence relationships, which can be divided into Encoder and Decoder. The acoustic model can be regarded as the Encoder part. Figure 5 is a schematic diagram of a model structure shown in an embodiment of the present invention, such as Figure 5As shown in the figure, MHA (MultiHeadAttention) represents the main component of the Attention module. The operations performed by MHA can be regarded as n independent Attention operations. The calculation method of MHA can be described as follows: MultiHead(Q,K,V)=Concat(head1,…,head n )W O Among them, head i =Attention(QW i Q , KW i K , VW i V ). Further, after the Attention module calculates the first attention weight matrix, the FFN module can further process it to obtain the output of the target acoustic model. The FFN module can be composed of a linear layer (Linear) and an activation function (Relu). Accordingly, the calculation method of the FFN module can be described as follows: FFN(x) = max(0,xW1+b1)W2+b2. Figure 5 In the structure shown, by setting the LayerNorm (LN) layer at the end, it is possible to avoid skipping the entire transformer layer completely during the calculation process to obtain better performance. At the same time, setting the LN layer at the beginning and in the middle (i.e., using PreLN as the basic structure) can make the model more stable during the training process. Of course, other structures of the Encoder can also be used, for example, using the PostLN structure, which is not limited in the embodiment of the present invention.

[0075] In one implementation, a recurrent network structure is often used to train the feature sequence of audio frames, for example, LSTM and BiLSTM are used for training. The recurrent network structure processes data sequentially, and the recurrent operation of the hidden layer allows it to retain previously learned content and expand it in time steps. Specifically, the calculation of the hidden layer at time t requires the output of the hidden layer at time t-1, and the recurrent network structure uses this to transfer information at different times. In this way, due to the limitations of the recurrent mechanism itself, the calculation at time t depends on the result at time t-1, which greatly limits the parallel capability of the model, and due to the loss of information during the calculation process, long dependencies cannot be well modeled.

[0076] In the embodiment of the present invention, an acoustic model including an attention module is used. Since Attention is used as a method for modeling the relationship between sequences, the correlation between any two frame features can be calculated, and longer distances can be modeled equally without suffering from information loss during the transmission process. At the same time, the calculation of the correlation between frame features can be performed in parallel, thereby improving the parallel computing capability of the model, enhancing its modeling capability for long-distance relationships, optimizing the accuracy of speech recognition tasks, and obtaining a faster speed, thereby ensuring the processing capability of the acoustic model in the embodiment of the present invention. Furthermore, compared to the method of using a recurrent network, the network structure of the Transformer is used as the basic structure of the acoustic model in the embodiment of the present invention, which can improve the processing performance of the model to a certain extent. For example, the error rates of Transformer and BiLSTM on different test sets can be shown in Table 3 below:

[0077]

[0078]

[0079] Table 3

[0080] It can be seen that the error rate of Transformer Encode as an acoustic model is lower than that of BiLSTM. It can be seen that Transformer Encode has better processing performance. In the embodiment of the present invention, the network structure of Transformer is used as the basic structure of the acoustic model, which can achieve performance improvement to a certain extent.

[0081] Furthermore, since the global frame features are used to calculate the attention weight matrix, the Attention module will simultaneously calculate the correlation between the current frame features and all other frame features. For the local correlation of the audio, there are many redundant calculations, which will cause the Attention module to have O(N) times the sentence length L. 2 ), which results in more computational and memory overheads for the acoustic model when processing long data, resulting in lower real-time rates. In addition, for the network structure of Transformer Encode, related technologies are often applied to data in other fields. Since the data also has different characteristics, there is still room for optimization for speech tasks. For example, after reclassifying the test set data originally classified by scene according to length and testing their real-time rates, the following table 4 can be obtained:

[0082]

[0083] Table 4

[0084] It can be seen that when the length of the data exceeds 5s, the real-time rate will begin to deteriorate, and there will be a more obvious decrease in recognition speed. In actual application scenarios, there are often a large number of audios exceeding 5s. Therefore, simple TransformerEncode is often not used for tasks with high speed requirements, and there are usage restrictions. And under extreme conditions, it will cause problems in system performance and experience. In an embodiment of the present invention, the network structure of Transformer is used as the basic structure of the acoustic model. Combined with the characteristics of speech data, the Attention operation is performed with the frame features within the second range during the training process. At the same time, when using the trained target acoustic model for recognition, the Attention operation is performed in combination with the frame features within the first range, and optimization suitable for acoustic modeling tasks in speech recognition is made. To a certain extent, while ensuring the recognition accuracy, the complexity can be reduced to ensure the recognition speed of long sentences.

[0085] Step S62: Based on the output of the target acoustic model, generate text corresponding to the audio to be recognized as the output of the online scoring library.

[0086] In an embodiment of the present invention, the output of the acoustic model is the acoustic score. Accordingly, the online scoring library deployed with the target acoustic model can obtain the acoustic score output by the target acoustic model. Furthermore, the online speech recognition system can further generate the text corresponding to the audio to be recognized based on the acoustic score, thereby realizing the online output of the text corresponding to the audio to be recognized based on the online scoring library. By way of example, the operation of obtaining the acoustic score can be a part of the speech recognition system. Further, the generation of the text corresponding to the audio to be recognized can be realized based on other links in the speech recognition system. By way of example, the language score can be obtained based on the corpus through the language model, and then the language score and the acoustic score are combined through the decoder to generate the final recognition result, which is the text corresponding to the audio to be recognized.

[0087] In an embodiment of the present invention, the target acoustic model is first deployed to a preset online scoring library, and then, through the online scoring library, based on the first attention weight matrix, the output of the target acoustic model is generated. Based on the output of the target acoustic model, the text corresponding to the audio to be recognized is generated as the output of the online speech recognition system. In this way, online audio recognition can be achieved by deploying the target acoustic model in the online scoring library and performing online scoring based on the deployed target acoustic model. At the same time, since the accuracy of the target acoustic model is higher during processing, the accuracy of online audio recognition can be ensured to a certain extent.

[0088] Optionally, the second range in the embodiment of the present invention may be positively correlated with the minimum processing efficiency that can be tolerated in the training phase. As the second range is larger, more adjacent frame features will be involved in the calculation each time, and accordingly, the processing performance will be better, but it will result in lower processing efficiency for the audio. Therefore, in the embodiment of the present invention, the minimum processing efficiency that can be tolerated in the training phase can be determined first, wherein the minimum processing efficiency can be pre-set according to actual conditions. Then, the processing performance and processing efficiency are tested when different values ​​are used as the second range. Finally, the second range is selected based on the comprehensive processing performance and processing efficiency. For example, a processing efficiency that is not less than the value with the best processing performance in the minimum processing efficiency is selected and set to the second range. Among them, the processing efficiency can be characterized by time consumption, and the processing performance can be characterized by accuracy.

[0089] For example, taking the values ​​20, 50, and 35 as examples and the global range as a reference, the accuracy of the model on different data sets can be shown in Table 5 below:

[0090]

[0091] Table 5

[0092] Among them, Context Size represents the value of the second range. It can be seen that when the value of Context Size is 50, the accuracy is already close to the accuracy of the model using adjacent frame features in the global range, that is, the processing performance is already close to the performance of Global Attention. Therefore, while meeting the minimum processing efficiency requirement, the value 50 can be selected as the second range. That is, when calculating the second attention weight matrix in the fine-tuning stage, the sample frame features can be used to take the 50 adjacent sample frame features on the left and right of the same row of the sample frame features.

[0093] Optionally, before the above operation of calculating the first attention weight matrix based on the frame features and the adjacent frame features within the first range of the frame features, the following operation may be further performed:

[0094] Step S71, reading multiple frame features and a first range of adjacent frame features of each of the frame features based on a preset sliding window unit; the total number of adjacent frame features within the first range and the frame features is an integer multiple of the frame features that can be covered by the preset sliding window unit in a coverable row, and the adjacent frame features and the frame features are located in the same row.

[0095] In an embodiment of the present invention, the preset sliding window unit may be pre-set, and for example, the preset sliding window unit may be a sliding window in the form of a chunk. The preset sliding window unit may cover multiple rows at the same time, and the frame feature row covered by the preset sliding window unit each time it is read is the coverable row, and the coverable row changes as the preset sliding window unit slides and reads, thereby ensuring reading efficiency. For each row, since the total number of adjacent frame features and frame features within the first range is an integer multiple of the frame features that can be covered by the preset sliding window unit in each row of frame features, after being read N times by the preset sliding window unit, the frame features read N times may be directly used to avoid performing unnecessary mask operations, thereby improving computing efficiency. For example, Figure 6 Schematic diagram of a reading method shown in an embodiment of the present invention. Figure 6 As shown, in order to obtain the same number of ( Figure 4 In the related art, it is necessary to perform mask operation on the read frame features after reading multiple times based on the preset sliding window unit 01, so that the number of frame features finally retained meets the requirements, that is, to achieve Figure 6 Further, Figure 7 is a schematic diagram of another reading method provided by an embodiment of the present invention, such as Figure 7 As shown, since the first range set in the recognition process is larger, and the total number of adjacent frame features and frame features within the first range is an integer multiple of the frame features that can be covered by the preset sliding window unit in each row of frame features, in the embodiment of the present invention, the frame features read by chunks are all frame features that can participate in the calculation, so the mask operation can be omitted, making the calculation more efficient and improving the processing speed. At the same time, compared with Figure 6 As shown in the figure, due to the additional adjacent frame features, the recognition accuracy can be further ensured to a certain extent, so that the target acoustic model can be improved in both recognition accuracy and processing speed. Figure 8 is a schematic diagram of an accuracy rate provided by an embodiment of the present invention, such as Figure 8 As shown in the figure, line a represents the accuracy of decoding directly using the Global model, and line b represents the accuracy of decoding directly based on the target acoustic model. Figure 6 The accuracy of the method shown in FIG. 1 is for decoding audio data with different average data lengths. The c line indicates that the method is directly based on the target acoustic model. Figure 7 The accuracy of the method shown in Figure 1 is to decode the audio data with different average data lengths. Figure 8 It can be seen that using Figure 6The accuracy of decoding in the manner shown is lower than that of line a, that is, this method has performance loss compared to the basic Global model. Figure 7 The decoding method shown has little loss in accuracy and even exceeds the performance of the basic Global model in some parts.

[0096] Further, Fig. 9 is a schematic diagram of a real-time rate provided by an embodiment of the present invention, such as Fig. 9 As shown, since the embodiment of the present invention combines the characteristics of voice data, the original complexity O(N 2 ) is reduced to O(N). Therefore, compared with the real-time rate of the basic Global model (indicated by the d line), Figure 7 The real-time rate (indicated by line e) of the decoding method shown is smaller, that is, a better real-time rate can be obtained in the embodiment of the present invention.

[0097] Furthermore, since the Local model (i.e., the target acoustic model) is obtained by Fine tuning on the Global model trained with global frame features, better performance than Local decoding can be obtained when global frame features are used for decoding. For example, "Context50 Local scoring" represents the method of using Local decoding based on the target acoustic model, and "Context50 Global scoring" represents the method of using global frame features for decoding based on the target acoustic model. The accuracy of these two methods on speech of different lengths can be shown in the following Table 6:

[0098]

[0099] Table 6

[0100] It can be seen that the overall performance of the "Context50 Global Scoring" method is better than that of the "Context50 Local Scoring" method. Therefore, the method of reading adjacent frame features within a larger first range to recognize the speech to be recognized in the embodiment of the present invention can improve the recognition accuracy while ensuring the processing performance.

[0101] The present invention is described below in conjunction with the background of speech recognition. With the birth of the first experimental system that can recognize the pronunciation of English numerals, the process of speech recognition began. In the first stage of the development of speech recognition, the HMM-GMM model was mainly used to model acoustics, and the accuracy rate was improved slowly. In the second stage of the development of speech recognition, the use of the DNN model significantly improved the accuracy of speech recognition. In the third stage, thanks to the development of end-to-end technology, the speech recognition network has become deeper and more complex, and the performance of speech recognition has been further improved by using end-to-end technology. However, the current acoustic model has a high processing complexity for long sentences. In a related technology, an approximate Attention calculation is performed by Performer. Specifically, Performer directly maps Q and K to Q' and K' of dimension L times m through a function φ, so that the product of Q' and K' is approximately equal to A, so that the multiplication associative law can be used to first multiply (K')T and V, and then multiply it with Q', and the Attention calculation is converted into a linear complexity of the sentence length L. However, there will be a large performance loss in this way. Therefore, m is often set very large, which is only suitable for very long sentences. For ordinary long sentences in actual application scenarios, it will increase the amount of calculation. If m is set to a small value, it will cause a large performance loss, and it is difficult to trade off the amount of calculation and performance loss. In another related technology, the frame features involved in the calculation are reduced by directly adding a mask. However, in this method, it is necessary to calculate the attention first and then add the mask to achieve it. Therefore, the actual processing speed is often low, and it often requires special implementation of the underlying layer, which is difficult to promote.

[0102] In the embodiment of the present invention, starting from the speech recognition task itself, combined with the characteristics of the audio data, the trained Global model is used as the initial acoustic model, and on the basis of the initial acoustic model, the local Attention calculation is performed in combination with the adjacent frame features within the second range to obtain the target acoustic model. Therefore, the amount of calculation can be reduced, the complexity of processing ordinary long sentences in actual application scenarios can be reduced, the processing speed can be improved, and a large performance loss can be avoided. At the same time, in the process of applying the target acoustic model, the adjacent frame features within the larger first range are used for decoding, so that the recognition accuracy can be further ensured while reducing the complexity, and the processing performance can be ensured.

[0103] Fig.10 is a structural diagram of a model processing device provided by an embodiment of the present invention. The device 20 may include:

[0104] An input module 201 is used to use the frame features of the audio to be recognized as input of the target acoustic model;

[0105] A first calculation module 202 is used to calculate a first attention weight matrix for any frame feature, using an attention module in the target acoustic model, based on the frame feature and adjacent frame features within a first range of the frame feature; the range of adjacent frame features used by the target acoustic model during training is a second range, and the second range is smaller than the first range;

[0106] The generation module 203 is used to generate the output of the target acoustic model based on the first attention weight matrix to determine the text corresponding to the audio to be recognized.

[0107] Optionally, the target acoustic model is obtained by training the following modules:

[0108] A first acquisition module is used to acquire a trained initial acoustic model; the range of adjacent frame features used by the attention module in the initial acoustic model during training is a third range, and the third range is larger than the second range;

[0109] An adjustment module is used to use the second range as the range of adjacent frame features in the adjustment stage, and adjust the initial acoustic model to obtain the target acoustic model.

[0110] Optionally, the adjustment module is specifically used to:

[0111] Using the sample frame features of the training audio as input of the initial acoustic model, and starting from a specified processing stage corresponding to the initial acoustic model, calculating a second attention weight matrix based on the sample frame features and adjacent sample frame features within a second range of the sample frame features;

[0112] Based on the second attention weight matrix, parameters of the initial acoustic model are adjusted to obtain the target acoustic model.

[0113] Optionally, the initial acoustic model is obtained through training in a first training stage and a second training stage, the second training stage is located after the first training stage, and the designated processing stage is the second training stage.

[0114] Optionally, the initial acoustic model and the specified processing stage are selected by the following modules:

[0115] A second acquisition module is used to acquire a trained candidate acoustic model; the range of adjacent frame features used by the attention module in the candidate acoustic model during training is the third range, and different candidate acoustic models go through different training stages;

[0116] a second calculation module, configured to calculate, for any candidate acoustic model, a third attention weight matrix based on frame features of a test sample and frame features of adjacent samples within a second range of the frame features, starting from each training stage of the candidate acoustic model, so as to obtain an optimized candidate acoustic model;

[0117] A selection module is used to select an initial acoustic model from the candidate acoustic models based on the performance parameters of the optimized candidate acoustic models, and determine a designated processing stage corresponding to the initial acoustic model.

[0118] Optionally, the device further comprises:

[0119] A reading module is used to read multiple frame features and a first range of adjacent frame features of each of the frame features based on a preset sliding window unit; the total number of adjacent frame features within the first range and the frame features is an integer multiple of the frame features that can be covered by the preset sliding window unit in a coverable row, and the adjacent frame features and the frame features are located in the same row.

[0120] Optionally, the device further comprises: a deployment module, configured to deploy the target acoustic model to a preset online scoring library;

[0121] The generating module is specifically used for:

[0122] Obtaining, through the online scoring library, an output of the target acoustic model based on the first attention weight matrix;

[0123] Based on the output of the target acoustic model, a text corresponding to the audio to be recognized is generated as the output of the online scoring library.

[0124] The model processing device provided by the embodiment of the present invention uses the frame features of the audio to be recognized as the input of the target acoustic model. For any frame feature, the attention module in the target acoustic model is used to calculate the first attention weight matrix based on the frame feature and the adjacent frame features within the first range of the frame feature. The range of the adjacent frame features used by the target acoustic model during the training process is the second range, and the second range is smaller than the first range; based on the first attention weight matrix, the output of the target acoustic model is generated to determine the text corresponding to the audio to be recognized. Compared with the method of directly using all other frame features in the whole sentence audio, in the present invention, since only the adjacent frame features within the second range are used during the training process, that is, only part of the adjacent frame features are used, the processing time of the acoustic model during the training process can be shortened to a certain extent, thereby improving the processing efficiency. At the same time, when processing the speech to be recognized, the first attention weight matrix is ​​calculated in combination with the adjacent frame features within the larger first range to realize speech recognition. Since more adjacent frame features are used, the accuracy of audio recognition can be improved to a certain extent.

[0125] The present invention also provides a method for acquiring a target acoustic model, which can be applied to a model acquisition end, and can include the above steps S21 to S22. The implementation of each step in the method and the technical effects that can be achieved can refer to the above-mentioned related descriptions, and will not be repeated here.

[0126] The present invention also provides an electronic device, see Fig.11 , including: a processor 301, a memory 302, and a computer program 3021 stored in the memory and executable on the processor, wherein the processor implements the method of the aforementioned embodiment when executing the program.

[0127] The present invention also provides a readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method of the aforementioned embodiment.

[0128] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0129] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0130] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0131] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0132] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0133] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0134] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0137] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A model processing method, characterized in that: The method comprises: Perform speech endpoint detection and extract frame features of the audio to be recognized based on Fourier transform operation; Using the frame features of the audio to be recognized as input of a target acoustic model; For any frame feature, using the attention module in the target acoustic model, a first attention weight matrix is ​​calculated based on the frame feature and adjacent frame features within a first range of the frame feature; the target acoustic model uses adjacent frame features within a second range during training, the second range is smaller than the first range, and the second range is positively correlated with the minimum processing efficiency that can be tolerated in the training stage; the attention weight matrix is ​​calculated using global frame features; Based on the first attention weight matrix, generating the output of the target acoustic model to obtain the pronunciation probability of each frame of audio; The pronunciation probability and the scores of the preset language model on different phrase combinations are combined to perform screening to determine the text corresponding to the audio to be recognized.

2. The method according to claim 1, characterized in that The target acoustic model is trained in the following way: Acquire a trained initial acoustic model; an attention module in the initial acoustic model uses adjacent frame features within a third range during training, wherein the third range is larger than the second range; The second range is used as the range of adjacent frame features in the adjustment stage, and the initial acoustic model is adjusted and trained to obtain the target acoustic model.

3. The method according to claim 2, characterized in that The step of taking the second range as the range of adjacent frame features in the adjustment stage and adjusting and training the initial acoustic model to obtain the target acoustic model includes: Using the sample frame features of the training audio as input of the initial acoustic model, and starting from a specified processing stage corresponding to the initial acoustic model, calculating a second attention weight matrix based on the sample frame features and adjacent sample frame features within a second range of the sample frame features; the specified processing stage is a stage involving calculating the attention weight matrix; Based on the second attention weight matrix, parameters of the initial acoustic model are adjusted to obtain the target acoustic model.

4. The method according to claim 3, characterized in that The initial acoustic model is obtained through training in a first training stage and a second training stage, the second training stage is located after the first training stage, and the designated processing stage is the second training stage.

5. The method according to claim 3 or 4, characterized in that: The initial acoustic model and the specified processing stage are selected in the following manner: Acquire a trained candidate acoustic model; the range of adjacent frame features used by the attention module in the candidate acoustic model during training is the third range, and different candidate acoustic models go through different training stages; For any candidate acoustic model, starting from each training stage of the candidate acoustic model, based on the frame features of the test sample and the adjacent sample frame features within the second range of the frame features, a third attention weight matrix is ​​calculated to obtain the optimized candidate acoustic model; Based on the performance parameters of the optimized candidate acoustic models, an initial acoustic model is selected from the candidate acoustic models, and a designated processing stage corresponding to the initial acoustic model is determined.

6. The method according to any one of claims 1 to 4, characterized in that: Before calculating the first attention weight matrix based on the frame features and adjacent frame features within the first range of the frame features, the method further includes: Based on a preset sliding window unit, multiple frame features and a first range of adjacent frame features of each of the frame features are read; the total number of adjacent frame features within the first range and the frame features is an integer multiple of the frame features that can be covered by the preset sliding window unit in a coverable row, and the adjacent frame features and the frame features are located in the same row.

7. The method according to any one of claims 1 to 4, characterized in that: Before using the frame features of the audio to be recognized as the input of the target acoustic model, the method further includes: deploying the target acoustic model to a preset online scoring library; The step of generating the output of the target acoustic model based on the first attention weight matrix to determine the text corresponding to the audio to be recognized includes: Obtaining, through the online scoring library, an output of the target acoustic model based on the first attention weight matrix; Based on the output of the target acoustic model, a text corresponding to the audio to be recognized is generated as the output of the online scoring library.

8. A model processing device, characterized in that: The device comprises: An input module is used to perform speech endpoint detection, extract frame features of the audio to be recognized based on Fourier transform operation; and use the frame features of the audio to be recognized as input of the target acoustic model; A first calculation module is used to calculate a first attention weight matrix for any frame feature, using an attention module in the target acoustic model, based on the frame feature and adjacent frame features within a first range of the frame feature; the target acoustic model uses adjacent frame features within a second range during training, and the second range is smaller than the first range; the attention weight matrix is ​​calculated using global frame features; A generation module is used to generate the output of the target acoustic model based on the first attention weight matrix to obtain the pronunciation probability of each frame of audio; and to perform screening based on the pronunciation probability and the scores of the preset language model on different phrase combinations to determine the text corresponding to the audio to be recognized.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

10. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 7.

Images