Method and device for estimating confidence coefficient of model recognition result, equipment and medium

Abstract

The invention relates to the technical field of machine learning, in particular to a confidence estimation method and device for a model recognition result, equipment and a medium. When to-be-recognized data is received, the to-be-recognized data is input into a preset forward recognition model; the forward identification model outputs a forward identification sequence which corresponds to the to-be-identified data and contains forward decoding information, and the forward identification sequence is input to a preset reverse identification model after reversing the direction, so that the reverse identification model outputs a reverse identification sequence which corresponds to the forward identification sequence and contains reverse decoding information; and inputting the forward decoding information and the reverse decoding information into a preset target confidence estimation model, so that the target confidence estimation model outputs a confidence value corresponding to the forward recognition model. The accuracy and reliability of the confidence value in the recognition model are improved, so that the overall performance of automatic recognition is further improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a confidence estimation method, device, equipment and medium for model recognition results. Background Art

[0002] Confidence values ​​are an important part of recognition model systems (e.g., Automatic Speech Recognition (ASR)), showing the system's confidence in the recognition results and playing a key role in many upstream and downstream tasks related to recognition, such as semi-supervised training, spoken language evaluation and feedback, etc.

[0003] In traditional recognition models, word posterior probability is a popular method to obtain reliable confidence values. However, for existing end-to-end recognition models, especially self-attention models represented by Transformer or Conformer, since the autoregressive decoder relies on the complete decoding history, it is difficult to generate a compact form similar to the decoding graph that can represent a large number of recognition hypotheses. This feature makes it extremely expensive to calculate word posterior probabilities in the recognition space, which is not practical. In view of this, an alternative method is to use the output of the decoder, that is, the softmax probability value, as a reference for the confidence value. However, since end-to-end models generally show too high confidence in the softmax probability value, its quality is often difficult to guarantee when used as a confidence value, and the effect is not good. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a confidence estimation method, device, equipment and medium for model recognition results, which improves the accuracy and reliability of the confidence value in the recognition model.

[0005] A first aspect of the present application provides a confidence estimation method for a model recognition result, the method comprising: receiving a task instruction including data to be identified; Inputting the data to be identified into a preset forward recognition model, and obtaining a forward identification sequence output by the forward recognition model; the forward identification sequence includes forward decoding information; Reversing the direction of the forward identification sequence and inputting it into a preset reverse recognition model, so that the reverse recognition model outputs a reverse identification sequence corresponding to the forward identification sequence; the reverse identification sequence includes reverse decoding information; The forward decoding information and the reverse decoding information are input into a preset target confidence estimation model so that the target confidence estimation model outputs a confidence value corresponding to the forward recognition model.

[0006] A second aspect of the present application provides a device for estimating the confidence of a model recognition result, the device comprising: A receiving module, used for receiving a task instruction containing data to be identified; A first output module, used to input the data to be identified into a preset forward recognition model, and obtain a forward identification sequence output by the forward recognition model; the forward identification sequence includes forward decoding information; A second output module is used to reverse the direction of the forward identification sequence and input it into a preset reverse recognition model, so that the reverse recognition model outputs a reverse identification sequence corresponding to the forward identification sequence; the reverse identification sequence includes reverse decoding information; The third output module is used to input the forward decoding information and the reverse decoding information into a preset target confidence estimation model, so that the target confidence estimation model outputs a confidence value corresponding to the forward recognition model.

[0007] A third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the confidence estimation method for the model recognition result when executing the computer program.

[0008] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the confidence estimation method for the above-mentioned model recognition result.

[0009] In summary, the confidence estimation method, device, equipment and medium of the model recognition result provided by the present application, when receiving a task instruction containing the data to be recognized, the forward recognition model will process the data to be recognized, and output a forward identification sequence containing forward decoding information, and then reverse the direction of the forward identification sequence and input it into the reverse recognition model, so that the reverse recognition model processes the reversed identification sequence, and outputs a reverse identification sequence containing reverse decoding information corresponding to the forward identification sequence, and based on the input forward decoding information and reverse decoding information, the target confidence estimation model outputs a confidence value, which reflects the reliability of the recognition result of the forward recognition model for the data to be recognized. By combining the forward recognition model, the reverse recognition model and the target confidence estimation model, two-way verification of the recognition results and acquisition of rich decoding information are achieved, the accuracy and reliability of the confidence value are improved, and the overall performance of the recognition task is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a training flow chart of a Transformer series end-to-end recognition model in the prior art; Figure 2It is a flow chart of a confidence estimation method of a model recognition result shown in an embodiment of the present application; Figure 3 It is a schematic diagram of a training process of a target confidence estimation model shown in an embodiment of the present application; Figure 4 is a structural schematic diagram of a target confidence value estimation model shown in an embodiment of the present application; Figure 5 It is a functional module diagram of a confidence estimation device for a model recognition result shown in an embodiment of the present application; Figure 6 It is a structural schematic diagram of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0012] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0013] Reference Figure 1 As shown, it is a training flowchart of a Transformer series end-to-end recognition model in the prior art.

[0014] The existing Transformer series end-to-end recognition models completely rely on the attention mechanism to establish global dependencies between input and output information, allowing more parallelization by placing the self-attention module in the encoder and decoder. In addition, the existing Transformer series end-to-end recognition models also usually use multiple encoder and decoder modules, so the cross-attention module connecting the encoder and decoder exists in each decoder module.

[0015] When the input layer of the existing Transformer series end-to-end recognition model receives the original speech signal or preprocessed speech features, such as Mel-frequency cepstral coefficients (MFCC), the embedding layer converts the feature vector of the input layer into a low-dimensional representation that is easier for the model to process. After the embedding layer, the existing Transformer series end-to-end recognition model also includes a linear transformation layer for adjusting the dimension of the embedding vector to match the input requirements of the subsequent layers. Next, the existing Transformer series end-to-end recognition model uses a multi-head attention mechanism to calculate in parallel through multiple independent attention heads, each head focusing on a different part of the input sequence to capture the relationship between different positions in the input sequence. After the multi-head attention mechanism, the existing Transformer series end-to-end recognition model also adds a residual connection (i.e., adding the input directly to the output) and then performs layer normalization to stabilize the training process and accelerate convergence. Secondly, the existing Transformer series end-to-end recognition model also provides two layers of fully connected neural networks, where the fully connected neural network includes a nonlinear activation function (such as ReLU) for further processing the features weighted by the attention mechanism. To prevent the model from seeing future information, a mask is applied to the multi-head attention mechanism so that the model can only rely on previous outputs and input information when generating each output.

[0016] Reference Figure 2 As shown, it is a flowchart of a confidence estimation method for a model recognition result shown in an embodiment of the present application, and the confidence estimation method for a model recognition result includes the following steps.

[0017] S21, receiving a task instruction including data to be identified.

[0018] In some embodiments, the electronic device may be provided with an interface for receiving task instructions, such as an API (application programming interface), a message queue, or a network request interface. When the interface receives a task instruction, the instruction is first parsed to extract the data to be identified and the task requirement information therein. The parsed data to be identified and the relevant task requirement information are distributed to the subsequent processing flow. In addition, when the user uploads the data to be identified, the task instruction corresponding to the data to be identified can also be automatically generated. Among them, the modality corresponding to the data to be identified is different in different application scenarios. For example, in a speech recognition scenario, the data to be identified may be the collected speech data; in an entity recognition scenario, the data to be identified may be text data.

[0019] S22, inputting the data to be recognized into a preset forward recognition model, and obtaining a forward identification sequence output by the forward recognition model.

[0020] The forward identification sequence includes forward decoding information. In the process of the forward recognition model generating the forward identification sequence, the forward recognition model can automatically mark or record the decoding key information (called forward decoding information) corresponding to the forward identification sequence, including the decoder output d, the cross attention c and the identification embedding.

[0021] In some embodiments, the data to be identified can be collected by a data acquisition device, and the data to be identified can be audio data, video data, image data, etc. Exemplarily, when the data to be identified is voice data in audio data, an audio input device can be used to collect voice signals, and the collected voice signals can be converted into digital form by sampling and quantization, etc. to obtain voice data. Wherein, the audio input device can be built into an electronic device, for example, a microphone of an electronic device, or it can be independent of the electronic device. When the audio input device is independent of the electronic device, the audio input device can transmit the collected voice data to the electronic device by wired or wireless means. When the data to be identified is video data, a video input device can be used to collect video signals, and the collected video signals can be converted into digital form by sampling and quantization. Wherein, the video input device can be built into an electronic device, for example, a camera of an electronic device, or it can be independent of the electronic device. When the video input device is independent of the electronic device, the video input device can transmit the collected video data to the electronic device by wired or wireless means. For other data to be identified, this application will not be illustrated one by one.

[0022] When the electronic device receives the data to be identified, the electronic device can input it into a preset forward recognition model so that the forward recognition model outputs a corresponding forward identification sequence based on the data to be identified, wherein the forward identification sequence includes multiple recognition results corresponding to the data to be identified.

[0023] Through the above optional implementation, by receiving the data to be recognized and inputting it into the pre-trained forward recognition model, the rapid conversion of data to text is achieved, and the corresponding forward identification sequence is output. At the same time, the forward recognition model automatically records the forward decoding information in the process of generating the forward identification sequence, providing data support for subsequent processing, and improving the efficiency and accuracy of recognition and processing.

[0024] In order to more clearly understand the inventive concept of the present application, the data to be recognized in the embodiments of the present application is explained by taking voice data as an example.

[0025] When it is necessary to identify the voice data and obtain the corresponding positive identification sequence, it is necessary to pre-train a forward recognition model for processing the voice data. In an optional embodiment, the electronic device can pre-train a forward recognition model based on the positive acoustic features and the positive identification sequence. Specifically, the electronic device first needs to prepare a large amount of voice data as a training sample set, wherein the voice data may include different speakers, speech speeds, intonations, background noise and other conditions to ensure the generalization ability of the forward recognition model. Then, the voice data is converted into positive acoustic features using existing cepstral coefficients such as Mel-Frequency Cepstral Coefficients (MFCC), and the corresponding positive identification sequence is generated according to the positive position information of the voice data. Among them, the positive acoustic feature refers to the abstract representation of the voice data, which enables the model to understand and analyze the specific content of the voice data, and the positive acoustic feature and the positive identification sequence are in a one-to-one correspondence, which is used for supervised learning during model training. Further, the electronic device can adopt the existing Transformer series end-to-end recognition model architecture, which completely relies on the attention mechanism to establish the global dependency between the input (forward acoustic features) and the output (forward identification sequence). The electronic device can select a loss function, such as a cross entropy loss function, to measure the difference between the model output and the actual forward identification sequence. The electronic device can also select an efficient optimizer, such as Adam or SGD, to update the weight parameters of the model to minimize the loss function. Then, the acoustic features are used as the model input, that is, the forward identification sequence is used as the model output, and the existing Transformer series end-to-end recognition model is trained. During the training process, the value of the loss function is calculated by forward propagation, and the weight parameters of the model are updated by back propagation. The training is iterated multiple times until the iteration termination condition is met, such as the loss function converges or the number of iterations meets the preset training rounds.

[0026] When the forward recognition model training is completed, the electronic device can use the test sample set to evaluate the performance of the forward recognition model, such as recognition accuracy, recall rate and other indicators. Based on the evaluation results, the electronic device can optimize the structure of the forward recognition model, such as increasing the number of encoder and decoder layers, adjusting the parameters of the attention mechanism, etc., and / or adjust the training strategy, such as learning rate, batch size, etc., according to the change in the performance of the forward recognition model, to further improve the performance of the forward recognition model.

[0027] It should be noted that the recognition model architecture includes multiple encoder modules and multiple decoder modules, wherein the encoder module is used to process the input acoustic features, and the decoder module is used to generate the output positive identification sequence. In addition, since the Transformer series model cannot directly understand the position information in the positive identification sequence, the electronic device can also introduce position encoding, wherein the position encoding is injected into the token embedding to provide the position information of the positive identification sequence.

[0028] In an optional embodiment, before inputting the data to be identified into a preset forward recognition model, the method further includes: Performing pre-emphasis processing on the data to be identified; Segmenting the data to be identified after the pre-emphasis processing according to a preset time interval to obtain a plurality of data segments; Performing a windowing operation and a Fourier transform on the multiple data segments to obtain a spectrogram feature corresponding to each data segment in the multiple data segments; The spectrogram features are input into the forward recognition model.

[0029] In some embodiments, when receiving the data to be identified, the electronic device may pre-process the data to be identified, such as noise reduction, pre-emphasis, frame processing, windowing and feature extraction. Specifically, the electronic device may first perform noise reduction on the data to be identified by using adaptive filtering, wavelet transform denoising, etc. to remove background noise and other non-target sound interference. Then, in order to compensate for the attenuation of the high-frequency part of the data to be identified, the electronic device may perform pre-emphasis on the data to be identified by using a first-order high-pass filter to improve the accuracy of subsequent spectrum analysis. Then, the electronic device may perform frame processing on the data to be identified after the pre-emphasis processing is completed, that is, to divide the continuous data to be identified into multiple data segments with a preset time interval (e.g., 10ms), wherein each data segment has a partial overlap. For each data segment, the electronic device may apply a window function (e.g., Hamming window, Hamming window, etc.) to each data segment to reduce the spectrum leakage caused by the signal mutation at the frame boundary and improve the smoothness and accuracy of the spectrum analysis. Furthermore, the electronic device can perform a Fast Fourier Transform (FFT) on each frame of data in each data segment, and after taking the square of the amplitude, perform logarithmic calculations on the amplitude of each spectral line to obtain a spectrogram, which contains spectrogram features such as the frequency and energy distribution of the data to be identified. Finally, the obtained spectrogram features are input into a preset forward recognition model for subsequent data recognition. In other embodiments, the electronic device can also use existing cepstrum coefficients such as MFCC to extract features of the data to be identified.

[0030] In some embodiments, when the received data to be identified includes multiple data segments, the electronic device can determine the sampling rate corresponding to each data segment, and control the data length of each data segment to be the same based on the sampling rate. Specifically, the electronic device can pre-set a sampling rate threshold (for example, 16kHz), compare the sampling rate of each data segment with the sampling rate threshold, and when the sampling rate of the data segment meets the sampling rate threshold, directly copy the data segment with a short data length to the maximum data length, otherwise use the toolkit sox to convert the data sampling rate so that it meets the sampling rate threshold, and then copy the data segment with a short data length to the maximum data length, so that the data length of each data segment is the same.

[0031] Through the above optional implementation, a series of preprocessing steps, such as noise reduction, pre-emphasis, framing, windowing, and feature extraction, improve the quality of the data to be recognized, and provide more accurate and smooth spectrogram features for subsequent recognition. Finally, the optimized spectrogram features are input into the forward recognition model, which improves the accuracy and robustness of recognition and provides users with a better recognition experience.

[0032] S23, reversing the direction of the forward identification sequence and inputting it into a preset reverse recognition model, so that the reverse recognition model outputs a reverse identification sequence corresponding to the forward identification sequence.

[0033] The reverse identification sequence includes reverse decoding information. In the process of generating the reverse identification sequence by the reverse recognition model, the reverse recognition model can automatically mark or record the decoding key information (called reverse decoding information) corresponding to the reverse identification sequence, including the decoder output d, the cross attention c and the identification embedding.

[0034] In some embodiments, according to the same embodiment as step S21, the electronic device can pre-train a reverse recognition model based on the reverse acoustic features and the reverse identification sequence. After obtaining the forward identification sequence output by the forward recognition model, the electronic device can reverse the direction of the forward identification sequence, that is, reverse the forward identification sequence, and then input the reversed sequence into the preset reverse recognition model, so that the reverse recognition model outputs the corresponding reverse identification sequence based on the reversed sequence. Exemplarily, assuming that the forward identification sequence is "ACCD", the reverse identification sequence is "DCCA".

[0035] It should be noted that the present application uses the reverse direction of the forward identification sequence as the input of the reverse recognition model, rather than directly using the reverse direction of the data to be identified as the input of the reverse recognition model. The purpose is to enable the reverse recognition model to utilize the information that has been processed by the forward recognition model (i.e., the forward decoding information in the forward identification sequence), but at the same time, by reversing the sequence, the reverse recognition model can maintain a certain degree of independence when processing data, avoiding complete overlap with the forward recognition model or producing the same results.

[0036] Through the above optional implementation, the complementarity of the forward recognition model and the reverse recognition model in the learning process can be promoted, and the robustness and accuracy of the entire recognition system can be improved. At the same time, since the forward recognition model and the reverse recognition model can process the same data from different angles and sequences, it helps the recognition model to better capture the bidirectional information or contextual information of the data to be recognized.

[0037] S24, inputting the forward decoding information and the reverse decoding information into a preset target confidence estimation model, so that the target confidence estimation model outputs a confidence value corresponding to the forward recognition model.

[0038] In some embodiments, the forward decoding information contained in the forward identification sequence output by the forward recognition model and the reverse decoding information contained in the reverse identification sequence output by the reverse recognition model are used as inputs of a preset target confidence estimation model, and the target confidence estimation model is used to process the forward decoding information and the reverse decoding information, and output a confidence value. The confidence value reflects the reliability or accuracy of the output result of the forward recognition model, and the confidence value ranges from 0 to 1. The closer to 1, the more the target confidence estimation model believes that the recognition result output by the forward recognition model is correct, and vice versa, the closer to 0, the more the confidence estimation model believes that the recognition result output by the forward recognition model is wrong.

[0039] In some embodiments, the electronic device may splice the forward decoding information and the reverse decoding information into a single input to obtain the spliced ​​decoding information, and then input the spliced ​​decoding information into the target confidence estimation model. Among them, the splicing method may include: direct splicing, separation using delimiters (such as commas, periods, or special marks), or converting the forward decoding information and the reverse decoding information into other representations (such as vectors or matrices). Exemplarily, assuming that the direct splicing method is selected, the characters of the forward decoding information and the reverse decoding information can be arranged in sequence. Assuming that the method of using delimiters is selected, one or more delimiters can be inserted between the forward decoding information and the reverse decoding information.

[0040] Refer to Figure 4, the forward decoding information and the reverse decoding information are input into the fully connected layer of the target confidence value estimation model, and then a value between 0 and 1 is generated through the Sigmoid output layer as the confidence value score of the current forward recognition model. The decoder output d as the forward decoding information, As the cross attention c of the forward decoding information, and As the identification embedding of the forward decoding information, or The decoder output d as the forward decoding information, As the cross attention c of the forward decoding information, and As the identifier embedding of the forward decoding information. The decoder output d as the forward decoding information, As the cross attention c of the forward decoding information, and As the identification embedding of the forward decoding information, The decoder output d as the reverse decoding information, As the cross attention c of reverse decoding information, and As the identification embedding of the reverse decoding information, , , , , and As the input information of the target confidence value estimation model, it is input into the fully-connected layer of the target confidence value estimation model.

[0041] Through the above-mentioned optional implementation, by introducing the reverse identification sequence output by the reverse recognition model, combining it with the forward identification sequence, and using the target confidence estimation model to output the confidence value, the reliability assessment accuracy of the output result of the forward recognition model is improved, which provides an effective means for the confidence judgment of the recognition system and enhances the stability and accuracy of the system.

[0042] In order to facilitate understanding of the inventive idea of ​​the present application, the corresponding upstream and downstream task examples are listed below to describe the practical application of the present application embodiment. In the upstream task, that is, when performing a model training task related to speech, the model training task often covers a large amount of data and may involve a supervised or unsupervised learning method. At this point, the confidence value estimation model provided by the present application can accurately score confidence for a large amount of training data, so as to effectively screen out training data with higher quality, thereby ensuring that the model training process can be based on a more reliable data set, thereby improving the efficiency and accuracy of model training. In addition, in the downstream task, especially when processing speech recognition tasks with different pronunciation features, the present application can accurately measure the accuracy of the speech recognition model when processing the speech data of a specific pronunciation by scoring confidence for the speech data of different pronunciations, and provides strong support for the performance evaluation and optimization of the speech recognition system in practical applications.

[0043] In an optional implementation, the electronic device may pre-train a target confidence estimation model based on the deeply fused bidirectional sequence information (i.e., the forward identification sequence and the reverse identification sequence). Figure 3 The specific training process of the target confidence estimation model is explained. Figure 3 A schematic diagram of the training process of a target confidence estimation model shown in an embodiment of the present application.

[0044] S31, obtaining a training data set.

[0045] The training data set includes a plurality of training data, and one training data corresponds to one original recognition sequence.

[0046] In some embodiments, the electronic device needs to prepare a large amount of recognized data as a training sample set of the initial confidence estimation model for subsequent training of the initial confidence estimation model to obtain a target confidence estimation model.

[0047] S32, obtaining a forward training sequence after recognizing the training data through the forward recognition model.

[0048] In some embodiments, for each training data in the training data set, the electronic device can pre-process each training data according to the same implementation method as step S22, and convert it into corresponding data features, and input the data features into the forward recognition model, thereby improving recognition accuracy and recognition efficiency while reducing the complexity and resource usage of the model.

[0049] After the preprocessing is completed, the electronic device can input the data features obtained by the preprocessing into the pre-trained forward recognition model for recognition processing. The forward recognition model can output multiple recognition results corresponding to each training data (called candidate recognition results). Among them, the forward recognition model includes multiple encoder modules and decoder modules. When the forward recognition model receives the data features corresponding to each input training data, the encoder module of the forward recognition model can encode the data features to obtain the corresponding encoding features. At this time, the decoder of the forward recognition model decodes the encoding features to obtain multiple candidate recognition results. Furthermore, the electronic device can evaluate the accuracy of the recognition results, that is, compare the candidate recognition results with the original recognition sequence corresponding to each training data, so as to select N candidate recognition results with the highest accuracy from multiple candidate recognition results as the target recognition results, where N is an integer greater than or equal to 1.

[0050] Since the original recognition sequence of the training data used for training is known, the electronic device can determine the difference between each candidate recognition result and the original recognition sequence based on the original recognition sequence, so as to determine the accuracy rate corresponding to each candidate recognition result. Specifically, the electronic device can construct a confusion matrix based on the candidate recognition results and the original recognition sequence, where the confusion matrix is ​​a table that displays the relationship between the candidate recognition results and the original recognition sequence. Then, the number of true positives (True Positive, TP), false positives (False Positive, FP), true negatives (True Negative, TN) and false negatives (False Negative, FN) is counted from the confusion matrix, and the accuracy rate is determined based on the following formula: Accuracy = (TP + TN) / (TP + TN + FP + FN).

[0051] After determining the accuracy rate corresponding to each candidate recognition result, the electronic device sorts the multiple candidate recognition results from large to small based on the accuracy rate, places the candidate recognition results with large accuracy rates in front, and places the candidate recognition results with small accuracy rates in the back, and then selects N candidate recognition results from the multiple candidate recognition results after sorting as target recognition results. Further, after selecting N target recognition results with the highest accuracy rates from multiple candidate recognition results, the electronic device can convert the target recognition results into a sequence form to obtain a forward training sequence for training. For example, the electronic device can splice the target identification results together as a forward training sequence, or perform word segmentation or character segmentation on the target recognition results, and construct the results after word segmentation or character segmentation in order as a forward training sequence.

[0052] Through the above optional implementation, the training data after preprocessing is input into the pre-trained forward recognition model, so as to achieve efficient and accurate recognition of the training data. By constructing a confusion matrix and calculating the accuracy, the accuracy of each candidate recognition result can be accurately evaluated, and the N candidate recognition results with the highest accuracy are selected as the target recognition results, which improves the accuracy and efficiency of recognition and reduces the complexity and resource usage of the model.

[0053] S33: Output a reverse training sequence corresponding to the forward training sequence through the reverse recognition model.

[0054] Similarly, according to the same embodiment of step S23, the forward training sequence is reversed and input into a preset reverse recognition model, so that the reverse recognition model outputs a reverse training sequence corresponding to the forward training sequence.

[0055] S34, obtaining a judgment label.

[0056] The judgment label is outputted by a preset initial confidence estimation model based on the forward training sequence and the reverse training sequence.

[0057] In some embodiments, the electronic device can build an initial confidence estimation model based on machine learning or deep learning (for example, convolutional neural network (CNN), recurrent neural network (RNN), etc.). The parameter information such as the level of the initial confidence estimation model can be designed according to the requirements of the training task. Figure 4 As shown, the initial confidence estimation model may include a fully-connected layer and a Sigmoid output layer, or may be an existing model structure.

[0058] In some embodiments, when the forward recognition model outputs the forward training sequence corresponding to the training data, it also outputs the forward decoding key information (referred to as the first decoding key information) corresponding to the forward training sequence, including the decoder output d, the cross attention c and the identifier embedding. Similarly, when the reverse recognition model outputs the reverse training sequence, it also outputs the reverse decoding key information (referred to as the second decoding key information) corresponding to the reverse training sequence, also including the decoder output d, the cross attention c and the identifier embedding. Then, the electronic device can splice the first decoding key information and the second decoding key information according to the same embodiment of step S24 to obtain a spliced ​​decoding key information. The spliced ​​decoding key information is then input into the initial confidence estimation model, so that the confidence estimation model determines whether the recognition result output by the forward recognition model is accurate based on the spliced ​​decoding key information, outputs the corresponding training confidence, and uses the training confidence as a judgment label.

[0059] Through the above optional implementation, the initial confidence assessment model is trained by combining the decoded key information of the forward recognition model and the reverse recognition model, which improves the assessment accuracy of the model on the confidence of the recognition result, thereby enhancing the reliability and accuracy of the recognition.

[0060] S35, determining an original label according to the original recognition sequence and the forward training sequence.

[0061] In some embodiments, after the initial confidence estimation model outputs the judgment label, the electronic device can compare the candidate recognition result output by the forward recognition model with the original recognition sequence corresponding to the training data to determine whether the output of the forward recognition model is accurate and obtain an original label. If the candidate recognition result corresponds to the original recognition sequence, the original label is 1; if the candidate recognition result does not correspond to the original recognition sequence, the original label is 0.

[0062] S36, training the initial confidence estimation model according to the judgment label and the original label to obtain the target confidence estimation model.

[0063] The judgment label is the score (i.e., a score between 0 and 1) that the initial confidence estimation model uses to judge whether the recognition result of the forward recognition model is accurate based on the bidirectional sequence, and the original label is determined based on the comparison between the candidate recognition result output by the forward recognition model and the original recognition sequence. The electronic device can use the gap between the judgment label and the original label to train the initial confidence estimation model, that is, adjust the model parameters in the loss function of the initial confidence estimation model based on the gap between the judgment label and the original label, for example, using the Adam optimizer to adjust the loss function by gradient descent.

[0064] Whenever the loss function of the initial confidence estimation model is adjusted once, the electronic device will determine whether the current training meets the preset iteration condition. When it is determined that the current training meets the preset iteration condition, the loss function of the initial confidence estimation model is no longer adjusted based on the gap between the judgment label and the original label, and the initial confidence estimation model training is determined to be completed, and the trained initial confidence estimation model is called the target confidence estimation model. Otherwise, the loss function of the initial confidence estimation model continues to be adjusted based on the gap between the judgment label and the original label until the preset iteration condition is met. In some embodiments, the electronic device can pre-set an iteration number threshold (for example, 100 times). When the number of iterations meets the iteration number threshold, it can be determined that the current training meets the preset iteration condition. In other embodiments, the electronic device can determine whether the loss function of the initial confidence estimation model has converged, that is, when it is determined that the loss function of the initial confidence estimation model no longer changes significantly after multiple iterations, it can be determined that the loss function of the initial confidence estimation model has converged, and it is determined that the current training meets the preset iteration condition.

[0065] Through the above optional implementation, the initial confidence estimation model is trained by utilizing the gap between the judgment label and the original label until a preset iteration condition (such as an iteration number threshold or loss function convergence) is met, and the target model is obtained, thereby improving the accuracy and efficiency of the confidence assessment of the recognition result.

[0066] In an optional embodiment, the method further comprises: Obtaining a preset score threshold, and comparing the confidence value with the preset score threshold to obtain a confidence comparison result; Acquire task requirement information from the task instruction, and determine the task type corresponding to the task instruction based on the task requirement information; When the task type is an upstream task type, and the confidence comparison result indicates that the confidence value is less than the preset score threshold, marking the data to be identified as discarded data; When the task type is an upstream task type, and the confidence comparison result indicates that the confidence value is greater than or equal to the preset score threshold, marking the data to be identified as target data; When the task type is a downstream task type, and the confidence comparison result indicates that the confidence value is less than the preset score threshold, the model parameters of the forward recognition model are updated based on the confidence value.

[0067] In some embodiments, the electronic device may pre-set a score threshold for comparison with the confidence value output by the target confidence value estimation model to obtain a confidence comparison result. Next, the electronic device determines the task type corresponding to the task instruction based on the extracted task requirement information based on the task requirement information obtained in step S21, and the task type may include an upstream task type and a downstream task type. When the task type is an upstream task type, the confidence comparison result is determined to characterize whether the confidence value is greater than or equal to a preset score threshold or less than a preset score threshold. When it is determined that the confidence value is less than the preset score threshold, the data to be identified is marked as rejected data, that is, the data to be identified is considered to be inaccurate or unreliable due to insufficient confidence value, and is not suitable for subsequent analysis or processing. When it is determined that the confidence value is greater than or equal to the preset score threshold, the data to be identified is marked as target data, that is, the data to be identified is considered to be accurate and reliable, and can be used for subsequent analysis or processing.

[0068] When the task type is a downstream task type, it is also determined whether the confidence comparison result is greater than or equal to the preset score threshold or less than the preset score threshold. When it is determined that the confidence value is less than the preset score threshold, that is, the forward recognition model performs poorly in recognizing the data to be recognized, the model parameters of the forward recognition model are adjusted based on the confidence value. Specifically, for the recognition result whose confidence value is less than the preset score threshold, the electronic device can collect the corresponding data to be recognized and the recognition result as an error sample, and analyze the error sample to determine the cause of the recognition error, such as poor voice quality, noise interference, speaker feature mismatch, etc. Then, according to the analysis result of the error cause, the model parameters of the forward recognition model are adjusted, such as adjusting the model structure, increasing training data, etc. Finally, the electronic device can retrain the forward recognition model using the adjusted model parameters and training data. When it is determined that the confidence value is greater than or equal to the preset score threshold, the recognition result output by the forward recognition model is considered to be reliable, and there is no need to update the forward recognition model. If the confidence value is less than the preset score threshold, it is considered that there may be a problem with the recognition result output by the forward recognition model, and the forward recognition model needs to be updated.

[0069] Exemplarily, in upstream tasks, that is, when performing model training tasks related to speech, model training tasks often cover a large amount of data and may involve supervised or unsupervised learning methods. At this time, the confidence value estimation model provided by the present application can accurately score the confidence of a large amount of training data, so as to effectively screen out higher quality training data, thereby ensuring that the model training process can be based on a more reliable data set, thereby improving the efficiency and accuracy of model training. In addition, in downstream tasks, especially when dealing with speech recognition tasks with different pronunciation characteristics, the present application can accurately measure the accuracy of the speech recognition model when processing speech data with specific pronunciations by scoring the confidence of speech data with different pronunciations, which provides strong support for the performance evaluation and optimization of speech recognition systems in practical applications.

[0070] Through the above optional implementation, the forward recognition model is accurately updated by comparing the confidence value with the preset score threshold, which effectively improves the recognition accuracy of the forward recognition model under low confidence conditions and enhances its recognition stability and adaptability.

[0071] Compared with the prior art, the present application, based on the bidirectional sequence, constructs a more effective confidence value estimation model by comprehensively and efficiently utilizing the complete context information of the current identification in the recognition system. It can accurately evaluate the reliability of the recognition results output by the forward recognition model, thereby providing a more solid and reliable confidence value basis for recognition-related upstream and downstream tasks.

[0072] Reference Figure 5 , which is a functional module diagram of a confidence estimation device for a model recognition result shown in an embodiment of the present application.

[0073] In some embodiments, the confidence estimation device 50 for model recognition results may include multiple functional modules composed of computer program segments. The computer program of each program segment of the confidence estimation device 50 for model recognition results may be stored in a memory of an electronic device and executed by at least one processor to perform (see Figure 2 Description) Confidence value estimation function.

[0074] According to the functions performed, it can be divided into multiple functional modules. The functional modules may include: a receiving module 501, a first output module 502, a second output module 503, a third output module 504, a preprocessing module 505, a model training module 506 and a model updating module 507. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0075] The receiving module 501 is used to receive a task instruction containing data to be identified.

[0076] The first output module 502 is used to input the data to be identified into a preset forward recognition model to obtain a forward identification sequence output by the forward recognition model; the forward identification sequence includes forward decoding information.

[0077] The second output module 503 is used to reverse the direction of the forward identification sequence and input it into a preset reverse recognition model, so that the reverse recognition model outputs a reverse identification sequence corresponding to the forward identification sequence; the reverse identification sequence includes reverse decoding information.

[0078] The third output module 504 is used to input the forward decoding information and the reverse decoding information into a preset target confidence estimation model, so that the target confidence estimation model outputs a confidence value corresponding to the forward recognition model.

[0079] The preprocessing module 505 is used to: perform pre-emphasis processing on the data to be identified; segment the data to be identified after the pre-emphasis processing according to a preset time interval to obtain multiple data segments; perform windowing and Fourier transform on the multiple data segments to obtain the spectrogram features corresponding to each data segment in the multiple data segments; and input the spectrogram features into the forward recognition model.

[0080] The model training module 506 is used to: obtain a training data set; the training data set includes multiple training data, one training data corresponds to an original recognition sequence; obtain a forward training sequence after recognizing the training data through the forward recognition model, and output a reverse training sequence corresponding to the forward training sequence through the reverse recognition model; obtain a judgment label; the judgment label is output based on the forward training sequence and the reverse training sequence through a preset initial confidence estimation model; determine the original label according to the original recognition sequence and the forward training sequence; train the initial confidence estimation model according to the judgment label and the original label to obtain the target confidence estimation model.

[0081] The model training module 506 is also specifically used to: input the training data into the forward recognition model, so as to output multiple candidate recognition results after the training data is recognized by the forward recognition model; verify each of the candidate recognition results according to the original recognition sequence to determine the accuracy of each candidate recognition result; sort the multiple candidate recognition results from large to small based on the accuracy, and determine a preset number of the candidate recognition results in the front order as target recognition results; generate a forward training sequence based on one target recognition result.

[0082] The model training module 506 is further specifically used to: reverse the direction of the forward training sequence to obtain a reverse input sequence; input the reverse input sequence into the reverse recognition model to output the reverse training sequence based on the reverse input sequence through the reverse recognition model.

[0083] The model training module 506 is also specifically used to: obtain the first decoding key information of each forward training sequence and the second decoding key information of each reverse training sequence; associate and record a forward training sequence and its corresponding reverse training sequence as a training sequence group; splice the first decoding key information of the forward training sequence and the second decoding key information of the reverse training sequence in the same training sequence group to obtain spliced ​​decoding key information; input the spliced ​​decoding key information into the initial confidence estimation model to output a training confidence value based on the spliced ​​decoding key information through the initial confidence estimation model; and determine the judgment label based on the training confidence value.

[0084] The model updating module 507 is used to: obtain a preset score threshold, and compare the confidence value with the preset score threshold to obtain a confidence comparison result; obtain task requirement information from the task instruction, and determine the task type corresponding to the task instruction based on the task requirement information; when the task type is an upstream task type, and the confidence comparison result indicates that the confidence value is less than the preset score threshold, mark the data to be identified as rejected data; when the task type is an upstream task type, and the confidence comparison result indicates that the confidence value is greater than or equal to the preset score threshold, mark the data to be identified as target data; when the task type is a downstream task type, and the confidence comparison result indicates that the confidence value is less than the preset score threshold, update the model parameters of the forward recognition model based on the confidence value.

[0085] It should be understood that the various variations and specific embodiments of the confidence estimation method for model recognition results provided in the above-mentioned embodiments are also applicable to the confidence estimation device for model recognition results in the present embodiment. Through the above-mentioned detailed description of the confidence estimation method for model recognition results, those skilled in the art can clearly know the implementation method of the confidence estimation device for model recognition results in the present embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0086] See also Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. In a preferred embodiment of the present application, the electronic device 6 includes a memory 61 , at least one processor 62 and at least one communication bus 63 .

[0087] Those skilled in the art should understand that Figure 6 The structure of the electronic device 6 shown does not constitute a limitation of the embodiments of the present application, and can be either a bus structure or a star structure. The electronic device 6 can also include more or less other hardware or software than shown in the figure, or a different component arrangement.

[0088] In some embodiments, the electronic device 6 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The electronic device 3 may also include user equipment, which includes but is not limited to any electronic product that can interact with a user through a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smart phone, digital camera, etc.

[0089] In the above embodiments provided in the present application, it should be understood that the disclosed methods, devices, computer-readable storage media, and electronic devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple components or modules can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or components or modules, which can be electrical, mechanical or other forms.

[0090] The components described as separate components may or may not be physically separated, and the components shown as components may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the components may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0091] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each component may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0092] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0093] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0094] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0095] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A confidence estimation method for model recognition results, characterized in that: The method comprises: receiving a task instruction including data to be identified; Inputting the data to be identified into a preset forward recognition model, and obtaining a forward identification sequence output by the forward recognition model; the forward identification sequence includes forward decoding information; Reversing the direction of the forward identification sequence and inputting it into a preset reverse recognition model, so that the reverse recognition model outputs a reverse identification sequence corresponding to the forward identification sequence; the reverse identification sequence includes reverse decoding information; The forward decoding information and the reverse decoding information are input into a preset target confidence estimation model so that the target confidence estimation model outputs a confidence value corresponding to the forward recognition model.

2. The confidence estimation method for model recognition results according to claim 1, characterized in that: Before inputting the data to be identified into a preset forward recognition model, the method further includes: Performing pre-emphasis processing on the data to be identified; Segmenting the data to be identified after the pre-emphasis processing according to a preset time interval to obtain a plurality of data segments; Performing a windowing operation and a Fourier transform on the multiple data segments to obtain a spectrogram feature corresponding to each data segment in the multiple data segments; The spectrogram features are input into the forward recognition model.

3. The confidence estimation method of the model recognition result according to claim 1, characterized in that: The method further comprises: Acquire a training data set; the training data set includes a plurality of training data, and one training data corresponds to an original recognition sequence; After identifying the training data through the forward recognition model, a forward training sequence is obtained, and a reverse training sequence corresponding to the forward training sequence is output through the reverse recognition model; Acquire a judgment label; the judgment label is output based on the forward training sequence and the reverse training sequence through a preset initial confidence estimation model; Determine an original label according to the original recognition sequence and the forward training sequence; The initial confidence estimation model is trained according to the judgment label and the original label to obtain the target confidence estimation model.

4. The confidence estimation method for model recognition results according to claim 3, characterized in that: The step of obtaining a positive training sequence after identifying the training data by using the positive recognition model comprises: Inputting the training data into the forward recognition model, so as to output a plurality of candidate recognition results after recognizing the training data through the forward recognition model; Verifying each of the candidate recognition results according to the original recognition sequence to determine the accuracy of each of the candidate recognition results; Sort the multiple candidate recognition results from largest to smallest based on the accuracy, and determine a preset number of the candidate recognition results ranked first as target recognition results; A forward training sequence is generated based on a target recognition result.

5. The confidence estimation method of the model recognition result according to claim 3, characterized in that: The step of outputting the reverse training sequence corresponding to the forward training sequence through the reverse recognition model includes: Reversing the direction of the forward training sequence to obtain a reverse input sequence; The reverse input sequence is input into the reverse recognition model, so as to output the reverse training sequence based on the reverse input sequence through the reverse recognition model.

6. The confidence estimation method for model recognition results according to claim 3, characterized in that: The obtaining of the judgment tag comprises: Acquire first decoding key information of each of the forward training sequences and second decoding key information of each of the reverse training sequences; Associating and recording one of the forward training sequences and its corresponding reverse training sequence as a training sequence group; splicing the first decoding key information of the forward training sequence and the second decoding key information of the reverse training sequence in the same training sequence group to obtain spliced ​​decoding key information; Inputting the splicing decoding key information into the initial confidence estimation model, so as to output a training confidence value based on the splicing decoding key information through the initial confidence estimation model; The judgment label is determined based on the training confidence value.

7. The confidence estimation method for model recognition results according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtaining a preset score threshold, and comparing the confidence value with the preset score threshold to obtain a confidence comparison result; Acquire task requirement information from the task instruction, and determine the task type corresponding to the task instruction based on the task requirement information; When the task type is an upstream task type, and the confidence comparison result indicates that the confidence value is less than the preset score threshold, marking the data to be identified as discarded data; When the task type is an upstream task type, and the confidence comparison result indicates that the confidence value is greater than or equal to the preset score threshold, marking the data to be identified as target data; When the task type is a downstream task type, and the confidence comparison result indicates that the confidence value is less than the preset score threshold, the model parameters of the forward recognition model are updated based on the confidence value.

8. A confidence estimation device for a model recognition result, characterized in that: The device comprises: A receiving module, used for receiving a task instruction containing data to be identified; A first output module, used to input the data to be identified into a preset forward recognition model, and obtain a forward identification sequence output by the forward recognition model; the forward identification sequence includes forward decoding information; A second output module is used to reverse the direction of the forward identification sequence and input it into a preset reverse recognition model, so that the reverse recognition model outputs a reverse identification sequence corresponding to the forward identification sequence; the reverse identification sequence includes reverse decoding information; The third output module is used to input the forward decoding information and the reverse decoding information into a preset target confidence estimation model, so that the target confidence estimation model outputs a confidence value corresponding to the forward recognition model.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the confidence estimation method for the model recognition result described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for estimating the confidence of the model recognition result according to any one of claims 1 to 7 are implemented.

Images