Speech recognition confidence estimation method, storage medium, and electronic device

Abstract

The application discloses a confidence estimation method for speech recognition, a storage medium and an electronic device. The method comprises the following steps: obtaining speech data and text data; performing feature extraction on the speech data to obtain acoustic features of the speech data; performing length prediction and feature extraction on the acoustic features to obtain text length and speech features, wherein the text length is used to represent the number of characters contained in the text corresponding to the speech data, and the length of the speech features is the same as the text length; matching the speech data and the text data based on the speech features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data and the text data. The application solves the technical problem that the accuracy of the confidence estimation based on speech recognition in the related art is not high, so that the downstream task can better perform targeted operation based on the confidence estimation result, and help the downstream task to make correct judgment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a confidence estimation method for speech recognition, a storage medium and an electronic device. BACKGROUND

[0002] In a complete human-computer interaction link, intelligent dialogue, machine translation and other speech recognition downstream tasks not only need a single recognition result, but also need to quantify the reliability of the recognition result. However, in the current mainstream AED (attention based encoder-decoder) end-to-end speech recognition model, there is a problem of excessive confidence and label synchronization, which seriously affects the quality of the likelihood probability of the output layer as the confidence.

[0003] At present, there is no effective solution to the above problems. SUMMARY

[0004] Embodiments of the present application provide a confidence estimation method for speech recognition, a storage medium and an electronic device to at least solve the technical problem of low accuracy of confidence estimation based on speech recognition in related technologies.

[0005] According to an aspect of an embodiment of the present application, a confidence estimation method for speech recognition is provided, comprising: obtaining speech data and text data; performing feature extraction on the speech data to obtain acoustic features of the speech data; performing length prediction and feature extraction on the acoustic features to predict text length and speech features, wherein the text length is used to represent the number of characters contained in the text corresponding to the speech data, and the length of the speech features is the same as the text length; matching the speech data and the speech text based on the speech features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data and the text data.

[0006] According to another aspect of the embodiments of the present application, a confidence estimation method for speech recognition is also provided, comprising: in response to an input instruction acting on an operation interface, driving a speech collection device to collect speech data emitted by a biological object and obtaining text data; in response to a confidence estimation instruction acting on the operation interface, displaying a confidence estimation result on the operation interface, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data with the text data, the confidence estimation result is obtained by matching the speech data and the text data based on speech features of the speech data, the length of the speech features is the same as the length of the text, the length of the text is used to represent the number of characters contained in the text corresponding to the speech data, the length of the speech features and the length of the text are obtained by length prediction and feature extraction prediction on acoustic features of the speech data, and the acoustic features are obtained by feature extraction on the speech data.

[0007] According to another aspect of the embodiments of the present application, a confidence estimation method for speech recognition is also provided, comprising: driving a virtual reality (VR) device or an augmented reality (AR) device to collect speech data emitted by a biological object and obtaining text data; performing feature extraction on the speech data to obtain acoustic features of the speech data; performing length prediction and feature extraction on the acoustic features to predict a length of the text and speech features, wherein the length of the text is used to represent the number of characters contained in the text corresponding to the speech data, and the length of the speech features is the same as the length of the text; matching the speech data and the text data based on the speech features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data with the text data; and driving the VR device or the AR device to render and display the confidence estimation result on a presentation screen.

[0008] According to another aspect of the embodiments of the present application, a confidence estimation method for speech recognition is also provided, comprising: obtaining speech data and text data by calling a first interface, wherein the first interface comprises a first parameter, and the parameter value of the first parameter is the speech data and the text data; performing feature extraction on the speech data to obtain acoustic features of the speech data; performing length prediction and feature extraction on the acoustic features to predict a length of the text and speech features, wherein the length of the text is used to represent the number of characters contained in the text corresponding to the speech data, and the length of the speech features is the same as the length of the text; matching the speech data and the text data based on the speech features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data with the text data; and outputting the confidence estimation result by calling a second interface, wherein the second interface comprises a second parameter, and the parameter value of the second parameter is the confidence estimation result.

[0009] According to a further aspect of the embodiments of the present application, a computer readable storage medium is also provided, which includes a stored program, wherein the program, when executed, controls the device where the computer readable storage medium is located to perform the method of any of the embodiments of the present application.

[0010] According to a further aspect of the embodiments of the present application, an electronic device is also provided, which includes a processor, and a memory connected to the processor, for providing the processor with instructions of the method of any of the embodiments of the present application.

[0011] In the embodiments of the present application, after the voice data and the text data are acquired, the voice data is first subjected to feature extraction to obtain acoustic features of the voice data, then the acoustic features are subjected to length prediction and feature extraction to predict the text length and the voice features of the same length as the text length, and finally the voice data and the text data are matched based on the voice features to obtain the matching probability of each character contained in the text corresponding to the voice data with the text data, that is, a character-level confidence estimation result of the same length as the text length is obtained, so as to achieve the purpose of confidence estimation. It is easy to note that, since the voice features and the confidence estimation result are both of the same number of characters contained in the text corresponding to the voice data, the problem of low reliability of the confidence estimation result in the case that the voice data and the text data are not of the same length is avoided, so as to achieve the technical effect of improving the accuracy of the confidence estimation result, and further solve the technical problem of low accuracy of the confidence estimation based on speech recognition in the related art, and make the downstream task better perform targeted operation based on the confidence estimation result, help the downstream task make correct judgment, and lay a foundation for stable human-computer interaction link. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and together with the description given below, serve to explain the present application. In the drawings:

[0013] Figure 1 FIG. 1 is a schematic diagram of a hardware environment of a virtual reality device according to the speech recognition confidence estimation method of the embodiments of the present application;

[0014] Figure 2 FIG. 2 is a structural block diagram of a computing environment according to the speech recognition confidence estimation method of the embodiments of the present application;

[0015] Figure 3 FIG. 3 is a flowchart of the speech recognition confidence estimation method according to Embodiment 1 of the embodiments of the present application;

[0016] Figure 4ais a schematic diagram of a model structure of a Paraformer model according to an embodiment of the present application;

[0017] Figure 4b is a schematic diagram of a model structure of a confidence estimation model according to an embodiment of the present application;

[0018] Figure 5 is a flowchart of a confidence estimation method for speech recognition according to Embodiment 2 of the present application;

[0019] Figure 6 is a schematic diagram of an operation interface according to an embodiment of the present application;

[0020] Figure 7 is a flowchart of a confidence estimation method for speech recognition according to Embodiment 3 of the present application;

[0021] Figure 8 is a flowchart of a confidence estimation method according to Embodiment 4 of the present application;

[0022] Figure 9 is a schematic diagram of a confidence estimation device according to Embodiment 5 of the present application;

[0023] Figure 10 is a schematic diagram of a confidence estimation device according to Embodiment 6 of the present application;

[0024] Figure 11 is a schematic diagram of a confidence estimation device according to Embodiment 7 of the present application;

[0025] Figure 12 is a schematic diagram of a confidence estimation device according to Embodiment 8 of the present application;

[0026] Figure 13 is a structural block diagram of an AR / VR device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and above-mentioned accompanying drawings are intended to distinguish similar objects and not necessarily describe a particular chronological or sequential order. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus comprising a list of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products, or apparatuses.

[0029] First, some of the nouns or terms appearing in the description of the embodiments of the present application are applicable to the following explanations:

[0030] CEM: Confidence Estimation Module, a model capable of predicting the degree of matching of a certain speech-text in a speech recognition task.

[0031] ASR: Automatic Speech Recognition.

[0032] CIF: Continuous Intergrate-and-Fire, a continuous integrate-and-fire algorithm.

[0033] NAR ASR: None-Autoregressive Automatic Speech Recognition, a type of speech recognition model that does not rely on historical results for prediction.

[0034] Paraformer: Parallel Transformer, a NAR ASR model that utilizes CIF.

[0035] ATTN: attention, attention mechanism, the core idea is to find the relevance between data based on the original data.

[0036] FFN: Feed-Forward Networks, a feed-forward neural network, i.e., a fully connected neural network, in which the neurons are arranged in layers (each layer contains a number of neurons), each neuron is connected only to the neurons of the previous layer, receives the output of the previous layer, and outputs to the next layer, and there is no feedback between layers.

[0037] In the human-computer interaction link (such as a digital human product) cascaded by downstream tasks such as machine translation and intelligent dialogue, simultaneously providing the confidence value corresponding to the recognition result can help the downstream tasks make correct judgments, for example, the machine translation system can avoid invalid calculation by rejecting the low-confidence recognition result, and the digital human can quantify whether a speech is "clearly heard" through the confidence value in the process of voice interaction.

[0038] The traditional speech recognition system can predict reliable confidence based on the lattice of frame-level acoustic likelihood, but in the current mainstream AED end-to-end speech recognition model, the overconfidence and label synchronization problems seriously affect the quality of the likelihood probability of the output layer as the confidence. Among them, the overconfidence problem refers to that the decoder of the end-to-end speech recognition directly models to token (or Chinese character), and the likelihood probability distribution of the output layer is seriously unbalanced, and the maximum value of the likelihood probability is often greater than 0.9, and the posterior probability synchronized with the token also cannot reflect the deletion error and insertion error in prediction. The label synchronization problem refers to that the AED model lacks explicit acoustic model modeling and can only obtain a likelihood sequence equal in length to the output sequence, and cannot display the deletion error and insertion error through the likelihood probability, for example, for a speech of "today the weather is very good", only inputting "today" will also get a high confidence, which is obviously unreasonable, and the existing evaluation indicators cannot describe this defect.

[0039] Under the above algorithm background and application background, how to realize a reliable and efficient confidence prediction module in the end-to-end speech recognition model to perfect the function of the human-computer interaction link has become a key algorithm problem.

[0040] The present application proposes a confidence estimation algorithm based on a non-autoregressive end-to-end speech recognition model, which realizes high-quality recognition result confidence estimation by introducing an additional confidence estimation module based on the original speech recognition model, and solves the prediction failure problem of the existing confidence estimation algorithm in the sequence non-equal length mode. In actual application scenarios, the confidence estimation model can be used as a solution to the spoken language evaluation task, and in the human-computer interaction link cascaded by downstream tasks such as machine translation and intelligent dialogue, simultaneously providing the confidence value corresponding to the recognition result can help the downstream tasks make correct judgments.

[0041] Embodiment 1

[0042] According to the embodiments of the present application, a confidence estimation method for speech recognition is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Figure 1 Fig. 1 is a schematic diagram of a hardware environment of a virtual reality device according to an embodiment of a confidence estimation method of speech recognition. As shown in Fig. 1, a virtual reality device 104 is connected with a terminal 106, and the terminal 106 is connected with a server 102 through a network. The virtual reality device 104 is not limited to a virtual reality helmet, a virtual reality glasses, a virtual reality all-in-one machine, etc., the terminal 106 is not limited to a PC, a mobile phone, a tablet computer, etc., and the server 102 can be a server corresponding to a media file operator. The network includes but is not limited to a wide area network, a metropolitan area network or a local area network. Figure 1

[0044] Optionally, the virtual reality device 104 according to the embodiment includes a memory, a processor and a transmission device. The memory is configured to store an application program, and the application program can be used to execute the following steps: obtaining speech data and text data; performing feature extraction on the speech data to obtain acoustic features of the speech data; performing length prediction and feature extraction on the acoustic features to predict text length and speech features, wherein the text length is used to represent the number of characters contained in the text corresponding to the speech data, and the length of the speech features is the same as the text length; and matching the speech data and the text data based on the speech features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data and the text data, thereby solving the technical problem that the accuracy of confidence estimation based on speech recognition is not high in the related art, and achieving the purpose of improving the accuracy of confidence estimation.

[0045] The terminal according to the embodiment can be used to drive a virtual reality (VR) device or an augmented reality (AR) device to collect speech data emitted by a biological object, and drive the VR device or the AR device to display text data on a presentation screen; perform feature extraction on the speech data to obtain acoustic features of the speech data; perform length prediction and feature extraction on the acoustic features to predict text length and speech features, wherein the text length is used to represent the number of characters contained in the text corresponding to the speech data, and the length of the speech features is the same as the text length; match the speech data and the text data based on the speech features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data and the text data; and drive the VR device or the AR device to render and display the confidence estimation result on the presentation screen.

[0046] ​Optionally, the virtual reality device 104 of this embodiment is an HMD (Head Mount Display) head set with eye tracking, and the eye tracking module has the same function as the eye tracking module in the above embodiment, that is, the screen in the HMD head set is used to display real-time images, and the eye tracking module in the HMD is used to obtain the real-time movement trajectory of the user's eyeballs. The terminal of this embodiment obtains the position information and movement information of the user in the real three-dimensional space through the tracking system, and calculates the three-dimensional coordinates of the user's head in the virtual three-dimensional space and the direction of the user's field of view in the virtual three-dimensional space.

[0047] Figure 1 The hardware structure diagram shown not only can be used as an exemplary block diagram of the AR / VR device (or mobile device) described above, but also can be used as an exemplary block diagram of the server described above, and in an optional embodiment, Figure 2 The block diagram shown above illustrates the use of the above-mentioned Figure 1 The AR / VR device (or mobile device) shown is an embodiment of a computing node in a computing environment 201. Figure 2 is a structural block diagram of a computing environment for a confidence estimation method of speech recognition according to an embodiment of the present application, as shown in Figure 2 As shown, the computing environment 201 includes a plurality of (shown in the figure as 210-1, 210-2, …) computing nodes (such as servers) running on a distributed network. Each computing node contains local processing and memory resources, and end users 202 can remotely run applications or store data in the computing environment 201. Applications can be provided as a plurality of services 220-1, 220-2, 220-3 and 220-4 in the computing environment 301, representing services “A”, “D”, “E” and “H” respectively.

[0048] End users 202 can provide and access services through web browsers or other software applications on the client, and in some embodiments, the provision and / or requests of end users 202 can be provided to an entry gateway 230. The entry gateway 230 can include a corresponding agent to process the provision and / or requests for services (one or more services provided in the computing environment 201).

[0049] Services are provided or deployed in accordance with various virtualization technologies supported by the computing environment 201. In some embodiments, services can be provided in accordance with virtual machine (VM)-based virtualization, container-based virtualization, and / or the like. VM-based virtualization can be emulating a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the virtual machine is virtualized, in accordance with container-based virtualization, a container can be enabled to virtualize an entire operating system (OS) so that multiple workloads can run on a single OS instance.

[0050] In one embodiment of container-based virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in Figure 2 Service 220-2 can be equipped with one or more Pods 240-1, 240-2, …, 240-N (collectively, Pods). Each Pod can include a proxy 245 and one or more containers 242-1, 242-2, …, 242-M (collectively, containers). The one or more containers in a Pod handle requests related to one or more respective functions of the service, and the proxy 245 generally controls network functions related to the service, such as routing, load balancing, and the like. Other services can also be Pods similar to the Pods.

[0051] In operation, executing a user request from the end user 202 can require invoking one or more services in the computing environment 201, and executing one or more functions of a service can require invoking one or more functions of another service. As shown in Figure 2 Service “A” 220-1 receives a user request from the end user 202 from the ingress gateway 230, service “A” 220-1 can invoke service “D” 220-2, and service “D” 220-2 can request service “E” 220-3 to execute one or more functions.

[0052] The computing environment described above can be a cloud computing environment, and the allocation of resources is managed by a cloud service provider, allowing the development of functions without considering the implementation, adjustment, or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be split into a set of functions that can automatically scale independently, rather than expanding a single hardware device to handle potential loads.

[0053] In the above-described operating environment, the present disclosure provides a confidence estimation method for speech recognition as shown in Figure 3 It should be noted that the confidence estimation method for speech recognition of the embodiment can be implemented by a speech recognition system as shown in Figure 1The AR / VR device of the illustrated embodiment performs. Figure 3 is a flowchart of a confidence estimation method for speech recognition according to Embodiment 1 of the present application. As shown in Figure 3 The method can include the following steps:

[0054] In step S302, speech data and text data are obtained.

[0055] The speech data in the above step can be speech data obtained by collecting the sound uttered by the user in the human-computer interaction link. By recognizing the speech data, downstream tasks of speech recognition such as intelligent dialogue and machine translation can be continued. For example, in the application scenario of an intelligent assistant, the user can have a dialogue with the client by uttering the voice "help me query how to take a transportation tool from place A to place B", so that the intelligent assistant can quickly feed back the information needed by the user.

[0056] Since the core purpose of speech recognition technology is to convert language data into corresponding text, the text data in the above step can be the to-be-recognized text that needs to be matched with the speech data. The text data can be set according to the actual application scenario, for example, it can be a complete sentence set according to the actual speech recognition needs, but is not limited thereto.

[0057] In an optional embodiment, the entire speech recognition process can be performed by a server, thereby reducing the computing amount of the mobile terminal. In order to achieve the purpose of speech recognition, the user can directly use the mobile terminal to collect the sound uttered by himself to obtain the speech data, and upload the speech data to the server, and the server obtains the text data from the database based on the recognition demand corresponding to the client.

[0058] In another optional embodiment, in order to avoid the influence of network state on the use of speech recognition function by the user, the user can also directly use the mobile terminal to collect the sound uttered by himself to obtain the speech data, and obtain the text data pre-cached from the server from the local storage device.

[0059] In step S304, the speech data is feature-extracted to obtain the acoustic feature of the speech data.

[0060] The acoustic feature in the above step can be a multi-dimensional feature vector obtained by converting the speech model in the speech data into a parameter form, so that the multi-dimensional feature vector can be processed by a computer, a server, etc.

[0061] In an optional embodiment, an existing speech feature extraction algorithm can be used to feature-extract the speech data to obtain the acoustic feature, for example, the Mel frequency cepstral coefficient algorithm is used to feature-extract the speech data, but is not limited thereto.

[0062] In another optional embodiment, an existing speech feature extraction model can be used to extract features from the speech data to obtain acoustic features.

[0063] In step S306, length prediction and feature extraction are performed on the acoustic features to obtain text length and speech features, wherein the text length is used to represent the number of characters contained in the text corresponding to the speech data, and the length of the speech features is the same as the text length.

[0064] It should be noted that the number of characters contained in the text corresponding to the speech data can be the same as or different from the number of characters contained in the text data. For the two different numbers, the mainstream AED end-to-end speech recognition model has the disadvantage of non-robust confidence estimation. For example, for the speech data representing "today the weather is good", if the text data is "today", the confidence estimation result output by the AED end-to-end speech recognition model is similar to [0.98 0.96], and the sentence-level confidence is 0.97. This confidence estimation result obviously does not conform to the true situation.

[0065] In an optional embodiment, while predicting the number of words of the speech data, i.e., predicting the number of characters contained in the text corresponding to the speech data to obtain the text length, the acoustic features can be processed through feature extraction to obtain a high-dimensional acoustic feature with the same length as the text length, i.e., the above-mentioned speech features.

[0066] In step S308, the speech data and the text data are matched based on the speech features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data and the text data.

[0067] The confidence estimation result in the above steps is a character-level confidence, and therefore, the length of the confidence estimation result is the same as the number of characters contained in the text corresponding to the speech data, i.e., the above-mentioned text length.

[0068] In an optional embodiment, the speech data and the text data can be matched according to characters through the speech features to obtain a confidence estimation result with the same length as the above-mentioned text length, which is different from the traditional AED end-to-end speech recognition model to obtain a confidence estimation result with the same length as the text data. Each element in the confidence estimation result represents the matching probability of each character in the text corresponding to the speech data and the text data. For example, still taking the speech data representing "today the weather is good" as an example, if the text data is "today", the confidence estimation result obtained by using the above-mentioned scheme of the present application is [0.98 0.96 0.10 0.11 0.15 0.07], and the sentence-level confidence is 0.395. This confidence estimation result conforms to the true situation.

[0069] It should be noted that the above two steps can be performed by a confidence estimation model obtained by machine learning, which is not an existing AED end-to-end speech recognition model, but a model obtained by improving the existing model, which can realize feature extraction of acoustic features to obtain a speech feature corresponding to the number of characters contained in the text corresponding to the speech data, and can obtain a confidence estimation result corresponding to the number of characters contained in the text corresponding to the speech data by matching the speech data and the text data. Optionally, in the present application, the confidence estimation model can be implemented based on the Paraformer model.

[0070] The present application will be described in detail below taking a machine translation system as an example. The digital human can collect speech data to be translated in the process of voice interaction, and perform speech recognition on the speech data based on the text data in the database. In the entire recognition process, first, the speech data can be feature extracted by using a speech extraction model to obtain acoustic features, then a text length equal length speech feature can be provided while predicting the number of words of the speech data, and finally the speech data and the text data are matched based on the speech feature to obtain the final character-level confidence estimation result, which is also equal in length to the text. Since the accuracy of the confidence estimation result is high, the digital human can quantify whether the speech data is "clear" based on the confidence estimation result. If the confidence estimation result is low, the downstream machine translation system can refuse to translate the speech data to avoid invalid calculation, and require the user to issue a clearer speech data.

[0071] Based on the scheme provided by the above embodiments of the present application, after obtaining the speech data and the text data, first, the speech data can be feature extracted to obtain the acoustic features of the speech data, then the acoustic features are length predicted and feature extracted to predict the text length and the text length equal length speech feature, and finally the speech data and the text data are matched based on the speech feature to obtain the matching probability of each character contained in the text corresponding to the speech data and the text data, that is, a character-level confidence estimation result equal in length to the text is obtained, thereby achieving the purpose of confidence estimation. It is easy to note that since the speech feature and the confidence estimation result are the same as the number of characters contained in the text corresponding to the speech data, the problem of low reliability of the confidence estimation result under the condition that the speech data and the text data are not equal in length is avoided, thereby achieving the technical effect of improving the accuracy of the confidence estimation result, thereby solving the technical problem of low accuracy of confidence estimation based on speech recognition in the related art, and enabling the downstream task to better perform targeted operation based on the confidence estimation result, helping the downstream task to make correct judgment, and laying a foundation for stable human-computer interaction link.

[0072] In the above embodiments of the present application, the acoustic features are subjected to length prediction and feature extraction, and the text length and the speech features are predicted, including: using a confidence estimation model to perform length prediction and feature extraction on the acoustic features to obtain the text length and the speech features.

[0073] The confidence estimation model in the above steps can be a model for estimating the confidence of the true length based on a Paraformer model, wherein the Paraformer model is a basic speech recognition model of the confidence estimation model, and the specific model structure is as shown in Figure 4a The specific model structure of the confidence estimation model is as shown in Figure 4b After the training of the Paraformer model is completed, the ALIGNER module and the CIF-ALIGN ESTIMATOR module can be added, and the confidence estimation model is obtained by continuing training.

[0074] In an optional embodiment, the ENCODER module and the PREDICTOR module in the confidence estimation model can be used to perform word number prediction and feature extraction on the speech data through CIF, to provide a high-dimensional feature equal in length to the text length while predicting the text length, that is, to obtain the speech features.

[0075] In the above embodiments of the present application, the confidence estimation model includes an encoder module and a prediction module, wherein the acoustic features are subjected to length prediction and feature extraction by using the confidence estimation model to obtain the text length and the speech features, including: using the encoder module to perform feature encoding on the acoustic features to obtain encoded features; and using the prediction module to perform length prediction and feature extraction on the encoded features to obtain the text length and the speech features.

[0076] In an optional embodiment, as shown in Figure 4b The confidence estimation model at least includes an encoder module ENCODER and a prediction module PREDICTOR, and the specific network structures of the two modules can adopt the network structure of the existing Paraformer model, which is not limited in the present application. The encoder module can perform feature encoding on the input acoustic features features to obtain encoded features, wherein the acoustic features are represented by x 1：τ , and the encoded features are represented by e 1:τ ; the prediction module can perform length prediction and feature extraction on the encoded features to obtain the text length and the speech features equal in length to the text length, wherein the text length is represented by L', and the speech features are represented by E 1:L′ .

[0077] In the above embodiments of the present application, the speech data and the text data are matched based on the speech features to obtain the confidence estimation result, including: matching the speech data and the text data based on the speech features by using the confidence estimation model to obtain the confidence estimation result.

[0078] In an optional embodiment, the EMBED module, the PARALLELDECODER module, the ALIGNER module and the CIF-ALIGNEDESTIMATOR module in the confidence estimation model can be used to perform feature alignment by introducing an additional cross-attention, thereby generating a hidden state with the same length as the text length (instead of the same length as the text data) and containing both the text information of the text data and the acoustic information of the speech data, and further obtaining a likelihood probability of whether the output token is correct, that is, obtaining a confidence estimation result with the same length as the text length.

[0079] In the above embodiments of the present application, the confidence estimation model further includes: a word embedding module, a decoder module, a cross-attention module and a confidence estimation module, wherein the speech data and the text data are matched based on the speech features by using the confidence estimation model to obtain the confidence estimation result, including: the text data is feature extracted by using the word embedding module to obtain text features of the text data; the speech features and the text features are feature decoded by using the decoder module to obtain first decoding features corresponding to the speech features and second decoding features corresponding to the text data, wherein the length of the first decoding features is the text length, and the length of the second decoding features is the length of the text data; the first decoding features and the second decoding features are feature aligned by using the cross-attention module to obtain first attention features corresponding to the first decoding features and second attention features corresponding to the second decoding features, wherein the length of the first attention features and the second attention features is the text length; the first attention features and the second attention features are matched by using the confidence estimation module to obtain the confidence estimation result.

[0080] In an optional embodiment, as Figure 4bAs shown, the confidence estimation model further includes a word embedding module EMBED, a decoder module PARALLELDECODER, a cross-attention module ALIGNER, and a confidence estimation module CIF-ALIGNEDESTIMATOR. The specific network structure of the two modules EMBED and PARALLELDECODER can adopt the network structure of the existing Paraformer model, which is not limited in the present application. The ALIGNER can include a CROSSATTN layer and a SELFATTN layer. The CIF-ALIGNEDESTIMATOR can be a binary classifier including an Attention Decoder and an FFN. The word embedding module can perform feature extraction on the input text data hypothesis to obtain text features, where the text data is represented by y 1：L , the encoding features are represented by c 1：L , L represents the length of the text data. The decoder module can perform feature decoding on the speech features and the text features to obtain two high-dimensional features, namely a first decoding feature acousticembedding and a second decoding feature char embedding, where the first decoding feature is represented by a , and the second decoding feature is represented by d . The cross-attention module can introduce an additional cross-attention to align the acousticembedding and the decoder embedding to obtain hidden states required for final classification, namely a first attention feature and a second attention feature, where the first attention feature is represented by a 1:L′ , and the second attention feature is represented by d 1：L . The confidence estimation module can match the first attention feature and the second attention feature to obtain the likelihood probability of whether the output token is correct, that is, to obtain the confidence estimation result, where the confidence estimation result is represented by p 1：L′ .

[0081] In the above embodiments of the present application, the cross-attention module includes a self-attention layer and a cross-attention layer. The first decoding feature and the second decoding feature are aligned by the cross-attention module to obtain a first attention feature corresponding to the first decoding feature and a second attention feature corresponding to the second decoding feature, including: performing attention processing on the first decoding feature by the self-attention layer to obtain the first attention feature; and performing feature alignment on the first attention feature and the second decoding feature by the cross-attention layer to obtain the second attention feature.

[0082] In an optional embodiment, as shown in FIG. 4, the confidence estimation model further includes a cross-attention module ALIGNER, and a confidence estimation module CIF-ALIGNEDESTIMATOR. The ALIGNER can include a CROSSATTN layer and a SELFATTN layer. The CIF-ALIGNEDESTIMATOR can be a binary classifier including an Attention Decoder and an FFN. Figure 4bAs shown, the cross-attention module includes a self-attention layer SELFATTN and a cross-attention layer CROSSATTN, where CROSSATTN and SELFATTN can both adopt an existing dot product self-attention mechanism. The self-attention layer can perform attention processing on the first decoding feature based on the attention mechanism to obtain a self-attention representation of the first decoding feature, thereby obtaining a first attention feature. Then, the cross-attention layer can perform feature alignment on the first attention feature and the second decoding feature to obtain an aligned feature of the same length as the first decoding feature, that is, a second attention feature.

[0083] In the above embodiments of the present application, during training of the confidence estimation model, the model parameters of the decoder module remain unchanged.

[0084] In an alternative embodiment, a trained Paraformer model can be selected as a base model to initialize the ALIGNER and CIF-ALIGN ESTIMATOR modules, and the gradient of the PARALLELDECODER part is canceled, that is, during training of the confidence estimation model, the parameters of the PARALLELDECODER are fixed and will not be trained.

[0085] It should be noted that for the above-mentioned method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0086] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platform, and of course, it can also be realized by hardware. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method described in each embodiment of the present application.

[0087] Embodiment 2

[0088] According to the embodiments of the present application, a confidence estimation method for speech recognition is also provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0089] Figure 5 is a flowchart of a confidence estimation method for speech recognition according to Embodiment 2 of the present application. As shown in Figure 5 , the method comprises the following steps:

[0090] In step S502, in response to an input instruction acting on the operation interface, a speech collection device is driven to collect speech data emitted by a biological object, and text data is obtained.

[0091] The operation interface in the above step can be an operation interface provided by a speech recognition system to a user, as shown in Figure 6 , the interface can be displayed on a mobile terminal used by the user, and the user can achieve the purpose of human-computer interaction with the operation interface by operating on the operation interface.

[0092] The input instruction in the above step can be an instruction generated by the user operating on the operation interface, for example, for the operation interface as shown in Figure 6 , the user can click the "voice recording" button on the operation interface to generate an input instruction, so as to drive the speech collection device to collect speech data emitted by the biological object.

[0093] The speech collection device in the above step can be a device capable of collecting speech data on the mobile terminal, for example, a microphone provided by the mobile terminal, but is not limited thereto. The biological object can be a living being capable of emitting sound, for example, a human being, other animals capable of emitting sound.

[0094] In an optional embodiment, when the user needs to perform speech recognition, the user can operate on the operation interface to generate an input instruction, so that the speech recognition system can drive the speech collection device to collect the sound emitted by the user to obtain speech data, and at the same time, the speech recognition system can obtain text data used for speech recognition from the database.

[0095] In response to the confidence estimation instruction acting on the operation interface, the confidence estimation result is displayed on the operation interface, where the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data and the text data, and the confidence estimation result is obtained by matching the speech data and the text data based on the speech features of the speech data, the length of the speech features is the same as the length of the text, the length of the text is used to represent the number of characters contained in the text corresponding to the speech data, and the length of the text and the speech features are obtained by length prediction and feature extraction prediction on the acoustic features of the speech data, and the acoustic features are obtained by feature extraction on the speech data.

[0096] The confidence estimation instruction in the above steps can be an instruction generated by the user operating on the operation interface. For example, for the operation interface as shown in Figure 6 , the user can click the "confidence estimation" button on the operation interface to generate the confidence estimation instruction.

[0097] In an optional embodiment, when the user wants to perform speech recognition on the speech data, the user can click the "confidence estimation" button on the operation interface to generate the confidence estimation instruction. After the server performs speech recognition on the speech data and obtains the confidence estimation result, the confidence estimation result can be displayed in the confidence estimation result area of the operation interface as shown in Figure 6 .

[0098] In the above embodiments of the present application, the length of the text and the speech features are obtained by length prediction and feature extraction on the acoustic characteristics by using the confidence estimation model.

[0099] In the above embodiments of the present application, the confidence estimation model comprises an encoder module and a prediction module, and the text features and the speech features are obtained by length prediction and feature extraction on the encoded features by using the prediction module, and the encoded features are obtained by feature encoding on the acoustic features by using the encoder module.

[0100] In the above embodiments of the present application, the confidence estimation result is obtained by matching the speech data and the text data based on the speech features by using the confidence estimation model.

[0101] In the foregoing embodiments of the present application, the confidence estimation model further comprises a word embedding module, a decoder module, a cross-attention module, and a confidence estimation module, the confidence estimation result is obtained by matching the first attention feature corresponding to the speech feature and the second attention feature corresponding to the text data by using the confidence estimation module, the first attention feature and the second attention feature are obtained by performing feature alignment on the first decoding feature corresponding to the speech feature and the second decoding feature corresponding to the text data by using the cross-attention module, the first decoding feature and the second decoding feature are obtained by performing feature decoding on the speech feature and the text feature of the text data by using the decoder module, the text feature of the text data is obtained by performing feature extraction on the text data by using the word embedding module, the length of the first decoding feature is the text length, the length of the second decoding feature is the length of the text data, and the length of the first attention feature and the length of the second attention feature are both the text length.

[0102] In the foregoing embodiments of the present application, the cross-attention module comprises a self-attention layer and a cross-attention layer, the second attention feature is obtained by performing feature alignment on the first attention feature and the second decoding feature by using the cross-attention layer, and the first attention feature is obtained by performing attention processing on the first decoding feature by using the self-attention layer.

[0103] In the foregoing embodiments of the present application, in the training process of the confidence estimation model, the model parameters of the decoder module remain unchanged.

[0104] It should be noted that the preferred embodiments involved in the foregoing embodiments of the present application have the same application scenarios and implementation processes as the scheme provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0105] Embodiment 3

[0106] According to the embodiments of the present application, a confidence estimation method for speech recognition in a virtual reality (VR) device or an augmented reality (AR) device is also provided. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0107] Figure 7 FIG. 3 is a flowchart of a confidence estimation method for speech recognition according to Embodiment 3 of the present application. As shown in FIG. 3, the method can include the following steps: Figure 7

[0108] In step S702, the speech data emitted by the biological object is collected by driving the virtual reality (VR) device or the augmented reality (AR) device, and the text data is obtained.

[0109] ​The biological object in the above steps can be a biological object capable of making sounds, such as a human or other animals capable of making sounds.

[0110] In an optional embodiment, a voice collection device of the VR device or the AR device, such as a microphone, can be driven to collect the sound uttered by the user to obtain voice data, and meanwhile, the voice recognition system can obtain text data for voice recognition from the database.

[0111] In step S704, acoustic features of the voice data are extracted to obtain voice data.

[0112] In step S706, length prediction and feature extraction are performed on the acoustic features to obtain text length and voice features, where the text length is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice features is the same as the text length.

[0113] In step S708, the voice data and the text data are matched based on the voice features to obtain a confidence estimation result, where the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data.

[0114] In step S710, the confidence estimation result is rendered and displayed on a presentation picture of the VR device or the AR device.

[0115] The presentation picture in the above steps can be a picture presented on a display device of the VR device or the AR device, such as a picture displayed in real time on a screen in an HMD head-mounted display.

[0116] In the above embodiments of the present application, the length prediction and feature extraction are performed on the acoustic features to obtain the text length and the voice features, including: performing length prediction and feature extraction on the acoustic features by using a confidence estimation model to obtain the text length and the voice features.

[0117] In the above embodiments of the present application, the confidence estimation model includes an encoder module and a prediction module, and the length prediction and feature extraction are performed on the acoustic features by using the confidence estimation model to obtain the text length and the voice features, including: performing feature encoding on the acoustic features by using the encoder module to obtain encoded features; and performing length prediction and feature extraction on the encoded features by using the prediction module to obtain the text length and the voice features.

[0118] In the above embodiments of the present application, the voice data and the text data are matched based on the voice features to obtain the confidence estimation result, including: matching the voice data and the text data based on the voice features by using the confidence estimation model to obtain the confidence estimation result.

[0119] In the foregoing embodiments of the present application, the confidence estimation model further comprises a word embedding module, a decoder module, a cross-attention module, and a confidence estimation module. The confidence estimation model is used to match the speech data and the text data based on the speech features to obtain a confidence estimation result, including: the word embedding module is used to extract features of the text data to obtain text features of the text data; the decoder module is used to decode the speech features and the text features to obtain first decoding features corresponding to the speech features and second decoding features corresponding to the text data, wherein the length of the first decoding features is the text length, and the length of the second decoding features is the length of the text data; the cross-attention module is used to align the first decoding features and the second decoding features to obtain first attention features corresponding to the first decoding features and second attention features corresponding to the second decoding features, wherein the length of the first attention features and the length of the second attention features are both the text length; and the confidence estimation module is used to match the first attention features and the second attention features to obtain the confidence estimation result.

[0120] In the foregoing embodiments of the present application, the cross-attention module comprises a self-attention layer and a cross-attention layer. The cross-attention module is used to align the first decoding features and the second decoding features to obtain the first attention features corresponding to the first decoding features and the second attention features corresponding to the second decoding features, including: the self-attention layer is used to perform attention processing on the first decoding features to obtain the first attention features; and the cross-attention layer is used to align the first attention features and the second decoding features to obtain the second attention features.

[0121] In the foregoing embodiments of the present application, during the training of the confidence estimation model, the model parameters of the decoder module remain unchanged.

[0122] Optionally, in the present embodiment, the foregoing confidence estimation method can be applied to a hardware environment composed of a server and a virtual reality device. The confidence estimation method is displayed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device. The server can be a server corresponding to a media file operator. The network includes but is not limited to a wide area network, a metropolitan area network, or a local area network. The virtual reality device is not limited to a virtual reality headset, virtual reality glasses, a virtual reality all-in-one machine, and the like.

[0123] Optionally, the virtual reality device comprises a memory, a processor and a transmission device. The memory is configured to store an application program, which can be used to perform the following steps: driving the virtual reality (VR) device or the augmented reality (AR) device to collect voice data emitted by a biological object, and driving the VR device or the AR device to display text data on a presentation screen; performing feature extraction on the voice data to obtain acoustic features of the voice data; performing length prediction and feature extraction on the acoustic features to obtain a text length and voice features, wherein the text length is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice features is the same as the text length; matching the voice data and the text data based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data; and driving the VR device or the AR device to render and display the confidence estimation result on the presentation screen.

[0124] It should be noted that the above-mentioned confidence estimation method applied in the VR device or the AR device in the embodiment can comprise Figure 7 The method of the embodiment is used to achieve the purpose of driving the VR device or the AR device to display the confidence estimation method.

[0125] Optionally, the processor of the embodiment can call the application program stored in the memory through the transmission device to perform the above-mentioned steps. The transmission device can receive the media file sent by the server through the network, and can also be used for data transmission between the processor and the memory.

[0126] Optionally, in the virtual reality device, a head-mounted display with eye tracking is provided. A screen in the HMD head-mounted display is used to display the displayed video screen. An eye tracking module in the HMD is used to obtain the real-time motion trajectory of the user's eyeball. A tracking system is used to track the position information and motion information of the user in the real three-dimensional space. A calculation processing unit is used to obtain the real-time position and motion information of the user from the tracking system, and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space and the direction of the user's field of view in the virtual three-dimensional space.

[0127] In the embodiment of the present application, the virtual reality device can be connected with a terminal, and the terminal is connected with a server through a network. The above-mentioned virtual reality device is not limited to a virtual reality helmet, a virtual reality glasses, a virtual reality all-in-one machine, etc. The above-mentioned terminal is not limited to a PC, a mobile phone, a tablet computer, etc. The server can be a server corresponding to a media file operator. The above-mentioned network includes but is not limited to a wide area network, a metropolitan area network or a local area network.

[0128] It should be noted that the preferred embodiments involved in the above-mentioned embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0129] Embodiment 4

[0130] According to the embodiments of the present application, a confidence estimation method is further provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0131] Figure 8 is a flowchart of a confidence estimation method according to Embodiment 4 of the present application. As shown in Figure 8 the method comprises the following steps:

[0132] In step S802, voice data and text data are obtained by calling a first interface, wherein the first interface comprises a first parameter, and the parameter value of the first parameter is the voice data and the text data.

[0133] The first interface in the above steps can be an interface for data interaction between the client and the server. Through the interface, the client can send the collected voice data and the text data requiring speech recognition to the server.

[0134] In step S804, acoustic features of the voice data are obtained by performing feature extraction on the voice data.

[0135] In step S806, the acoustic features are subjected to length prediction and feature extraction, and the text length and the voice features are predicted, wherein the text length is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice features is the same as the text length.

[0136] In step S808, the voice data and the text data are matched based on the voice features, and a confidence estimation result is obtained, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data.

[0137] In step S810, the confidence estimation result is output by calling a second interface, wherein the second interface comprises a second parameter, and the parameter value of the second parameter is the confidence estimation result.

[0138] The second interface in the above steps can be an interface for data interaction between the client and the server. Through the interface, the server can send the confidence estimation result obtained by performing speech recognition and matching on the voice data and the text data to the client.

[0139] In the foregoing embodiments of the present application, the acoustic features are subjected to length prediction and feature extraction, and text length and speech features are predicted, comprising: using a confidence estimation model to perform length prediction and feature extraction on the acoustic features to obtain the text length and the speech features.

[0140] In the foregoing embodiments of the present application, the confidence estimation model comprises: an encoder module and a prediction module, wherein the confidence estimation model is used to perform length prediction and feature extraction on the acoustic features to obtain the text length and the speech features, comprising: using the encoder module to perform feature encoding on the acoustic features to obtain encoded features; and using the prediction module to perform length prediction and feature extraction on the encoded features to obtain the text length and the speech features.

[0141] In the foregoing embodiments of the present application, the speech data and the text data are matched based on the speech features to obtain the confidence estimation result, comprising: using the confidence estimation model to match the speech data and the text data based on the speech features to obtain the confidence estimation result.

[0142] In the foregoing embodiments of the present application, the confidence estimation model further comprises: a word embedding module, a decoder module, a cross-attention module and a confidence estimation module, wherein the confidence estimation model is used to match the speech data and the text data based on the speech features to obtain the confidence estimation result, comprising: using the word embedding module to extract features of the text data to obtain text features of the text data; using the decoder module to decode the speech features and the text features to obtain first decoding features corresponding to the speech features and second decoding features corresponding to the text data, wherein the length of the first decoding features is the text length, and the length of the second decoding features is the length of the text data; using the cross-attention module to align the first decoding features and the second decoding features to obtain first attention features corresponding to the first decoding features and second attention features corresponding to the second decoding features, wherein the length of the first attention features and the length of the second attention features are both the text length; and using the confidence estimation module to match the first attention features and the second attention features to obtain the confidence estimation result.

[0143] In the foregoing embodiments of the present application, the cross-attention module comprises: a self-attention layer and a cross-attention layer, wherein the cross-attention module is used to align the first decoding features and the second decoding features to obtain the first attention features corresponding to the first decoding features and the second attention features corresponding to the second decoding features, comprising: using the self-attention layer to perform attention processing on the first decoding features to obtain the first attention features; and using the cross-attention layer to align the first attention features and the second decoding features to obtain the second attention features.

[0144] In the above embodiments of the present application, during the training of the confidence estimation model, the model parameters of the decoder module remain unchanged.

[0145] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0146] Embodiment 5

[0147] According to the embodiments of the present application, a confidence estimation device for implementing the above confidence estimation method is further provided, as shown in the figure, the device 900 includes an acquisition module 902, a feature extraction module 904, a prediction and extraction module 906 and a matching module 908. Figure 9

[0148] The acquisition module 902 is configured to acquire speech data and text data; the feature extraction module 904 is configured to perform feature extraction on the speech data to obtain acoustic features of the speech data; the prediction and extraction module 906 is configured to perform length prediction and feature extraction on the acoustic features to obtain a text length and speech features, wherein the text length is used to represent the number of characters contained in the text corresponding to the speech data, and the length of the speech features is the same as the text length; and the matching module 908 is configured to match the speech data and the text data based on the speech features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data with the text data.

[0149] It should be noted that the above acquisition module 902, feature extraction module 904, prediction and extraction module 906 and matching module 908 correspond to steps S302 to S308 in Embodiment 1, and the four modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules as part of the device can run in the AR / VR device provided in Embodiment 1.

[0150] In the above embodiments of the present application, the prediction and extraction module is further configured to perform length prediction and feature extraction on the acoustic features by using a confidence estimation model to obtain a text length and speech features.

[0151] In the above embodiments of the present application, the confidence estimation model includes an encoder module and a prediction module, and the prediction and extraction module includes an encoding unit configured to perform feature encoding on the acoustic features by using the encoder module to obtain encoded features, and a prediction and extraction unit configured to perform length prediction and feature extraction on the encoded features by using the prediction module to obtain a text length and speech features.

[0152] ​In the above embodiments of the present application, the matching module is further configured to match the speech data and the text data based on the speech features by using the confidence estimation model to obtain a confidence estimation result.

[0153] In the above embodiments of the present application, the confidence estimation model further includes a word embedding module, a decoder module, a cross-attention module, and a confidence estimation module, and the matching module includes a feature extraction unit configured to extract features of the text data by using the word embedding module to obtain text features of the text data; a decoding unit configured to decode the speech features and the text features by using the decoder module to obtain first decoding features corresponding to the speech features and second decoding features corresponding to the text data, wherein the length of the first decoding features is the text length, and the length of the second decoding features is the length of the text data; a feature alignment unit configured to align the first decoding features and the second decoding features by using the cross-attention module to obtain first attention features corresponding to the first decoding features and second attention features corresponding to the second decoding features, wherein the length of the first attention features and the length of the second attention features are both the text length; and a matching unit configured to match the first attention features and the second attention features by using the confidence estimation module to obtain the confidence estimation result.

[0154] In the above embodiments of the present application, the cross-attention module includes a self-attention layer and a cross-attention layer, and the feature alignment unit includes a self-attention subunit configured to perform attention processing on the first decoding features by using the self-attention layer to obtain the first attention features, and a cross-attention subunit configured to align the first attention features and the second decoding features by using the cross-attention layer to obtain the second attention features.

[0155] In the above embodiments of the present application, during the training of the confidence estimation model, the model parameters of the decoder module remain unchanged.

[0156] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0157] Embodiment 6

[0158] According to the embodiments of the present application, a confidence estimation device for implementing the above confidence estimation method is further provided, as shown in Figure 10 The device 1000 includes a driving module 1002 and a display module 1004.

[0159] The driving module 1002 is configured to drive the voice collection device to collect voice data of the biological object in response to an input instruction acting on the operation interface, and obtain text data.

[0160] It should be noted that the driving module 1002 and the display module 1004 correspond to steps S502 to S504 in Embodiment 2, and the two modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules can run in the AR / VR device provided in Embodiment 1 as part of the device.

[0161] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same scheme, application scenario, and implementation process as the scheme provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0162] Embodiment 7

[0163] According to the embodiments of the present application, a confidence estimation method is also provided, as shown in Figure 11 The device 1100 includes a first driving module 1102, a feature extraction module 1104, a prediction and extraction module 1106, a matching module 1108, and a second driving module 1110.

[0164] Among them, the first driving module 1102 is used to drive the virtual reality VR device or the augmented reality AR device to collect voice data emitted by the biological object and obtain text data; the feature extraction module 1104 is used to extract features from the voice data to obtain the acoustic features of the voice data; the prediction and extraction module 1106 is used to perform length prediction and feature extraction on the acoustic features to predict the text length and voice features, wherein the text length is used to characterize the number of characters contained in the text corresponding to the voice data, and the length of the voice feature is the same as the text length; the matching module 1108 is used to match the voice data and text data based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to characterize the matching probability of each character contained in the text corresponding to the voice data and the text data; the second driving module 1110 is used to drive the VR device or AR device to render and display the confidence estimation result on the presentation screen.

[0165] It should be noted that the first driving module 1102, feature extraction module 1104, prediction and extraction module 1106, matching module 1108, and second driving module 1110 described above correspond to steps S702 to S710 in Example 3. The examples and application scenarios implemented by the five modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the apparatus, can be run in the AR / VR device provided in Example 1.

[0166] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0167] Example 8

[0168] According to an embodiment of the present application, a confidence estimation device for implementing the above confidence estimation method is also provided, such as Figure 12 As shown, the apparatus 1200 includes: a first calling module 1202 , a feature extraction module 1204 , a prediction and extraction module 1206 , a matching module 1208 and a second calling module 1210 .

[0169] The first calling module 1202 is configured to obtain voice data and text data by calling a first interface, the first interface includes a first parameter, and a parameter value of the first parameter is the voice data and the text data; the feature extraction module 1204 is configured to perform feature extraction on the voice data to obtain acoustic features of the voice data; the prediction and extraction module 1206 is configured to perform length prediction and feature extraction on the acoustic features to obtain a text length and voice features, wherein the text length is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice features is the same as the text length; the matching module 1208 is configured to match the voice data and the text data based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data; and the second calling module 1210 is configured to output the confidence estimation result by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the confidence estimation result.

[0170] It should be noted that the first calling module 1202, the feature extraction module 1204, the prediction and extraction module 1206, the matching module 1208 and the second calling module 1210 correspond to steps S802 to S810 in Embodiment 4, and the five modules have the same instances and application scenarios as the corresponding steps, but are not limited to the solutions disclosed in Embodiment 1. It should be noted that the above modules can run in the AR / VR device provided in Embodiment 1 as part of the device.

[0171] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios, implementation processes as the solutions provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

[0172] Embodiment 9

[0173] The embodiments of the present application can provide an electronic device, which can include an AR / VR device, a server, a client, and the AR / VR device can be any one of the AR / VR devices in the AR / VR device group. Optionally, the electronic device includes a processor and a memory connected to the processor, and the memory is configured to provide instructions for the processor to process the following processing steps: obtaining voice data and text data; performing feature extraction on the voice data to obtain acoustic features of the voice data; performing length prediction and feature extraction on the acoustic features to obtain a text length and voice features, wherein the text length is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice features is the same as the text length; matching the voice data and the voice text based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data.

[0174] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the scheme provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0175] Embodiment 10

[0176] The embodiments of the present application can provide an AR / VR device, which can be any AR / VR device in a group of AR / VR devices. Alternatively, in the embodiments, the AR / VR device can be replaced by a terminal device such as a mobile terminal.

[0177] Alternatively, in the embodiments, the AR / VR device can be located in at least one network device of a plurality of network devices of a computer network.

[0178] In the embodiments, the AR / VR device can execute program codes of the following steps in the confidence estimation method: obtaining voice data and text data; performing feature extraction on the voice data to obtain acoustic features of the voice data; performing length prediction and feature extraction on the acoustic features to predict text length and voice features, wherein the text length is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice features is the same as the text length; and matching the voice data and the voice text based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data.

[0179] Alternatively, Figure 13 is a structural block diagram of an AR / VR device according to an embodiment of the present application. As shown in the figure, the AR / VR device A can include one or more (only one is shown in the figure) processors 1302 and a memory 1304. Figure 13

[0180] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the confidence estimation method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, i.e., implements the confidence estimation method described above. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the terminal A through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0181] ​The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining voice data and text data; performing feature extraction on the voice data to obtain acoustic features of the voice data; performing length prediction and feature extraction on the acoustic features to obtain a text length and speech features, wherein the text length is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the speech features is the same as the text length; and matching the voice data and the voice text based on the speech features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data.

[0182] Optionally, the processor can further execute program codes of the following steps: performing length prediction and feature extraction on the acoustic features by using the confidence estimation model to obtain the text length and the speech features.

[0183] Optionally, the confidence estimation model comprises an encoder module and a prediction module, and the processor can further execute program codes of the following steps: performing feature encoding on the acoustic features by using the encoder module to obtain encoded features; and performing length prediction and feature extraction on the encoded features by using the prediction module to obtain the text length and the speech features.

[0184] Optionally, the processor can further execute program codes of the following steps: matching the voice data and the text data based on the speech features by using the confidence estimation model to obtain the confidence estimation result.

[0185] Optionally, the confidence estimation model further comprises a word embedding module, a decoder module, a cross-attention module and a confidence estimation module, and the processor can further execute program codes of the following steps: performing feature extraction on the text data by using the word embedding module to obtain text features of the text data; performing feature decoding on the speech features and the text features by using the decoder module to obtain first decoding features corresponding to the speech features and second decoding features corresponding to the text data, wherein the length of the first decoding features is the text length, and the length of the second decoding features is the length of the text data; performing feature alignment on the first decoding features and the second decoding features by using the cross-attention module to obtain first attention features corresponding to the first decoding features and second attention features corresponding to the second decoding features, wherein the length of the first attention features and the second attention features is the text length; and matching the first attention features and the second attention features by using the confidence estimation module to obtain the confidence estimation result.

[0186] Optionally, the cross-attention module comprises a self-attention layer and a cross-attention layer, and the processor can further execute program codes for: performing attention processing on the first decoding feature by using the self-attention layer to obtain first attention features; and performing feature alignment on the first attention features and the second decoding feature by using the cross-attention layer to obtain second attention features.

[0187] Optionally, during the training of the confidence estimation model, the model parameters of the decoder module remain unchanged.

[0188] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: in response to an input instruction acting on the operation interface, driving the voice collection device to collect voice data emitted by the biological object and obtaining text data; and in response to a confidence estimation instruction acting on the operation interface, displaying a confidence estimation result on the operation interface, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data, and the confidence estimation result is obtained by matching the voice data and the text data based on voice features of the voice data, the length of the voice features being the same as the length of the text, the length of the text being used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice features and the length of the text being obtained by length prediction and feature extraction prediction on acoustic features of the voice data, the acoustic features being obtained by feature extraction on the voice data.

[0189] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: driving a virtual reality (VR) device or an augmented reality (AR) device to collect voice data emitted by the biological object and obtaining text data; performing feature extraction on the voice data to obtain acoustic features of the voice data; performing length prediction and feature extraction on the acoustic features to predict the length of the text and the voice features, wherein the length of the text is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice features is the same as the length of the text; matching the voice data and the text data based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data; and driving the VR device or the AR device to render and display the confidence estimation result on a presentation screen.

[0190] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain voice data and text data by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the voice data and text data; perform feature extraction on the voice data to obtain the acoustic features of the voice data; perform length prediction and feature extraction on the acoustic features to predict the text length and voice features, wherein the text length is used to characterize the number of characters contained in the text corresponding to the voice data, and the length of the voice feature is the same as the text length; match the voice data and text data based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to characterize the matching probability of each character contained in the text corresponding to the voice data with the text data; output the confidence estimation result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the confidence estimation result.

[0191] By using an embodiment of the present application, a scheme for a confidence estimation method is provided. Because the speech features and confidence estimation results are the same as the number of characters contained in the text corresponding to the speech data, the problem of low reliability of the confidence estimation results when the speech data and text data are of unequal lengths is avoided, thereby achieving the technical effect of improving the accuracy of the confidence estimation results, thereby solving the technical problem of low accuracy of confidence estimation based on speech recognition in related technologies, and enabling downstream tasks to better perform targeted operations based on the confidence estimation results, helping downstream tasks make correct judgments, and laying the foundation for a stable human-computer interaction link.

[0192] It can be understood by those skilled in the art that Figure 13 The structure shown is for illustration only. The AR / VR device can also be replaced by a smartphone (such as an Android phone, iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, and other terminal devices. Figure 13 It does not limit the structure of the above AR / VR device. For example, AR / VR device A may also include Figure 13 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 13 Figure 13 Different configurations shown.

[0193] Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by instructing the terminal device related hardware through a program, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0194] Embodiment 11

[0195] The embodiments of the present application also provide a computer readable storage medium. Optionally, in the embodiment, the computer readable storage medium can be used to save the program code executed by the confidence estimation method provided in the embodiment 1.

[0196] Optionally, in the embodiment, the computer readable storage medium can be located in any one of the computer terminals in the AR / VR device terminal group in the AR / VR device network, or in any one of the mobile terminals in the mobile terminal group.

[0197] Optionally, in the embodiment, the computer readable storage medium is configured to store program code for performing the following steps: obtaining voice data and text data; performing feature extraction on the voice data to obtain acoustic features of the voice data; performing length prediction and feature extraction on the acoustic features to obtain a text length and voice features, wherein the text length is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice features is the same as the text length; and matching the voice data and the voice text based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data.

[0198] Optionally, the computer readable storage medium is further configured to store program code for performing the following steps: performing length prediction and feature extraction on the acoustic features by using the confidence estimation model to obtain the text length and the voice features.

[0199] Optionally, the confidence estimation model includes an encoder module and a prediction module, and the computer readable storage medium is further configured to store program code for performing the following steps: performing feature encoding on the acoustic features by using the encoder module to obtain encoded features; and performing length prediction and feature extraction on the encoded features by using the prediction module to obtain the text length and the voice features.

[0200] Optionally, the computer readable storage medium is further configured to store program code for performing the following steps: matching the voice data and the text data based on the voice features by using the confidence estimation model to obtain the confidence estimation result.

[0201] Optionally, the confidence estimation model further comprises a word embedding module, a decoder module, a cross-attention module and a confidence estimation module, and the computer readable storage medium is further configured to store program codes for performing the following steps: performing feature extraction on the text data by using the word embedding module to obtain text features of the text data; performing feature decoding on the speech features and the text features by using the decoder module to obtain first decoding features corresponding to the speech features and second decoding features corresponding to the text data, wherein the length of the first decoding features is the text length, and the length of the second decoding features is the length of the text data; performing feature alignment on the first decoding features and the second decoding features by using the cross-attention module to obtain first attention features corresponding to the first decoding features and second attention features corresponding to the second decoding features, wherein the length of the first attention features and the length of the second attention features are both the text length; and performing matching on the first attention features and the second attention features by using the confidence estimation module to obtain the confidence estimation result.

[0202] Optionally, the cross-attention module comprises a self-attention layer and a cross-attention layer, and the computer readable storage medium is further configured to store program codes for performing the following steps: performing attention processing on the first decoding features by using the self-attention layer to obtain the first attention features; and performing feature alignment on the first attention features and the second decoding features by using the cross-attention layer to obtain the second attention features.

[0203] Optionally, during the training of the confidence estimation model, the model parameters of the decoder module remain unchanged.

[0204] Optionally, in the embodiment, the computer readable storage medium is configured to store program codes for performing the following steps: in response to an input instruction acting on the operation interface, driving the speech collection device to collect speech data emitted by the biological object, and obtaining the text data; and in response to a confidence estimation instruction acting on the operation interface, displaying the confidence estimation result on the operation interface, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data with the text data, the confidence estimation result is obtained by matching the speech data and the text data based on speech features of the speech data, the length of the speech features is the same as the text length, the text length is used to represent the number of characters contained in the text corresponding to the speech data, the speech features and the text length are obtained by performing length prediction and feature extraction prediction on acoustic features of the speech data, and the acoustic features are obtained by performing feature extraction on the speech data.

[0205] Optionally, in the embodiment, the computer readable storage medium is configured to store program code for performing the following steps: driving a virtual reality (VR) device or an augmented reality (AR) device to collect voice data emitted by a biological object and to obtain text data; performing feature extraction on the voice data to obtain acoustic features of the voice data; performing length prediction and feature extraction on the acoustic features to predict a text length and voice features, wherein the text length is used to represent a number of characters contained in the text corresponding to the voice data, and the voice features have the same length as the text length; matching the voice data and the text data based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent a matching probability of each character contained in the text corresponding to the voice data with the text data; and driving the VR device or the AR device to render and display the confidence estimation result on a presentation screen of the VR device or the AR device.

[0206] Optionally, in the embodiment, the computer readable storage medium is configured to store program code for performing the following steps: obtaining voice data and text data by calling a first interface, wherein the first interface comprises a first parameter, and a parameter value of the first parameter is the voice data and the text data; performing feature extraction on the voice data to obtain acoustic features of the voice data; performing length prediction and feature extraction on the acoustic features to predict a text length and voice features, wherein the text length is used to represent a number of characters contained in the text corresponding to the voice data, and the voice features have the same length as the text length; matching the voice data and the text data based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent a matching probability of each character contained in the text corresponding to the voice data with the text data; and outputting the confidence estimation result by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter is the confidence estimation result.

[0207] The above sequence numbers of the embodiments of the present application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0208] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0209] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only a schematic and illustrative, for example, the division of units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0210] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0211] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0212] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various program codes that can be stored in the medium.

[0213] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A confidence estimation method for speech recognition, characterized by, The method comprises: acquiring voice data and text data; performing feature extraction on the voice data to obtain acoustic features of the voice data; performing length prediction and feature extraction on the acoustic features to predict text length and voice features, wherein the text length is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice features is the same as the text length; matching the voice data and the text data based on the voice features to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data, and the length of the confidence estimation result is the same as the text length.

2. The method of claim 1, wherein, Performing length prediction and feature extraction on the acoustic features to predict text length and voice features, comprising: performing length prediction and feature extraction on the acoustic features by using a confidence estimation model to obtain the text length and the voice features.

3. The method of claim 2, wherein, The confidence estimation model comprises an encoder module and a prediction module, wherein performing length prediction and feature extraction on the acoustic features by using a confidence estimation model to obtain the text length and the voice features comprises: performing feature encoding on the acoustic features by using the encoder module to obtain encoded features; performing length prediction and feature extraction on the encoded features by using the prediction module to obtain the text length and the voice features.

4. The method of claim 1, wherein, Matching the voice data and the text data based on the voice features to obtain a confidence estimation result, comprising: matching the voice data and the text data based on the voice features by using a confidence estimation model to obtain the confidence estimation result.

5. The method of claim 4, wherein, The confidence estimation model further comprises a word embedding module, a decoder module, a cross-attention module, and a confidence estimation module, wherein matching the voice data and the text data based on the voice features by using a confidence estimation model to obtain a confidence estimation result comprises: performing feature extraction on the text data by using the word embedding module to obtain text features of the text data; performing feature decoding on the voice features and the text features by using the decoder module to obtain first decoding features corresponding to the voice features and second decoding features corresponding to the text data, wherein the length of the first decoding features is the text length, and the length of the second decoding features is the length of the text data; performing feature alignment on the first decoding features and the second decoding features by using the cross-attention module to obtain first attention features corresponding to the first decoding features and second attention features corresponding to the second decoding features, wherein the length of the first attention features and the length of the second attention features are both the text length; matching the first attention features and the second attention features by using the confidence estimation module to obtain the confidence estimation result.

6. The method of claim 5, wherein, The cross-attention module comprises a self-attention layer and a cross-attention layer, wherein the cross-attention module is used to perform feature alignment on the first decoding feature and the second decoding feature, to obtain a first attention feature corresponding to the first decoding feature and a second attention feature corresponding to the second decoding feature, which comprises: The self-attention layer is used to perform attention processing on the first decoding feature to obtain the first attention feature; The cross-attention layer is used to perform feature alignment on the first attention feature and the second decoding feature to obtain the second attention feature.

7. The method of claim 5, wherein, During the training of the confidence estimation model, the model parameters of the decoder module remain unchanged.

8. A confidence estimation method for speech recognition, characterized by, Comprising: In response to an input instruction acting on the operation interface, driving a voice collection device to collect voice data emitted by a biological object, and obtaining text data; In response to a confidence estimation instruction acting on the operation interface, displaying a confidence estimation result on the operation interface, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data, the confidence estimation result is obtained by matching the voice data and the text data based on the voice feature of the voice data, the length of the voice feature is the same as the text length, the text length is used to represent the number of characters contained in the text corresponding to the voice data, the voice feature and the text length are obtained by length prediction and feature extraction prediction on the acoustic feature of the voice data, the acoustic feature is obtained by feature extraction on the voice data, and the length of the confidence estimation result is the same as the text length.

9. A confidence estimation method for speech recognition, characterized by, Comprising: Driving a virtual reality (VR) device or an augmented reality (AR) device to collect voice data emitted by a biological object, and obtaining text data; Performing feature extraction on the voice data to obtain acoustic features of the voice data; Performing length prediction and feature extraction on the acoustic features to predict a text length and a voice feature, wherein the text length is used to represent the number of characters contained in the text corresponding to the voice data, and the length of the voice feature is the same as the text length; Matching the voice data and the text data based on the voice feature to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the voice data and the text data, and the length of the confidence estimation result is the same as the text length; Driving the VR device or the AR device to render and display the confidence estimation result on a presentation screen.

10. A confidence estimation method for speech recognition, characterized by, Comprising: Obtaining voice data and text data by calling a first interface, wherein the first interface comprises a first parameter, and the parameter value of the first parameter is the voice data and the text data; Performing feature extraction on the voice data to obtain acoustic features of the voice data; The acoustic feature is subjected to length prediction and feature extraction, and text length and speech features are predicted, wherein the text length is used to represent the number of characters contained in the text corresponding to the speech data, and the length of the speech feature is the same as the text length; The speech data and the text data are matched based on the speech feature to obtain a confidence estimation result, wherein the confidence estimation result is used to represent the matching probability of each character contained in the text corresponding to the speech data and the text data, and the length of the confidence estimation result is the same as the text length; The confidence estimation result is output by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the confidence estimation result.

11. A computer readable storage medium, characterized in that, The computer readable storage medium includes a stored program, wherein the program controls the device where the computer readable storage medium is located to execute the method of any one of claims 1-10 when the program is running.

12. An electronic device, comprising: It includes: a processor; a memory connected to the processor, used to provide the processor with instructions for processing the method of any one of claims 1-10.

Images