Speech recognition method and speech recognition apparatus, electronic device, storage medium

Abstract

A speech recognition method, a speech recognition device, an electronic device, and a storage medium are provided. The speech recognition method includes obtaining a speech recognition model, the speech recognition model including a first encoding module, a first alignment module, a first shielding module, a first correction module, and a first decoding module. Speech information is input into the first encoding module for encoding processing to obtain a first encoding vector. The first encoding vector is input into the first decoding module for decoding processing to obtain a decoded text sequence. The first encoding vector is input into the first alignment module for alignment processing to obtain a first text sequence aligned with the decoded text sequence. At least one first text label in the first text sequence is shielded by the first shielding module to obtain a target text sequence. The decoded text sequence and the target text sequence are input into the first correction module for correction processing to obtain a speech recognition text, which can improve the efficiency of automatic speech recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech recognition method, a speech recognition device, an electronic device, and a storage medium. BACKGROUND

[0002] Automatic speech recognition (ASR) is a technology for converting human speech into text. Current automatic speech recognition models usually adopt an encoder-decoder architecture, however, this model architecture needs to use the generated words to predict the next position of the words in the sentence, resulting in low decoding efficiency of speech recognition. Therefore, how to improve the efficiency of automatic speech recognition has become a technical problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a speech recognition method, a speech recognition device, an electronic device, and a storage medium, which can improve the efficiency of automatic speech recognition.

[0004] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides a speech recognition method, which comprises: acquiring speech information; acquiring a speech recognition model, the speech recognition model comprising a first encoding module, a first alignment module, a first shielding module, a first correction module, and a first decoding module; inputting the speech information into the first encoding module for encoding processing to obtain a first encoding vector; inputting the first encoding vector into the first decoding module for decoding processing to obtain a decoded text sequence; inputting the first encoding vector into the first alignment module for alignment processing to obtain a first text sequence, the first text sequence being aligned with the decoded text sequence, and the first text sequence comprising at least two first text labels; inputting the first text sequence into the first shielding module to shield at least one first text label through the first shielding module to obtain a target text sequence; and inputting the decoded text sequence and the target text sequence into the first correction module for correction processing to obtain a speech recognition text.

[0005] In some optional embodiments, the inputting the first text sequence into the first shielding module to shield at least one first text label through the first shielding module to obtain a target text sequence comprises: acquiring a confidence degree corresponding to each first text label; inputting the first text sequence into the first shielding module, and acquiring a first text label with a confidence degree less than a preset value from the first text sequence as a to-be-shielded label through the first shielding module; and shielding the to-be-shielded label in the first text sequence through the first shielding module to obtain a target text sequence.

[0006] In some optional embodiments, the training step of the speech recognition model comprises:

[0007] constructing an initial generation model comprising a second encoding module, a second alignment module, a second masking module, a second correction module and a second decoding module; obtaining a speech sample and a reference recognition text corresponding to the speech sample; inputting the speech sample into the second encoding module for encoding processing to obtain a second encoding vector; inputting the second encoding vector into the second alignment module for alignment processing to obtain a second text sequence, the second text sequence comprising at least two second text labels; inputting the second text sequence into the second masking module to mask at least one of the second text labels by the second masking module to obtain a third text sequence; inputting the second encoding vector into the second decoding module for decoding processing to obtain a fourth text sequence; inputting the third text sequence and the fourth text sequence into the second correction module for correction processing to obtain a target recognition text; adjusting the initial generation model according to the reference recognition text and the target recognition text to obtain a speech recognition model.

[0008] In some optional embodiments, the adjusting the initial generation model according to the reference recognition text and the target recognition text to obtain a speech recognition model comprises:

[0009] calculating a loss function value of the initial generation model according to a preset loss function, the reference recognition text and the target recognition text; if the loss function value satisfies a preset training end condition, determining a speech recognition model according to the initial generation model; if the loss function value does not satisfy the training end condition, continuing to perform the step of obtaining a speech sample and a reference recognition text corresponding to the speech sample;

[0010] The method further comprises:

[0011] if the loss function value satisfies the training end condition, determining the loss function value as a preset value, the preset value being used for the first masking module to determine the to-be-masked label from the first text sequence.

[0012] In some optional embodiments, the masking at least one of the second text labels by the second masking module to obtain a third text sequence comprises:

[0013] masking, by the second masking module, a second text label randomly obtained from the second text sequence as a target text label to obtain a third text sequence.

[0014] In some optional embodiments, the decoded text sequence includes at least one decoded text label, and the inputting the decoded text sequence and the target text sequence into the first correction module for correction processing to obtain the speech recognition text comprises:

[0015] The decoded text sequence and the target text sequence are input into the first correction module; the first correction module is used to identify a sequence position corresponding to each of the to-be-screened labels in the decoded text sequence, and then obtain a decoded text label corresponding to the sequence position from the decoded text sequence as a matching label; and the first correction module is used to replace the to-be-screened labels with the matching labels to obtain the speech recognition text.

[0016] In some optional embodiments, the inputting the speech information into the first encoding module for encoding processing to obtain the first encoding vector comprises:

[0017] A plurality of speech segmentation points are analyzed from the speech information; the speech information is segmented according to the plurality of speech segmentation points to obtain a speech feature sequence, the speech feature sequence including a plurality of speech segments; and the speech feature sequence is input into the first encoding module for encoding processing to obtain the first encoding vector.

[0018] To achieve the above object, a second aspect of the embodiment of the present application provides a speech recognition device, which comprises:

[0019] An acquisition module is configured to acquire speech information, and a speech recognition model is acquired, the speech recognition model including a first encoding module, a first alignment module, a first screening module, a first correction module and a first decoding module;

[0020] An encoding processing module is configured to input the speech information into the first encoding module for encoding processing to obtain a first encoding vector;

[0021] A decoding processing module is configured to input the first encoding vector into the first decoding module for decoding processing to obtain a decoded text sequence;

[0022] An alignment processing module is configured to input the first encoding vector into the first alignment module for alignment processing to obtain a first text sequence, the first text sequence being aligned with the decoded text sequence, and the first text sequence including at least two first text labels;

[0023] A screening processing module is configured to input the first text sequence into the first screening module, and the first screening module is used to screen at least one of the first text labels to obtain a target text sequence;

[0024] The correction processing module is configured to input the decoded text sequence and the target text sequence into the first correction module for correction processing, to obtain speech recognition text.

[0025] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech recognition method of the first aspect when executing the computer program.

[0026] To achieve the above object, a fourth aspect of the embodiments of the present application further provides a computer readable storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the speech recognition method of the first aspect.

[0027] The speech recognition method and the speech recognition device, the electronic device and the storage medium provided by the embodiments of the present application can pre-train a speech recognition model, and the speech recognition model comprises a first encoding module, a first alignment module, a first shielding module, a first correction module and a first decoding module. The speech recognition model can input the speech information into the first encoding module for encoding processing to obtain a first encoding vector. The first decoding module decodes the first encoding vector into a decoded text sequence, the first alignment module performs alignment processing on the first encoding vector to obtain a first text sequence aligned with the decoded text sequence, and the first shielding module shields at least one first text label in the first text sequence to obtain a target text sequence. The speech recognition method can dynamically filter abnormal labels and reduce the inference time. Based on this, the first correction module corrects the decoded text sequence and the target text sequence into speech recognition text, can focus on matching the text content, relaxes the restriction on the text order, improves the efficiency of automatic speech recognition, and in addition, not only alleviates the decoding error problem of the second decoding module, but also can replace and correct the text labels combined with the data alignment processing result, which is conducive to improving the accuracy of speech recognition. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flowchart of a speech recognition method provided by the embodiments of the present application;

[0029] Figure 2 is Figure 1 is a specific flowchart of step S106 in the method;

[0030] Figure 3 is a flowchart of a training step of a speech recognition model in the embodiments of the present application;

[0031] Figure 4 is a module block diagram of a speech recognition device provided by the embodiments of the present application;

[0032] Figure 5 Fig. 1 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0034] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0036] First, the terms involved in the present application are analyzed:

[0037] Artificial intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0038] Natural language processing (NLP): NLP uses computers to process, understand and use human languages (such as Chinese, English, etc.), and NLP is a branch of artificial intelligence, an interdisciplinary subject of computer science and linguistics, and is also commonly referred to as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistic research related to language computing.

[0039] Automatic speech recognition (ASR): A technology that takes speech as the research object and converts human speech into text through speech signal processing and pattern recognition. ASR is suitable for voice-controlled remote controls of household appliances and electronic devices such as televisions, computers, cars, sound systems, air conditioners, etc., voice-controlled name dialing on telephones, mobile phones or PDAs, voice-controlled speech search tags for digital recorders, voice-controlled children's toys, etc.; it can also be used in personal, call center, and telecommunications-level application information query and service fields.

[0040] Current automatic speech recognition models usually use an encoder-decoder architecture, however, this model architecture needs to use the generated word to predict the next position word in the sentence, resulting in low decoding efficiency of speech recognition. Therefore, how to improve the efficiency of automatic speech recognition has become a technical problem to be solved.

[0041] Based on this, the embodiments of the present application provide a speech recognition method and a speech recognition device, an electronic device and a storage medium, which can improve the efficiency of automatic speech recognition.

[0042] The embodiments of the present application provide a speech recognition method and a speech recognition device, an electronic device and a storage medium, which are specifically explained by the following embodiments. First, the speech recognition method in the embodiments of the present application is described.

[0043] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0044] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, and the like. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning / deep learning, and the like.

[0045] The voice recognition method provided by the embodiments of the present application relates to the technical field of artificial intelligence, in particular to the technical field of data mining. The voice recognition method provided by the embodiments of the present application can be applied to a terminal, can be applied to a server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart watch or the like; the server can be a stand-alone server, or a cloud server providing cloud services, a cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content delivery network (CDN), and basic cloud computing services such as big data and artificial intelligence platforms; and the software can be an application implementing the voice recognition method, but is not limited to the above forms. Hereinafter, the terminal will be taken as an example for description.

[0046] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0047] In a first aspect, refer to Figure 1 , Figure 1 is a flowchart of a voice recognition method provided by the embodiments of the present application. The voice recognition method includes steps S100 to S107. It should be understood that the voice recognition method of the embodiments of the present application includes but is not limited to steps S100 to S107, which will be described in detail below in combination with Figure 1 .

[0048] Step S101: Acquire voice information.

[0049] In the embodiments of the present application, the voice information can be directly collected by the audio collection device of the terminal, or the collected voice information can be received from a database or other terminals, and no specific limitation is made thereto.

[0050] Step S102: Obtain a voice recognition model, which includes a first encoding module, a first alignment module, a first shielding module, a first correction module, and a first decoding module.

[0051] In the embodiments of the present application, the voice recognition model can adopt a non-autoregressive (non-autoregressive translation, NART) end-to-end automatic speech recognition framework. In machine translation, unlike the autoregressive (autoregressive translation, ART) model which needs to use the generated word to predict the word at the next position in the sentence, the non-autoregressive model breaks the serial order when generating a text sequence, and can decode the entire target sentence at one time, thereby solving the problems brought by the ART model. Among them, the first encoding module is used for voice encoding, the first alignment module is used for modeling of the text sequence, the first shielding module is used for shielding processing of specified data in the text sequence, the first correction module is used for comparison and supplement of the text sequence, and the first decoding module is used for data decoding.

[0052] Step S103: Input the voice information into the first encoding module for encoding processing to obtain a first encoding vector.

[0053] In the embodiments of the present application, the first encoding module can adopt a Transform module or a recurrent neural networks (RNN) module, and no specific limitation is made thereto.

[0054] In some optional embodiments, step S103 can include but is not limited to the following steps:

[0055] A plurality of voice segmentation points are analyzed from the voice information, and the voice information is segmented according to the plurality of voice segmentation points to obtain a voice feature sequence, which includes a plurality of voice segments. The voice feature sequence is input into the first encoding module for encoding processing to obtain the first encoding vector. As can be seen, the voice information is first segmented into a plurality of voice segments, and then the plurality of voice segments are encoded into the first encoding vector, which can improve the accuracy of subsequent alignment of the first encoding vector with a specific text sequence.

[0056] Among them, the voice information is an audio segment, and the plurality of voice segmentation points can be analyzed from the voice information in any one of the following ways:

[0057] In an implementation, a time axis corresponding to the speech information can be acquired, and a plurality of time points can be determined from the time axis as different speech segmentation points according to a preset time interval, for example, one speech segmentation point can be taken from the time axis every 10 milliseconds or 15 milliseconds.

[0058] In another implementation, a detection window [wStart, wEnd] can be initialized, wStart is a starting time point of the window, and wEnd is a terminal time point of the window. Whether a segmentation point exists in the detection window can be detected by running a bayesian information criterion (BIC) algorithm. If the segmentation point exists in the detection window, a speech segmentation point is recorded, and the detection window is moved in the speech information, so that the step of detecting the segmentation point is continued to be executed. If the segmentation point does not exist in the detection window, the position of the detection window is not moved, wEnd is adjusted to obtain a new detection window, that is, [wStart, wEnd+wGrow], wGrow is a window adjustment amount, and then the step of detecting the segmentation point is continued to be executed. When wEnd exceeds an ending time point of the entire speech information, the process of detecting the segmentation point is ended.

[0059] Further, after the speech information is subjected to speech segmentation to obtain the speech feature sequence, voice activity detection (VAD) can be performed on each speech segment. If it is detected that no speech endpoint exists in the speech segment, the speech segment is removed, and finally a new speech feature sequence is obtained, so that a long silent period is recognized and eliminated, and the problem of illegal segmentation points is solved. The speech endpoint is used to represent a starting position or an ending position of the audio signal.

[0060] Step S104: The first encoding vector is input into the first decoding module for decoding processing to obtain a decoding text sequence.

[0061] In the embodiments of the present application, the first decoding module can adopt a Transform module or an RNN module, etc., and the first decoding module is trained and used in cooperation with the first encoding module, that is, the encoding algorithm of the first encoding module and the decoding algorithm of the first decoding module have a corresponding relationship, which is not limited.

[0062] Step S105: The first encoding vector is input into the first alignment module for alignment processing to obtain a first text sequence.

[0063] In the embodiment of the present application, the first text sequence is aligned with the decoded text sequence, and the first text sequence includes at least two first text labels. Specifically, the first alignment module can adopt a connectionist temporal classification (CTC) module, an attention-based encoder-decoder (AED) module, or an RNN Transducer (RNN-T), without limitation. Specifically, the CTC module is used to map the encoding vector to the text sequence. Since the length of the text label in the text sequence is less than the length of the speech frame corresponding to the encoding vector, a specific label is inserted between the repeated text labels to construct a CTC path with the same length as the speech frame, so that the sequence data can be directly learned without pre-labeling the mapping relationship between the input sequence and the output sequence in the training data, breaking the data dependency constraint and reducing the redundant work of label pre-planning, thereby achieving better application effect in sequence learning tasks.

[0064] Step S106: input the first text sequence into the first masking module, mask at least one first text label through the first masking module, and obtain a target text sequence.

[0065] Optionally, step S106 can specifically be: determining a to-be-masked label from the first text sequence through the first masking module, and replacing the to-be-masked label with a label symbol, so as to facilitate more accurate sequence alignment and identification and differentiation of repeated labels. The label symbol can be a blank symbol or other specified symbol, without specific limitation.

[0066] Step S107: input the decoded text sequence and the target text sequence into the first correction module for correction processing, and obtain a speech recognition text.

[0067] It can be understood that the target text sequence is the sequence construction and alignment result of the first alignment module on the first encoding vector, and the decoded text sequence is the decoding result of the first decoding module on the first encoding vector, so the decoded text sequence can also satisfy the alignment relationship with the target text sequence. In step S107, specifically, by constructing the first correction module, the text label in the target text sequence that is subjected to the masking processing by the decoded text sequence can be replaced or completed with a label, and the final speech recognition text is obtained.

[0068] The speech recognition method provided in the embodiments of the present application can shield at least one first text label in the first text sequence through the first shielding module to obtain a target text sequence, can dynamically filter abnormal labels, and reduce reasoning time. Based on this, the first correction module corrects the decoding text sequence and the target text sequence into speech recognition text, can focus on matching the text content, relaxes the restriction on the text sequence, improves the efficiency of automatic speech recognition, and in addition, can not only alleviate the decoding error problem of the second decoding module, but also replace and correct the text label in combination with the data alignment processing result, which is conducive to improving the accuracy of speech recognition.

[0069] Please refer to Figure 2 , Figure 2 is Figure 1 a specific flowchart of step S106 in the method. In some embodiments of the present application, step S106 includes but is not limited to steps S201 to S203, which will be described in detail below in combination with Figure 2 .

[0070] Step S201: Obtain the confidence corresponding to each first text label.

[0071] In the embodiments of the present application, the confidence corresponding to each first text label is used to represent the probability that the speech information corresponding to the speech recognition text contains the first text label. Specifically, by inputting the first encoding vector into the first alignment module, the first alignment module can also output the confidence corresponding to each first text label.

[0072] Step S202: Input the first text sequence into the first shielding module, and obtain the first text label with a confidence less than a preset value from the first text sequence as a to-be-shielded label through the first shielding module.

[0073] In the embodiments of the present application, the preset value can represent a confidence threshold value artificially specified and adjusted, which is not specifically limited. As can be seen, through steps S201 and S202, the first text label with low confidence is shielded from the first text sequence, the dynamic replacement of the low-confidence label is completed, and the reasoning time is reduced.

[0074] Step S203: Shield the to-be-shielded label in the first text sequence through the first shielding module to obtain a target text sequence.

[0075] Please refer to Figure 3 , Figure 3 is a flowchart of the training steps of a speech recognition model in the embodiments of the present application. In some embodiments of the present application, the training steps of the speech recognition model include but are not limited to steps S301 to S308. The details will be described below in combination with Figure 3 .

[0076] Step S301: constructing an initial generation model, the initial generation model comprising a second encoding module, a second alignment module, a second shielding module, a second correction module, and a second decoding module.

[0077] It can be understood that the speech recognition model in the embodiments of the present application is obtained by training the initial generation model, and therefore the initial generation model can refer to the above description of each module in the speech recognition model, which will not be repeated here. In actual application, after step S301, the speech recognition model obtained by the last training can be taken as a new initial generation model, and further model training processing is performed through steps S302 to S308, so as to continuously optimize the accuracy of the speech recognition model.

[0078] Step S302: obtaining a speech sample and a reference recognition text corresponding to the speech sample.

[0079] In the embodiments of the present application, the reference recognition text is the text content obtained by text conversion of the speech sample.

[0080] Step S303: inputting the speech sample into the second encoding module for encoding processing to obtain a second encoding vector.

[0081] Step S304: inputting the second encoding vector into the second alignment module for processing to obtain a second text sequence, the second text sequence comprising at least two second text labels.

[0082] Step S305: inputting the second text sequence into the second shielding module, shielding at least one second text label through the second shielding module to obtain a third text sequence.

[0083] In an optional embodiment, step S305 can be specifically: inputting the second text sequence into the second shielding module, randomly obtaining a second text label from the second text sequence as a target text label through the second shielding module, and shielding the target text label to obtain the third text sequence, so as to traverse more diverse and comprehensive weight adjustment conditions in the model training process. Wherein, the target text label represents the second text label to be shielded.

[0084] In another optional embodiment, after step S304, the confidence corresponding to each second text label can also be obtained through the second alignment module, and then step S305 can be: obtaining m second text labels with the lowest confidence from the second text sequence as target text labels through the second shielding module, wherein m is a positive integer artificially set and adjusted, and is not specifically limited.

[0085] Step S306: inputting the second encoding vector into the second decoding module for decoding processing to obtain a fourth text sequence.

[0086] Step S307: inputting the third text sequence and the fourth text sequence into the second correction module for correction processing to obtain a target recognition text.

[0087] Step S308: adjusting the initial generation model according to the reference recognition text and the target recognition text to obtain a speech recognition model.

[0088] In step S308, specifically, a first similarity between the reference recognition text and the target recognition text can be calculated through a loss function of the initial generation model, the loss function of the initial generation model is optimized according to the first similarity, the model loss of the loss function is back propagated, the model parameters are continuously adjusted until the first similarity is greater than or equal to a first similarity threshold, the optimization of the initial generation model is stopped, and a speech recognition model meeting the requirements is obtained. The model parameters include, but are not limited to, the module parameters corresponding to at least one of the second encoding module, the second alignment module, the second masking module, the second correction module and the second decoding module. It should be noted that the loss function of the initial generation model can be any one of a 0-1 loss function, an absolute value loss function, a logarithmic loss function, an exponential loss function and a Hinge loss function, and can be selected and adjusted according to actual needs, which will not be described here.

[0089] In step S308 of some embodiments, step S308 can further include, but is not limited to, the following steps:

[0090] According to the preset loss function, the reference recognition text and the target recognition text, the loss function value of the initial generation model is calculated. The preset loss function can include cross entropy or aligned cross entropy (AXE), without limitation. In some optional implementation manners, the loss function value of the initial generation model can include a first loss value corresponding to the second alignment module and a second loss value corresponding to the second decoding module, and the loss function value of the initial generation model is the sum of the first loss value and the second loss value. The first loss value and the second loss value satisfy:

[0091] wherein Y bnk = θ bnk (Y), Y rec = θ rec (Y bnk ), L ctc is the first loss value, L axe is the second loss value, Y is the second text sequence, X is the speech sample, Y rec is the target recognition text, Y bnk is the third text sequence, θ bnk represents a masking function corresponding to the second masking module, and θ recThe second correction module corresponds to a correction function.

[0092] Based on this, in one aspect, if the loss function value of the initial generation model satisfies the training end condition, the speech recognition model is determined according to the initial generation model. The training end condition can be specified and adjusted by a person, for example, the loss function value of the initial generation model is less than a specified value, or in the case of using alignment cross entropy for the loss function of the initial generation model, the loss function value of the initial generation model reaches a minimum value, which is not limited. It can be understood that the alignment cross entropy of the initial generation model reaches a minimum value, that is, the monotonic alignment that minimizes the cross entropy loss is found by dynamic programming, which means that the initial generation model can focus on content matching of the text sequence rather than sequential matching, improving the recognition accuracy of the model.

[0093] On the other hand, if the loss function value of the initial generation model does not satisfy the training end condition, step S302 is continued to be executed, thereby entering the next round of model training process.

[0094] It can be seen that using alignment cross entropy as the loss function will neither limit the model convergence condition too strictly, nor will it appear text generation results that match the text content but do not completely match the text position, resulting in a large difference between the text generation result and the actual reference text sequence.

[0095] In some optional implementations, in order to improve the accuracy of the second alignment module in sequence alignment, the second alignment module can also be pre-trained. The pre-training step of the second alignment module can include: after step S304, calculating the second similarity between the second text sequence and the reference recognition text through the loss function of the second alignment module, optimizing the loss function of the second alignment module according to the second similarity, performing back propagation on the model loss of the loss function, and constantly adjusting the module parameters until the second similarity is greater than or equal to a second similarity threshold, stopping the optimization of the second alignment module, and obtaining the required second alignment module.

[0096] Specifically, the loss function L of the second alignment module is ctc which can be:

[0097] L ctc = -ln P ctc (Y|X), wherein, q

[0098] indicates T mapping paths of mapping the second text sequence to the reference recognition text, q t indicates the t-th mapping path, t is a positive integer and t∈[1, T], and T is a positive integer greater than 1.

[0099] In an optional implementation, if the loss function value of the initially generated model satisfies the training end condition, the loss function value is determined as a preset value. It can be understood that the preset value is used by the first shielding module to determine the to-be-shielded label from the first text sequence. As can be seen, taking the loss function value when the model converges as the confidence threshold for screening labels in the actual inference stage can dynamically program to find the minimum loss function value, alleviate the problem of input mismatch between the training stage and the inference stage of the decoding module, and solve the problem of the influence of retaining high-confidence error labels on the filling of other blank labels.

[0100] In other optional implementations, the loss function value corresponding to the second alignment module can also be taken as the preset value when the second alignment module satisfies the convergence constraint condition. Alternatively, the loss function value corresponding to the second alignment module can be taken as the preset value when the second decoding module satisfies the convergence constraint condition. Alternatively, the loss function value corresponding to the second alignment module can be taken as the preset value when the second alignment module and the second decoding module each satisfy their respective convergence constraint conditions. This is not limited.

[0101] In step S107 of some embodiments, the decoded text sequence includes at least one decoded text label, and step S107 can include but is not limited to the following steps:

[0102] The decoded text sequence and the target text sequence are input into the first correction module. Through the first correction module, the sequence position corresponding to each to-be-shielded label in the decoded text sequence is identified, and the decoded text label corresponding to the sequence position is obtained from the decoded text sequence as a matching label. Through the first correction module, the matching label is replaced with the to-be-shielded label to obtain the speech recognition text.

[0103] For example, assuming that the target text sequence is [a1, blank2, a3, blank4], blank2 represents the to-be-shielded label of the second sequence position, blank4 represents the to-be-shielded label of the fourth sequence position, and the decoded text sequence is [a1, a2, a3, a4], the decoded text label a2 can be taken as the matching label corresponding to the second sequence position, and the decoded text label a4 can be taken as the matching label corresponding to the fourth sequence position, so that a2 is used to replace blank2 and a4 is used to replace blank4, and finally the speech recognition text a1+a2+a3+a4 is obtained. As can be seen, based on the alignment relationship between the decoded text sequence and the target text sequence, the shielded text label can be quickly corrected.

[0104] Please refer to Figure 4 , Figure 4is a module block diagram of a speech recognition device provided by an embodiment of the present application. In some embodiments, the speech recognition device comprises an acquisition module 401, an encoding processing module 402, a decoding processing module 403, an alignment processing module 404, a shielding processing module 405, and a correction processing module 406.

[0105] The acquisition module 401 is configured to acquire speech information, and acquire a speech recognition model, wherein the speech recognition model comprises a first encoding module, a first alignment module, a first shielding module, a first correction module, and a first decoding module.

[0106] The encoding processing module 402 is configured to input the speech information into the first encoding module for encoding processing, to obtain a first encoding vector.

[0107] The decoding processing module 403 is configured to input the first encoding vector into the first decoding module for decoding processing, to obtain a decoded text sequence.

[0108] The alignment processing module 404 is configured to input the first encoding vector into the first alignment module for alignment processing, to obtain a first text sequence, wherein the first text sequence is aligned with the decoded text sequence, and the first text sequence comprises at least two first text labels.

[0109] The shielding processing module 405 is configured to input the first text sequence into the first shielding module, to shield at least one first text label through the first shielding module, to obtain a target text sequence.

[0110] The correction processing module 406 is configured to input the decoded text sequence and the target text sequence into the first correction module for correction processing, to obtain a speech recognition text.

[0111] It should be noted that the speech recognition device of the embodiment of the present application corresponds to the speech recognition method described above, and will not be described here.

[0112] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech recognition method described above when executing the computer program. The electronic device can be any intelligent terminal, including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0113] The following will be described in combination with Figure 5 The electronic device of the embodiment of the present application will be described in detail.

[0114] As Figure 5 , Figure 5 The hardware structure of the electronic device of another embodiment is shown, which comprises:

[0115] The processor 501 can be implemented by a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0116] The memory 502 can be implemented by a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), and the like. The memory 502 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 502 and are called and executed by the processor 501 to implement the voice recognition method of the embodiments of the present application.

[0117] The input / output interface 503 is configured to implement information input and output.

[0118] The communication interface 504 is configured to implement the communication interaction between the device and other devices. The communication can be implemented by a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0119] The bus 505 is configured to transmit information between various components (for example, the processor 501, the memory 502, the input / output interface 503, and the communication interface 504) of the device.

[0120] The processor 501, the memory 502, the input / output interface 503, and the communication interface 504 are connected to each other by the bus 505 to realize the communication connection between the device.

[0121] The embodiments of the present application further provide a storage medium, which is a computer readable storage medium, and stores computer executable instructions. The computer executable instructions are configured to make a computer execute the voice recognition method.

[0122] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0123] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0124] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps or different steps.

[0125] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0126] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0127] The terms "first", "second", "third", "fourth" and the like used in the specification of the present application and the above-described drawings, if any, are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0128] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0129] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0131] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0132] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0133] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A speech recognition method, characterized in that, The method includes: Obtain voice information; A speech recognition model is obtained, the speech recognition model including a first encoding module, a first alignment module, a first masking module, a first correction module and a first decoding module; The voice information is input into the first encoding module for encoding processing to obtain the first encoding vector; The first encoded vector is input into the first decoding module for decoding processing to obtain a decoded text sequence; The first encoded vector is input into the first alignment module for alignment processing to obtain a first text sequence. The first text sequence is aligned with the decoded text sequence, and the first text sequence includes at least two first text tags, each of which has a confidence level. The first text sequence is input into the first blocking module, and at least one tag to be blocked in the first text tag is blocked by the first blocking module to obtain the target text sequence; wherein the confidence level of the tag to be blocked is less than a preset value; The decoded text sequence and the target text sequence are input into the first correction module for correction processing to obtain speech recognition text; The decoded text sequence includes at least one decoded text tag. The step of inputting the decoded text sequence and the target text sequence into the first correction module for correction processing to obtain speech recognition text includes: The decoded text sequence and the target text sequence are input into the first correction module; The first correction module identifies the sequence position corresponding to each tag to be blocked in the decoded text sequence, and obtains the decoded text tag corresponding to the sequence position from the decoded text sequence as a matching tag. The first correction module uses the matching tags to replace the tags to be blocked to obtain speech recognition text.

2. The method according to claim 1, characterized in that, The step of inputting the first text sequence into the first blocking module, and using the first blocking module to block at least one tag to be blocked in the first text tag to obtain the target text sequence includes: Obtain the confidence score corresponding to each of the first text tags; The first text sequence is input into the first blocking module, and the first blocking module obtains a first text tag with a confidence level less than a preset value from the first text sequence as the tag to be blocked. The first blocking module performs blocking processing on the tags to be blocked in the first text sequence to obtain the target text sequence.

3. The method according to claim 1, characterized in that, The training steps of the speech recognition model include: An initial generation model is constructed, which includes a second encoding module, a second alignment module, a second masking module, a second correction module, and a second decoding module; Acquire speech samples and the corresponding reference recognition text; The speech sample is input into the second encoding module for encoding processing to obtain the second encoding vector; The second encoding vector is input into the second alignment module for alignment processing to obtain a second text sequence, which includes at least two second text labels; The second text sequence is input into the second blocking module, and at least one of the second text tags is blocked by the second blocking module to obtain the third text sequence; The second encoded vector is input into the second decoding module for decoding processing to obtain the fourth text sequence; The third and fourth text sequences are input into the second correction module for correction processing to obtain the target recognition text. The initial generation model is adjusted based on the reference recognition text and the target recognition text to obtain a speech recognition model.

4. The method according to claim 3, characterized in that, The step of adjusting the initial generation model based on the reference recognition text and the target recognition text to obtain a speech recognition model includes: The loss function value of the initial generation model is calculated based on the preset loss function, the reference recognition text, and the target recognition text; If the loss function value satisfies the preset training termination condition, then the speech recognition model is determined based on the initial generation model; If the loss function value does not meet the training termination condition, then continue to execute the steps of obtaining speech samples and the corresponding reference recognition text. The method further includes: If the loss function value satisfies the training termination condition, the loss function value is determined as a preset value, which is used by the first blocking module to determine the tag to be blocked from the first text sequence.

5. The method according to claim 3, characterized in that, The step of blocking at least one of the second text tags through the second blocking module to obtain the third text sequence includes: The second masking module randomly selects a second text tag from the second text sequence as a target text tag and masks the target text tag to obtain a third text sequence.

6. The method according to any one of claims 1 to 5, characterized in that, The step of inputting the voice information into the first encoding module for encoding processing to obtain the first encoding vector includes: Multiple speech segmentation points were analyzed from the speech information; Based on the multiple speech segmentation points, the speech information is segmented to obtain a speech feature sequence, which includes multiple speech segments. The speech feature sequence is input into the first encoding module for encoding processing to obtain the first encoding vector.

7. A voice recognition device, characterized in that, The device includes: An acquisition module is used to acquire voice information; and to acquire a voice recognition model, wherein the voice recognition model includes a first encoding module, a first alignment module, a first masking module, a first correction module, and a first decoding module; An encoding processing module is used to input the voice information into the first encoding module for encoding processing to obtain a first encoding vector; A decoding processing module is used to input the first encoded vector into the first decoding module for decoding processing to obtain a decoded text sequence; An alignment processing module is used to input the first encoded vector into the first alignment module for alignment processing to obtain a first text sequence. The first text sequence is aligned with the decoded text sequence, and the first text sequence includes at least two first text tags, each of which has a confidence level. A masking processing module is used to input the first text sequence into the first masking module, and to mask at least one tag to be masked in the first text tag through the first masking module to obtain a target text sequence; wherein the confidence level of the tag to be masked is less than a preset value; The correction processing module is used to input the decoded text sequence and the target text sequence into the first correction module for correction processing to obtain speech recognition text; The decoded text sequence includes at least one decoded text tag. The correction processing module is used to input the decoded text sequence and the target text sequence into the first correction module for correction processing to obtain speech recognition text, including: The decoded text sequence and the target text sequence are input into the first correction module; The first correction module identifies the sequence position corresponding to each tag to be blocked in the decoded text sequence, and obtains the decoded text tag corresponding to the sequence position from the decoded text sequence as a matching tag. The first correction module uses the matching tags to replace the tags to be blocked to obtain speech recognition text.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method according to any one of claims 1 to 6.

Images