Speech recognition method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of voice recognition, in particular to a voice recognition method and device, electronic equipment and a storage medium, and aims to improve the voice recognition efficiency. The method comprises the following steps: acquiring to-be-recognized voice, encoding the to-be-recognized voice through a target encoder, and acquiring candidate encoding features corresponding to the to-be-recognized voice; obtaining a blank frame probability corresponding to each voice frame in the to-be-recognized voice according to the frame distribution information of the to-be-recognized voice, and screening out at least one target frame from the to-be-recognized voice based on the obtained blank frame probability; the frame distribution information represents probability distribution of each voice frame in a dictionary space; screening out a target candidate coding feature corresponding to at least one target frame from the candidate coding features; and decoding based on the target candidate coding features to obtain a text recognition result corresponding to the to-be-recognized voice. According to the invention, the frame rate compression is carried out through the frame distribution information, only part of important frames are decoded, and the speech recognition efficiency can be effectively improved.

Description

Background Art

[0002] Speech recognition technology refers to the technology that converts speech signals into corresponding text by recognizing them. Real-time speech recognition refers to combining speech recognition technology to give text recognition results synchronously while the subject is speaking.

[0003] The speech recognition system in the related technology often uses an end-to-end model to convert speech signals into text. However, under high concurrency conditions, multiple requests are simultaneously initiated to the speech recognition system in a very short unit time, resulting in low decoding efficiency of the model.

[0004] Therefore, how to improve the efficiency of speech recognition is an urgent problem to be solved. Summary of the invention

[0005] Embodiments of the present application provide a speech recognition method, device, electronic device and storage medium to improve speech recognition efficiency.

[0006] A speech recognition method provided in an embodiment of the present application includes:

[0007] Acquire a speech to be recognized, and encode the speech to be recognized by a target encoder to obtain candidate coding features corresponding to the speech to be recognized;

[0008] According to the frame distribution information of the speech to be recognized, the blank frame probability corresponding to each speech frame in the speech to be recognized is obtained, and based on the obtained blank frame probability, at least one target frame is screened out from the speech to be recognized; the frame distribution information represents the probability distribution of each speech frame in the speech to be recognized in the dictionary space; the blank frame probability represents the probability that a speech frame is a blank frame;

[0009] Filtering out a target candidate coding feature corresponding to the at least one target frame from the candidate coding features;

[0010] Decoding is performed based on the target candidate coding features to obtain a text recognition result corresponding to the speech to be recognized.

[0011] An embodiment of the present application provides a speech recognition device, comprising:

[0012] An encoding unit, used to obtain a speech to be recognized, and encode the speech to be recognized through a target encoder to obtain candidate encoding features corresponding to the speech to be recognized;

[0013] A screening unit is used to obtain the blank frame probability corresponding to each speech frame in the speech to be recognized according to the frame distribution information of the speech to be recognized, and based on the obtained blank frame probability, screen out at least one target frame from the speech to be recognized; the frame distribution information represents the probability distribution of each speech frame in the speech to be recognized in the dictionary space; the blank frame probability represents the probability that the speech frame is a blank frame; and screen out a target candidate coding feature corresponding to the at least one target frame from the candidate coding features;

[0014] A decoding unit is used to decode based on the target candidate coding features to obtain a text recognition result corresponding to the speech to be recognized.

[0015] Optionally, the screening unit is specifically used for:

[0016] Determining blank frames and non-blank frames in the speech to be recognized according to the obtained blank frame probability;

[0017] Non-blank frames, blank frames between candidate frames with repeated content, and blank frames with a blank frame probability less than a preset threshold are taken as target frames; wherein the candidate frames with repeated content are: non-blank frames with adjacent positions and repeated contents among the non-blank frames contained in the speech to be recognized.

[0018] Optionally, the screening unit is specifically used for:

[0019] The blank frames between the candidate frames with repeated contents are used as target frames;

[0020] Except for the blank frames between the candidate frames with repeated contents, blank frames whose corresponding blank frame probabilities are less than a preset threshold are taken as target frames.

[0021] Optionally, if the candidate frames with repeated content contain at least two consecutive blank frames;

[0022] The screening unit is specifically used for:

[0023] Part or all of the at least two consecutive blank frames are used as the target frame.

[0024] Optionally, in a real-time speech recognition scenario, the target encoder is a streaming encoder in a trained first real-time speech model;

[0025] Then the encoding unit is specifically used for:

[0026] When the acquired length of the speech to be recognized reaches a first preset duration, feature extraction is performed on the speech to be recognized based on the first real-time speech recognition model to obtain a speech feature sequence;

[0027] Encoding the speech feature sequence by the streaming encoder to obtain the candidate coding features and cache them;

[0028] And, the decoding unit is also used for:

[0029] Based on the first real-time speech recognition model, the candidate coding features are decoded, and the real-time recognition text obtained by decoding is used as the text recognition result corresponding to the speech to be recognized.

[0030] Optionally, the encoding unit is further used for:

[0031] Before the screening unit obtains the blank frame probability corresponding to each speech frame in the speech to be recognized according to the frame distribution information of the speech to be recognized, the following operations are performed:

[0032] When the acquired length of the speech to be recognized reaches a second preset duration, the multiple candidate coding features encoded within the second preset duration are subjected to feature extraction through the classification linear layer and activation function in the first real-time speech recognition model to obtain the frame distribution information; the second preset duration is greater than the first preset duration.

[0033] Optionally, the first real-time speech recognition model further includes a cascade encoder;

[0034] The screening unit is specifically used for:

[0035] Filtering a target candidate coding feature corresponding to the at least one target frame from the multiple candidate coding features;

[0036] Then the decoding unit is specifically used for:

[0037] Based on the cascade encoder, encoding the target candidate coding feature to obtain a target coding feature;

[0038] Based on the first real-time speech recognition model, the target coding feature is decoded, and the decoded target recognition text is used as the text recognition result corresponding to the speech to be recognized.

[0039] Optionally, in a real-time speech recognition scenario, the target encoder is a streaming encoder in a trained second real-time speech model;

[0040] Then the encoding unit is specifically used for:

[0041] When the acquired length of the speech to be recognized reaches a first preset duration, feature extraction is performed on the speech to be recognized based on a second real-time speech recognition model to obtain a speech feature sequence;

[0042] Encoding the speech feature sequence by the streaming encoder to obtain frame distribution information output by an intermediate layer of the streaming encoder and candidate coding features outputted finally, and caching the candidate coding features;

[0043] And, the decoding unit is also used for:

[0044] Based on the second real-time speech recognition model, the candidate coding features are decoded, and the real-time recognition text obtained by decoding is used as the text recognition result corresponding to the speech to be recognized.

[0045] Optionally, the second real-time speech recognition model further includes a cascade encoder;

[0046] The screening unit is specifically used for:

[0047] When the acquired length of the speech to be recognized reaches a second preset duration, a target candidate coding feature corresponding to the at least one target frame is screened out from a plurality of candidate coding features encoded within the second preset duration; the second preset duration is greater than the first preset duration;

[0048] Then the decoding unit is specifically used for:

[0049] Based on the cascade encoder, encoding the target candidate coding feature to obtain a target coding feature;

[0050] Based on the second real-time speech recognition model, the target coding feature is decoded, and the decoded target recognition text is used as the text recognition result corresponding to the speech to be recognized.

[0051] Optionally, in a non-real-time speech recognition scenario, the target encoder is a non-real-time encoder in a trained non-real-time speech model;

[0052] Then the encoding unit is specifically used for:

[0053] Extracting features of the speech to be recognized based on a non-real-time speech recognition model to obtain a speech feature sequence;

[0054] Encoding the speech to be recognized by the non-real-time encoder to obtain frame distribution information and candidate coding features output by the intermediate layer of the non-real-time encoder;

[0055] And, the decoding unit is specifically used for:

[0056] Obtaining a target coding feature obtained by encoding the target candidate coding feature by the non-real-time encoder;

[0057] The target coding feature is decoded through the non-real-time speech recognition model to obtain a text recognition result corresponding to the speech to be recognized.

[0058] Optionally, when training the first real-time speech recognition model, the corresponding loss function includes:

[0059] Connectionist temporal classification (CTC) loss and cross entropy loss constructed based on the difference between the text recognition result output by the first real-time speech recognition model and the annotated text recognition result;

[0060] The CTC loss is constructed based on the difference between the frame distribution information obtained by feature extraction through the classification linear layer and the activation function in the first real-time speech recognition model and the labeled frame distribution information.

[0061] Optionally, when training the second real-time speech recognition model, the corresponding loss function includes:

[0062] Based on the difference between the text recognition result output by the second real-time speech recognition model and the annotated text recognition result, a CTC loss and a cross entropy loss are constructed;

[0063] The CTC loss is constructed based on the difference between the frame distribution information output by the intermediate layer of the streaming encoder and the labeled frame distribution information.

[0064] Optionally, when training the non-real-time speech recognition model, the corresponding loss function includes:

[0065] A cross entropy loss constructed based on the difference between the text recognition result obtained by the non-real-time speech recognition model and the annotated text recognition result;

[0066] The CTC loss is constructed based on the difference between the frame distribution information output by the intermediate layer of the non-real-time encoder and the labeled frame distribution information.

[0067] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned speech recognition methods.

[0068] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of any one of the above-mentioned speech recognition methods.

[0069] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned speech recognition methods.

[0070] The beneficial effects of this application are as follows:

[0071] The embodiments of the present application provide a speech recognition method, device, electronic device and storage medium. Since the speech recognition method in the present application does not directly decode after encoding the speech to be recognized, but determines the probability of blank frames corresponding to each speech frame based on the frame distribution information of the speech to be recognized, and based on this probability, deletes some blank frames with lower importance, so as to filter out at least one target frame from the speech to be recognized, thereby achieving compression of the output frame; when decoding, decoding is only based on the target candidate coding features corresponding to the filtered target frame. In this way, the decoding efficiency can be improved under the premise of very little performance loss, so as to effectively improve the speech recognition efficiency. Moreover, in the case of high concurrency, each request can improve the speech recognition efficiency of the request based on the above method.

[0072] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0074] Figure 1 An optional schematic diagram of an application scenario in an embodiment of the present application;

[0075] Figure 2 A flow chart of a speech recognition method in an embodiment of the present application;

[0076] Figure 3A This is a schematic diagram of the first method of screening target frames in an embodiment of the present application;

[0077] Figure 3B This is a schematic diagram of the second method of screening target frames in an embodiment of the present application;

[0078] Figure 3CThis is a schematic diagram of a third method of screening target frames in an embodiment of the present application;

[0079] Figure 4 This is a schematic diagram of the structure of a first real-time speech recognition model in an embodiment of the present application;

[0080] Figure 5 A flowchart of a speech recognition method based on a first real-time speech recognition model in an embodiment of the present application;

[0081] Figure 6 This is a schematic diagram of the encoding timing of a speech to be recognized in an embodiment of the present application;

[0082] Figure 7 A schematic diagram of the structure of a second real-time speech recognition model in an embodiment of the present application;

[0083] Figure 8 A process of a speech recognition method based on a second real-time speech recognition model in an embodiment of the present application;

[0084] Fig. 9 A schematic diagram of the structure of a non-real-time speech recognition model in an embodiment of the present application;

[0085] Fig.10 A flowchart of a speech recognition method based on a non-real-time speech recognition model in an embodiment of the present application;

[0086] Fig.11 A logic diagram of a speech recognition process in an embodiment of the present application;

[0087] Fig.12 A schematic diagram of the structure of a speech recognition device in an embodiment of the present application;

[0088] Fig.13 A schematic diagram of a hardware structure of an electronic device to which an embodiment of the present application is applied;

[0089] Fig.14 A schematic diagram of the hardware structure of another electronic device to which the embodiments of the present application are applied. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0091] The following is an introduction to some concepts involved in the embodiments of the present application.

[0092] Frame distribution information: the probability distribution of each speech frame in the speech to be recognized in the dictionary space. The dictionary space specifically refers to the space where the pronunciation dictionary (also called vocabulary) is located.

[0093] Pronunciation dictionary / vocabulary: The pronunciation dictionary contains a collection of words that the system can handle, and indicates their pronunciation. The pronunciation dictionary is used to obtain the mapping relationship between the modeling unit of the acoustic model and the modeling unit of the language model, thereby connecting the acoustic model and the language model to form a search state space for the decoder to perform decoding work. In the dictionary generation process, the pinyin-phoneme mapping determines the conversion rules / mapping relationship from pinyin to phoneme. The Chinese word-pinyin mapping is used to list as many Chinese words and their corresponding pinyin as possible. The word-to-phoneme (Grapheme-to-Phoneme, g2p) tool is used to implement the function of inputting Chinese words and outputting corresponding phonemes. Chinese words also need to be collected because the pronunciation dictionary needs to cover as many word-phonemes as possible. Finally, the Chinese word list is used as input, and the word-phoneme mapping obtained through the g2p tool is the pronunciation dictionary.

[0094] Real-time speech recognition: refers to the real-time speech recognition of the input speech, which is a speech recognition task with time constraints. Real-time speech recognition technology can be applied to many occasions, such as when chatting, the voice input is accurate and efficient, freeing your hands, and the spoken content is displayed on the screen in real time, making the chat smooth; in the meeting scene, the voice of each speaker can be recorded in real time to improve the efficiency of meeting recording; the teacher's classroom content is recorded in real time, and the school can record the teaching content and evaluate the teaching quality; the conference speech can display the guest speech subtitles on the screen in real time, display them word by word and correct them intelligently, and so on.

[0095] Non-real-time speech recognition: Speech recognition can be performed on the input speech at any time. It is a speech recognition task without time constraints.

[0096] Encoder: A device that compiles and converts signals (such as bit streams) or data into a signal form that can be used for communication, transmission and storage. The encoders in the embodiments of the present application relate to streaming encoders, cascade encoders and non-real-time encoders. Among them, a streaming encoder refers to an encoder that can process streaming input, and a cascade encoder refers to an encoder formed by connecting modules of the same (or similar) structure. A non-real-time encoder is an encoder that can process non-real-time input, such as directly inputting a piece of speech into the model for encoding through the encoder.

[0097] Chunk: A streaming scheme for the attention mechanism that implements information truncation by chunking the sequence. Specifically, information truncation is implemented by chunking the feature sequence, and is often used in streaming schemes for the attention mechanism.

[0098] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0099] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0100] Among them, machine learning (ML) is the core of artificial intelligence, a branch of artificial intelligence, and a large field. It allows computers to have the same learning ability as humans, simulate and realize human learning behavior and ability, and have the ability to recognize and judge like humans, which can be regarded as bionics. The core of machine learning is data, algorithm (model), and computing power (computer computing power). It is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or realize human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. As in the embodiment of the present application, the relevant speech recognition model can be trained by machine learning technology, such as the first real-time speech recognition model, the second real-time speech recognition model, the non-real-time speech recognition model, etc.

[0101] The present application further relates to speech recognition technology in speech processing technology, also known as Automatic Speech Recognition (ASR), which is a technology for converting human speech into text, and its goal is to convert the vocabulary content in human speech into computer-readable input, such as keystrokes, binary codes or character sequences. It is different from speaker recognition and speaker verification, which attempt to identify or verify the speaker who made the speech rather than the vocabulary content contained in it.

[0102] Speech recognition is a multidisciplinary field, which is closely related to acoustics, phonetics, linguistics, digital signal processing theory, information theory, computer science and many other disciplines. Due to the diversity and complexity of speech signals, current speech recognition systems can only achieve satisfactory performance under certain restrictions, or can only be applied to certain specific occasions. The performance of speech recognition systems generally depends on the following four factors: the size of the recognition vocabulary and the complexity of the speech; the quality of the speech signal; single speaker or multiple speakers; hardware.

[0103] Applications of speech recognition technology include voice dialing, voice navigation, indoor equipment control, voice document retrieval, simple dictation data entry, etc. Speech recognition technology combined with other natural language processing technologies such as machine translation and speech synthesis technology can build more complex applications, such as speech-to-speech translation.

[0104] In an embodiment of the present application, a speech recognition model can be trained based on the machine learning, automatic speech recognition and other technologies listed above, so as to perform real-time or non-real-time speech recognition based on the trained speech recognition model, obtain the text of the speech to be recognized and display it.

[0105] The following is a brief introduction to the design concept of the embodiment of the present application:

[0106] With the rapid development of artificial intelligence technology, speech recognition systems have entered multiple fields such as industry, communications, home services, and consumer electronics. At present, speech recognition systems are widely used in various scenarios such as AI subtitles, audio and video content security review, voice wake-up, meeting records, and voice input. How to improve the efficiency of speech recognition has become the focus of research in the industry.

[0107] As the entrance to human-computer interaction, speech recognition is an important research direction in the field of artificial intelligence. End-to-end speech recognition discards the pronunciation dictionary, language model, and decoding network that the hybrid speech recognition model relies on, and realizes the direct conversion of audio feature sequence to text sequence. The model uses the entire speech as input, and the encoder encodes the input speech into a high-level feature representation; the decoder starts from the start symbol and gradually predicts the corresponding text sequence based on the editor output until the end mark is predicted.

[0108] In the related technology, the cascade speech recognition system based on CTC and attention-based encoder-decoder (AED) requires two forward passes and two decodings. Although the system has made some efficiency optimizations, the decoding efficiency is still low under high concurrency.

[0109] In view of this, the embodiments of the present application propose a speech recognition method, device, electronic device and storage medium. Due to the speech recognition method in the present application, after encoding the speech to be recognized, it is not directly decoded, but the probability of blank frames corresponding to each speech frame is determined according to the frame distribution information of the speech to be recognized, and based on this probability, some blank frames with lower importance are deleted to filter out at least one target frame from the speech to be recognized, thereby achieving compression of the output frame; when decoding, it is only based on the target candidate coding features corresponding to the filtered target frame. Under this method, the decoding efficiency can be improved under the premise of very little performance loss, so as to effectively improve the speech recognition efficiency. Moreover, in the case of high concurrency, each request can improve the speech recognition efficiency of the request based on the above method.

[0110] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0111] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120.

[0112] In the embodiment of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; a voice recognition-related client may be installed on the terminal device, which may be software (such as a browser, chat software, office software, voice recognition software, etc.), or a web page, a small program, etc. The server 120 is a background server corresponding to the software or web page, small program, etc., or a server specifically used for voice recognition, which is not specifically limited in the present application. The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0113] It should be noted that the speech recognition method in each embodiment of the present application can be executed by an electronic device, which can be a terminal device 110 or a server 120, that is, the method can be executed by the terminal device 110 or the server 120 alone, or can be executed by the terminal device 110 and the server 120 together. For example, when the method is executed by the terminal device 110 and the server 120 together, it is assumed that the terminal device 110 is installed with speech recognition software, and the object inputs a speech through the software (it can also be a local voice, etc., which is not specifically limited in this article), and the terminal device 110 sends the voice input by the object as the speech to be recognized to the server 120, and the server 120 encodes the speech to be recognized through the target encoder to obtain the candidate coding features corresponding to the speech to be recognized; according to the frame distribution information of the speech to be recognized, the blank frame probability corresponding to each speech frame in the speech to be recognized is obtained, and based on the obtained blank frame probability, at least one target frame is selected from the speech to be recognized; then, from the candidate coding features, a target candidate coding feature corresponding to at least one target frame is selected; decoding is performed based on the target candidate coding feature to obtain a text recognition result corresponding to the speech to be recognized. During the speech recognition process, the server 120 will also send the obtained text recognition results to the terminal device 110 in a timely manner to display them to the object.

[0114] In an optional implementation, the terminal device 110 and the server 120 may communicate with each other via a communication network.

[0115] In an optional implementation, the communication network is a wired network or a wireless network.

[0116] It should be noted that Figure 1The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0117] In an embodiment of the present application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as in the speech recognition method disclosed in the embodiment of the present application, the speech data involved can be stored on the blockchain, for example, the speech to be recognized, various related coding features (such as candidate coding features), frame distribution information, text recognition results, etc.

[0118] In addition, the embodiments of the present application can be applied to various fields, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, signal processing, pattern recognition, probability theory and information theory, vocal mechanism and auditory mechanism.

[0119] The following describes the speech recognition method provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this regard.

[0120] See also Figure 2 As shown, it is a flow chart of an implementation of a speech recognition method provided in an embodiment of the present application. Taking the server as the execution subject as an example, the specific implementation process of the method is as follows S21-S24:

[0121] S21: The server obtains the speech to be recognized, and encodes the speech to be recognized through a target encoder to obtain candidate coding features corresponding to the speech to be recognized.

[0122] The speech recognition method in the embodiment of the present application can be applied to real-time speech recognition scenarios or non-real-time speech recognition scenarios. In different speech recognition scenarios, the target encoder can be different, which is not specifically limited herein.

[0123] S22: The server obtains the blank frame probability corresponding to each speech frame in the speech to be recognized according to the frame distribution information of the speech to be recognized, and selects at least one target frame from the speech to be recognized based on the obtained blank frame probability.

[0124] In speech recognition, the length of the actual output is much smaller than the length of the input. Therefore, blank characters can be introduced to fill the gap, such as blank. Blank is introduced in systems such as CTC or Transducer (an end-to-end speech recognition system framework) to solve alignment problems.

[0125] For speech, the basic unit of choice is the frame, and the span of a frame is adjustable to accommodate different text units. For text, characters (or letters, phonemes) form words, words form sentences, etc. Characters are the first unit that comes to mind.

[0126] The frame distribution information represents the probability distribution of each speech frame in the speech to be recognized in the dictionary space.

[0127] Taking characters (or phonemes, etc.) as constituent units to divide the frames as an example, it is assumed that the dictionary space in the embodiment of the present application includes 26 English letters and 1 blank character (which can be written as _), for a total of 27 characters.

[0128] Specifically, the frame distribution information indicates the probability that each speech frame belongs to the above 27 characters, wherein the blank frame probability indicates the probability that the speech frame is a blank frame.

[0129] In the embodiment of the present application, for a speech frame, if the probability that the speech frame corresponds to a certain character exceeds a first threshold, the speech frame can be considered to be the certain character. The sum of the probabilities that a speech frame is the above 27 characters is a fixed value, such as 1. That is, a speech frame will correspond to 27 probability values, and the sum of these 27 probability values ​​is 1.

[0130] Assuming that the first threshold is set to 0.8, if among the 27 probability values ​​corresponding to a certain speech frame, the probability value corresponding to the character a is 0.9, which exceeds the first threshold of 0.8, it indicates that the speech frame is the character a. For another example, among the 27 probability values ​​corresponding to a certain speech frame, the probability value corresponding to the character blank is 0.85, which exceeds the first threshold of 0.8, it indicates that the speech frame is a blank character.

[0131] Specifically, if the probability distribution of a speech frame indicates that the speech frame is any character other than the above-mentioned blank character, the speech frame is a non-blank frame and can be recorded as a peak frame; conversely, if the probability distribution of the speech frame indicates that the speech frame is a blank character, the speech frame is a blank frame and can be recorded as a blank frame.

[0132] In the embodiment of the present application, some blank frames with lower importance can be deleted according to the probability values ​​of peak and blank, so as to achieve compression of output frames and improve the efficiency of speech recognition.

[0133] When at least one target frame is selected from the speech to be recognized based on the determined blank frame probability, an optional implementation is as follows:

[0134] Firstly, according to the obtained blank frame probability, the blank frames and non-blank frames in the speech to be recognized are determined.

[0135] For example, for a certain speech frame, if the blank frame probability corresponding to the speech frame exceeds a first threshold, it means that the speech frame is a blank frame; otherwise, the speech frame is a non-blank frame.

[0136] By setting the above-mentioned first threshold, the positions of blank frames and non-blank frames in the frame sequence corresponding to the speech to be recognized can be analyzed, and then non-blank frames, blank frames between candidate frames with repeated content, and blank frames with a blank frame probability less than a preset threshold are taken as target frames.

[0137] The candidate frames with repeated contents are: non-blank frames that are adjacent in position and have repeated contents among all non-blank frames included in the speech to be recognized.

[0138] It should be noted that, for a sequence of speech frames, if the speech frames in the sequence only meet some of the above three conditions: "non-blank frames", "blank frames between candidate frames with repeated content", and "blank frames with a probability of blank frames less than a preset threshold", then the speech frames that meet some of the conditions can be used as target frames. For example, if there are no candidate frames with repeated content in the sequence, then only non-blank frames and blank frames with a probability of blank frames less than a preset threshold need to be used as target frames. This article does not make specific restrictions.

[0139] In the embodiment of the present application, the greater the blank frame probability, the higher the authenticity of the speech frame being Blank, and the smaller the blank frame probability, the lower the authenticity of the speech frame being Blank. It is possible that the speech frame is currently recognized as a blank frame due to recognition errors and other reasons. Therefore, in order to ensure the accuracy of the speech recognition results, the present application selects a blank frame with a blank frame probability less than a preset threshold as the target frame.

[0140] Take the above-mentioned non-blank frame, the blank frame between candidate frames with repeated content, and the blank frame with a blank frame probability less than a preset threshold as Condition 1, Condition 2, and Condition 3, respectively, as an example.

[0141] Among them, condition 1 is for filtering non-blank frames, and conditions 2 and 3 are for filtering blank frames. In general, non-blank frames are generally recognized real texts, so they need to be retained and marked as target frames, while blank frames mean that the frame has no prediction value, so some can be filtered to be retained and marked as target frames.

[0142] In the embodiment of the present application, condition 2 and condition 3 can be used in combination to filter blank frames. An optional implementation is as follows:

[0143] First, part or all of the blank frames between candidate frames with repeated content are taken as target frames; then, other blank frames except the blank frames between candidate frames with repeated content, whose corresponding blank frame probabilities are less than a preset threshold, are taken as target frames.

[0144] That is, first filter based on condition 2, and then filter other blank frames based on condition 3.

[0145] For example, a speech frame sequence corresponding to a speech to be recognized is: a blank a blank b blank c blankblank c, and the blank frames are the second frame, the fourth frame, the sixth frame, the eighth frame, and the ninth frame. Among them, the blank frames between the candidate frames with repeated contents are: the eighth frame and the ninth frame, and the other blank frames except the blank frames between the candidate frames with repeated contents are: the second frame, the fourth frame, and the sixth frame.

[0146] First, based on condition 2, both the eighth frame and the ninth frame are taken as target frames, or one frame is selected as the target frame.

[0147] Optionally, when selecting some blank frames from candidate frames with repeated content as target frames, in addition to the random selection method, the blank frame probability can also be combined for screening, such as selecting a blank frame with a smaller blank frame probability as the target frame. Alternatively, it can also be implemented in combination with condition 3, that is, selecting some blank frames whose corresponding blank frame probability is less than a preset threshold as the target frame, etc. Any selection method is applicable to the embodiments of the present application, and is not specifically limited herein.

[0148] Furthermore, based on condition 3, the frames with a blank frame probability less than a preset threshold are selected from the second frame, the fourth frame and the sixth frame as target frames.

[0149] In addition to the above-mentioned implementation methods, screening can also be performed based on condition 2 and condition 3 separately, such as:

[0150] Part or all of the blank frames between candidate frames with repeated contents are taken as target frames; and the blank frames whose corresponding blank frame probability is less than a preset threshold among the blank frames included in the speech to be recognized are taken as target frames.

[0151] In this embodiment, when screening is performed based on condition 2, part or all of the blank frames between candidate frames with repeated contents are used as target frames, that is, when there are multiple blank frames between candidate frames with repeated contents, only one of them may be used as the target frame, or multiple or even all of them may be used as the target frames. In the case where only part of them are used as the target frames, the remaining blank frames that are not used as the target frames may be further screened based on condition 3.

[0152] Still taking the above speech frame sequence: a blank a blank b blank c blank blank c as an example, the blank frames between the candidate frames with repeated contents are: the eighth frame and the ninth frame.

[0153] When filtering based on condition 2, one frame can be selected from the eighth frame and the ninth frame as the target frame. If the eighth frame is selected, then when filtering based on condition 3, the target frame can be further filtered based on the blank frame probability from the remaining blank frames except the eighth frame, i.e., the second frame, the fourth frame, the sixth frame and the ninth frame.

[0154] It should be noted that the above-mentioned candidate frames with repeated content specifically refer to speech frames that are adjacent in position and have repeated content among the non-blank frames contained in the speech to be recognized. That is, in the embodiment of the present application, either there are no speech frames between the "candidate frames with repeated content" or there are only blank frames without other non-blank frames. Only in the second case will the "blank frames between candidate frames with repeated content" be retained.

[0155] In an embodiment of the present application, some blank frames with lower importance are deleted according to the probability values ​​of peak and blank after the target encoder, thereby achieving compression of the output frames, which can effectively improve the efficiency of speech recognition.

[0156] Based on the above ideas, the specific process of target frame screening is illustrated below:

[0157] Example 1: For example, the voice frame sequence is: a blank blank b blank cc blank, then in this sequence, there are a total of 8 voice frames.

[0158] For condition 1, the non-blank frames are the first frame "a", the fourth frame "b", the sixth frame "c" and the seventh frame "c" except blank. These frames can be retained and recorded as target frames.

[0159] For condition 2, the candidate frames with repeated content are the sixth frame "c" and the seventh frame "c", and there is no blank frame between these two frames, so no processing is performed.

[0160] For condition 3, the second frame, the third frame, the fifth frame and the eighth frame are blank frames, and for these blank frames, the blank frame probability is greater than the first threshold. If there is a blank frame whose blank frame probability is less than the second threshold (i.e., the preset threshold) among these blank frames, the blank frame can also be retained. The second threshold is greater than the first threshold.

[0161] For example, the blank frame probabilities corresponding to the second frame, the third frame, the fifth frame and the eighth frame are 0.95, 0.87, 0.85 and 0.99 respectively. If the second threshold is set to 0.9, the third frame and the fifth frame are retained.

[0162] In summary, through the above screening, the target frames are: the first frame "a", the third frame "blank", the fourth frame "b", the fifth frame "blank", the sixth frame "c" and the seventh frame "c".

[0163] It should be noted that the first threshold and the second threshold in the embodiment of the present application can be determined according to actual needs and can be any value, but the second threshold must be greater than the first threshold.

[0164] See also Figure 3A As shown, it is a schematic diagram of the first type of target frame screening in an embodiment of the present application, and the schematic diagram specifically lists the target frame screening results corresponding to Example 1.

[0165] like Figure 3A In the figure, the first line is the speech frames to be screened, and the second line is the target frame screening result, wherein the target frame is the black filled part. Specifically, 4 target frames are screened based on condition 1, and 2 target frames are screened based on condition 3.

[0166] Example 2: For example, the speech frame sequence is: ca blank a blank b blank c blank blank, then in this sequence, there are a total of 10 speech frames.

[0167] For condition 1, the non-blank frames are the first frame "c", the second frame "a", the fourth frame "a", the sixth frame "b" and the eighth frame "c" except blank. These frames can be retained and recorded as target frames.

[0168] For condition 2, in the above five non-blank frames, the characters "c" and "a" are repeated, but the character "c" corresponds to the first frame and the eighth frame, and between these two characters "c", there are two characters a and one character b. In this case, no processing is required. The character "a" corresponds to the second frame and the fourth frame, and between these two characters "a", there is only a blank frame, and there is only one blank frame, so the blank frame needs to be retained and recorded as the target frame.

[0169] Based on condition 2, the third frame "blank" is retained. In addition, the remaining blank frames are: the fifth frame, the seventh frame, the ninth frame and the tenth frame. On this basis, the target frame can be further filtered from the remaining blank frames based on condition 3.

[0170] For condition 3, the blank frame probabilities corresponding to the fifth, seventh, ninth and tenth frames are 0.95, 0.97, 0.99 and 0.81 respectively. If the second threshold is set to 0.9, the tenth frame is retained.

[0171] In summary, through the above screening, the target frames are: the first frame "c", the second frame "a", the third frame "blank", the fourth frame "a", the sixth frame "b", the eighth frame "c" and the tenth frame "blank".

[0172] See also Figure 3B As shown, it is a schematic diagram of the second method of filtering target frames in an embodiment of the present application, and the schematic diagram specifically lists the target frame filtering results corresponding to Example 2.

[0173] like Figure 3B In the figure, the first line is the speech frames to be screened, and the second line is the target frame screening result, wherein the target frame is the black filled part. Specifically, 5 target frames are screened based on condition 1, 1 target frame is screened based on condition 2, and 1 target frame is screened based on condition 3.

[0174] Example 3: For example, the speech frame sequence is: blank a blank blank abbc, then in this sequence, there are 8 speech frames in total.

[0175] For condition 1, the non-blank frames are the second frame "a", the fifth frame "a", the sixth frame "b", the seventh frame "b" and the eighth frame "c" except blank. These frames can be retained and recorded as target frames.

[0176] For condition 2, among the above five non-blank frames, characters "a" and "b" have repeated contents, but there is no blank frame between the two characters "b". In this case, no processing is required. Between the two characters "a", there is only a blank frame and no other non-blank frames. In this case, the blank frame between the two characters "a" can be retained.

[0177] An optional implementation is that if at least two consecutive blank frames are included between candidate frames with repeated content, part or all of the at least two consecutive blank frames are used as target frames.

[0178] That is, as in the above “blank a blank blank abbc”, there are two consecutive blank frames (the third frame and the fourth frame respectively), and only one blank frame may be retained, or all of them may be retained.

[0179] For example, only one blank frame is retained, such as randomly selecting the third frame from the third frame and the fourth frame (namely recorded as the target frame), and removing the fourth blank frame (namely recorded as the non-target frame). Then, based on condition 2, the remaining blank frame also includes the first frame.

[0180] For condition 3, for example, the blank frame probabilities corresponding to the first frame are 0.91, which is greater than the second threshold 0.9, and the first frame is removed (ie, recorded as a non-target frame).

[0181] In summary, through the above screening, the target frames are: the second frame "a", the third frame "blank", the fifth frame "a", the sixth frame "b", the seventh frame "b" and the eighth frame "c".

[0182] See also Figure 3C As shown, it is a schematic diagram of the third method of filtering target frames in an embodiment of the present application, and the schematic diagram specifically lists the target frame filtering results corresponding to Example 2.

[0183] like Figure 3C In the figure, the first line is the speech frames to be screened, and the second line is the target frame screening result, wherein the target frame is the black filled part. Specifically, 5 target frames are screened based on condition 1, and 1 target frame is screened based on condition 2.

[0184] It should be noted that the above-listed methods for filtering target frames are only simple examples, and any method for filtering target frames based on the obtained blank frame probability is applicable to the embodiments of the present application, and will not be described one by one here. In addition, the above-listed one voice frame corresponding to a single character is only a simple example, and it can also be multiple characters, which is not specifically limited in this article.

[0185] In the above implementation, the continuous blank frames between repeated tokens can be set to be compressed and retained partially or completely, wherein the retained part can not only improve the speech recognition efficiency, but also improve the speech recognition accuracy by retaining some blank frames. When retaining all, the speech recognition efficiency can be effectively improved.

[0186] S23: The server selects a target candidate coding feature corresponding to at least one target frame from the candidate coding features.

[0187] S24: The server decodes based on the target candidate coding features to obtain a text recognition result corresponding to the speech to be recognized.

[0188] In the embodiment of the present application, after encoding the speech to be recognized, decoding is not performed directly, but the probability of blank frames corresponding to each speech frame is determined based on the frame distribution information of the speech to be recognized, and based on this probability, some blank frames with lower importance are deleted to filter out at least one target frame from the speech to be recognized, thereby achieving compression of the output frame; when decoding, decoding is performed based only on the target candidate encoding features corresponding to the filtered target frame. In this way, the decoding efficiency can be improved with little performance loss, so as to effectively improve the speech recognition efficiency. Moreover, in the case of high concurrency, each request can improve the speech recognition efficiency of the request based on the above method.

[0189] The specific speech recognition method in the embodiment of the present application is applicable not only to real-time speech recognition scenarios, but also to non-real-time speech recognition scenarios.

[0190] The following examples are given for real-time speech recognition scenarios and non-real-time speech recognition scenarios:

[0191] (A) Real-time speech recognition scenario.

[0192] In the real-time speech recognition scenario, the target encoder may be a streaming encoder, through which streaming modeling may be implemented.

[0193] In the present application, in order to realize the prediction of frame distribution information, an additional frame distribution prediction module can be added after the streaming encoder in the existing real-time speech model, and an intermediate connection timing classification (intermediate CTC) strategy can be added in the middle of the streaming encoder. The frame rate compression strategy is applied to the streaming forward to simplify the encoder structure used to determine the compressed frame, thereby further improving the speech recognition efficiency of the speech recognition system.

[0194] The following is an explanation of the two prediction methods of the above two frame distribution information:

[0195] Method 1: Add an additional frame distribution prediction module.

[0196] Optionally, the frame distribution prediction module includes a classification linear layer and an activation function.

[0197] See also Figure 4 As shown, it is a schematic diagram of the structure of a first real-time speech recognition model in an embodiment of the present application. The first real-time speech recognition model includes two encoders and two decoding processing modules, as well as a frame distribution prediction module, such as Figure 4The S400 part includes a classification linear layer Norm and an activation function Softmax. The classification linear layer is used for normalization processing, and the frame distribution information corresponding to the speech to be recognized can be determined through the activation function. The two encoders are a multi-layer streaming encoder and a few-layer cascade encoder. The two decoding processing modules are: a first decoding processing module and a second decoding processing module.

[0198] in, Figure 4 The encoder shown in FIG. 1 uses a convolution-augmented Transformer (Conformer) structure, which is a Transformer structure combined with convolution. For example, the streaming encoder is Figure 4 The Short Masked Conformer Encoder in , also known as the short masked Conformer encoder, is a Conformer structure. For example, the streaming encoder is a 16-layer Conformer structure. Figure 4 The LongMasked Conformer Encoder in , also called the long masked Conformer encoder, is also a Conformer structure. For example, the cascade encoder is a 2-layer Conformer structure.

[0199] Among them, the first decoding processing module decodes the encoding features based on the CTC framework, and therefore, it can also be called a CTC decoding processing module, such as Figure 4 The CTC branch in , including Norm and Softmax; the Transformer (a deep learning network structure based on the attention mechanism) structure used in the second decoding processing module, which decodes the encoded features based on the attention mechanism, can also be called the Transformer decoding processing module, such as Figure 4 The Transformer decoder branch in . Figure 4 As shown, the second decoding processing module is a Transformer network structure, including 6 Transformer decoders (Transformer Decoder) and Softmax.

[0200] During the decoding process, the first decoding processing module is used to decode in a non-autoregressive manner. In other words, there is no dependency between each character in the text recognition result obtained based on the first decoding processing module. Each character in the entire text recognition result is predicted in parallel and synchronously. Specifically, the first decoding processing module includes a Softmax layer, through which the probability distribution corresponding to the coding feature can be determined. The server uses the character corresponding to the maximum probability in the probability distribution as the recognition result corresponding to the coding feature. Among them, the coding feature can correspond to a single character, multiple characters or blank characters, etc., which is not specifically limited in this article.

[0201] In the decoding process, the second decoding processing module is used for decoding in an autoregressive manner. In the text recognition result obtained by the second decoding processing module, each character has a dependency relationship, that is, the current character is determined based on the characters obtained before the character in time sequence.

[0202] Based on the above Figure 4 The structure of the first real-time speech recognition model listed above, when performing speech recognition, an optional implementation process is as follows:

[0203] See also Figure 5 As shown, it is a flow chart of a speech recognition method based on a first real-time speech recognition model in an embodiment of the present application, taking a server as an execution subject as an example, including S501 to S511:

[0204] S501: Acquire the speech to be recognized in real time.

[0205] In the embodiment of the present application, in the real-time speech recognition scenario, the subject can input speech in real time through the relevant client installed on the terminal device, and the terminal device can collect the speech fragments spoken by the subject through the microphone, and send the speech fragments input by the subject to the server in real time as the speech to be recognized by sending a packet from the client through communication with the server. The speech recognition model is used to recognize the speech to be recognized, that is, to convert the speech to be recognized into corresponding text.

[0206] S502: Determine whether the acquired speech to be recognized reaches a first preset duration. If yes, execute step S503; otherwise, return to step S501.

[0207] Specifically, S21 is performed in a streaming encoding manner, and the voice segments are divided into dynamic chunks in the streaming encoding, wherein the size of the dynamic chunk can be 8 to 22 frames, and the corresponding voice length can be 320 ms to 880 ms.

[0208] In this case, it is necessary to judge the duration of the currently acquired speech to be recognized and analyze whether it reaches the first preset duration.

[0209] Assuming that the set chunk size is 500ms (ie, 0.5s), that is, the first preset duration is 500ms, when the speech length accumulates to the set chunk size, the streaming encoder encoding starts.

[0210] It should be noted that the first preset duration in the embodiment of the present application can be flexibly set according to actual needs (such as business scenarios) and can be set to any value representing time, as long as the first preset duration is less than the second preset duration. For example, the first preset duration can also be set to 0.8 seconds or 1 second, etc., which is not specifically limited herein.

[0211] That is, the target encoder is a streaming encoder in the trained first real-time speech model. An optional implementation is as follows S503-S504:

[0212] S503: Whenever the length of the acquired speech to be recognized reaches a first preset duration, feature extraction is performed on the acquired speech to be recognized based on the first real-time speech recognition model to obtain a speech feature sequence.

[0213] S504: Encode the speech feature sequence through a streaming encoder to obtain candidate coding features and cache them.

[0214] In an embodiment of the present application, a first real-time speech recognition model that has been trained is deployed on the server side. When the length of the speech to be recognized currently received by the server reaches the first preset duration, the server can extract features of the speech to be recognized that has been received within the current first preset duration to obtain a speech feature sequence. The speech features in the speech feature sequence can be spectral features, Mel filter features, or Fbank features, etc., which are not specifically limited in this article.

[0215] Then, the speech feature sequence can be encoded through a streaming encoder to obtain the encoding results of the speech features of the speech to be recognized (essentially a speech segment) within a first preset time length, which are recorded as candidate encoding features in this article.

[0216] S505: Based on the CTC branch in the first real-time speech recognition model, the candidate coding features are decoded, and the real-time recognition text obtained by decoding is used as the text recognition result corresponding to the speech to be recognized.

[0217] In the embodiment of the present application, the CTC branch used in S505 is the first decoding processing module, that is, Figure 4 The CTC branch in the decoder uses CTC plus Token Lexicon Grammar (TLG) to decode and display it on the screen in real time.

[0218] It should be noted that the Transformer decoder branch in the embodiment of the present application is not used in the actual speech recognition process. It only plays an auxiliary role in the training process of the speech recognition model.

[0219] In this case, the server can promptly feed back the real-time recognized text to the terminal device, and display it on the screen in real time through the client.

[0220] Specifically, the first decoding processing module is used to decode the candidate coding features obtained in S504 to obtain the text corresponding to the voice segment within the current first preset time length, which is recorded as real-time recognition text in this article and displayed to the object in real time through the terminal device.

[0221] In an optional embodiment, affected by the speaking rhythm of the subject, the coding features in the candidate coding features may correspond to a single character, multiple characters or blank characters, etc., which is not specifically limited in this application. This makes the server decode the candidate coding features to obtain multiple characters including blank characters and repeated characters. The server removes the blank characters and repeated characters from the multiple characters, and then obtains the final real-time recognition text. Correspondingly, the process of the server displaying the real-time recognition text is as follows: the server decodes the candidate coding features based on the first decoding processing module to obtain multiple characters arranged in time sequence. Then, the server removes the repeated characters from the multiple characters based on the position of the blank characters in the multiple characters and the grammatical rules.

[0222] Then, the server deletes the blank characters from the multiple characters, and displays the remaining characters as real-time recognition text. The grammar rule is used to indicate the rules that need to be followed between consecutively arranged characters. The grammar rule can be determined by the TLG network.

[0223] The TLG network includes a language model G, a dictionary model L and an output mapping model T. The language model G is used to determine whether an input word sequence conforms to the grammar of the language, as well as the probability of the occurrence of the word sequence. The dictionary model L is used to map a word sequence into a character sequence. The output mapping model T is used to map the character sequence output by the output layer to a single character. The rule is to map multiple consecutive identical outputs of the model without blank characters in between to a single character, and to map consecutive identical outputs with blank characters in between to multiple characters. The solution provided in the embodiment of the present application can reduce the influence of the subject's speaking rhythm by removing blank characters and repeated characters from multiple characters through grammatical rules, thereby improving the accuracy of speech recognition.

[0224] For example, multiple characters are arranged in time sequence as “_today today_day_weather_sunny_bright”, where “_” represents a blank character. Based on the above grammar rules, the real-time recognition text corresponding to the multiple characters is “Today’s weather is sunny”.

[0225] In an embodiment of the present application, the speech features of a speech segment within a first preset time length are encoded and decoded to obtain the recognized real-time recognition text. Since the first preset time length is short, the speech feature sequence of the speech segment within the first preset time length contains less feature information, so that the existing context information can be obtained faster. The speech feature sequence is encoded and decoded based on the context information, thereby reducing the delay in speech recognition. Since the real-time recognition text can be displayed faster, the delay time on the screen can be reduced, thereby enhancing the object experience.

[0226] S506: Determine whether the total duration of the to-be-recognized speech obtained each time reaches a second preset duration. If yes, execute step S507; otherwise, return to step S501.

[0227] It should be noted that the above-listed steps S503 to S505 can be executed multiple times, that is, as time goes by, at every first preset time, the server will encode the speech feature sequence of the speech segment within the current first preset time through the streaming encoder. If the first preset time length is set to 500ms and the second preset time length is set to 5s, the above S503~S505 need to be executed in a loop 10 times, that is, when the speech to be recognized obtained by the server is 0.5s, S503~S505 are executed on the speech segment of 0~0.5s obtained this time, and candidate coding feature 1 and real-time recognition text 1 are obtained; then, when the speech to be recognized obtained by the server reaches 1s, S503~S505 are executed on the speech segment of 0.5~1s obtained this time, and candidate coding feature 2 and real-time recognition text 2 are obtained;…; when the speech to be recognized obtained by the server reaches 5s, S503~S505 are executed on the speech segment of 4.5~5s obtained this time, and candidate coding feature 10 and real-time recognition text 10 are obtained.

[0228] like Figure 6 As shown, it is a schematic diagram of the encoding timing of a speech to be recognized in an embodiment of the present application. Figure 6 An example of the encoding logic within the first preset time length and the second preset time length is briefly listed. When the first preset time length is 500ms and the second preset time length is 5s, the encoding is performed ten times by the streaming encoder within the first preset time length, and then S506 is satisfied and encoding is performed by the cascade encoder. Figure 6 The black filled rectangle indicates the part that needs to be encoded each time.

[0229] The server can store the candidate coding features obtained each time.

[0230] Afterwards, it is determined through the judgment of S506 that the total duration of the speech to be recognized obtained in the above-mentioned times is 5 seconds, which reaches the second preset duration, and S507 is executed. The specific process is as follows:

[0231] S507: When the length of the acquired speech to be recognized reaches a second preset duration, multiple candidate coding features encoded within the second preset duration are subjected to feature extraction through a classification linear layer and an activation function in the first real-time speech recognition model to obtain frame distribution information.

[0232] S508: Obtain the blank frame probability corresponding to each speech frame in the speech to be recognized according to the frame distribution information of the speech to be recognized, and select at least one target frame from the speech to be recognized based on the obtained blank frame probability.

[0233] S509: Filter out a target candidate coding feature corresponding to at least one target frame from the multiple candidate coding features.

[0234] It should be noted that the second preset duration in the embodiment of the present application can be flexibly set according to actual needs (such as business scenarios), and can be set to any value representing time, as long as the second preset duration is greater than the first preset duration. For example, it can be set to 3 to 10 seconds according to the business scenario, for example, it can be 3s, 5s or 10s, etc., which is not specifically limited in this article.

[0235] The multiple candidate coding features within the second preset duration refer to the candidate coding features obtained by executing S503 to S504 within the first preset duration. Still taking the above-mentioned example that the second preset duration is 5s, the multiple candidate coding features encoded within the second preset duration can be the candidate coding features 1 to 10 listed above.

[0236] Specifically, the cascade encoder is used to encode the output of the streaming encoder. When the length of the currently received speech segment reaches the second preset duration, the server outputs a plurality of candidate encoding features through the streaming encoder.

[0237] In the embodiment of the present application, after reaching the second preset duration, the multiple candidate coding features within the second preset duration are not directly decoded based on the cascade encoder, but based on S507 to S509, the portion corresponding to the target frame is screened out from the multiple stored candidate coding features, and recorded as the target candidate coding feature. The server inputs the screened multiple target candidate coding features into the cascade encoder, and encodes the multiple target candidate coding features through the cascade encoder to obtain the target coding feature. Then, the target coding feature is decoded by the first decoding processing module to obtain the target recognition text, and the real-time recognition text is replaced with the target recognition text for display.

[0238] Taking the candidate coding features 1 to 10 listed above as an example, here, the target candidate coding features corresponding to the target frame are selected from multiple candidate coding features. This does not mean that several candidate coding features (such as candidate coding features 1 to 5) are selected from the candidate coding features 1 to 10 and recorded as target candidate coding features, but rather that the part corresponding to the target frame is selected from each candidate coding feature.

[0239] Specifically, each candidate coding feature can be understood as including coding features corresponding to multiple speech frames, and among these speech frames, by screening through the target frame screening method listed above, a target frame that meets the conditions can be determined, and the coding feature corresponding to the target frame is the target candidate coding feature in the embodiment of the present application.

[0240] In the above implementation, by adding a frame rate compression strategy after streaming decoding, the efficiency of the secondary forward pass can be improved, and the overall reasoning efficiency and tail packet delay of the speech recognition system can be effectively optimized.

[0241] S510: Encode the target candidate coding feature based on the cascade encoder to obtain the target coding feature.

[0242] S511: Based on the CTC branch in the first real-time speech recognition model, the target coding feature is decoded, and the decoded target recognition text is used as the text recognition result corresponding to the speech to be recognized.

[0243] Specifically, the cascade encoder in the embodiment of the present application is used to encode the output of the streaming encoder. When the length of the currently received voice segment reaches the second preset duration, the server outputs multiple candidate coding features through the streaming encoder. From the multiple stored candidate coding features, the part corresponding to the target frame is screened out and recorded as the target candidate coding feature. The server inputs the screened multiple target candidate coding features into the cascade encoder, and encodes the multiple target candidate coding features through the cascade encoder to obtain the target coding feature. Then, the target coding feature is decoded by the first decoding processing module to obtain the target recognition text, and the real-time recognition text is replaced with the target recognition text for display.

[0244] Similarly, affected by the speaking rhythm of the object, the coding features in the target candidate coding features may also correspond to a single character, multiple characters or blank characters, etc., which is not limited in the embodiments of the present application. This makes the server decode the target candidate coding features to obtain multiple characters including blank characters and repeated characters. The server removes the blank characters and repeated characters in the multiple characters, and then obtains the final target recognition text. The server can remove the blank characters and repeated characters in the multiple characters based on grammatical rules to obtain the target recognition text. The way in which the server obtains the target recognition text is similar to the way in which the server obtains the real-time recognition text in step S505, which will not be repeated here. The solution provided in the embodiments of the present application, through grammatical rules, removes blank characters and repeated characters in multiple characters, which can reduce the influence of the speaking rhythm of the object, thereby improving the accuracy of speech recognition.

[0245] Then, the server may feed back the target recognition text to the terminal device, and the terminal device may replace the multiple real-time recognition texts currently displayed with the target recognition text.

[0246] In the embodiment of the present application, the server replaces the displayed multiple real-time recognition texts with the target recognition text, which is the final recognition result of the voice segment within the current second preset time length.

[0247] In the above-mentioned embodiment, based on the frame rate compression strategy, some of the multiple candidate coding features of the speech segment within the second preset time length are selected as target candidate coding features for encoding and decoding to obtain the recognized target recognition text. Since the second preset time length is longer, the multiple target candidate coding features of the speech segment within the second preset time length contain more feature information, so the target recognition text obtained is more accurate. Multiple real-time recognition texts are replaced with target recognition text, thereby improving the efficiency and accuracy of speech signal recognition.

[0248] Specifically, when training the first real-time speech recognition model, in order to reduce the delay of speech recognition and ensure the accuracy of speech recognition by the speech recognition model, the server can train the speech recognition model through sample speech segments. The sample speech segments can be speech segments of various people such as men's speech segments, women's speech segments, or children's speech segments, and can also be speech segments of various languages ​​such as Chinese speech segments, English speech segments, or Korean speech segments, and this application does not limit this.

[0249] Similarly, these sample speech segments may also be used when training a second real-time speech recognition model or a non-real-time speech recognition model, which is not specifically limited herein.

[0250] Based on the above, the training process of the first real-time speech recognition model is as follows:

[0251] First, a plurality of feature subsequences in a sample feature sequence of a sample speech segment are respectively encoded by a streaming encoder in a first real-time speech recognition model to be trained to obtain a plurality of first encoded features.

[0252] The streaming encoder is used to encode a sample feature sequence of a sample speech segment. The sample feature sequence is obtained by extracting features from the sample speech segment. The sample feature sequence can be divided into multiple feature subsequences, and the embodiment of the present application does not limit the length and number of the feature subsequences. The server can encode the multiple feature subsequences respectively through the streaming encoder to obtain multiple first coding features corresponding to the multiple feature subsequences. Among them, the feature subsequences correspond to the first coding features one by one.

[0253] Furthermore, the multiple first coding features are divided into blocks to obtain multiple feature blocks, each feature block includes at least two first coding features.

[0254] In an embodiment of the present application, the multiple first coding features are arranged in sequence in time. The server can divide the multiple first coding features into multiple feature blocks. For any feature block, the feature block includes at least two first coding features, that is, the feature information contained in a single feature block is more than the feature information contained in a single first coding feature, so that in the process of encoding and decoding the features, more feature information can be used to train the speech recognition model, thereby improving the accuracy of the speech recognition model. Among them, the number of first coding features contained in any two feature blocks in the multiple feature blocks can be equal or unequal, and the embodiment of the present application does not limit this.

[0255] Then, the plurality of feature blocks are subjected to feature extraction through the classification linear layer and activation function in the first real-time speech recognition model to obtain frame distribution information of greedy search. According to the frame distribution information of the sample speech segment, the blank frame probability corresponding to each speech frame in the sample speech segment is obtained, and based on the obtained blank frame probability, at least one sample frame is screened out from the sample speech segment according to the compressed and sorted index. From the plurality of feature blocks, a target feature block corresponding to at least one sample frame is screened out.

[0256] Furthermore, multiple target feature blocks are encoded respectively through the cascade encoder in the first real-time speech recognition model to obtain multiple second encoded features.

[0257] In an embodiment of the present application, the cascade encoder is used to encode the output of the streaming encoder. The server can first divide the multiple first coding features output by the streaming encoder into multiple feature blocks. Then, the server inputs the multiple feature blocks into the cascade encoder, and the cascade encoder encodes the multiple feature blocks respectively to obtain multiple second coding features corresponding to the multiple feature blocks. Alternatively, the server can also first input the multiple first coding features output by the streaming encoder into the cascade encoder. Then, the server divides the multiple first coding features into multiple feature blocks. Then, the server encodes the multiple feature blocks respectively through the cascade encoder to obtain multiple second coding features corresponding to the multiple feature blocks. Among them, the feature blocks correspond one to one to the second coding features.

[0258] Furthermore, the plurality of first coding features and the plurality of second coding features are decoded respectively to obtain a plurality of text recognition results.

[0259] In the embodiment of the present application, the server decodes the plurality of first coding features to obtain decoded texts corresponding to the plurality of first coding features. The server decodes the plurality of second coding features to obtain decoded texts corresponding to the plurality of second coding features.

[0260] Finally, based on the differences between multiple text recognition results and the annotated text of the sample speech fragment, as well as the difference between the frame distribution information output by the model and the frame distribution information annotated by the sample speech fragment, a loss function is constructed, and based on the loss function, parameters of the first real-time speech recognition model to be trained are adjusted.

[0261] The annotated text is the real text corresponding to the sample speech segment, and the difference is the difference between the characters in the text recognition result and the characters in the annotated text. The annotated frame distribution information is the real frame distribution information corresponding to the sample speech segment, and the difference is the difference between the probability in the frame distribution information output by the model and the probability in the annotated frame distribution information.

[0262] Optionally, when training the first real-time speech recognition model, the corresponding loss function includes:

[0263] Based on the difference between the text recognition result output by the first real-time speech recognition model and the annotated text recognition result, a CTC loss and a cross entropy (CE) loss are constructed; and based on the difference between the frame distribution information obtained by feature extraction through the classification linear layer and the activation function in the first real-time speech recognition model and the annotated frame distribution information, a CTC loss is constructed.

[0264] Wherein, when a plurality of first coding features and a plurality of second coding features are decoded respectively to obtain a plurality of text recognition results, the specific process is as follows:

[0265] Through the first decoding processing module, multiple first coding features and multiple second coding features are decoded respectively to obtain a first text and a second text; through the second decoding processing module, multiple first coding features and multiple second coding features are decoded respectively to obtain a third text and a fourth text.

[0266] That is, the first real-time speech recognition model outputs four text recognition results, namely the first text, the second text, the third text and the fourth text mentioned above.

[0267] Based on this, based on the difference between the text recognition results output by the first real-time speech recognition model and the annotated text recognition results, the specific process of constructing the CTC loss and the cross entropy loss is as follows:

[0268] The outputs of the streaming encoder and the cascade encoder are passed through the Softmax layer of the CTC branch to calculate the CTC loss function respectively, and two loss function values ​​are obtained, namely: based on the difference between the first text and the annotated text, the first loss is determined, and the first loss is the CTC loss; based on the difference between the second text and the annotated text, the second loss is determined, and the second loss is the CTC loss; in addition, Figure 4 The outputs of the Softmax layer of the Transformer decoding branch are respectively used to calculate the cross entropy loss function, and two loss function values ​​are obtained, namely: based on the difference between the third text recognition result and the annotated text, the third loss is determined, and the third loss is the cross entropy loss. Based on the difference between the fourth text recognition result and the annotated text, the fourth loss is determined, and the fourth loss is the cross entropy loss.

[0269] If, based on the difference between the frame distribution information obtained by feature extraction through the classification linear layer and the activation function in the first real-time speech recognition model and the labeled frame distribution information, the constructed CTC loss is the fifth loss. Then, the loss function of the present application when training the first real-time speech recognition model includes the five parts of the first loss to the fifth loss, which can be specifically calculated by weighted summation (using different weights) of these five parts.

[0270] Optionally, the training process listed above can be executed in a loop multiple times to obtain a trained first real-time speech recognition model; alternatively, multiple rounds of iterative training can be performed on the first real-time speech recognition model to be trained based on the first loss to the fourth loss, and the training result can be used as the initialization model. On this basis, the compression strategy listed in this application is introduced after the streaming encoder, and the model obtained after a small number of fine-tuning steps is used as the final trained first real-time speech recognition model.

[0271] Method 2: Add an intermediate ctc strategy in the middle of the streaming encoder.

[0272] In this way, there is no need to add an additional frame distribution prediction module. Instead, an intermediate CTC strategy is added in the middle of the streaming encoder (which can be understood as adding a CTC loss). The frame rate compression strategy is applied to the streaming forward to simplify the encoder structure used to determine the compressed frame, further improving the system efficiency.

[0273] In the real-time speech recognition scenario, the target encoder is a streaming encoder in the trained second real-time speech model, and the second real-time speech recognition model also includes a cascade encoder and two decoding processing modules. The second real-time speech model has a basically the same structure as the first real-time speech model, except that it does not have a frame distribution prediction module.

[0274] See also Figure 7 As shown, it is a schematic diagram of the structure of a second real-time speech recognition model in the embodiment of the present application. The second real-time speech recognition model includes two encoders and two decoding processing modules. For details, please refer to the above embodiment, and the repeated parts will not be repeated.

[0275] Among them, the middle layer of the multi-layer streaming encoder can output frame distribution information. Taking the streaming encoder as a 16-layer Conformer structure as an example, the frame distribution information can be output at the 8th layer.

[0276] Based on the above Figure 7 The structure of the first real-time speech recognition model listed above, when performing speech recognition, an optional implementation process is as follows:

[0277] See also Figure 8 As shown, it is a flow chart of a speech recognition method based on a second real-time speech recognition model in an embodiment of the present application, taking the server as the execution subject as an example, including S801 to S810:

[0278] S801: Acquire the speech to be recognized in real time.

[0279] S802: Determine whether the acquired speech to be recognized reaches a first preset duration. If yes, execute step S803; otherwise, return to step S801.

[0280] The specific implementation method can refer to the above S501 to S502, and the repeated parts will not be repeated.

[0281] Specifically, S21 is performed in a streaming encoding manner, and the target encoder is a streaming encoder in the trained second real-time speech model. An optional implementation is as follows S803-S804:

[0282] S803: Whenever the length of the acquired speech to be recognized reaches a first preset duration, feature extraction is performed on the speech to be recognized based on the second real-time speech recognition model to obtain a speech feature sequence.

[0283] The specific implementation method can refer to the above S503, and the repeated parts will not be repeated.

[0284] S804: Encode the speech feature sequence through a streaming encoder to obtain frame distribution information output by an intermediate layer of the streaming encoder and candidate coding features outputted finally, and cache the candidate coding features.

[0285] The difference from S504 is that the intermediate ctc strategy is added to the intermediate layer of the streaming encoder of the second real-time speech recognition model. In the embodiment of the present application, the intermediate layer of the streaming encoder can output frame distribution information, and the last layer of the streaming encoder can output candidate coding features. The target candidate coding features can be screened from the candidate coding features based on the frame distribution information, and the specific implementation method is as follows S805:

[0286] S805: Obtaining the blank frame probability corresponding to each speech frame in the speech to be recognized according to the frame distribution information, and selecting at least one target frame from the speech to be recognized based on the obtained blank frame probability.

[0287] The specific screening process can be found in the above embodiments, and the repeated parts will not be repeated here.

[0288] S806: Based on the CTC branch in the second real-time speech recognition model, the candidate coding features are decoded, and the real-time recognition text obtained by decoding is used as the text recognition result corresponding to the speech to be recognized.

[0289] In this case, the real-time recognized text can be displayed on the screen in real time. The specific screening process can be referred to the above embodiment, and the repeated parts will not be repeated.

[0290] S807: Determine whether the total duration of the to-be-recognized speech obtained each time reaches a second preset duration. If yes, execute step S808; otherwise, return to step S801.

[0291] The specific determination method may refer to the above S506, and the repeated parts will not be repeated.

[0292] S808: When the length of the acquired speech to be recognized reaches a second preset duration, a target candidate coding feature corresponding to at least one target frame is screened out from a plurality of candidate coding features encoded within the second preset duration.

[0293] S809: Encode the target candidate coding feature based on the cascade encoder to obtain the target coding feature;

[0294] S810: Based on the CTC branch in the second real-time speech recognition model, the target coding feature is decoded, and the decoded target recognition text is used as the text recognition result corresponding to the speech to be recognized.

[0295] Specifically, the specific process of screening the target candidate coding features, encoding the target candidate coding features, and decoding the target coding features can be referred to the above embodiments, and the repeated parts will not be repeated.

[0296] Optionally, in S804, in addition to obtaining the frame distribution information output by the intermediate layer of the streaming encoder, the coding features output by the intermediate layer of the streaming encoder can also be used as candidate coding features, and the candidate coding features are screened based on the frame distribution information to obtain target candidate coding features, and then the target candidate coding features are encoded through the subsequent layers of the streaming encoder to obtain intermediate coding features and cache them.

[0297] In this case, the real-time text recognition result is obtained by decoding the intermediate coding features. Moreover, after reaching the second preset time length, the multiple intermediate coding features within the second preset time length are directly encoded by the cascade encoder to obtain the target coding features.

[0298] In this way, frame compression begins after the middle layer of the streaming encoder, further improving the efficiency of speech recognition.

[0299] Similar to the training process of the first real-time speech recognition model, the training process of the second real-time speech recognition model is as follows:

[0300] First, a streaming encoder in a second real-time speech recognition model to be trained is used to encode multiple feature subsequences in a sample feature sequence of a sample speech segment, respectively, to obtain frame distribution information output by an intermediate layer, and finally to output multiple first encoding features.

[0301] Furthermore, the multiple first coding features are divided into blocks to obtain multiple feature blocks, each feature block includes at least two first coding features.

[0302] Furthermore, according to the frame distribution information of the sample speech segment, the blank frame probability corresponding to each speech frame in the sample speech segment is obtained, and based on the obtained blank frame probability, at least one sample frame is screened out from the sample speech segment according to the compressed and sorted index. From the multiple feature blocks, a target feature block corresponding to at least one sample frame is screened out.

[0303] Furthermore, multiple target feature blocks are encoded respectively through the cascade encoder in the second real-time speech recognition model to obtain multiple second encoded features.

[0304] Furthermore, the plurality of first coding features and the plurality of second coding features are decoded respectively to obtain a plurality of text recognition results.

[0305] In the embodiment of the present application, the server decodes the plurality of first coding features to obtain decoded texts corresponding to the plurality of first coding features. The server decodes the plurality of second coding features to obtain decoded texts corresponding to the plurality of second coding features.

[0306] Finally, based on the differences between multiple text recognition results and the annotated text of the sample speech fragment, as well as the difference between the frame distribution information output by the model and the frame distribution information annotated by the sample speech fragment, a loss function is constructed, and based on the loss function, parameters of the second real-time speech recognition model to be trained are adjusted.

[0307] Optionally, when training the second real-time speech recognition model, the corresponding loss function includes:

[0308] Based on the difference between the text recognition result output by the second real-time speech recognition model and the annotated text recognition result, a CTC loss and a cross entropy loss are constructed; and based on the difference between the frame distribution information output by the intermediate layer of the streaming encoder and the annotated frame distribution information, a CTC loss is constructed.

[0309] The multiple text recognition results obtained by respectively decoding the multiple first encoding features and the multiple second encoding features include the first to fourth texts listed above. The CTC loss and cross entropy loss constructed based on the difference between the text recognition result output by the second real-time speech recognition model and the annotated text recognition result are the same as the first to fourth losses listed above.

[0310] Therefore, the loss function of the present application when training the second real-time speech recognition model includes the five parts of the first loss to the fifth loss mentioned above, which can be specifically calculated by weighted summation (using different weights) of these five parts.

[0311] Optionally, the training process listed above can be executed in a loop multiple times to obtain a trained second real-time speech recognition model; alternatively, multiple rounds of iterative training can be performed on the second real-time speech recognition model to be trained based on the first loss to the fourth loss, and the training result can be used as the initialization model. On this basis, the compression strategy listed in this application is introduced after the streaming encoder, and the model obtained after a small number of fine-tuning steps is used as the final trained second real-time speech recognition model.

[0312] It should be noted that in the embodiment of the present application, in the model training stage of the first real-time speech recognition model or the second real-time speech recognition model, the cascade model is first trained, and the speech features are extracted and input into the streaming encoder in batches. In the streaming encoding, a dynamic chunk plus dynamic history mask training method is used. The size of the dynamic chunk is 8 to 22 frames, and the corresponding speech length is 320ms to 880ms.

[0313] Specifically, when encoding a feature subsequence, the server encodes the feature subsequence through the feature subsequence that precedes the feature subsequence in time, so that the feature encoding of the feature subsequence contains the context information of the feature subsequence, so there is no need to introduce the context information, and speech recognition can be achieved only through the context information. Based on this, the speech recognition model is trained to obtain the context information required for recognition more quickly, thereby reducing the delay of speech recognition.

[0314] The server may also encode the feature subsequence by using a feature subsequence that is before the feature subsequence in time and a feature subsequence that is after the feature subsequence in time, so that the feature encoding of the feature subsequence includes part of the preceding information and part of the following information of the feature subsequence, thereby not needing to introduce too much context information. Based on this, the speech recognition model is trained to obtain the context information required for recognition more quickly, thereby reducing the delay in speech recognition.

[0315] In the embodiment of the present application, the server can obtain the characteristic subsequence whose timing is before the characteristic subsequence or the characteristic subsequence whose timing is after the characteristic subsequence by means of dynamic masking. In other words, in the process of encoding any characteristic subsequence, the server can determine the characteristic subsequence whose timing is before the characteristic subsequence or the characteristic subsequence whose timing is after the characteristic subsequence.

[0316] Then, the server masks the feature subsequences in the sample feature sequence that are useless for the encoding operation through a mask. Then, the server encodes the feature subsequence through the feature subsequences with uncovered timing before and after the feature subsequence. The solution provided in the embodiment of the present application determines, by means of a dynamic mask, some feature subsequences with timing before the feature subsequence or some feature subsequences with timing after the feature subsequence, so that in the process of training the speech recognition model, there is no need to introduce too much context information. Based on this, the speech recognition model is trained to obtain the context information required for recognition more quickly, thereby reducing the delay in speech recognition.

[0317] It should be noted that the sample speech segment comes from a sample training set, and the sample training set includes multiple sample speech segments. The server can train the speech recognition model in batches. A batch of samples includes multiple sample speech segments. The server can encode a batch of sample speech segments in a whole sentence training manner, or divide the sample feature sequence of the sample speech segment in a chunk manner to obtain multiple feature subsequences, and encode the multiple feature subsequences respectively. The server can determine the length of the speech feature corresponding to the feature subsequence by the following formula.

[0318]

[0319] Among them, l max It represents the maximum feature length corresponding to the sample feature sequence of sample speech segments in a batch; U represents uniform distribution; x represents a random number used to indicate the batch of sample speech segments.

[0320] In the final training process, half of the batches of sample speech segments will be trained as whole sentences, that is, the sample feature sequences of the sample speech segments of the batch will be encoded and decoded as a whole, and the sample feature sequences will not be divided; the feature lengths corresponding to the sample feature sequences of the sample speech segments of the other half of the batches are randomly selected between 8 and 22, that is, the sample feature sequences of the sample speech segments of the batch are divided based on the values ​​corresponding to the batch, and the multiple feature subsequences after the division are encoded and decoded respectively. The history dynamic selection starts from zero to view the entire history.

[0321] The input batch passes through the streaming encoder and then is input into the cascade encoder. In the cascade encoder, this application uses the same mask training method as the streaming encoder, but the chunk range is set to 50 to 250 frames, and the corresponding speech length is 2s to 10s. For details, please refer to the above embodiment, and the repeated parts will not be repeated.

[0322] It should be noted that real-time speech recognition usually differs in different business scenarios. For example, in input method products, the object needs to appear on the screen immediately and the recognition result can be sent quickly after the speaker finishes speaking. This places strict restrictions on the first packet delay and the last packet delay.

[0323] In the embodiment of the present application, the cascade model can set low-latency chunks in streaming decoding to ensure real-time screen display, and the cascade decoding sets a truncation length of about 3s. The tail packet is increased within 80ms relative to streaming decoding, and the object experience is basically unchanged. In order to further improve the efficiency of decoding, the present application adds a frame rate compression strategy after streaming decoding, improves the efficiency of the secondary forward pass, and optimizes the overall reasoning efficiency and tail packet delay of the system.

[0324] The present application achieves 30% to 60% compression of the frame rate by adding a frame rate compression scheme after the streaming encoder, thereby improving the forward efficiency of the cascade encoder with little performance loss (within 0.3% to 1.5%), thereby improving the efficiency of the entire system.

[0325] (2) Non-real-time speech recognition scenario

[0326] Optionally, in a non-real-time speech recognition scenario, the target encoder is a non-real-time encoder and a decoder in a trained non-real-time speech model.

[0327] See also Fig. 9 As shown, it is a structural diagram of a non-real-time speech recognition model in an embodiment of the present application. The non-real-time speech recognition model includes a non-real-time encoder and a decoder. Specifically, an intermediate CTC strategy is added in the middle of the non-real-time encoder, and the frame rate compression strategy is applied to the forward direction, and training is performed to obtain the frame distribution of the intermediate output. During speech decoding, a segment of speech input can pass through the first few layers of encoders to obtain a segment of frame distribution, which is compressed and then passed through the next few layers of encoders.

[0328] See also Fig.10 As shown, it is a flowchart of a speech recognition method based on a non-real-time speech recognition model in an embodiment of the present application, taking the server as the execution subject as an example, including S1001 to S1007:

[0329] S1001: Acquire speech to be recognized.

[0330] S1002: extracting features of the speech to be recognized based on the non-real-time speech recognition model to obtain a speech feature sequence;

[0331] Specifically, S21 is performed in a non-real-time encoding manner, and the target encoder is a non-real-time encoder in a trained non-real-time speech model. An optional implementation is as follows S1003 to S1006:

[0332] S1003: Encode the speech to be recognized by a non-real-time encoder to obtain frame distribution information and candidate coding features output by an intermediate layer of the non-real-time encoder.

[0333] Specifically, the intermediate layer of the non-real-time encoder of the non-real-time speech recognition model adds an intermediate CTC strategy. In the embodiment of the present application, the intermediate layer of the non-real-time encoder can output frame distribution information and candidate coding features.

[0334] S1004: Obtaining the blank frame probability corresponding to each speech frame in the speech to be recognized according to the frame distribution information, and selecting at least one target frame from the speech to be recognized based on the obtained blank frame probability.

[0335] S1005: Filter out target candidate coding features corresponding to at least one target frame from the candidate coding features.

[0336] S1006: Encode the target candidate coding features based on the subsequent layers of the non-real-time encoder to obtain the target coding features finally output by the non-real-time encoder.

[0337] In the embodiment of the present application, after the candidate coding features are compressed with S1004 and S1005 (the specific compression method can be found in the above embodiment, and the repeated parts will not be repeated), the target coding features are obtained by continuing to encode through the subsequent layers of the intermediate layer of the non-real-time encoder.

[0338] If the non-real-time encoder has 12 layers and the middle layer is the 6th layer, it is necessary to encode the target candidate coding features through the 7th to 12th layers to obtain the target coding features.

[0339] S1007: Based on the decoder in the non-real-time speech recognition model, the target coding feature is decoded, and the decoded target recognition text is used as the text recognition result corresponding to the speech to be recognized.

[0340] For the specific decoding method, please refer to the decoding method of the second decoding processing module listed above, and the repeated parts will not be repeated.

[0341] In the embodiment of the present application, the training process of the non-real-time speech recognition model is as follows:

[0342] First, a non-real-time encoder in a non-real-time speech recognition model to be trained is used to encode multiple feature subsequences in a sample feature sequence of a sample speech segment, respectively, to obtain frame distribution information and a first encoding feature output by an intermediate layer of the non-real-time encoder.

[0343] Furthermore, according to the frame distribution information of the sample speech segment, the blank frame probability corresponding to each speech frame in the sample speech segment is obtained, and based on the obtained blank frame probability, at least one sample frame is screened out from the sample speech segment according to the compressed and sorted index. From the first coding features, a target first coding feature corresponding to at least one sample frame is screened out.

[0344] Furthermore, the target first coding feature is encoded based on the subsequent layers of the non-real-time speech encoder, and then the second coding feature is obtained.

[0345] Furthermore, a decoder based on the non-real-time speech recognition model decodes the second encoding feature to obtain a text recognition result.

[0346] Finally, based on the difference between the text recognition results and the annotated text of the sample speech fragment, as well as the difference between the frame distribution information output by the model and the frame distribution information annotated by the sample speech fragment, a loss function is constructed, and based on the loss function, the parameters of the non-real-time speech recognition model to be trained are adjusted.

[0347] Optionally, when training a non-real-time speech recognition model, the corresponding loss function includes:

[0348] The cross entropy loss is constructed based on the difference between the text recognition results obtained by the non-real-time speech recognition model and the annotated text recognition results. The CTC loss is constructed based on the difference between the frame distribution information output by the intermediate layer of the non-real-time encoder and the annotated frame distribution information.

[0349] The text recognition result obtained based on the non-real-time speech recognition model is the output result of the final decoding layer. The annotated text is the real text corresponding to the sample speech segment, and the difference is the difference between the characters in the text recognition result and the characters in the annotated text. The annotated frame distribution information is the real frame distribution information corresponding to the sample speech segment, and the difference is the difference between the probability in the frame distribution information output by the model and the probability in the annotated frame distribution information.

[0350] In the above implementation, the distribution of a frame is obtained through the first several layers of encoders of the non-real-time encoder, which is compressed and then passed through the following several layers of encoders, thereby effectively improving the efficiency of speech signal recognition.

[0351] In addition, it should be noted that the speech recognition method in the embodiment of the present application can also be applied to the Transducer system or other speech recognition systems, which will not be described in detail here.

[0352] See also Fig.11 As shown, it is a logical diagram of a speech recognition process in an embodiment of the present application. Take the deployment of the speech recognition model on the server side as an example. Speech recognition software can be installed on the terminal device, and the object inputs speech through the software. The terminal device sends the speech input by the object as the speech to be recognized to the server. The server encodes the speech to be recognized through the target encoder in the speech recognition model, and obtains the candidate coding features corresponding to the speech to be recognized; according to the frame distribution information of the speech to be recognized, the blank frame probability corresponding to each speech frame in the speech to be recognized is obtained, and based on the obtained blank frame probability, at least one target frame is screened out from the speech to be recognized; then, from the candidate coding features, the target candidate coding features corresponding to at least one target frame are screened out; through the speech recognition model, decoding is performed based on the target candidate coding features to obtain the text recognition result corresponding to the speech to be recognized. During the speech recognition process, the server will also send the obtained text recognition results to the terminal device in a timely manner to display them to the object, such as Fig.11 As shown. Among them, Fig.11 The speech recognition model shown can be any one of the first real-time speech recognition model, the second real-time speech recognition model, the non-real-time speech recognition model listed in this article, or other speech recognition models with the addition of a frame rate compression strategy, which is not specifically limited in this article.

[0353] It should be noted that the speech recognition model in the embodiment of the present application can also be deployed on the terminal device side, that is, speech recognition software can be installed on the terminal device, and the subject inputs speech through the software. The terminal device can encode the speech to be recognized through the target encoder in the speech recognition model to obtain candidate coding features corresponding to the speech to be recognized; according to the frame distribution information of the speech to be recognized, the blank frame probability corresponding to each speech frame in the speech to be recognized is obtained, and based on the obtained blank frame probability, at least one target frame is screened out from the speech to be recognized; then, from the candidate coding features, a target candidate coding feature corresponding to at least one target frame is screened out; through the speech recognition model, decoding is performed based on the target candidate coding feature to obtain the text recognition result corresponding to the speech to be recognized, and displayed on the screen, etc., which is not specifically limited in this article.

[0354] Based on the same inventive concept, the present application also provides a speech recognition device. Fig.12 As shown, it is a schematic diagram of the structure of the speech recognition device 1200, which may include:

[0355] The encoding unit 1201 is used to obtain the speech to be recognized, and encode the speech to be recognized through a target encoder to obtain candidate encoding features corresponding to the speech to be recognized;

[0356] The screening unit 1202 is used to obtain the blank frame probability corresponding to each speech frame in the speech to be recognized according to the frame distribution information of the speech to be recognized, and screen out at least one target frame from the speech to be recognized based on the obtained blank frame probability; the frame distribution information represents the probability distribution of each speech frame in the speech to be recognized in the dictionary space; the blank frame probability represents the probability that the speech frame is a blank frame; and screen out at least one target candidate coding feature corresponding to the target frame from the candidate coding features;

[0357] The decoding unit 1203 is used to perform decoding based on the target candidate coding features to obtain a text recognition result corresponding to the speech to be recognized.

[0358] Optionally, the screening unit 1202 is specifically used for:

[0359] Determine the blank frames and non-blank frames in the speech to be recognized according to the obtained blank frame probability;

[0360] Non-blank frames, blank frames between candidate frames with repeated content, and blank frames with a blank frame probability less than a preset threshold are taken as target frames; wherein the candidate frames with repeated content are: non-blank frames with adjacent positions and repeated content among all non-blank frames contained in the speech to be recognized.

[0361] Optionally, the screening unit 1202 is specifically used for:

[0362] The blank frames between the candidate frames with repeated contents are used as target frames;

[0363] Except for the blank frames between candidate frames with repeated contents, blank frames whose corresponding blank frame probabilities are less than a preset threshold are taken as target frames.

[0364] Optionally, if the candidate frames with repeated content contain at least two consecutive blank frames;

[0365] The screening unit 1202 is specifically used for:

[0366] Part or all of at least two consecutive blank frames are used as target frames.

[0367] Optionally, in a real-time speech recognition scenario, the target encoder is a streaming encoder in a trained first real-time speech model;

[0368] The encoding unit 1201 is specifically used for:

[0369] When the length of the acquired speech to be recognized reaches a first preset duration, feature extraction is performed on the speech to be recognized based on the first real-time speech recognition model to obtain a speech feature sequence;

[0370] Encode the speech feature sequence through a streaming encoder to obtain candidate coding features and cache them;

[0371] And, the decoding unit 1203 is further used for:

[0372] Based on the first real-time speech recognition model, the candidate coding features are decoded, and the real-time recognition text obtained by decoding is used as the text recognition result corresponding to the speech to be recognized.

[0373] Optionally, the encoding unit 1201 is further used for:

[0374] Before the screening unit 1202 obtains the blank frame probability corresponding to each speech frame in the speech to be recognized according to the frame distribution information of the speech to be recognized, the following operations are performed:

[0375] When the length of the acquired speech to be recognized reaches a second preset duration, multiple candidate coding features encoded within the second preset duration are subjected to feature extraction through the classification linear layer and activation function in the first real-time speech recognition model to obtain frame distribution information; the second preset duration is greater than the first preset duration.

[0376] Optionally, the first real-time speech recognition model further includes a cascade encoder;

[0377] The screening unit 1202 is specifically used for:

[0378] Selecting a target candidate coding feature corresponding to at least one target frame from a plurality of candidate coding features;

[0379] The decoding unit 1203 is specifically used for:

[0380] Based on the cascade encoder, the target candidate encoding features are encoded to obtain the target encoding features;

[0381] Based on the first real-time speech recognition model, the target coding feature is decoded, and the decoded target recognition text is used as the text recognition result corresponding to the speech to be recognized.

[0382] Optionally, in a real-time speech recognition scenario, the target encoder is a streaming encoder in a trained second real-time speech model;

[0383] The encoding unit 1201 is specifically used for:

[0384] When the length of the acquired speech to be recognized reaches a first preset time length, feature extraction is performed on the speech to be recognized based on the trained second real-time speech recognition model to obtain a speech feature sequence;

[0385] Encode the speech feature sequence through the streaming encoder to obtain frame distribution information output by the intermediate layer of the streaming encoder and candidate coding features outputted finally, and cache the candidate coding features;

[0386] And, the decoding unit 1203 is further used for:

[0387] Based on the second real-time speech recognition model, the candidate coding features are decoded, and the real-time recognition text obtained by decoding is used as the text recognition result corresponding to the speech to be recognized.

[0388] Optionally, the second real-time speech recognition model further includes a cascade encoder;

[0389] The screening unit 1202 is specifically used for:

[0390] When the length of the acquired speech to be recognized reaches a second preset duration, a target candidate coding feature corresponding to at least one target frame is screened out from a plurality of candidate coding features encoded within the second preset duration; the second preset duration is greater than the first preset duration;

[0391] The decoding unit 1203 is specifically used for:

[0392] Based on the cascade encoder, the target candidate encoding features are encoded to obtain the target encoding features;

[0393] Based on the second real-time speech recognition model, the target coding features are decoded, and the decoded target recognition text is used as the text recognition result corresponding to the speech to be recognized.

[0394] Optionally, in a non-real-time speech recognition scenario, the target encoder is a non-real-time encoder in a trained non-real-time speech model;

[0395] The encoding unit 1201 is specifically used for:

[0396] Extract features of the speech to be recognized based on the non-real-time speech recognition model to obtain a speech feature sequence;

[0397] Encode the speech to be recognized by a non-real-time encoder to obtain frame distribution information and candidate coding features output by an intermediate layer of the non-real-time encoder;

[0398] And, the decoding unit 1203 is specifically used for:

[0399] Obtaining a target coding feature obtained by encoding the target candidate coding feature by a non-real-time encoder;

[0400] Through the non-real-time speech recognition model, the target encoding features are decoded to obtain the text recognition results corresponding to the speech to be recognized.

[0401] Optionally, when training the first real-time speech recognition model, the corresponding loss function includes:

[0402] Based on the difference between the text recognition result output by the first real-time speech recognition model and the annotated text recognition result, a CTC loss and a cross entropy loss are constructed;

[0403] The CTC loss is constructed based on the difference between the frame distribution information obtained by feature extraction through the classification linear layer and the activation function in the first real-time speech recognition model and the labeled frame distribution information.

[0404] Optionally, when training the second real-time speech recognition model, the corresponding loss function includes:

[0405] Based on the difference between the text recognition result output by the second real-time speech recognition model and the annotated text recognition result, a CTC loss and a cross entropy loss are constructed;

[0406] The CTC loss is constructed based on the difference between the frame distribution information output by the intermediate layer of the streaming encoder and the labeled frame distribution information.

[0407] Optionally, when training the non-real-time speech recognition model, the corresponding loss function includes:

[0408] A cross entropy loss constructed based on the difference between the text recognition result obtained by the non-real-time speech recognition model and the annotated text recognition result;

[0409] The CTC loss is constructed based on the difference between the frame distribution information output by the intermediate layer of the non-real-time encoder and the labeled frame distribution information.

[0410] Since the speech recognition method in this application does not directly decode the speech to be recognized after encoding, but determines the probability of blank frames corresponding to each speech frame based on the frame distribution information of the speech to be recognized, and based on this probability, deletes some blank frames with lower importance, so as to filter out at least one target frame from the speech to be recognized, thereby achieving compression of the output frame; when decoding, decoding is only based on the target candidate coding features corresponding to the filtered target frame. In this way, the decoding efficiency can be improved with little performance loss, so as to effectively improve the speech recognition efficiency. Moreover, in the case of high concurrency, each request can improve the speech recognition efficiency of the request based on the above method.

[0411] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0412] After introducing the speech recognition method and apparatus according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0413] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0414] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server, such as Figure 1 In this embodiment, the structure of the electronic device can be as follows: Fig.13 As shown, it includes a memory 1301 , a communication module 1303 and one or more processors 1302 .

[0415] The memory 1301 is used to store computer programs executed by the processor 1302. The memory 1301 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0416] The memory 1301 may be a volatile memory, such as a random-access memory (RAM); the memory 1301 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1301 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1301 may be a combination of the above memories.

[0417] The processor 1302 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1302 is used to implement the above-mentioned speech recognition method when calling the computer program stored in the memory 1301 .

[0418] The communication module 1303 is used to communicate with terminal devices and other servers.

[0419] The specific connection medium between the memory 1301, the communication module 1303 and the processor 1302 is not limited in the embodiment of the present application. Fig.13 In the embodiment, the memory 1301 and the processor 1302 are connected via a bus 1304. The bus 1304 is connected to the processor 1302 via a bus 1304. Fig.13 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1304 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.13The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0420] The memory 1301 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the speech recognition method of the embodiment of the present application. The processor 1302 is used to execute the above-mentioned speech recognition method, such as Figure 2 shown.

[0421] In another embodiment, the electronic device may also be other electronic devices, such as Figure 1 The terminal device 110 shown in FIG. 1 is a terminal device 110 shown in FIG. 1 . In this embodiment, the structure of the electronic device can be as follows: Fig.14 As shown, it includes: a communication component 1410, a memory 1420, a display unit 1430, a camera 1440, a sensor 1450, an audio circuit 1460, a Bluetooth module 1470, a processor 1480 and other components.

[0422] The communication component 1410 is used to communicate with the server. In some embodiments, a circuit wireless fidelity (Wireless Fidelity, WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help the user to send and receive information through the WiFi module.

[0423] The memory 1420 can be used to store software programs and data. The processor 1480 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 1420. The memory 1420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 1420 stores an operating system that enables the terminal device 110 to run. In the present application, the memory 1420 can store an operating system and various application programs, and can also store a computer program for executing the speech recognition method of the embodiment of the present application.

[0424] The display unit 1430 can also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 1430 may include a display screen 1432 disposed on the front of the terminal device 110. The display screen 1432 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 1430 may be used to display the user interface, text recognition results, etc. in the embodiments of the present application.

[0425] The display unit 1430 can also be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device 110. Specifically, the display unit 1430 may include a touch screen 1431 arranged on the front of the terminal device 110, which can collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

[0426] The touch screen 1431 can be covered on the display screen 1432, or the touch screen 1431 and the display screen 1432 can be integrated to realize the input and output functions of the terminal device 110, and the integrated display screen can be referred to as a touch display screen. In this application, the display unit 1430 can display the application and the corresponding operation steps.

[0427] The camera 1440 can be used to capture still images, and the user can publish the images taken by the camera 1440 through the application. The camera 1440 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the processor 1480 to convert it into a digital image signal.

[0428] The terminal device may further include at least one sensor 1450, such as an acceleration sensor 1451, a distance sensor 1452, a fingerprint sensor 1453, and a temperature sensor 1454. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0429] The audio circuit 1460, the speaker 1461, and the microphone 1462 can provide an audio interface between the user and the terminal device 110. The audio circuit 1460 can transmit the electrical signal converted from the received audio data to the speaker 1461, which is converted into a sound signal for output. The terminal device 110 can also be equipped with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1460 and converted into audio data, and then the audio data is output to the communication component 1410 to be sent to, for example, another terminal device 110, or the audio data is output to the memory 1420 for further processing.

[0430] The Bluetooth module 1470 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1470 to exchange data.

[0431] The processor 1480 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1420 and calling data stored in the memory 1420. In some embodiments, the processor 1480 may include one or more processing units; the processor 1480 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1480. In the present application, the processor 1480 can run the operating system, application programs, user interface display and touch response, as well as the voice recognition method of the embodiment of the present application. In addition, the processor 1480 is coupled to the display unit 1430.

[0432] In some possible implementations, various aspects of the speech recognition method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of the speech recognition method according to various exemplary implementations of the present application described above in this specification. For example, the electronic device may perform the following steps: Figure 2 Follow the steps shown in .

[0433] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0434] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0435] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0436] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0437] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user electronic device, partially on the user electronic device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0438] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0439] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0440] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0441] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0442] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0443] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0444] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0445] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A speech recognition method, characterized in that: The method comprises: Acquire a speech to be recognized, and encode the speech to be recognized by a target encoder to obtain candidate coding features corresponding to the speech to be recognized; Obtaining, based on the frame distribution information of the speech to be recognized, a blank frame probability corresponding to each speech frame in the speech to be recognized, and selecting at least one target frame from the speech to be recognized based on the obtained blank frame probability; the frame distribution information represents the probability distribution of each speech frame in the speech to be recognized in a dictionary space; and the blank frame probability represents the probability that a speech frame is a blank frame; Filtering the target candidate coding feature corresponding to the at least one target frame from the candidate coding features; Decoding is performed based on the target candidate coding features to obtain a text recognition result corresponding to the speech to be recognized.

2. The method according to claim 1, wherein The step of selecting at least one target frame from the speech to be recognized based on the determined blank frame probability includes: Determining blank frames and non-blank frames in the speech to be recognized according to the obtained blank frame probability; Non-blank frames, blank frames between candidate frames with repeated content, and blank frames with a blank frame probability less than a preset threshold are taken as target frames; wherein the candidate frames with repeated content are: non-blank frames with adjacent positions and repeated content among the non-blank frames contained in the speech to be recognized.

3. The method according to claim 2, wherein The blank frames between candidate frames with repeated content and blank frames with a blank frame probability less than a preset threshold are used as target frames, including: The blank frames between candidate frames with repeated content are used as target frames; Except for the blank frames between the candidate frames with repeated content, blank frames whose corresponding blank frame probabilities are less than a preset threshold are used as target frames.

4. The method according to claim 2 or 3, wherein: If the candidate frames with repeated content contain at least two consecutive blank frames; Then, the blank frames between the candidate frames with repeated content are used as target frames, including: Part or all of the at least two consecutive blank frames are used as the target frame.

5. The method according to any one of claims 1 to 3, wherein: In a real-time speech recognition scenario, the target encoder is a streaming encoder in a trained first real-time speech model; The step of encoding the speech to be recognized by a target encoder to obtain candidate coding features corresponding to the speech to be recognized includes: When the acquired length of the speech to be recognized reaches a first preset duration, feature extraction is performed on the speech to be recognized based on the first real-time speech recognition model to obtain a speech feature sequence; Encoding the speech feature sequence by the streaming encoder to obtain the candidate coding features and cache them; And, the method further comprises: Based on the first real-time speech recognition model, the candidate coding features are decoded, and the real-time recognition text obtained by decoding is used as the text recognition result corresponding to the speech to be recognized.

6. The method according to claim 5, wherein Before obtaining the blank frame probability corresponding to each speech frame in the speech to be recognized according to the frame distribution information of the speech to be recognized, the method further includes: When the length of the speech to be recognized reaches a second preset duration, the multiple candidate coding features encoded within the second preset duration are subjected to feature extraction through the classification linear layer and activation function in the first real-time speech recognition model to obtain the frame distribution information; the second preset duration is greater than the first preset duration.

7. The method according to claim 6, wherein The first real-time speech recognition model also includes a cascade encoder; Then, screening out the target candidate coding feature corresponding to the at least one target frame from the candidate coding features includes: Filtering a target candidate coding feature corresponding to the at least one target frame from the multiple candidate coding features; The decoding based on the target candidate coding feature to obtain the text recognition result corresponding to the speech to be recognized includes: Encoding the target candidate coding feature based on the cascade encoder to obtain a target coding feature; Based on the first real-time speech recognition model, the target coding feature is decoded, and the decoded target recognition text is used as the text recognition result corresponding to the speech to be recognized.

8. The method according to any one of claims 1 to 3, wherein: In a real-time speech recognition scenario, the target encoder is a streaming encoder in a trained second real-time speech model; The step of encoding the speech to be recognized by a target encoder to obtain candidate coding features corresponding to the speech to be recognized includes: When the acquired length of the speech to be recognized reaches a first preset duration, feature extraction is performed on the speech to be recognized based on the trained second real-time speech recognition model to obtain a speech feature sequence; Encoding the speech feature sequence by the streaming encoder to obtain frame distribution information output by an intermediate layer of the streaming encoder and candidate coding features outputted finally, and caching the candidate coding features; And, the method further comprises: Based on the second real-time speech recognition model, the candidate coding features are decoded, and the real-time recognition text obtained by decoding is used as the text recognition result corresponding to the speech to be recognized.

9. The method according to claim 8, wherein The second real-time speech recognition model further includes a cascade encoder; Then, screening out the target candidate coding feature corresponding to the at least one target frame from the candidate coding features includes: When the acquired length of the speech to be recognized reaches a second preset duration, selecting a target candidate coding feature corresponding to the at least one target frame from a plurality of candidate coding features obtained by encoding within the second preset duration; the second preset duration is greater than the first preset duration; The decoding based on the target candidate coding feature to obtain the text recognition result corresponding to the speech to be recognized includes: Encoding the target candidate coding feature based on the cascade encoder to obtain a target coding feature; Based on the second real-time speech recognition model, the target coding feature is decoded, and the decoded target recognition text is used as the text recognition result corresponding to the speech to be recognized.

10. The method according to any one of claims 1 to 3, characterized in that: In a non-real-time speech recognition scenario, the target encoder is a non-real-time encoder in a trained non-real-time speech model; The step of encoding the speech to be recognized by a target encoder to obtain candidate coding features corresponding to the speech to be recognized includes: Extracting features of the speech to be recognized based on a non-real-time speech recognition model to obtain a speech feature sequence; Encoding the speech to be recognized by the non-real-time encoder to obtain frame distribution information and candidate coding features output by an intermediate layer of the non-real-time encoder; Furthermore, decoding based on the target candidate coding features to obtain a text recognition result corresponding to the speech to be recognized includes: Obtaining a target coding feature obtained by encoding the target candidate coding feature by the non-real-time encoder; The target coding feature is decoded by the non-real-time speech recognition model to obtain a text recognition result corresponding to the speech to be recognized.

11. The method according to claim 7, wherein When training the first real-time speech recognition model, the corresponding loss function includes: Constructing a connectionist temporal classification (CTC) loss and a cross entropy loss based on the difference between the text recognition result output by the first real-time speech recognition model and the annotated text recognition result; The CTC loss is constructed based on the difference between the frame distribution information obtained by feature extraction through the classification linear layer and the activation function in the first real-time speech recognition model and the labeled frame distribution information.

12. The method according to claim 9, wherein When training the second real-time speech recognition model, the corresponding loss function includes: Based on the difference between the text recognition result output by the second real-time speech recognition model and the annotated text recognition result, a CTC loss and a cross entropy loss are constructed; The CTC loss is constructed based on the difference between the frame distribution information output by the intermediate layer of the streaming encoder and the labeled frame distribution information.

13. The method according to claim 10, wherein When training the non-real-time speech recognition model, the corresponding loss function includes: A cross entropy loss constructed based on the difference between the text recognition result obtained by the non-real-time speech recognition model and the annotated text recognition result; The CTC loss is constructed based on the difference between the frame distribution information output by the intermediate layer of the non-real-time encoder and the labeled frame distribution information.

14. A speech recognition device, characterized in that: include: An encoding unit, configured to obtain a speech to be recognized, and encode the speech to be recognized through a target encoder to obtain candidate coding features corresponding to the speech to be recognized; a screening unit, configured to obtain, based on the frame distribution information of the speech to be recognized, a blank frame probability corresponding to each speech frame in the speech to be recognized, and screen out at least one target frame from the speech to be recognized based on the obtained blank frame probability; the frame distribution information represents the probability distribution of each speech frame in the speech to be recognized in the dictionary space; the blank frame probability represents the probability that a speech frame is a blank frame; and screen out a target candidate coding feature corresponding to the at least one target frame from the candidate coding features; A decoding unit is used to decode based on the target candidate coding features to obtain a text recognition result corresponding to the speech to be recognized.

15. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor is enabled to perform the steps of any one of the methods of claims 1 to 13.

16. A computer-readable storage medium, characterized in that The method comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the methods according to claims 1 to 13.

17. A computer program product, characterized in that The method comprises a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the methods described in claims 1 to 13.

Images