Voice recognition method, training method of deep layer learning model, and device and apparatus
By decoding and extracting word-level audio features from initial recognition results, the method addresses the inconsistency in feature representation lengths, improving speech recognition accuracy and efficiency.
Patent Information
- Application Number
- JP2024148016
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-08-29
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-29
AI Technical Summary
Existing speech recognition methods face challenges due to inconsistent feature representation lengths caused by framing, leading to reduced accuracy and computational inefficiency.
A method that decodes first audio features to obtain initial recognition results, extracts word-level audio features, and uses these as prior information to achieve unified audio feature representations of equal length, improving accuracy and efficiency by addressing the length mismatch in traditional speech framing.
This approach enhances speech recognition accuracy and computational efficiency by ensuring consistent feature representation lengths, overcoming the limitations of traditional framing methods.
Smart Images

Figure 2024167341000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as speech recognition and deep learning, and specifically to a speech recognition method, a method for training a deep learning model for speech recognition, a speech recognition device, an apparatus for training a deep learning model for speech recognition, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]
[0002] Artificial intelligence is a field that studies how computers can imitate some human thought processes and intelligent behaviors (e.g., learning, reasoning, thinking, planning, etc.), and includes both hardware and software technologies. AI hardware technology generally includes sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, and other technologies, while AI software technology mainly includes natural language processing technology, computer vision technology, voice recognition technology, and several major directions such as machine learning / deep learning, big data processing technology, and knowledge graph technology.
[0003] Automatic speech recognition (ASR) is a technology that automatically converts voice signals input through a computer into corresponding text. With the in-depth research of deep learning technology in the field of speech recognition, especially the submission of end-to-end speech recognition technology, the accuracy rate of speech recognition has been greatly improved while reducing the modeling complexity. With the increasing popularity of various intelligent devices, large vocabulary online speech recognition systems are widely applied in various scenarios such as speech transcription, intelligent customer service, in-vehicle navigation, smart home, etc. In these speech recognition tasks, users usually expect to be able to get quick and accurate responses and feedback from the system after completing voice input, so the requirements for the accuracy rate and real-time factor of the speech recognition model are very high.
[0004] The methods described in this section are not necessarily methods that have been previously conceived or adopted. Unless otherwise noted, any methods described in this section should not be considered prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise noted, any problem addressed in this section should not be considered an admission of prior art. Summary of the Invention
[0005] The present disclosure provides a speech recognition method, a method for training a deep learning model for speech recognition, a speech recognition device, an apparatus for training a deep learning model for speech recognition, an electronic device, a computer-readable storage medium, and a computer program product.
[0006] According to one aspect of the present disclosure, there is provided a speech recognition method, the speech recognition method including: acquiring first speech features of a speech to be recognized, the first speech features including a plurality of speech segment features corresponding to a plurality of speech segments in the speech to be recognized; decoding the first speech features using a first decoder to acquire a plurality of first decoding results corresponding to a plurality of words in the speech to be recognized, the first decoding results indicating a first recognition result of the corresponding words; extracting and acquiring a second speech feature from the first speech features based on first a priori information, the first a priori information including the plurality of first decoding results, the second speech features including a plurality of first word-level audio features corresponding to the plurality of words; and decoding the second speech features using a second decoder to acquire a plurality of second decoding results corresponding to the plurality of words, the second decoding results indicating a second recognition result of the corresponding words.
[0007] According to another aspect of the present disclosure, there is provided a method for training a deep learning model for speech recognition, the deep learning model including a first decoder and a second decoder, the training method includes: obtaining an actual recognition result of a sample speech and a plurality of words in the sample speech; obtaining first sample speech features of the sample speech, the first sample speech features including a plurality of sample speech segment features corresponding to a plurality of sample speech segments in the sample speech; decoding the first sample speech features using the first decoder to obtain a plurality of first sample decoding results corresponding to the plurality of words in the sample speech, the first sample decoding results indicating a first recognition result of the corresponding word; The method includes: extracting and obtaining second sample speech features from the first sample speech features based on sample a priori information, where the first sample a priori information includes a plurality of first sample decoding results, and the second sample speech features include a plurality of first sample word-level audio features corresponding to a plurality of words; decoding the second sample speech features using a second decoder to obtain a plurality of second sample decoding results corresponding to the plurality of words, where the second sample decoding results indicate second recognition results of the corresponding words; and adjusting parameters of the deep learning model based on the actual recognition results of the plurality of words, the first recognition results and the second recognition results to obtain a trained deep learning model.
[0008] According to another aspect of the present disclosure, there is provided a speech recognition apparatus, the apparatus including: a speech feature encoding module configured to obtain first speech features of a speech to be recognized, the first speech features including a plurality of speech segment features corresponding to a plurality of speech segments in the speech to be recognized; a first decoder configured to decode the first speech features and obtain a plurality of first decoding results corresponding to a plurality of words in the speech to be recognized, the first decoding results indicating a first recognition result of the corresponding words; a word-level feature extraction module configured to extract and obtain a second speech feature from the first speech features based on first a priori information, the first a priori information including the plurality of first decoding results, the second speech features including a plurality of first word-level audio features corresponding to the plurality of words; and a second decoder configured to decode the second speech features and obtain a plurality of second decoding results corresponding to the plurality of words, the second decoding results indicating a second recognition result of the corresponding words.
[0009] According to another aspect of the present disclosure, there is provided an apparatus for training a deep learning model for speech recognition, the deep learning model including a first decoder and a second decoder, the training apparatus including: an acquisition module configured to acquire an actual recognition result of a sample speech and a plurality of words in the sample speech; an audio feature encoding module configured to acquire first sample speech features of the sample speech, the first sample speech features including a plurality of sample speech segment features corresponding to a plurality of sample speech segments in the sample speech; a first decoder configured to decode the first sample speech features and acquire a plurality of first sample decoding results corresponding to the plurality of words in the sample speech, the first sample decoding results indicating a first recognition result of the corresponding word; the word-level feature extraction module configured to extract and obtain second sample speech features from the first sample speech features based on actual prior information, where the first sample prior information includes a plurality of first sample decoding results, and the second sample speech features include a plurality of first sample word-level audio features corresponding to a plurality of words; a second decoder configured to decode the second sample speech features and obtain a plurality of second sample decoding results corresponding to the plurality of words, where the second sample decoding results indicate second recognition results of the corresponding words; and a parameter adjustment module configured to adjust parameters of a deep learning model based on the actual recognition results of the plurality of words, the first recognition results and the second recognition results, to obtain a trained deep learning model.
[0010] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively coupled to the at least one processor, where the memory stores instructions executable by the at least one processor, which instructions, when executed by the at least one processor, cause the at least one processor to perform the method set forth above.
[0011] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method described above.
[0012] According to another aspect of the present disclosure there is provided a computer program product comprising a computer program which, when executed by a processor, implements the above-mentioned method.
[0013] According to one or more embodiments of the present disclosure, the present disclosure obtains and decodes first speech features including multiple speech segment features of a speech to be recognized, obtains an initial recognition result for the speech to be recognized, further uses the initial recognition result to extract word-level audio features from the first speech features, and then decodes the word-level audio features to obtain a final recognition result.
[0014] By taking the initial recognition result for the speech to be recognized as a priori, extracting and obtaining a unified audio feature representation of equal length at the word level from the speech feature information of unequal length in the frame-level audio information, and decoding the word-level audio features to obtain the final recognition result, the problem of inconsistent lengths of feature representations of speech framing in the conventional technology is solved, and the accuracy of speech recognition is improved and the calculation efficiency is increased.
[0015] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, and are not intended to limit the scope of protection of the present disclosure. Other features of the present disclosure will be easily understood from the following description. [Brief description of the drawings]
[0016] The drawings illustratively illustrate the examples, constitute a part of the specification, and together with the written description serve to explain exemplary embodiments of the examples. The illustrated examples are for illustrative purposes only and are not intended to limit the scope of the claims. In all the drawings, the same reference numerals refer to similar, but not necessarily identical, elements.
[0017] [Figure 1] FIG. 1 is a schematic diagram illustrating an example system capable of implementing various methods described herein, according to embodiments of the present disclosure. [Diagram 2] FIG. 2 is a flow chart diagram illustrating a speech recognition method according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 is a flow chart diagram illustrating obtaining a first speech feature of a speech to be recognized according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram illustrating a Conformer streaming multi-layer disconnected attention model based on historical feature abstraction, according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 is a flow chart diagram of extracting a second speech feature from a first speech feature according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a flow chart diagram illustrating a speech recognition method according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a flow chart diagram illustrating a speech recognition method according to an embodiment of the present disclosure. [Figure 8] FIG. 2 is a schematic diagram illustrating an end-to-end voice large model according to an embodiment of the present disclosure. [Figure 9] FIG. 2 is a flow chart diagram illustrating a method for training a deep learning model for speech recognition according to an embodiment of the present disclosure. [Figure 10] 1 is a configuration block diagram showing a voice recognition device according to an embodiment of the present disclosure. [Figure 11] FIG. 1 is a block diagram illustrating a configuration of an apparatus for training a deep learning model for speech recognition according to an embodiment of the present disclosure. [Figure 12] FIG. 1 is a block diagram illustrating an example electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] Hereinafter, exemplary embodiments of the present disclosure will be described in conjunction with the drawings, and various details in the embodiments of the present disclosure contained therein are to be considered as merely illustrative, in order to facilitate understanding. Therefore, it should be recognized that those skilled in the art can make various changes and modifications to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the following description omits descriptions of known functions and structures.
[0019] In this disclosure, unless otherwise stated, the terms "first," "second," and the like, used to describe various elements are not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of an element, or in some cases, may refer to different instances based on the contextual description.
[0020] The terms used in the description of various examples of the present disclosure are intended only to describe specific examples and are not intended to be limiting. Unless otherwise clearly indicated by the context, an element may be one or more, unless the number of elements is specifically limited. In addition, the term "and / or" used in the present disclosure covers any and all possible combinations of the listed items.
[0021] In the related art, some speech recognition methods use framing speech features to learn audio feature representations. However, the content information contained in speech changes constantly with the speaker's speaking speed, intonation, tone, etc., and different speakers will have completely different expressions for the same content. Therefore, this feature representation method will result in the length of the representation obtained by the framing speech features being inconsistent, which will affect the accuracy of speech recognition, and the obtained feature representation will contain a large amount of redundant features, resulting in low computation efficiency.
[0022] In order to solve the above problem, the present disclosure obtains and decodes a first speech feature of a speech to be recognized, which includes a plurality of speech segment features, to obtain an initial recognition result for the speech to be recognized, and further uses the initial recognition result to extract word-level audio features from the first speech feature, and then decodes the word-level audio features to obtain a final recognition result. Taking the initial recognition result for the speech to be recognized as a priori, extracts and obtains equal-length unified audio feature representations at the word level from unequal-length speech feature information in frame-level audio information, and decodes the word-level audio features to obtain a final recognition result, thereby solving the problem of inconsistent lengths of feature representations of speech framing in the past, and improving the accuracy of speech recognition and computation efficiency.
[0023] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. 1 illustrates a schematic diagram of an example system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. With reference to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.
[0024] In an embodiment of the present disclosure, the server 120 operates to execute one or more services or software applications of the speech recognition method and / or the method of training a deep learning model for speech recognition of the present disclosure. In one exemplary embodiment, a complete speech recognition system or an assembly of a part of a speech recognition system, such as a speech large model, can be deployed on the server.
[0025] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software-as-a-Service (SaaS) model.
[0026] In the configuration shown in FIG. 1, the server 120 may include one or more assemblies that implement the functions performed by the server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof that may be executed on one or more processors. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 may interact with the server 120 using one or more client applications to utilize the services provided by these assemblies. It should be understood that a variety of different system configurations are possible and may differ from the system 100. Thus, FIG. 1 is an example of a system for implementing various methods described herein and is not intended to be limiting.
[0027] A user may input speech to be recognized using client devices 101, 102, 103, 104, 105, and / or 106. The client devices may provide an interface through which a user of the client device interacts with the client device. The client devices may also output information to the user through the interface, such as, for example, outputting speech recognition results to the user. Although only six client devices are shown in FIG. 1, one skilled in the art will appreciate that the present disclosure may support any number of client devices.
[0028] The client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general purpose computers (e.g., personal computers or laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (e.g., GOOGLE Chrome OS), and may include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android, and the like. Portable handheld devices may include mobile phones, intelligent phones, tablets, personal digital assistants (PDAs), and the like. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices may run a variety of applications, such as Internet-related applications, communication applications (eg, email applications), Short Message Service (SMS) applications, and may use a variety of communication protocols.
[0029] Network 110 may be any type of network known to those skilled in the art that may use any one of a number of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.
[0030] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may also include one or more virtual machines running a virtual operating system, or other computing architecture involving virtualization (e.g., one or more flexible pools of virtualized logical storage devices to maintain the server's virtual storage). In various embodiments, server 120 may execute one or more services or software applications that provide the functionality described below.
[0031] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercial server operating system. Server 120 may also run any one of a variety of additional server applications and / or mid-tier applications, including an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.
[0032] In some embodiments, server 120 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.
[0033] In some embodiments, the server 120 may be a server of a distributed system or a server incorporating blockchain. The server 120 may be a cloud server, or an intelligent cloud computing server or an intelligent cloud host equipped with artificial intelligence technology. The cloud server is a host product in a cloud computing service system, which solves the defects of high management difficulty and low business scalability that exist in traditional physical hosts and virtual private server (VPS) services.
[0034] System 100 may include one or more databases 130. In some embodiments, these databases may be used to store data or other information. For example, one or more of databases 130 may be used to store information such as audio files or video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be remote from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of a variety of types. In some embodiments, a database used by server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data from the database and from the database in response to instructions.
[0035] In some embodiments, one or more of databases 130 may be used by an application to store data for the application. The databases used by the application may be various types of databases, such as a key-value repository, an object repository, a general-purpose repository supported by a file system, etc.
[0036] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the application of the various methods and apparatus described in accordance with this disclosure. According to one aspect of the present disclosure, a speech recognition method is provided. As shown in Fig. 2, the speech recognition method includes: step S201, acquiring a first speech feature of a speech to be recognized, the first speech feature including a plurality of speech segment features corresponding to a plurality of speech segments in the speech to be recognized; step S202, using a first decoder to decode the first speech feature and acquire a plurality of first decoding results corresponding to a plurality of words in the speech to be recognized, the first decoding results indicating a first recognition result of the corresponding words; step S203, extracting and acquiring a second speech feature from the first speech feature based on first a priori information, the first a priori information including a plurality of first decoding results, the second speech feature including a plurality of first word-level audio features corresponding to the plurality of words; step S204, using a second decoder to decode the second speech feature and acquire a plurality of second decoding results corresponding to the plurality of words, the second decoding results indicating a second recognition result of the corresponding words.
[0037] In this way, the initial recognition result for the speech to be recognized is taken as a priori, and a unified audio feature representation of equal length at the word level is extracted and obtained from the speech feature information of unequal length in the frame-level audio information, and the word-level audio features are decoded to obtain the final recognition result, thereby solving the problem of inconsistency in the length of the feature representation of speech framing in the conventional technology, thereby improving the accuracy of speech recognition and increasing the calculation efficiency.
[0038] To facilitate the description of the technical concept, the speech to be recognized in the embodiment of the present disclosure includes speech content corresponding to multiple words. In step S201, various existing audio feature extraction methods can be adopted to obtain a first audio feature of the speech to be recognized. The multiple speech segments may be obtained by cutting the speech to be recognized in a fixed length, or by other cutting methods. The multiple speech segment features may correspond one-to-one to the multiple speech segments, and the same speech segment may correspond to multiple speech segment features (as described below), but this is not limited thereto.
[0039] According to some embodiments, as shown in FIG. 3, step S201, obtaining a first speech feature of the speech to be recognized, may include step S301, obtaining original speech features of the speech to be recognized; step S302, determining a plurality of spikes in the speech to be recognized based on the original speech features; and step S303, truncating the original speech features to obtain a plurality of speech segment features that correspond one-to-one to the plurality of spikes.
[0040] Since the spike signal usually corresponds to each word in the speech to be recognized, first, the spike signal of the speech to be recognized is obtained, and then a number of speech segment features that correspond one-to-one to the multiple spikes are obtained based on the spike information, so that the first decoder decodes the first speech feature driven by the spike information to obtain an accurate initial recognition result.
[0041] In step S301, speech features are extracted for a plurality of speech frames included in the speech to be recognized, and original speech features including a plurality of speech frame features can be obtained.
[0042] In step S302, a binary CTC (Connectionist Temporal Classification) module modeled on a Causal Conformer is used to process the original speech features, so as to obtain CTC spike information and determine multiple spikes in the speech to be recognized. As can be understood, multiple spikes in the speech to be recognized can also be determined by other methods, which are not limited here.
[0043] In step S303, the cutting into original speech features may be to cut a plurality of speech frame features corresponding to a plurality of speech frames into a plurality of sets of speech frame features, where each set of speech frame / speech frame features constitutes one speech segment / speech segment feature.
[0044] According to some embodiments, step S303, cutting the original speech features and obtaining a plurality of speech segment features corresponding to the plurality of spikes one-to-one, may include cutting the original speech features according to a preset time length, and taking the speech segment feature of the speech segment in which each of the plurality of spikes exists as the speech segment feature corresponding to the spike, so that the speech segment feature corresponding to each spike has the same length in the above manner. It should be noted that in this manner, if a speech segment contains more than one spike, the speech segment feature of the speech segment will be the speech segment corresponding to each of the spikes at the same time.
[0045] As can be appreciated, the preset time length d can be set according to needs. In the embodiment described in Figure 4, the preset time length d is five audio frames.
[0046] According to some embodiments, step S303, cutting the original speech features and obtaining a plurality of speech segment features corresponding one-to-one to the plurality of spikes, may include cutting the original speech features according to the plurality of spikes, and taking the feature of the speech segment between each two adjacent spikes as the speech segment feature corresponding to one of the spikes, so that in the above manner, the speech segment feature corresponding to each spike contains the complete speech information of the speech segment between two adjacent spikes.
[0047] In some embodiments, the original speech features may be downsampled (eg, convolutional downsampled) before any use is made on them (either by the CTC module or initial speech recognition).
[0048] According to some embodiments, the plurality of speech segment features may be sequentially obtained by streaming truncating the original speech features. Step S202, decoding the first speech feature using the first decoder, may include streaming decoding the plurality of speech segment features sequentially using the first decoder. Thus, by streaming truncating the original speech features and streaming decoding the first speech features, an initial recognition result for the speech to be recognized can be quickly obtained.
[0049] According to some embodiments, the speech segment features can be further encoded using a scheme based on historical feature abstraction, thereby enhancing the descriptive ability of the speech segment features and improving the accuracy of the initial recognition results obtained after decoding the speech segment features. As shown in Fig. 3, step S201, obtaining a first speech feature of the speech to be recognized, can include step S304, obtaining corresponding historical feature abstraction information for the currently obtained speech segment feature, where the historical feature abstraction information is obtained by attention modeling the previous speech segment feature using a first decoding result corresponding to the previous speech segment feature, and step S305, obtaining the corresponding enhanced speech segment feature by encoding the currently obtained speech segment feature in combination with the historical feature abstraction information using a first encoder.
[0050] In some embodiments, the historical feature abstraction information corresponding to the currently obtained speech segment feature includes historical feature abstraction information corresponding to each of a plurality of previous speech segment features, and the historical feature abstraction information of each previous speech segment feature is obtained by attention modeling the previous speech segment feature using a first decoding result corresponding to the previous speech segment feature. In one exemplary embodiment, the first decoding result is used as a query feature Q, and the previous speech segment features are used as key features K and value features V to perform calculation of an attention mechanism, thereby obtaining the historical feature abstraction information of the previous speech segment feature. The calculation process of the attention mechanism can be shown as follows:
[0051]
number
[0052] where d kis the dimension of the feature. It can be understood that the calculation of other feature acquisition and attention mechanism based on the query feature, key feature, and value feature in this disclosure can all refer to this formula. It should be noted that the number of features obtained by this method is the same as the number of features included in the query feature.
[0053] According to some embodiments, step S305, using a first encoder to encode the currently obtained speech segment features in combination with the historical feature abstraction information to obtain corresponding enhanced speech segment features, may include obtaining the corresponding enhanced speech segment features output by the first encoder by using the currently obtained speech segment features as query features Q of the first encoder and the splicing results of the historical feature abstraction information and the currently obtained speech segment features as key features K and value features V of the first encoder.
[0054] This enables the method to fully discover more timing and linguistic relationships in speech features, greatly improving the history abstraction ability of the model, and also improving the accuracy of the decoding results for the enhanced speech segment features.
[0055] In some embodiments, the first encoder and the first decoder together can form a Streaming Multi-Layer Truncated Attention model (SMLTA) model based on historical feature abstraction. As shown in FIG. 4, the Conformer SMLTA model 400 mainly includes two parts, a Streaming Truncated Conformer Encoder 402 (Streaming Truncated Conformer Encoder), i.e., the first encoder, and a Transformer Decoder 404 (Transformer Decoder), i.e., the first decoder. The Streaming Truncated Conformer Encoder includes N stacked Conformer modules, and each Conformer module includes a feedforward module 406, a multi-head self-attention module 408, a convolution module 410, and a feedforward module 412. The Conformer module encodes the speech segment features layer by layer and obtains corresponding hidden features (i.e., the speech segment features after being enhanced). The Transformer decoder contains M stacked Transformer modules, and uses a streaming attention mechanism to screen the hidden features output by the encoder and output a first decoded result that indicates the initial recognition result.
[0056] Figure 4 further illustrates the principle of Conformer SMLTA based on historical feature abstraction. The input original speech features 414 are first cut into speech segment features of the same length, and then the streaming Conformer encoder performs feature encoding on each speech segment feature. The Transformer decoder counts the number of spikes contained in each audio segment according to the spike information 416 of the binary CTC model, and decodes and outputs the recognition result of the current segment according to the number of spikes. Finally, according to the decoding result of the current segment, the hidden features of each layer of the Conformer encoder are modeled with correlation attention to obtain the historical feature abstraction contained in the corresponding speech segment, and the historical feature abstraction information obtained by the abstraction of each layer and the currently obtained speech segment feature are spliced to calculate the next segment.
[0057] According to some embodiments, as shown in FIG. 5 , step S203, extracting and obtaining a second speech feature from the first speech feature based on the first a priori information, may include step S501, for each word of the plurality of words, obtaining a first word-level audio feature corresponding to the word, which is output by the attention module, by taking the first decoding result corresponding to the word as a query feature Q of the attention module, and the first speech feature as a key feature K and a value feature V of the attention module.
[0058] In this way, by using the first decoding results corresponding to each of the multiple words as query features Q and the first speech features as key features K and value features V, the initial recognition results for the speech to be recognized can be effectively used as a priori information, and word-level audio features corresponding to each word can be obtained.
[0059] In some embodiments, the first word-level audio features output by the attention module can be obtained by substituting the corresponding Q, K, and V into the formula of the attention mechanism described above and calculating them.
[0060] According to some embodiments, as shown in FIG. 5 , step S203, extracting and obtaining a second speech feature from the first speech features based on the first a priori information, may include step S502, obtaining an enhanced second speech feature by globally encoding the plurality of first word-level audio features corresponding to the plurality of words using a second encoder.
[0061] In this way, by globally encoding multiple first word-level audio features corresponding to multiple words, the first encoder can effectively compensate for the lack of being able to encode global feature information due to the need to meet streaming recognition, and the description ability of the equal-length unified feature representation is significantly improved.
[0062] In some embodiments, the second encoder may be a conformer encoder that may include N layers of stacked conformer modules. The conformer module simultaneously fuses an attention model and a convolutional model, so that it can effectively model long-range and local relationships in audio features simultaneously, greatly improving the descriptive power of the model.
[0063] As can be understood, extracting and obtaining the second speech feature from the first speech feature based on the first a priori information can be achieved by methods other than the attention mechanism and the conformer encoder, but this is not limited thereto.
[0064] According to some embodiments, step S204, decoding the second audio features using a second decoder to obtain a plurality of second decoded results corresponding to the plurality of words may include, for each word of the plurality of words, obtaining a second decoded result corresponding to the word output by the second decoder by taking the first decoded result corresponding to the word as a query feature Q of the second decoder and the second audio features as a key feature K and a value feature V of the second decoder.
[0065] In this way, by using the first decoded results corresponding to each of the multiple words as query features Q and the second speech features as key features K and value features V, the initial recognition results for the speech to be recognized can be effectively used as a priori information, and the second decoded results corresponding to each word can be obtained.
[0066] In addition, the conventional encoder-decoder structure or decoder-only structure faces the cache loading problem during decoding. Currently, the calculation speed of GPU has improved significantly, but due to the limited by the development of computer hardware resources, the speed at which the decoder loads model parameters into the cache during calculation has not improved significantly, which severely restricts the decoding efficiency of the speech recognition model. In addition, whether the speech recognition model has an encoder-decoder structure or a decoder-only structure, during decoding, both need to rely on the decoding result of the previous point before the next point can be calculated, and this recursive calculation method results in the model needing to be repeatedly loaded into the cache, which causes a certain degree of calculation delay. In particular, with the increase in parameters of the speech large model, the problem of calculation delay caused by the cache loading is even more prominent, and the requirement for the decoding real-time performance of online decoding cannot be met. However, by using the already obtained first decoding results corresponding to each of multiple words as the query features of the second decoder, the final recognition result can be obtained by performing parallel calculations only once, thereby effectively solving the cache load problem faced by large-scale models.
[0067] According to some embodiments, the second decoder may include a forward decoder and a backward decoder, both of which may be configured for each word of the plurality of words to take the first decoding result of the word as the input query feature Q and the second audio features as the input key feature K and value feature V, and the forward decoder may be configured to temporally mask the input features from left to right and the backward decoder may be configured to temporally mask the input features from right to left.
[0068] This enables language modeling in two different directions by installing a forward decoder that time-masks the input features from left to right, and a backward decoder that time-masks the input features from right to left, realizing simultaneous modeling of the linguistic context and further improving the predictive ability of the model.
[0069] In some embodiments, the forward decoder may be referred to as a Left-Right Transformer decoder and the reverse decoder may be referred to as a Right-Left Transformer decoder. Both the forward and reverse decoders may include K stacked time mask Transformer modules.
[0070] According to some embodiments, for each word of a plurality of words, obtaining a second decoded result corresponding to the word output by the second decoder by using the first decoded result of the word as a query feature Q of the second decoder and the second audio features as a key feature K and a value feature V of the second decoder may include fusing a plurality of forward decoded features corresponding to the plurality of words output by the forward decoder and a plurality of backward decoded features corresponding to the plurality of words output by the backward decoder to obtain a plurality of fused features corresponding to the plurality of words; and obtaining a plurality of second decoded results based on the plurality of fused features.
[0071] In some embodiments, the forward decoded features and the backward decoded features can be directly added to obtain the corresponding fused features, which can then be processed, such as Softmax, to obtain the final recognition result.
[0072] After obtaining the second decoding result, the second decoding result can be used as a priori information of the recognition result to re-extract word-level audio features or reuse the second decoder for decoding.
[0073] According to some embodiments, as shown in Fig. 6, the speech recognition method may further include: step S605, for each word of the plurality of words, obtaining an N+1th decoding result corresponding to the word output by the second decoder by taking the Nth decoding result of the word as a query feature Q of the second decoder, and the second speech features as a key feature K and a value feature V of the second decoder, where N is an integer equal to or greater than 2. As can be understood, the operations of steps S601 to S604 in Fig. 6 are similar to those of steps S201 to S204 in Fig. 2, so description thereof will be omitted here.
[0074] Therefore, by using the second decoder to perform multiple iterative decoding, the accuracy rate of speech recognition can be improved. According to some embodiments, as shown in FIG. 7, the speech recognition method includes step S705, extracting and obtaining a third speech feature from the first speech feature based on second a priori information, where the second a priori information includes a plurality of second decoding results, and the third speech feature includes a plurality of second word-level audio features corresponding to a plurality of words; and step S706, decoding the third speech feature using a second decoder to obtain a plurality of third decoding results corresponding to the plurality of words, where the third decoding results indicate a third recognition result of the corresponding word.
[0075] Therefore, the accuracy of speech recognition can be further improved by re-extracting word-level audio features using the second decoding result as a priori for the recognition result, and then using a second decoder to decode the new word-level audio features.
[0076] As can be understood, the operations of steps S701 to S704 in FIG. 7 are similar to the operations of steps S201 to S204 in FIG. 2, and therefore will not be described here.
[0077] According to some embodiments, the second decoder may be a speech large model. The model size of the second decoder can reach billions of parameters, so that the linguistic information contained in the speech can be fully mined and the modeling ability of the model can be greatly improved. In some exemplary embodiments, the parameter amount of the speech large model, which is the second decoder, can be 2B, but can also be other parameter amounts of 1 billion levels or more.
[0078] In some embodiments, the model size of the first decoder (or the model that the first encoder and the first decoder form) may be, for example, hundreds of megabytes. Since its function is to stream out the initial recognition results for the speech to be recognized, large parameters are not required.
[0079] In some embodiments, as shown in FIG. 8 , a first encoder 810 (SMLTA2 Encoder), a first decoder 820 (SMLTA2 Decoder), an attention module 830, a second encoder 840 (Conformer Encoder), and a second decoder 850 (including a forward decoder 860 (Left-Right Transformer Decoder) and a backward decoder 870 (Right-Left Transformer Decoder)) can together constitute an end-to-end speech large model 800.
[0080] According to another aspect of the present disclosure, a method for training a deep learning model for speech recognition is provided. The deep learning model includes a first decoder and a second decoder. As shown in FIG. 9, the training method includes: step S901, obtaining a sample speech and an actual recognition result of a plurality of words in the sample speech; step S902, obtaining a first sample speech feature of the sample speech, where the first sample speech feature includes a plurality of sample speech segment features corresponding to a plurality of sample speech segments in the sample speech; step S903, using the first decoder to decode the first sample speech feature and obtain a plurality of first sample decoding results corresponding to a plurality of words in the sample speech, where the first sample decoding results indicate a first recognition result of the corresponding word; step S904, decoding the first sample speech feature based on the first sample a priori information. 9, the first sample a priori information includes a plurality of first sample decoding results, and the second sample speech features include a plurality of first sample word-level audio features corresponding to a plurality of words; step S905; using a second decoder to decode the second sample speech features and obtain a plurality of second sample decoding results corresponding to the plurality of words, the second sample decoding results indicating second recognition results of the corresponding words; step S906; adjusting parameters of the deep learning model according to the actual recognition results of the plurality of words, the first recognition results and the second recognition results to obtain a trained deep learning model. As can be understood, the operations of steps S902 to S905 in FIG. 9 are similar to the operations of steps S201 to S204 in FIG. 2, and therefore will not be described here.
[0081] As a result, the deep learning model trained by the above method takes the initial recognition result for the speech to be recognized as a priori, extracts and obtains equal-length unified audio feature representations at the word level from the unequal-length speech feature information in the frame-level audio information, and decodes the word-level audio features to obtain the final recognition result, thereby solving the problem of inconsistent lengths of feature representations of speech framing in the conventional method, improving the accuracy of speech recognition and improving computational efficiency.
[0082] In some embodiments, the deep learning model may also include other modules related to the above-mentioned speech recognition method, such as a first encoder, a second encoder, an attention module, etc. For the operation of each module in the deep learning model, the operation of the corresponding module in the above-mentioned speech recognition method may also be referred to.
[0083] In some embodiments, in step S906, a first loss value may be determined based on the actual recognition result and the second recognition result, and the parameters of the deep learning model may be adjusted based on the first loss value. In some embodiments, a second loss value may be determined based on the actual recognition result and the first recognition result, and the parameters of the deep learning model may be adjusted based on the first loss value and the second loss value. In some embodiments, the second loss value may be used to adjust the parameters of the first decoder (and the first encoder), and the first loss value may be used to adjust the parameters of the second decoder (and the attention module, the second encoder), and may be used to adjust the parameters of the deep learning model end-to-end. In addition, some modules of the deep learning model may be individually trained or pre-trained in advance. As can be understood, the parameters of the deep learning model may be adjusted by other manners, and are not limited here.
[0084] As can be appreciated, the speech recognition method described above can be performed using a deep learning model trained according to the training method described above. According to another aspect of the present disclosure, a speech recognition device is provided. As shown in Fig. 10, the device 1000 includes: a speech feature encoding module 1010 configured to obtain a first speech feature of a speech to be recognized, the first speech feature including a plurality of speech segment features corresponding to a plurality of speech segments in the speech to be recognized; a first decoder 1020 configured to decode the first speech feature and obtain a plurality of first decoding results corresponding to a plurality of words in the speech to be recognized, the first decoding results indicating a first recognition result of the corresponding word; a word-level feature extraction module 1030 configured to extract and obtain a second speech feature from the first speech feature based on first a priori information, the first a priori information including the plurality of first decoding results, the second speech feature including a plurality of first word-level audio features corresponding to the plurality of words; and a second decoder 1040 configured to decode the second speech feature and obtain a plurality of second decoding results corresponding to the plurality of words, the second decoding results indicating a second recognition result of the corresponding word. As can be understood, the operations of modules 1010 to 1040 of the apparatus 1000 are similar to the operations of steps S201 to S204 of FIG. 2, so the description will be omitted here.
[0085] According to some embodiments, the audio feature encoding module 1010 can be configured to obtain original audio features of the audio to be recognized, determine a plurality of spikes in the audio to be recognized based on the original audio features, and truncate the original audio features to obtain a plurality of audio segment features that correspond one-to-one to the plurality of spikes.
[0086] According to some embodiments, truncating the original audio features to obtain a plurality of audio segment features that correspond one-to-one to the plurality of spikes may include truncating the original audio features based on a predetermined length of time, and setting the audio segment feature of the audio segment in which each spike among the plurality of spikes is present as the audio segment feature corresponding to the spike.
[0087] According to some embodiments, cutting the original audio features to obtain a plurality of audio segment features corresponding one-to-one to the plurality of spikes may include cutting the original audio features based on the plurality of spikes, and taking the feature of the audio segment between each two adjacent spikes as the audio segment feature corresponding to one of the spikes.
[0088] According to some embodiments, the plurality of speech segment features may be obtained in sequence by streaming truncating the original speech features, and the first decoder may be configured to stream-decode the plurality of speech segment features in sequence.
[0089] According to some embodiments, the speech feature encoding module may be configured to obtain corresponding historical feature abstraction information for the currently obtained speech segment feature, where the historical feature abstraction information is obtained by attention modeling the previous speech segment feature using a first decoding result corresponding to the previous speech segment feature. The speech feature encoding module may include a first encoder configured to encode the currently obtained speech segment feature in combination with the historical feature abstraction information and output a corresponding enhanced speech segment feature.
[0090] According to some embodiments, the first encoder may be configured to receive currently derived speech segment features as first encoder query features, and output corresponding enhanced speech segment features by receiving splicing results of the historical feature abstraction information and the currently derived speech segment features as first encoder key features and value features.
[0091] According to some embodiments, the word-level feature extraction module may include an attention module configured to, for each word of the plurality of words, receive a first decoding result corresponding to the word as a query feature of the attention module, and receive first speech features as key features and value features of the attention module, thereby outputting a first word-level audio feature corresponding to the word.
[0092] According to some embodiments, the word-level feature extraction module may include a second encoder configured to obtain enhanced second speech features by globally encoding a plurality of first word-level audio features corresponding to a plurality of words.
[0093] According to some embodiments, the second decoder may be configured to, for each word of the plurality of words, output a second decoded result corresponding to the word by receiving a first decoded result corresponding to the word as a query feature of the second decoder and receiving second audio features as key features and value features of the second decoder.
[0094] According to some embodiments, the second decoder may include a forward decoder and a reverse decoder, both of which are configured to receive, for each word of the plurality of words, a first decoded result of the word as an input query feature and a second audio feature as an input key feature and value feature, and the forward decoder is configured to temporally mask the input features from left to right and the reverse decoder is configured to temporally mask the input features from right to left.
[0095] According to some embodiments, the second decoder may be configured to fuse a plurality of forward decoding features corresponding to the plurality of words output by the forward decoder and a plurality of backward decoding features corresponding to the plurality of words output by the backward decoder to obtain a plurality of fused features corresponding to the plurality of words, and to obtain a plurality of second decoding results based on the plurality of fused features.
[0096] According to some embodiments, the second decoder may be configured to, for each word of the plurality of words, output an N+1th decoded result corresponding to the word by receiving an Nth decoded result of the word as a query feature of the second decoder and receiving second audio features as key features and value features of the second decoder, where N is an integer greater than or equal to 2.
[0097] According to some embodiments, the word-level feature extraction module may be configured to extract and obtain a third speech feature from the first speech features based on second a priori information, where the second a priori information includes a plurality of second decoding results, where the third speech feature includes a plurality of second word-level audio features corresponding to the plurality of words. The second decoder may be configured to decode the third speech features and obtain a plurality of third decoding results corresponding to the plurality of words, where the third decoding results indicate a third recognition result of the corresponding word.
[0098] According to some embodiments, the second decoder may be a speech large model. According to another aspect of the present disclosure, a training apparatus for a deep learning model for speech recognition is provided. The deep learning model includes a first decoder and a second decoder. As shown in FIG. 11, the training apparatus 1100 includes an acquisition module 1110 configured to acquire an actual recognition result of a sample speech and a plurality of words in the sample speech; an audio feature encoding module 1120 configured to acquire first sample speech features of the sample speech, the first sample speech features including a plurality of sample speech segment features corresponding to a plurality of sample speech segments in the sample speech; a first decoder 1130 configured to decode the first sample speech features and acquire a plurality of first sample decoding results corresponding to a plurality of words in the sample speech, the first sample decoding results indicating a first recognition result of the corresponding word; and a second decoder 1140 configured to select a second sub-sub ... The device 1100 further includes a word-level feature extraction module 1140 configured to extract and obtain sample speech features, where the first sample a priori information includes a plurality of first sample decoding results, and the second sample speech features include a plurality of first sample word-level audio features corresponding to a plurality of words; a second decoder 1150 configured to decode the second sample speech features and obtain a plurality of second sample decoding results corresponding to the plurality of words, where the second sample decoding results indicate second recognition results of the corresponding words; and a parameter adjustment module 1160 configured to adjust parameters of a deep learning model based on the actual recognition results of the plurality of words, the first recognition results and the second recognition results to obtain a trained deep learning model. As can be understood, the operations of the modules 1110 to 1160 of the device 1100 are similar to those of steps S901 to S906 of FIG. 9, and therefore will not be described here.
[0099] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of relevant user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and morals.
[0100] According to embodiments of the present disclosure, an electronic device, a readable storage medium, and a computer program product are further provided. Referring to FIG. 12, a block diagram of an electronic device 1200 that can operate as a server or client of the present disclosure, which is an example of a hardware device applicable to each aspect of the present disclosure, will be described. The electronic device represents various forms of digital electronic computing devices, such as laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, mobile phones, intelligent phones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.
[0101] 12, the electronic device 1200 includes a computing unit 1201, which can perform various suitable operations and processes according to a computer program stored in a read-only memory (ROM) 1202 or loaded from a storage unit 1208 into a random access memory (RAM) 1203. The RAM 1203 may further store various programs and data necessary for operating the electronic device 1200. The computing unit 1201, the ROM 1202 and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.
[0102] A number of components in the electronic device 1200, such as an input unit 1206, an output unit 1207, a storage unit 1208, and a communication unit 1209, are connected to an input / output (I / O) interface 1205. The input unit 1206 may be any type of device capable of inputting information to the electronic device 1200, and the input unit 1206 may generate input numeric or character information and key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track board, a track ball, a control lever, a microphone, and / or a remote control. The output unit 1207 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1208 may include, but is not limited to, a magnetic disk, an optical disk. The communications unit 1209 enables the electronic device 1200 to exchange information / data with other devices via a computer network, e.g., the Internet, and / or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communications device, a wireless communications transceiver, and / or a chipset, e.g., a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communications device, and / or the like.
[0103] The computing unit 1201 may be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 1201 may include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 executes each of the methods and processes described in the preceding paragraph, such as the speech recognition method and / or the method of training a deep learning model for speech recognition. For example, in some embodiments, the speech recognition method and / or the method of training a deep learning model for speech recognition may be implemented as a computer software program tangibly included in a machine-readable medium, such as the storage unit 1208. In some embodiments, some or all of the computer programs may be loaded and / or installed in the electronic device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, it may perform one or more steps of the speech recognition method and / or the method for training a deep learning model for speech recognition described above. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the speech recognition method and / or the method for training a deep learning model for speech recognition in any other suitable manner (e.g., by firmware).
[0104] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), software hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may receive data and instructions from a storage system, at least one input device, at least one output device, and may transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0105] The program code implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, so that when executed by the processor or controller, the program code performs the functions / operations specified in the flow charts and / or block diagrams. The program code may be entirely executed on a machine, partly executed on a machine, partly executed on a machine as a separate software package and partly executed on a remote machine, or entirely executed on a remote machine or server.
[0106] In the context of the present disclosure, a machine-readable medium may be a tangible medium, which may include or store a program for use in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection with one or more leads, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0107] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer that includes a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user may provide input to the computer. Other types of devices can be used to provide for interaction with a user, for example, where feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and input from the user can be received in any form (including acoustic input, speech input, or tactile input).
[0108] The systems and techniques described herein may be implemented in a computing system including a backstage component (e.g., as a data server), a middleware component (e.g., an application server), a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique embodiment), or any combination of the backstage component, middleware component, or front-end component. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include, for example, a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0109] The computer system may include a client side and a server. The client side and the server are generally remote from each other and usually interact with each other via a communication network. The relationship between the client side and the server is generated by running computer programs having a client side-server relationship with each other on corresponding computers. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0110] It should be understood that the above-mentioned various forms of flow may be used, and steps may be rearranged, added, or removed. For example, each step described in the present disclosure may be performed in parallel, sequentially, or in a different order, and the present disclosure is not limited thereto, as long as the technical solution disclosed in the present disclosure can achieve the desired results.
[0111] Although the embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above methods, systems, and devices are merely illustrative embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the scope of the claims and their equivalents after grant. Various elements of the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in a different order from the order described in this disclosure. Furthermore, various elements of the embodiments or examples may be combined in various ways. It is important to note that with the evolution of technology, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.
Claims
1. 1. A method for speech recognition, comprising: obtaining first speech features of a speech to be recognized, the first speech features including a plurality of speech segment features corresponding to a plurality of speech segments in the speech to be recognized; Decoding the first speech feature using a first decoder to obtain a plurality of first decoding results corresponding to a plurality of words in the speech to be recognized, the first decoding results indicating first recognition results of the corresponding words; Extracting and acquiring second speech features from the first speech features based on first a priori information, the first a priori information including the first decoding results, and the second speech features including a plurality of first word level audio features corresponding to the plurality of words; and decoding the second speech features using a second decoder to obtain a plurality of second decoding results corresponding to the plurality of words, the second decoding results indicating second recognition results for the corresponding words.
2. Extracting and acquiring a second speech feature from the first speech feature based on first a priori information, 2. The method of claim 1, comprising: for each word of the plurality of words, obtaining a first word-level audio feature corresponding to the word output by the attention module by taking a first decoding result corresponding to the word as a query feature of an attention module and the first speech feature as a key feature and a value feature of the attention module.
3. Extracting and acquiring a second speech feature from the first speech feature based on first a priori information, 3. The method of claim 2, further comprising: utilizing a second encoder to globally encode a plurality of first word-level audio features corresponding to the plurality of words to obtain enhanced second speech features.
4. Decoding the second speech feature using a second decoder to obtain a plurality of second decoding results corresponding to the plurality of words includes:
4. A method according to any one of claims 1 to 3, comprising, for each word of the plurality of words, obtaining a second decoded result corresponding to the word, which is output by the second decoder, by taking the first decoded result corresponding to the word as a query feature of the second decoder and the second audio features as key features and value features of the second decoder.
5. 5. The method of claim 4, wherein the second decoder comprises a forward decoder and a backward decoder, both of which are configured, for each word of the plurality of words, to take a first decoding result of the word as an input query feature and the second audio features as input key features and value features, the forward decoder is configured to temporally mask the input features from left to right, and the backward decoder is configured to temporally mask the input features from right to left.
6. obtaining, for each word of the plurality of words, a second decoded result corresponding to the word output by the second decoder by using the first decoded result of the word as a query feature of the second decoder and the second speech features as key features and value features of the second decoder; fusing a plurality of forward decoded features corresponding to the plurality of words output by the forward decoder and a plurality of backward decoded features corresponding to the plurality of words output by the backward decoder to obtain a plurality of fused features corresponding to the plurality of words; and obtaining the second decoding results based on the fusion features.
7. 5. The method of claim 4, further comprising: for each word of the plurality of words, obtaining an N+1-th decoded result corresponding to the word output by the second decoder by taking the N-th decoded result of the word as a query feature of the second decoder and the second audio features as key features and value features of the second decoder, where N is an integer greater than or equal to 2.
8. Extracting third speech features from the first speech features based on second a priori information, the second a priori information including the second decoding results, and the third speech features including a plurality of second word level audio features corresponding to the words; 4. The method of claim 1, further comprising: utilizing the second decoder to decode the third speech features to obtain a plurality of third decoding results corresponding to the plurality of words, the third decoding results indicating third recognition results for corresponding words.
9. Obtaining a first speech feature of a speech to be recognized includes: obtaining original speech features of the speech to be recognized; determining a plurality of spikes in the speech to be recognized based on the original speech features; and truncating the original audio features to obtain the audio segment features that correspond one-to-one to the spikes.
10. The plurality of speech segment features are obtained in turn by streaming truncating the original speech features, and decoding the first speech features using a first decoder includes:
10. The method of claim 9, further comprising utilizing the first decoder to stream-decode the plurality of audio segment features in sequence.
11. Obtaining a first speech feature of a speech to be recognized includes: obtaining corresponding historical feature abstraction information for a currently obtained speech segment feature, the historical feature abstraction information being obtained by attention modeling the previous speech segment feature using a first decoding result corresponding to the previous speech segment feature; and encoding the currently derived speech segment feature in combination with the historical feature abstraction information using a first encoder to obtain a corresponding enhanced speech segment feature.
12. obtaining corresponding enhanced speech segment features by encoding the currently obtained speech segment features in combination with the historical feature abstraction information using a first encoder; The method of claim 11, further comprising obtaining the corresponding enhanced speech segment features output by the first encoder by taking the currently obtained speech segment features as query features of the first encoder and taking splicing results of the historical feature abstraction information and the currently obtained speech segment features as key features and value features of the first encoder.
13. truncating the original speech features to obtain the speech segment features that correspond one-to-one to the spikes; 10. The method of claim 9, further comprising: truncating the original speech features based on a predetermined length of time; and taking the speech segment feature of the speech segment in which each spike of the plurality of spikes exists as the speech segment feature corresponding to the spike.
14. truncating the original speech features to obtain the speech segment features that correspond one-to-one to the spikes; 10. The method of claim 9, further comprising: cutting the original audio feature based on the spikes; and taking the audio segment feature between each two adjacent spikes as the audio segment feature corresponding to one of the spikes.
15. The method according to any one of claims 1 to 3, wherein the second decoder is a speech large model.
16. 1. A method for training a deep learning model for speech recognition, the deep learning model including a first decoder and a second decoder, the training method comprising: obtaining a sample speech and an actual recognition result of a plurality of words in the sample speech; obtaining first sample speech features of the sample speech, the first sample speech features including a plurality of sample speech segment features corresponding to a plurality of sample speech segments in the sample speech; utilizing a first decoder to decode the first sample speech features to obtain a plurality of first sample decoding results corresponding to a plurality of words in the sample speech, the first sample decoding results indicating first recognition results for the corresponding words; Extracting and obtaining second sample speech features from the first sample speech features based on first sample a priori information, the first sample a priori information including the first sample decoding results, and the second sample speech features including a plurality of first sample word level audio features corresponding to the plurality of words; utilizing a second decoder to decode the second sample speech features to obtain a plurality of second sample decoding results corresponding to the plurality of words, the second sample decoding results indicating second recognition results for the corresponding words; and adjusting parameters of the deep learning model based on the actual recognition results of the plurality of words, the first recognition result and the second recognition result to obtain a trained deep learning model.
17. A speech recognition device, comprising: a speech feature encoding module configured to obtain first speech features of a speech to be recognized, the first speech features including a plurality of speech segment features corresponding to a plurality of speech segments in the speech to be recognized; a first decoder configured to decode the first speech features to obtain a plurality of first decoding results corresponding to a plurality of words in the speech to be recognized, the first decoding results indicating first recognition results for the corresponding words; a word-level feature extraction module configured to extract second speech features from the first speech features based on first a priori information, the first a priori information including the first decoding results, and the second speech features including a plurality of first word-level audio features corresponding to the plurality of words; a second decoder configured to decode the second speech features to obtain a plurality of second decoding results corresponding to the plurality of words, the second decoding results indicating second recognition results for the corresponding words.
18. The word level feature extraction module:
20. The apparatus of claim 17, further comprising an attention module configured to, for each word of the plurality of words, output a first word-level audio feature corresponding to the word by receiving a first decoding result corresponding to the word as a query feature of the attention module and receiving the first speech feature as a key feature and a value feature of the attention module.
19. The word level feature extraction module:
20. The apparatus of claim 18, comprising: a second encoder configured to globally encode a plurality of first word-level audio features corresponding to the plurality of words to obtain enhanced second speech features.
20. 20. The apparatus of claim 17, wherein the second decoder is configured to, for each word of the plurality of words, output a second decoded result corresponding to the word by receiving a first decoded result corresponding to the word as a query feature of the second decoder and receiving the second audio features as key features and value features of the second decoder.
21. 21. The apparatus of claim 20, wherein the second decoder comprises a forward decoder and a reverse decoder, both of which are configured to receive, for each word of the plurality of words, a first decoded result of the word as input query features and the second audio features as input key features and value features, the forward decoder configured to temporally mask the input features from left to right and the reverse decoder configured to temporally mask the input features from right to left.
22. The second decoder comprises: Fusing a plurality of forward decoded features corresponding to the plurality of words output by the forward decoder and a plurality of backward decoded features corresponding to the plurality of words output by the backward decoder to obtain a plurality of fused features corresponding to the plurality of words; The apparatus of claim 21 , configured to obtain the second decoding results based on the fusion features.
23. The second decoder comprises:
21. The apparatus of claim 20, configured to, for each word of the plurality of words, receive an Nth decoding result of the word as a query feature of the second decoder, and receive the second audio features as key features and value features of the second decoder, thereby outputting an N+1th decoding result corresponding to the word, where N is an integer greater than or equal to 2.
24. The word-level feature extraction module is configured to extract third speech features from the first speech features based on second a priori information, the second a priori information including the second decoding results, and the third speech features including a plurality of second word-level audio features corresponding to the words; 20. The apparatus of claim 17, wherein the second decoder is configured to decode the third speech features to obtain a plurality of third decoding results corresponding to the plurality of words, the third decoding results indicating third recognition results for corresponding words.
25. The speech feature encoding module includes: Obtaining original speech features of the speech to be recognized; determining a plurality of spikes in the speech to be recognized based on the original speech features; 20. An apparatus according to any one of claims 17 to 19, configured to truncate the original audio features to obtain the audio segment features in one-to-one correspondence with the spikes.
26. 26. The apparatus of claim 25, wherein the plurality of speech segment features are obtained in sequence by streaming truncating the original speech features, and the first decoder is configured to stream-decode the plurality of speech segment features in sequence.
27. The speech feature encoding module includes: For a currently obtained speech segment feature, a corresponding historical feature abstraction information is obtained, the historical feature abstraction information being obtained by attention modeling the previous speech segment feature using a first decoding result corresponding to the previous speech segment feature; wherein the speech feature encoding module:
27. The apparatus of claim 26, comprising a first encoder configured to output a corresponding enhanced speech segment feature by encoding a currently derived speech segment feature in combination with the historical feature abstraction information.
28. The first encoder comprises:
28. The apparatus of claim 27, configured to output the corresponding enhanced speech segment features by receiving the currently obtained speech segment features as query features of the first encoder and receiving splicing results of the historical feature abstraction information and the currently obtained speech segment features as key features and value features of the first encoder.
29. truncating the original speech features to obtain the speech segment features that correspond one-to-one to the spikes; 26. The apparatus of claim 25, further comprising: truncating the original audio features based on a predetermined length of time; and determining an audio segment feature of a audio segment in which each spike of the plurality of spikes exists as the audio segment feature corresponding to the spike.
30. truncating the original speech features to obtain the speech segment features that correspond one-to-one to the spikes; 26. The apparatus of claim 25, further comprising: truncating the original audio features based on the spikes; and taking the audio segment feature between each two adjacent spikes as the audio segment feature corresponding to one of the spikes.
31. Apparatus according to any one of claims 17 to 19, wherein the second decoder is a speech large model.
32. 1. An apparatus for training a deep learning model for speech recognition, the deep learning model including a first decoder and a second decoder, the training apparatus comprising: a capture module configured to capture a sample speech and an actual recognition result of a plurality of words in the sample speech; a speech feature encoding module configured to obtain first sample speech features of the sample speech, the first sample speech features including a plurality of sample speech segment features corresponding to a plurality of sample speech segments in the sample speech; a first decoder configured to decode the first sample speech features to obtain a plurality of first sample decoding results corresponding to a plurality of words in the sample speech, the first sample decoding results indicating first recognition results for the corresponding words; A word-level feature extraction module configured to extract second sample speech features from the first sample speech features based on first sample a priori information, the first sample a priori information including the first sample decoding results, and the second sample speech features including a plurality of first sample word-level audio features corresponding to the plurality of words; a second decoder configured to decode the second sample speech features to obtain a plurality of second sample decoding results corresponding to the plurality of words, the second sample decoding results indicative of second recognition results for corresponding words; and a parameter adjustment module configured to adjust parameters of the deep learning model based on the actual recognition results of the plurality of words, the first recognition result and the second recognition result, to obtain a trained deep learning model.
33. An electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor, wherein: An electronic device, wherein the memory stores instructions executable by at least one processor, the instructions being executed by the at least one processor such that the at least one processor can perform a method according to any one of claims 1 to 16.
34. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to carry out the method of any one of claims 1 to 16.
35. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 16.
Citation Information
Patent Citations
System and method for end-to-end speech recognition with trigger door tension
JP2022522379A
Speech recognition method, codec method and apparatus, electronic device and storage medium
JP2023041610A
Derivation Model-Based Two-Pass End-to-End Speech Recognition
JP2023041867A
Predicting Word Boundaries for On-Device Batching of End-To-End Speech Recognition Models
US20230107493A1
Cited By
Data processing method, data processing device, electronic equipment and storage medium
CN121096313A
