Voice activity detection integration for improved automatic speech detection

Abstract

The present disclosure relates to voice activity detection integration to improve automatic speech detection. Systems, computer-implemented methods, and computer program products are provided for facilitating multi-task training of a recurrent neural network transducer (RNN-T) using automatic speech recognition (ASR) information. According to embodiments, a system can include a memory that stores computer executable components and a processor that executes the computer executable components stored in the memory. The computer executable components can include a RNN-T that can receive ASR information. The computer executable components can include a voice activity detection (VAD) model that trains the RNN-T using the ASR information, where the RNN-T can further include an encoder and a joint network. One or more outputs of the encoder can be integrated with one or more outputs of the joint network and the VAD model.

Description

Background Technology

[0001] One or more embodiments in this document generally relate to ASR systems typically deployed with voice activity detection (VAD) models to run automatic speech recognition (ASR), and more specifically to multi-task training of recurrent neural network transducers (RNN-T) that receive ASR information. Summary of the Invention

[0002] The following overview is presented to provide a basic understanding of one or more embodiments described herein. This overview is not intended to identify key or important elements, or to depict any scope of a particular embodiment or any scope of the claims. The sole purpose of this summary is to present concepts in a simplified form as a prelude to the more detailed description that follows. In one or more embodiments described herein, devices, systems, computer-implemented methods, apparatuses, and / or computer program products are described that can facilitate updates (such as training) to one or more statistical sets.

[0003] According to an embodiment, the system may include a memory storing computer-executable components and a processor executing the computer-executable components stored in the memory. The computer-executable components may include a recurrent neural network transducer (RNN-T) that receives automatic speech recognition information via multi-task training. The RNN-T may include an encoder, wherein one or more outputs of the encoder may be integrated with a joint network of the RNN-T. Further, the one or more outputs of the encoder may be integrated with the one or more outputs of a voice activity detection (VAD) model.

[0004] According to another embodiment, a computer-implemented method may include receiving Automatic Speech Recognition (ASR) information by a system operatively coupled to a processor. The computer-implemented method may also include multi-task training of an RNN-T by the processor using the ASR information. The computer-implemented method may involve the processor integrating one or more outputs of the RNN-T's encoder with a joint network of the RNN-T and / or a VAD model to improve noise robustness.

[0005] According to another embodiment, a computer program product for training an RNN-T using a VAD model may include a computer-readable storage medium having program instructions embodied therein, which, when executed by a processor, cause the processor to receive ASR information. The program product enables the processor to use the ASR information to multitask train the RNN-T and / or integrate the encoder of the RNN-T with one or more outputs of the joint network of the RNN-T and the VAD model. Attached Figure Description

[0006] Figure 1A block diagram of an exemplary, non-limiting system that facilitates the integration of VAD information into ASR according to one or more embodiments described herein is shown.

[0007] Figure 2A A block diagram of an exemplary, non-limiting system that can facilitate the integration of VAD information into ASR according to one or more embodiments described herein is shown.

[0008] Figure 2B A block diagram of another example non-limiting system that facilitates the integration of VAD information into ASR according to one or more embodiments described herein is shown.

[0009] Figure 3A A block diagram of another example non-limiting system that facilitates the integration of VAD information into ASR according to one or more embodiments described herein is shown.

[0010] Figure 3B A block diagram of another example non-limiting system that facilitates the integration of VAD information into ASR according to one or more embodiments described herein is shown.

[0011] Figure 3C A block diagram of another example non-limiting system that facilitates the integration of VAD information into ASR according to one or more embodiments described herein is shown.

[0012] Figure 3D A block diagram of another example non-limiting system that facilitates the integration of VAD information into ASR according to one or more embodiments described herein is shown.

[0013] Figure 4 Various speech signal diagrams of example non-limiting systems, as inputs to RNN-T and / or VAD models according to one or more embodiments described herein, are shown.

[0014] Figure 5A A flowchart of an example non-limiting system that facilitates the integration of VAD information with RNN-T according to one or more embodiments described herein is shown.

[0015] Figure 5B A flowchart of an example non-limiting system that facilitates the integration of VAD information with RNN-T according to one or more embodiments described herein is shown.

[0016] Figure 6 A flowchart is shown of an exemplary, non-limiting computer implementation of a method for facilitating the integration of VAD information into ASR according to one or more embodiments described herein.

[0017] Figure 7A table is shown that illustrates experimental results of an exemplary, non-limiting computer implementation of a method for facilitating the integration of VAD information into ASR, according to one or more embodiments described herein.

[0018] Figure 8A Several tables are shown, illustrating experimental results of exemplary, non-limiting computer-implemented methods for facilitating the integration of VAD information into ASR according to one or more embodiments described herein.

[0019] Figure 8B Several tables are shown illustrating experimental results of exemplary, non-limiting computer-implemented methods for facilitating the integration of VAD information into ASR according to one or more embodiments described herein.

[0020] Figure 8C Several tables are shown, illustrating experimental results of exemplary, non-limiting computer-implemented methods for facilitating the integration of VAD information into ASR according to one or more embodiments described herein.

[0021] Figure 8D Several tables are shown, illustrating experimental results of exemplary, non-limiting computer-implemented methods for facilitating the integration of VAD information into ASR according to one or more embodiments described herein.

[0022] Figure 9 A block diagram is shown that illustrates an example non-limiting operating environment that can facilitate one or more embodiments described herein.

[0023] Figure 10 A block diagram of an example, non-limiting cloud computing environment according to one or more embodiments described herein is shown.

[0024] Figure 11 Block diagrams of several examples, non-limiting abstract model layers according to one or more embodiments described herein are shown. Detailed Implementation

[0025] The following detailed description is illustrative only and is not intended to limit the embodiments and / or their application or use. Furthermore, one is not intended to be bound by any express or implied information presented in the preceding background or overview section or this detailed description section.

[0026] Referring to Automatic Speech Recognition (ASR) systems, it will be understood that ASR systems can be deployed alongside Voice Activity Detection (VAD) systems to operate ASR on acoustic signals containing speech. ASR systems maintain performance by removing unwanted non-speech segments from the input audio signal during inference; however, errors can propagate if the VAD system fails to properly separate speech from non-speech segments. ASR systems are typically constructed using segmented speech utterances. Therefore, unexpected errors can occur when VAD-segmented utterances contain long non-speech segments and / or consist entirely of non-speech. VAD systems are more likely to fail in noisy environments and / or in unknown acoustic domains, which more significantly triggers insertion errors in ASR.

[0027] Furthermore, referring to ASR systems, it will be understood that ASR systems have been widely used in human-computer communication. Recent ASR systems with end-to-end neural network architectures have been implemented with significantly lower development costs than conventional hybrid ASR systems. In well-known neural ASR systems, recurrent neural network transducers (RNN-T) are widely used due to their lower computational cost compared to systems designed on other architectures, offering competitive performance and the ability to recognize speech online. ASR can be paired with VAD systems, which extract the actual speech portion from the input audio signal by removing non-speech portions before the ASR decoding process begins. Recently, neural network-based VADs have been implemented to capture the unique properties of speech under varying noise conditions. By combining multi-task learning methods with feature integration architectures, the system can achieve a relative improvement of up to 10% under very low signal-to-noise ratio (SNR) conditions compared to systems simply trained on mixed data consisting of speech and long non-speech segments.

[0028] VAD models support speech recognition in realistic scenarios where speakers are speaking at different times and locations; however, serious recognition errors can occur if VAD fails to segment spoken and non-speaking segments in the input audio. For example, if the VAD system identifies a non-speaking audio segment as speech, ASR attempts to output text from an empty speech input. ASR systems are typically trained with well-segmented speech data (e.g., data that may include short silent regions before and / or after actual speech segments, and / or long silent regions that may have been removed beforehand from the training data), leading to errors in speech detection and / or determination. Therefore, ASR accuracy can deteriorate under noisy conditions where VAD execution is inaccurate.

[0029] Previous instances of integrating end-to-end ASR and VAD to improve speech segmentation and recognition performance have primarily focused on ASR in unsegmented long audio under clean conditions. These instances include: a) Connectionist Mechanism Temporal Classification (CTC)-based ASR with a VAD task, where speech can be segmented by assuming blank markers from the CTC soft-max output as speech boundaries; b) multi-task learning frameworks for ASR and VAD, where ASR and VAD share a common layer for extracting latent representations from raw waveform inputs (e.g., further optimized via ASR and VAD criteria); and / or c) multi-task learning between audio-visual ASR and VAD for noisy speech inputs to leverage visual information.

[0030] By at least partially addressing these issues, explicitly utilizing VAD information in training RNN-T-based ASR can improve robustness to speech recognition under noisy conditions. Improving VAD accuracy is a common approach to reducing weaknesses in ASR caused by speech segmentation failures under noisy conditions.

[0031] In view of these problems, one or more embodiments described herein can be implemented to produce solutions to one or more of these problems in the form of systems, computer-implemented methods, and / or computer program products that can facilitate the processes of: a) receiving Automatic Speech Recognition (ASR) information by a processor; b) performing multi-task training of the RNN-T using the ASR information by the processor; and / or c) integrating the encoder of the RNN-T with one or more outputs of the joint network of the RNN-T and the VAD model by the processor to improve the robustness of speech recognition under noisy conditions. That is, the embodiments described herein include one or more systems, computer-implemented methods, apparatuses, and / or computer program products that can facilitate one or more of the foregoing processes.

[0032] Furthermore, improving the accuracy of VADs has become a common approach to improving the robustness of ASRs. However, one or more embodiments described herein improve the robustness of ASRs against VAD errors. That is, the embodiments described herein include one or more systems, computer-implemented methods, apparatuses, and / or computer program products that can integrate VAD information into ASRs via feature-level integration and / or multi-task learning.

[0033] Previously, multi-task learning in RNN-T has been proposed, where keyword localization and language modeling are subtasks of rare word recognition. To improve ASR performance in noisy environments, multi-task learning jointly minimizes the ASR RNN-T loss and VAD error when predicting frame-level speech / non-speech tags generated by a convolutional neural network (CNN)-based VAD system using spectral time.

[0034] Furthermore, untranscribed audio data within a multi-task learning framework can be used as auxiliary training data for VAD components via knowledge distillation. Typically, preparing transcribed data for ASR training covering a large number of speech domains is very expensive; therefore, the amount of training data available for ASR is usually limited. Further, in embodiments, VAD systems trained using data from different environmental domains perform well under varying acoustic conditions. The VAD model contains information representing the surrounding acoustic environment that can be used to improve the robustness of the ASR model (e.g., information representing the surrounding acoustic environment to distinguish between speaking and non-speaking). In one or more embodiments described herein, the system can extract rich acoustic environmental knowledge contained in a well-trained VAD model into the ASR network without using additional transcribed data to improve ASR robustness; for example, experimental results from English telephone conversations with long silent portions indicate that by using auxiliary VAD training data, the system improves ASR noise robustness in new acoustic domains.

[0035] One or more embodiments will now be described with reference to the accompanying drawings, wherein like reference numerals are used throughout to refer to like elements. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a more thorough understanding of one or more embodiments. However, it will be apparent in different circumstances that one or more embodiments may be practiced without these specific details.

[0036] Figure 1 A block diagram of an example non-limiting system 100 including an RNN-T comprising a joint network component 106, an encoder component 108, and / or a predictor component 110 is shown. The VAD integration system 100 may integrate the encoder component 108 of an RNN-T 102 (e.g., an ASR RNN-T) with the joint network 106 of the RNN-T 102 and one or more outputs of a VAD model 104 to improve robustness to noise. For brevity, repeated descriptions of similar elements employed in other embodiments described herein are omitted. Aspects of systems (e.g., VAD integration system 100, etc.), devices, or processes in various embodiments of the invention may constitute one or more machine-executable components embodied within one or more machines (e.g., embodied in one or more computer-readable media (or media) associated with one or more machines). Such components, when executed by one or more machines (e.g., computers, computing devices, virtual machines, combinations thereof, and / or the like), cause the machines to perform the described operations.

[0037] Now turn specifically to one or more figures, and first turn to... Figure 1This figure illustrates a block diagram of an example non-limiting system 100 that facilitates the reception of Automatic Speech Recognition (ASR) information by a processor 124 according to one or more embodiments described herein. The VAD integration system 100 may include components compatible with a cloud computing environment 1000 (…). Figure 10 The associated recurrent neural network transducer (RNN-T) 102 and / or voice activity detection (VAD) model 104. For example, but not limited to, the RNN-T 102 may be an RNN-T 102 configured for automatic speech recognition. The RNN-T may further include a predictor component 110 connected to a joint network component 106 (e.g., a joint network). The VAD integration system 100 may include one or more RNN-T 102s, one or more VAD models 104, one or more networks 130, one or more input devices 132, and / or one or more computer applications 134. The RNN-T 102 and / or VAD model 104 may be connected to one or more machines included in the VAD integration system 100. As used herein, the one or more machines may include one or more of the following: computing devices, general-purpose computers, special-purpose computers, quantum computing devices (e.g., quantum computers), tablet computing devices, handheld devices, server-type computing machines and / or databases, laptop computers, notebook computers, desktop computers, cellular phones, smartphones, consumer appliances and / or instruments, industrial and / or commercial equipment, digital assistants, multimedia internet-enabled telephones and / or another type of device.

[0038] In some examples, the VAD integrated system 100 may be compatible with the following (see below) Figure 10 The cloud computing environment described is 1000 and / or see below. Figure 11 One or more functional abstraction layers (e.g., hardware and software layer 1160, virtualization layer 1170, management layer 1180, and / or workload layer 1190) are described in relation to each other.

[0039] The VAD integrated system 100 and / or its components (e.g., RNN-T 102 and / or VAD model 104) may be implemented as described below. Figure 10 And / or refer to the following text. Figure 11The one or more functional abstraction layers (e.g., quantum software and / or the like) described herein describe one or more computing resources of a cloud computing environment to perform one or more operations according to one or more embodiments described herein. For example, one or more of the cloud computing environment and / or functional abstraction layers 1160, 1170, 1180 and / or 1190 may include one or more classical computing devices (e.g., classical computers, classical processors, virtual machines, servers and / or the like), quantum hardware and / or quantum software (e.g., quantum computing devices, quantum computers, quantum processors, quantum circuit simulation software, superconducting circuits and / or the like), which may be used by the system and / or its components to perform one or more operations according to one or more embodiments described herein. For example, the system and / or its components may employ one or more classical and / or quantum computing resources to perform one or more classical and / or quantum: mathematical functions, calculations and / or equations; calculation and / or processing scripts; algorithms; models (e.g., artificial intelligence (AI) models, machine learning (ML) models and / or similar models); and / or another operation according to one or more embodiments described herein.

[0040] It should be understood that while one or more embodiments described herein include a detailed description of cloud computing, the implementation of the teachings stated herein is not limited to a cloud computing environment. Rather, one or more embodiments described herein can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0041] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0042] The features are as follows:

[0043] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0044] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0045] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but are able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0046] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0047] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0048] The service model is as follows:

[0049] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0050] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0051] Infrastructure as a Service (IaaS): This provides consumers with the capability to offer processing, storage, networking, and / or other basic computing resources, where consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and / or potentially limited control over selected networking components (e.g., host firewalls).

[0052] The deployment model is as follows:

[0053] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0054] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and / or compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0055] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0056] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0057] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and / or semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0058] Furthermore, the VAD integrated system 100 can be associated with or included in data analysis systems, data processing systems, graph analysis systems, graph processing systems, big data systems, social networking systems, speech recognition systems, image recognition systems, graphics modeling systems, bioinformatics systems, data compression systems, artificial intelligence systems, authentication systems, syntactic pattern recognition systems, medical systems, health monitoring systems, network systems, computer network systems, communication systems, router systems, server systems, high-availability server systems (e.g., telecommunications server systems), web server systems, file server systems, data server systems, disk array systems, power strip systems, cloud-based systems, etc. Accordingly, the VAD integrated system 100 can be used to solve problems that are inherently highly technical, non-abstract, and / or cannot be performed by humans as a set of mental actions, using hardware and / or software.

[0059] Turning now to aspects of the VAD integrated system 100, the system 100 may include one or more RNN-T 102, one or more VAD models 104, one or more system buses 120, one or more memory / storage components 122, one or more processors 124, one or more networks 130, one or more input devices 132 and / or one or more computer applications 134.

[0060] It should be understood that the embodiments depicted in the various figures disclosed herein are for illustrative purposes only, and therefore, the architecture of the embodiments is not limited to the systems, devices, and / or components depicted therein, nor to any particular order, connection, and / or coupling of the systems, devices, and / or components depicted therein. For example, in some embodiments, the VAD integrated system 100 may further include the operating environment 900 referenced herein and Figure 9 The various computers and / or computing-based components described herein. In several embodiments, the computers and / or computing-based components may be combined to achieve integration. Figure 1 Or used in conjunction with one or more of the systems, apparatuses, components and / or computer-implemented operations shown and described in the other accompanying drawings disclosed herein.

[0061] Memory 122 may store one or more computer and / or machine-readable, writable, and / or executable components and / or instructions that, when executed by processor 124 (e.g., a classical processor, a quantum processor, and / or a similar processor), facilitate the execution of operations defined by this or these executable components and / or instructions. For example, memory 122 may store computer and / or machine-readable, writable, and / or executable components and / or instructions that, when executed by processor 124, facilitate the execution of various functions described herein related to RNN-T 102, VAD model 104, and / or another component associated with VAD integrated system 100, as described herein with or without reference to the various accompanying drawings of one or more embodiments.

[0062] Memory 122 may include volatile memory (e.g., random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), etc.) and / or non-volatile memory (e.g., read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), etc.) employing one or more memory architectures. Reference is made below to system memory 906 and... Figure 9 Other examples of memory 122 are described. These examples of memory 122 can be used to implement any one or more embodiments described herein.

[0063] Processor 124 may include one or more types of processors and / or electronic circuitry (e.g., classical processors, quantum processors, and / or similar processors) that can implement one or more computer- and / or machine-readable, writable, and / or executable components and / or instructions that can be stored in memory 122. For example, processor 124 may perform various operations that can be specified by computer- and / or machine-readable, writable, and / or executable components and / or instructions, including but not limited to logic, control, input / output (I / O), arithmetic, etc. In some embodiments, processor 124 may include one or more central processing units, multi-core processors, microprocessors, dual microprocessors, microcontrollers, system-on-a-chip (SoC), array processors, vector processors, quantum processors, and / or another type of processor. Reference is made to processing unit 904 and... Figure 9 Additional examples of processor 124 are described below. Examples of processor 124 can be used to implement any one or more embodiments described herein.

[0064] The VAD integrated system 100, RNN-T 102, VAD model 104, joint network component 106, encoder component 108, prediction component 110, processor 124, and / or another component of system 100, memory 122, as described herein, may be communicatively, electrically, operatively, and / or optically coupled to each other via bus 120 to perform the functions of system 100 and / or any component coupled thereto. Bus 120 may include one or more memory buses, memory controllers, peripheral buses, external buses, local buses, quantum buses, and / or another type of bus that may employ a different bus architecture.

[0065] The following references the system bus 908 and Figure 9 Other examples of bus 120 are described. Any one or more embodiments described herein may be implemented using examples of bus 120.

[0066] VAD integrated system 100 may include any type of component, machine, device, facility, equipment, and / or instrument, including a processor and / or capable of effectively and / or operatively communicating with wired and / or wireless networks. All suitable such embodiments are contemplated. For example, VAD integrated system 100 may include server devices, computing devices, general-purpose computers, special-purpose computers, quantum computing devices (e.g., quantum computers), tablet computing devices, handheld devices, server-type computing machines and / or databases, laptop computers, notebook computers, desktop computers, cellular phones, smartphones, consumer appliances and / or instruments, industrial and / or commercial devices, digital assistants, multimedia internet-enabled phones, multimedia players, and / or other types of devices.

[0067] The VAD integrated system 100 can be coupled (e.g., communication, electrical, operational, optical, and / or similar) to one or more external systems, sources, and / or devices (e.g., classical and / or quantum computing devices, communication devices, and / or similar) via data cables (e.g., High Definition Multimedia Interface (HDMI), Recommended Standard (RS) 232, Ethernet cables, and / or the like). In some embodiments, the VAD integrated system 100 can be coupled (e.g., communication, electrical, operational, optical, and / or similar) to one or more external systems, sources, and / or devices (e.g., classical and / or quantum computing devices, communication devices, and / or similar) via a network.

[0068] In some embodiments, the network may include one or more wired and / or wireless networks, including but not limited to cellular networks, wide area networks (WANs) (e.g., the Internet), or local area networks (LANs). For example, the VAD integrated system 100, RNN-T 102, and / or VAD model 104 may communicate with, for example, one or more external systems, sources, and / or devices, computing devices using virtually any desired wired or wireless technology (and vice versa), including but not limited to: Wi-Fi, Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS); WiMAX, Enhanced General Packet Radio Service (Enhanced GPRS), 3GPP Long Term Evolution (LTE), 3GPP2 Ultra Mobile Broadband (UMB), High Speed ​​Packet Access (HSPA), Zigbee and other 802.XX wireless technologies and / or traditional telecommunications technologies. Session Initiation Protocol (SIP) RF4CE protocol, wireless HART protocol, 6LoWPAN (IPv6 over low-power wireless local area networks), Z-Wave, ANT, ultra-wideband (UWB) standard protocols and / or other proprietary and / or non-proprietary communication protocols. In related examples, VAD integrated system 100 may include hardware (e.g., central processing unit (CPU), transceiver, decoder, quantum hardware, quantum processor and / or the like), software (e.g., set of threads, set of processes, executing software, quantum pulse scheduling, quantum circuits, quantum gates and / or the like) and / or combinations of hardware and software that facilitate the transfer of information between VAD integrated system 100 and external systems, sources and / or devices (e.g., computing devices, communication devices, etc.).

[0069] The VAD integrated system 100 may include one or more computer and / or machine-readable, writable, and / or executable components and / or instructions that, when executed by the processor 124 (e.g., a classical processor, a quantum processor, and / or the like), facilitate the execution of one or more operations defined by such components and / or instructions. Furthermore, in many embodiments, as described herein with or without reference to the accompanying drawings of one or more embodiments, any component associated with the VAD integrated system 100 may include one or more computer and / or machine-readable, writable, and / or executable components and / or instructions that, when executed by the processor 124, facilitate the execution of one or more operations defined by such components and / or instructions. For example, the network component 106 (e.g., a connection to the network and / or the network connection layer of the RNN-T), the encoder component 108 (e.g., the network connection layer of the RNN-T), the connection to the encoder network and / or encoder connection layer of the RNN-T), the predictor component 110 (e.g., a connection to the predictor network and / or predictor layer of the RNN-T), and / or any other component associated with the VAD integrated system 100 as disclosed herein (e.g., communicating with, electronically, operatively, and / or optically coupled to, and / or employed by the system 100), may include such computer and / or machine-readable, writable, and / or executable components and / or instructions. Therefore, according to numerous embodiments, the VAD integrated system 100 as disclosed herein and / or any components associated therewith may employ processor 124 to execute such computer and / or machine-readable, writable, and / or executable components and / or instructions to facilitate the performance of one or more operations described herein with reference to system 100 and / or any such components associated therewith.

[0070] As disclosed herein, the VAD integration system 100 can facilitate (e.g., via processor 124) the execution of operations performed by and / or associated with the RNN-T 102, VAD model 104, joint network component 106, encoder component 108, prediction component 110, and / or another component associated with the system 100. For example, as described in detail below, the VAD integration system 100 can facilitate via processor 124 (e.g., a classical processor, a quantum processor, and / or the like) the following: receiving automatic speech recognition (ASR) information; using the ASR information to multi-task train the RNN-T 102; and / or integrating the encoder component 108 of the RNN-T 102 with one or more outputs of the joint network component 106 of the RNN-T 102 and / or the VAD model 104 to improve robustness to noise. As will be apparent from below, one or more systems can be used to transmit VAD-based knowledge to the RNN-T 102. For example, but not limited to, the VAD ensemble system 100 can be used to improve noise robustness through feature ensemble and / or multi-task training.

[0071] Turn now Figure 1 Additional aspects described, such as Figure 1 The components of the VAD integrated system 100 described herein will be further described in terms of system functionality. The following will refer to... Figure 2A , Figure 2B as well as Figures 3A to 3D The exemplary embodiments further describe additional descriptions of the functionality, wherein, for the sake of brevity, repeated descriptions of similar elements and / or processes used in the corresponding embodiments are omitted.

[0072] In embodiments, such as in Figure 2A and Figure 2B As generally shown, the VAD integrated system 100 may include a speech frame input sequence 400 (see, for example, Figure 4 Feature integration (e.g., feature extraction). The VAD integration system 100 may include feature integration from one or more of various connectivity / orientation methods. The VAD integration system 100 may include integration at a pre-encoder location (see, e.g., Figure 2A ) and / or encoder post-position (see, for example, Figure 2B The VAD model 204 is further described. RNN-T 202 may include a joint network component 206 (e.g., connections to the joint network), an encoder component 208 (e.g., connections to the encoder network), and / or a predictor component 210 (e.g., connections to the predictor network). Given a sequence of length T, x = (x1,...,x...). T The input sequence of speech features is used by RNN-T 202 to output a text token sequence y = (y1,...,y2) of length U by modeling the conditional distribution p(y|x) of the input and output. U In this embodiment, feature integration can occur by concatenating the speech activity class probabilities generated from a separately trained VAD model with ASR features. The VAD model 204 can predict a speech activity class v = (v1,...,v2) of length T from a sequence of speech frames x of the same length. T The sequence of VAD integrated system 100. The characteristics between the VAD output probability p(v|x) and the ASR features of the corresponding speech frame.

[0073] like Figure 2A As shown, the system may include a single-task ASR with cascaded VAD features integrated at the pre-encoder location. For example, but not limited to, VAD model 204 may be connected before encoder component 208 and thus fed into encoder component 208. VAD integration system 100 may concatenate the VAD output probability p(v|x) with the input features of RNN-Tx before encoder component 208, as represented by the following Equation 1, where xc It refers to the encoder input features after concatenation.

[0074] Equation 1: x c =((x1,p(v1|x)),...,(x T ,p(v T |x))

[0075] like Figure 2B As shown, the VAD integration system 100 may include a single-task ASR with VAD feature integration via a connection at a post-encoder location. For example, but not limited to, VAD model 204 may be connected after encoder component 208, such that VAD information from VAD model 204 and the output of encoder component 208 can be connected and sent to joint network 206. Feature-level integration can occur between the VAD output probability p(v|x) and the RNN-T encoder output h = (h1,...,h...). T The relationship between h is represented by equations 2 and 3 below, where h c It is the integral characteristic that will be passed to the joint network.

[0076] Equation 2: h = Encoder(x)

[0077] Equation 3: h c =((h1,p(v1|x)),...,(h T ,p(v T |x))

[0078] Now switch to... Figure 3A , 3B In various aspects of the systems described in 3C and / or 3D, the VAD integration system 100 can multitask train the RNN-T 302 and / or the VAD model 304 to improve ASR robustness. In the embodiments generally described herein, the VAD integration system 100 can train the RNN-T 302 as the primary task using ASR information (e.g., ASR criteria), and / or the system 100 can train the VAD model 304 as a subtask. For example, but not limited to, the VAD integration system 100 can simultaneously and / or optionally train the RNN-T 302 and the VAD model 304. Further, in one or more of the various embodiments (e.g., Figure 3A , 3B In 3C and 3D, the encoder layer of the RNN-T used for ASR processing can be shared with the encoder layer used for VAD processing (e.g., as a subtask).

[0079] In some implementations, such as Figure 3AAs generally illustrated, the VAD integration system 100 may include a pre-encoder sharing architecture (e.g., MTL 1) for implementing multi-task training. Accordingly, an additional network may be appended to the RNN-T 302 before the encoder component 308 for operations and / or processes to occur at the encoder component 308. The RNN-T 302 comprises a joint network component 306, an encoder component 308 and / or a predictor component 310. The VAD integration system 100 may generate an input representing ASR information to be passed to the encoder component 308 and the VAD model 304. In an embodiment, the VAD integration system 100 comprises a shared network consisting of a stack of fully connected neural network 312 (e.g., FC) layers with hyperbolic tangent functions. The VAD model 304 may operate as a subtask, and / or the VAD model 304 may consist of a convolutional neural network (CNN).

[0080] Furthermore, in other implementations, such as Figure 3B As generally illustrated, the VAD integration system 100 may include a partial encoder sharing architecture (e.g., MTL 2) for implementing multi-task training. Accordingly, the RNN-T 302 (e.g., ASR RNN-T) and the VAD model 304 share at least a part of the RNN-T 302 from an underlying layer (e.g., a physical connection layer of the VAD integration system 100, whereby the VAD model 304 and the RNN-T 302 may be physically connected to the encoder component 308 of the RNN-T 302, and / or may employ processes adopted by the encoder component 308 for both the ASR RNN-T 302 function and the VAD model 304 function). The RNN-T 302 may comprise a joint network component 306, an encoder component 308 and / or a predictor component 310 (e.g., an encoder layer of the RNN-T shared with the VAD model 304). The VAD model 304 (e.g., a VAD branch) is followed by a stack of fully connected layers for VAD classes that predict an input (e.g., a speech frame input sequence). The encoder component 308 may include one or more long short-term memory (LSTM) components / processors 314, 316 to implement multi-task processing for the RNN-T 302 and the VAD model 304.

[0081] In some implementations, such as in Figure 3C As generally illustrated, the VAD integration system 100 may include a full encoder sharing architecture (e.g., MTL 3) for implementing multi-task learning. Accordingly, the ASR and the VAD model 304 may use all encoder layers in the RNN-T 302 as a shared network. The RNN-T 302 may comprise a joint network component 306, a predictor component 310 and / or an encoder component 320 (e.g., which may be fully shared with the VAD model for multi-task learning).

[0082] Furthermore, in other embodiments, such as in Figure 3D As shown in the overall diagram, the VAD ensemble system 100 may include a full encoder shared with a feature ensemble architecture (e.g., MTL4) to enable multi-task training. Thus, the VAD soft output can be projected into a vector having the same dimensions as the RNN-T encoder output using a fully connected layer 322 (e.g., connected to a VAD model 304 outside of an RNN-T 302). Figure 3D In one embodiment, encoder component layer 320 can be used as a shared layer for VAD model 304 and RNN-T 302 to train the entire network (e.g., VAD ensemble system 100). This system can combine the VAD soft output with the RNN-T encoder output via element-wise summation.

[0083] It should be understood that the VAD integrated system 100 can be used independently and / or in combination with Figure 1-3D The system architecture is trained using auxiliary VAD data. In an embodiment, the VAD integration system 100 may utilize auxiliary, untranscribed, audio-only data (e.g., auxiliary VAD data) to optimize the network related to the VAD task to improve ASR performance. The VAD integration system 100 may utilize auxiliary VAD data through one or more training actions / procedures.

[0084] First, regarding the non-component aspect, the VAD integrated system 100 has a speech frame input sequence 400 associated with it (e.g., an audio input signal and / or stream). The speech frame input sequence 400 may include one or more of a variety of speech signal modes, such as... Figure 4 As generally illustrated. For example, but not limited to, the speech frame input sequence 400 may include: a speech-only segment 402, a non-speech-only segment 404, and / or multiple speech and non-speech segments 406. The speech frame input sequence 400 received by the VAD model 104 may contain short non-speech segments at both ends of the input audio (e.g., 408). Under typical noise conditions, the VAD model 104 can easily generate prediction errors of the actual speech segments of the speech frame input sequence. Furthermore, to improve the noise robustness of the system, VAD information may be incorporated into the ASR training based on RNN-T 102. VAD information can be integrated into ASR through feature-level ensemble and / or multi-task learning.

[0085] In an embodiment, the VAD integration system 100 can actively train the ASR and VAD models 104 using spoken and non-spoken segments. For example, but not limited to, the spoken-only training data can be augmented by artificially adding non-spoken segments before and / or after each side of a spoken segment. The VAD integration system 100 can utilize not only the augmented training data but also the spoken-only segment 402 and the non-spoken-only segment 404 (e.g., spaces). Furthermore, the VAD integration system 100 can handle one or more spoken + non-spoken contexts. For example, but not limited to, the VAD integration system 100 can handle pre-silence 406, bilateral silence 408, post-silence 410, and / or mid-silence 412 on the spoken frame input sequence 400.

[0086] Next, see Figure 5A and 5B ,These Figure 1 The flowchart illustrates an example, non-limiting computer-implemented method 500, that facilitates multi-task training of RNN-T 302 and VAD models 304 according to one or more embodiments described herein. Further, in cases such as Figure 5A and 5B In the overall implementation, the method for multi-task training of the RNN-T 302 and VAD model 304 may include receiving automatic speech recognition information (ASR) by a processor (502). The method may also include training the RNN-T 302 using the ASR information by the processor (504). Further, in an embodiment, the method may include integrating the encoder component 308 of the RNN-T 302 with the joint network component 306 of the RNN-T 302 and one or more outputs of the VAD model 304 by the processor to improve robustness to noise (506).

[0087] Turn now Figure 5B The action of integrating the encoder component 308 of the RNN-T 302 with the VAD model 304 (506) is shown in more detail. At 508, the computer-implemented method may include operating the VAD model 304 as a subtask of operating the RNN-T 302. Further, the computer-implemented method may include classifying one or more inputs of the RNN-T 302 into one or more speech activity categories associated with the pre-trained VAD model 304 via the encoder component (510).

[0088] At 512, the computer-implemented method 500 may include integrating one or more outputs of encoder component 308 into the same dimension as one or more outputs of VAD model 304 via a processor. Further, in an embodiment, the computer-implemented method 500 may include applying an element-wise sum operation at 514 between one or more outputs of encoder 308 and one or more outputs of VAD model 304.

[0089] Next, refer to Figure 6 The computer-implemented method 500 may further include multi-task training using auxiliary VAD data. According to one or more embodiments, ASR performance can be further improved / optimized by freezing one or more parameters of the system (e.g., utilizing auxiliary VAD data, as shown in process 600). At 602, the computer-implemented method 600 may include optimizing RNN-T 302 for the ASR task. RNN-T 302 can be optimized by freezing only the VAD parameters and using transcribed data. Further, the VAD ensemble system 100 can be trained for an ASR-specific task by freezing the parameters of the VAD model 304. The backpropagation loss calculation can be represented by Equation 4 below.

[0090] Equation 4: Loss = LossASR

[0091] At 604, the computer-implemented method 600 may include optimizing the VAD model 304 by freezing ASR parameters and shared layers (e.g., between RNN-T302 and VAD model 304) and using transcribed and auxiliary untranscribed data. In an embodiment, freezing ASR parameters may include freezing the joint network component 306, encoder component 308, and / or predictor component 310. The backpropagation loss calculation may be represented by Equation 5 below.

[0092] Equation 5: Loss = LossVAD

[0093] At 606, the computer-implemented method 600 may include joint optimization of ASR (e.g., ASR RNN-T 302) and VAD model 304. For example, but not limited to, the computer-implemented method 600 may include updating VAD model 304 and shared layer parameters (e.g., updating VAD model 304 and / or encoder component 308) to optimize the VAD ensemble system 100. For each training epoch of the computer-implemented method 600, the VAD ensemble system 100 may be optimized using VAD-specific data and / or the system 100 may be optimized using ASR-VAD joint optimization. In embodiments, the VAD ensemble system 100 may be optimized only for the ASR task (e.g., freezing VAD parameters), in response, the VAD ensemble system 100 may be optimized only for VAD optimization (e.g., freezing ASR parameters), and / or in a further response, the VAD ensemble system 100 may jointly optimize the ASR and VAD models 304. System 100 can use untranscribed VAD training data with weighted loss to update VAD model 304 and shared layer parameters. The weighted loss calculation can be represented by Equation 6 below, where β is a hyperparameter of the VAD task loss weights used for joint optimization.

[0094] Equation 6: Loss = β·LossVAD

[0095] In an embodiment, updating the ASR and VAD parameters may include updating the parameters of the VAD model 304, the joint network component 306, the encoder component 308, and / or the predictor component 310. Further, in an embodiment, the ASR and VAD parameters may be updated using shared transcriptional data and a joint weighted loss, as shown in Equation 7 below, where α is a hyperparameter of the ASR task loss weights.

[0096] Equation 7: Loss=α·LossASR+β·LosVAD

[0097] With the help of examples, the system can utilize pseudo-tags generated from a separately well-trained VAD model. This pseudo-tag process can be viewed as knowledge distillation from VAD to ASR. Knowledge distillation is a technique that can be implemented to mimic a complex teacher network using a simple student network. Separate VAD models (e.g., teacher VAD models) can be trained with a large amount of training data to generate better pseudo-soft tags for student VAD models, where the student is a sub-task VAD network connected to an RNN-TASR network. Pseudo-VAD tags can be obtained from acoustic features x in the training material. To perform distillation. The distillation technique can be represented by the following equation 8, where... It is the VAD class predicted after the teacher's VAD. Given X, V can then be used as the target label for the student's VAD in a multi-task framework.

[0098] Equation 8:

[0099] Turn now Figure 7 The results provided in Figure 8, and four tables 700, 800, 802, and / or 804, illustrate the results of one or more various experiments. In the first experiment, a Switchboard (SWB) corpus of approximately 300 hours of multi-speaker American English speech from telephone conversations was used. The data could be augmented by including non-speaking segments longer than 2 seconds in the training material (e.g., such segments are typically discarded in a common training setting). In addition to the standard 267 hours of SWB data, augmentation produced approximately 88 hours of non-speaking data only and approximately 599 hours of speech data mixed with speaking segments and non-speaking segments.

[0100] In the second experiment, it can be prepared in Figure 6This dataset contains updated auxiliary VAD training data with VAD standards within a multi-task learning framework. The data includes approximately 200 hours of English call center (CC) conversations and 200 hours of non-speech conversations without available transcripts. Ambient noise was added to the training set used for model training in this dataset. The resulting average speech-to-noise ratio (SNR) is 14 dB.

[0101] In an embodiment, for example at least partially in Figure 3B As illustrated, the RNN-T can comprise six bidirectional Long Short-Term Memory (Bi-LSTM) encoder layers (640 units per direction per layer) and a single unidirectional LSTM prediction layer (with only 1024 units). The joint network embeds the 1280-dimensional stacked encoder vector from the last layer and the 1024-dimensional prediction network into a 256-dimensional and / or combined projection vector. After applying hyperbolic tangent, the output is projected onto 42 logits, followed by a softmax layer corresponding to 41 characters. 40-dimensional speaker-independent log-Mel filter bank features can be extracted every 10ms as ASR features. These features are enhanced with incremental and double-increment coefficients after speech level mean and global variance normalization. Independent CNN-based VADs generate frame-level VAD markers, which can be utilized via feature ensemble and / or multi-task learning systems.

[0102] Furthermore, in the implementation, when classifying speech frames into speech, non-speech, and music categories, the isolated VAD model may include four convolutional layers, where the input channels are 3, 16, 32, and 2 from the first layer to the last layer, and / or the output class dimension is 3. The isolated VAD can be trained (e.g., in experiments) on over 2000 hours of English speech data comprising different ASR domains (including both spontaneous and read-aloud speech). On the other hand, the VAD component is a subtask of MTL2, MTL3, and MTL4 (consisting of three stacks of FC layers). In MTL2, the first three encoder layers can be shared between the ASR and VAD tasks. The learning rate for ASR includes a value of 2e-4, and the learning rate for VAD in the multi-task learning framework includes a value of 2e-5. Both systems can be optimized using stochastic gradient descent (SGD) with a batch size of 64.

[0103] Furthermore, such as Figure 7As shown, experiments were conducted using only speech segments and those combined with non-speech segments that simulated VAD prediction errors. The results of these experiments are shown in Tables 700 and 702, where the ASR word error rate is abbreviated as WER%, and the frame-level VAD equal error rate is abbreviated as EER%. The baseline and the proposed system were evaluated on modified SWB and Callhome (CH) test sets created by manually adding non-speech segments of varying lengths before, after, or between two speech segments. The average length of the non-speech segment added to each test speech was 5.5 seconds. Real-world ambient noise was also added to these test sets. In addition, only speech and only noise (non-speech) tests were performed.

[0104] The experimental results in Tables 700 and 800 show that the standard RNN-T, labeled "no augmentation (aug)" trained with the original SWB training corpus, performs well under the ideal, speech-only input condition with manual segmentation, but its performance drops sharply in the more realistic case of a combination of speech and long non-speech parts. In this case, the text decoded by "standard RNN-T (no augmentation)" contains many unexpected insertion errors in the non-speech parts. Conversely, adding non-speech segments to the training dataset ("standard RNN-T (augmented)") significantly reduces the number of those errors, as can also be seen in Table 800.

[0105] In different test conditions, post-encoder feature integration performs better than pre-encoder feature integration (see, for example, Table 700). Although the WER by post-encoder integration in the “speaking + non-speaking” and “speaking only” input cases is similar to the “standard RNN-T (August)”, this technique significantly reduces the insertion error in the non-speaking only test case, as shown in 800.

[0106] Next, we move on to multi-task learning, and the experimental results are also listed in 700 and 800. When VAD-assisted data is not utilized, MTL4 (β = 0.01) provides the greatest improvement in the “speaking + non-speaking” test case. Further improvements are obtained by using auxiliary VAD training data (VAD-assisted data) in MTL2, MTL3, and MTL4. Compared to the model without auxiliary VAD data, the model and / or non-limiting implementations improve not only the WER in the “speaking + non-speaking” test case but also in the “speaking only” test case. “MTL 4 + VAD-assisted data” shows that the best performance yields a relative improvement of 21.5% and 3.6% respectively in the “speaking + non-speaking” test case compared to the standard RNN-T without and with data augmentation. Moreover, “MTL 4 + VAD-assisted data” provides a relative improvement of 4.6% compared to the “standard RNN-T (augmented)” in the speaking only test case.

[0107] In an embodiment, experiments were conducted using audio signals automatically segmented by a separate VAD system (e.g., the audio signals could then be used to generate VAD tags). The results of these experiments are shown in Figure 8 below Table 802. Because different types of segmentation errors in the low SNR VAD results include classifications that categorize speech segments as noise, the average absolute WER in Table 802 is greater than the absolute WER in Table 700. However, the VAD integrated system 100 exhibits a consistent gain exceeding that of the baseline system.

[0108] Turning now to Table 804 in Figure 8, which indicates the results of acoustic customization optimized via the VAD subtask, with additional test data (CC) in the same domain as the VAD-assisted data (containing WER at different SNRs). CC is an out-of-domain acoustic test set, and therefore the absolute WER is high. By adding VAD-assisted data to the training data, “MTL 4+VAD-assisted data” produces improvements in both the CC and SWB / CH test sets at each SNR setting. As the SNR decreases, the WER gap between “Standard RNN-T (Enhanced)” and “MTL 4+VAD-assisted data” tends to widen.

[0109] In this embodiment, further testing and / or performance improvements of the VAD integrated system 100 can be achieved via... Figure 8B , 8C And / or 8D to illustrate. The VAD integrated system 100 can be tested via test sets “CH” and “internal CC” for mono recording, split channel A or split channel B. Table 806 contains experimental results indicating the effectiveness of the VAD integrated system 100 for manually segmented speech-only audio with noise in an average SNR of 4dB mixed with car noise. In addition, Table 808 (e.g., Figure 8C This includes experimental results indicating the effectiveness of the VAD integrated system 100 on manually segmented speaking and non-speaking segments. For example, but not limited to, non-speaking segments can be longer than or equal to approximately 2 seconds.

[0110] Now go to Figure 8C Table 810 indicates additional results from experiments conducted on the VAD integrated system 100 via the SWB and CH test sets. Further, Table 812 illustrates experimental results from manually segmenting noisy speech at various SNR levels (e.g., SNR levels of 14.2 dB, 4.2 dB, and / or 1.4 dB). Additionally, experiments may be illustrated by Tables 814 and / or 816. Figure 8DAs shown, Table 814 illustrates experimental results achieved by manually segmenting noisy speech and non-speech at various SNR levels (e.g., SNR levels of 14 dB, 4.2 dB, and / or 1.4 dB). Table 816 demonstrates the effectiveness of the VAD integrated system at various speech segments. Speech segments may include non-speech only 404, pre-silence 406, post-silence 410, and / or mid-silence 412 (see, for example...). Figure 4 ).

[0111] In cases where unsupervised and semi-supervised training of ASR is not feasible on acoustically challenging data, computer-implemented methods can show promising improvements due to their relative computational cost. Experimental results in Tables 700, 800, 802, 804, 806, 808, 810, 812, 814, and / or 816 demonstrate the ability to enhance the system's robustness to noisy environments.

[0112] For simplicity of explanation, the computer-implemented method is depicted and described as a series of actions. It should be understood and recognized that the subject matter innovation is not limited to the actions shown and / or the order of the actions; for example, actions may occur in different orders and / or simultaneously, and may occur with other actions not presented and described herein. Furthermore, not all actions shown are necessary to implement the computer-implemented method according to the disclosed subject matter. Moreover, those skilled in the art will understand and appreciate that the computer-implemented method may alternatively be represented as a series of interrelated states via state diagrams or events. Furthermore, it should be understood that the computer-implemented method disclosed below and throughout this specification can be stored on an article of art to facilitate the transfer and assignment of the computer-implemented method to a computer. As used herein, the term article of art is intended to encompass a computer program accessible from any computer-readable device or storage medium.

[0113] To provide further background for the various implementations described herein, Figure 9 The following discussion is intended to provide a general description of a suitable operating environment 900 in which the various embodiments described herein may be implemented. Although embodiments have been described above in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that embodiments may also be implemented in combination with other program modules and / or as a combination of hardware and software.

[0114] Typically, program modules include routines, programs, components, data structures, etc., that perform specific tasks and / or implement specific abstract data types. Furthermore, those skilled in the art will recognize that the methods of this invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, and personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, each of which can be operatively coupled to one or more associated devices.

[0115] The embodiments illustrated in this document can also be implemented in a distributed computing environment, where some tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules can reside on both local and remote storage devices.

[0116] Computing devices typically include a variety of media, which may include computer-readable storage media, machine-readable storage media, and / or communication media, these two terms being used differently from each other herein. A computer-readable storage medium or a machine-readable storage medium can be any available storage medium accessible by a computer, and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable storage media and / or machine-readable storage media can be implemented using any method or technique for storing information such as computer-readable and / or machine-readable instructions, program modules, structured data, and / or unstructured data.

[0117] Computer-readable storage media may include, but are not limited to: random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital universal disc (DVD), Blu-ray disc (BD) and / or other optical disc storage, magnetic tape cassettes, magnetic tape, disc storage and / or other magnetic storage devices, solid-state drives and / or other solid-state storage devices, and / or other tangible and / or non-transient media that can be used to store desired information. In this regard, the terms “tangible” or “non-transient” as used herein with respect to storage, memory, or computer-readable media shall be understood to exclude only the propagation of transient signals themselves as a modifier, and shall not waive the rights to all standard storage, memory, and / or computer-readable media that do not only propagate transient signals themselves.

[0118] Computer-readable storage media can be accessed by one or more local or remote computing devices, for example via access requests, queries and / or other data retrieval protocols, for various operations relative to the information stored in the media.

[0119] Communication media typically embody computer-readable instructions, data structures, program modules, or other structured or unstructured data as data signals such as modulated data signals (e.g., carrier waves or other transmission mechanisms), and include any information delivery or transmission medium. The term "modulated data signal" refers to a signal whose characteristics are set or altered in a manner that encodes information in one or more signals. By way of example and not limitation, communication media may include wired media, such as wired networks and / or direct wired connections, and / or wireless media, such as acoustic, RF, infrared, and / or other wireless media.

[0120] Refer again Figure 9 An exemplary operating environment 900 for implementing various embodiments of the aspects described herein includes a computer 902, which includes a processing unit 904, a system memory 906, and / or a system bus 908.

[0121] System bus 908 can couple system components, including but not limited to system memory 906, to processing unit 904. Processing unit 904 can be any of different commercially available processors. Dual microprocessors and / or other multiprocessor architectures can be used as processing unit 904.

[0122] System bus 908 can be any of several types of bus structures that can be further interconnected to a memory bus (with or without a memory controller), peripheral bus, and / or local bus using any of a variety of commercially available bus architectures. System memory 906 may include ROM 910 and / or RAM 912. The Basic Input / Output System (BIOS) may be stored in non-volatile memory such as ROM, erasable programmable read-only memory (EPROM), and / or EEPROM, and the BIOS contains basic routines such as those that help transfer information between components within computer 902 during startup.

[0123] RAM912 may also include high-speed RAM, such as static RAM for caching data.

[0124] Computer 902 may further include an internal hard disk drive (HDD) 914 (e.g., EIDE, SATA), one or more external storage devices 916 (e.g., floppy disk drive (FDD), memory stick or flash drive reader, memory card reader, etc.) and / or a drive 920 (e.g., a solid-state drive or optical disc drive), which can read from or write to a disk 922 (e.g., CD-ROM, DVD, BD, etc.). Alternatively, in cases involving solid-state drives, disk 922 may not be included unless it is separate. Although the internal HDD 914 is shown as being located within computer 902, the internal HDD 914 may also be configured for external use in a suitable rack (not shown). Additionally, although not shown in operating environment 900, a solid-state drive (SSD) may be used as a supplement to or replacement for HDD 914. HDD 914, external storage devices 916, and drive 920 may be connected to system bus 908 via HDD interface 924, external storage interface 926, and drive interface 928, respectively. The HDD interface 924 for external drive implementation may include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external drive connection technologies are contemplated in the embodiments described herein.

[0125] The drive and its associated computer-readable storage medium provide non-volatile storage of data, data structures, computer-executable instructions, etc. For computer 802, the drive and storage medium accommodate any data stored in a suitable digital format. Although the above description of computer-readable storage media refers to corresponding types of storage devices, those skilled in the art will understand that other types of computer-readable storage media (whether currently existing or developed in the future) may also be used in the example operating environment, and further, any such storage medium may contain computer-executable instructions for performing the methods described herein.

[0126] Multiple program modules can be stored in the drive and RAM 912, including an operating system 930, one or more applications 932, other program modules 934, and / or program data 936. All or part of the operating system, applications, modules, and / or data can also be cached in RAM 912. The systems and methods described herein can be implemented using different commercially available operating systems and / or combinations of operating systems.

[0127] Computer 902 may optionally include emulation technology. For example, a hypervisor (not shown) or other intermediary may emulate the hardware environment of operating system 930, and the emulated hardware may optionally be different from the hardware shown in FIG8. In a related embodiment, operating system 930 may include a virtual machine (VM) among a plurality of VMs hosted at computer 902. Furthermore, operating system 930 may provide a runtime environment for application 932, such as the Java Runtime Environment or the .NET Framework. A runtime environment is a consistent execution environment that allows application 932 to run on any operating system that includes a runtime environment. Similarly, operating system 930 may support containers, and application 932 may be in the form of a container, which is a lightweight, standalone, executable software package including, for example, code, runtime, system tools, system libraries, and / or application settings.

[0128] Furthermore, the computer 902 can be equipped with security modules, such as a Trusted Processing Module (TPM). For example, using a TPM, the boot component hashes the boot component in time and waits for the result to match a security value before loading the boot component. This process can occur at any layer of the computer 902's code execution stack, such as at the application execution level and / or the operating system (OS) kernel level, thereby achieving security at any level of code execution.

[0129] User entities can input commands and information into computer 902 through one or more wired / wireless input devices (e.g., keyboard 938, touchscreen 940, and / or pointing devices such as mouse 942). Other input devices (not shown) may include microphones, infrared (IR) remote controls, radio frequency (RF) remote controls, or other remote controls, joysticks, virtual reality controllers and / or virtual reality headsets, game controllers, styluses, image input devices (e.g., cameras), gesture sensor input devices, visual motion sensor input devices, emotion or face detection devices, biometric input devices (e.g., fingerprint or iris scanners), or the like. These and other input devices can be connected to processing unit 904 through input device interface 944, which can be coupled to system bus 908, but can also be connected through other interfaces such as parallel ports, IEEE 1394 serial ports, game ports, USB ports, IR interfaces, etc. Interfaces and / or similar things.

[0130] The monitor 946 or other type of display device can also be connected to the system bus 808 via an interface such as a video adapter 948. In addition, the computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

[0131] Computer 902 can operate in a networked environment using logical connections via wired and / or wireless communications to one or more remote computers, such as remote computer 950. Remote computer 950 can be a workstation, server computer, router, personal computer, laptop computer, microprocessor-based entertainment device, peer-to-peer device, and / or other public network node, and typically includes many or all of the elements described relative to computer 902; however, for brevity, only memory / storage device 952 is shown. The depicted logical connections include wired / wireless connections to a local area network (LAN) 954 and / or a larger network (e.g., a wide area network (WAN) 956). LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can connect to global communication networks, such as the Internet.

[0132] When used in a LAN networking environment, computer 902 can connect to local network 954 via a wired and / or wireless communication network interface or adapter 958. Adapter 958 facilitates wired or wireless communication to LAN 954, which may also include a wireless access point (AP) deployed thereon for communicating with adapter 958 in wireless mode.

[0133] When used in a WAN networking environment, computer 902 may include modem 960 and / or a communication server connected to WAN 956 via other means (such as via the Internet) for establishing communication over WAN 956. Modem 960, which may be internal or external and wired and / or wireless, may be connected to system bus 908 via input device interface 944. In a networking environment, program modules depicted relative to computer 902 or parts thereof may be stored in remote memory / storage device 952. It should be understood that the network connection shown is an example, and other means for establishing communication links between computers may be used.

[0134] When used in a LAN or WAN networking environment, computer 902 can access cloud storage systems or other network-based storage systems as a supplement to or replacement of external storage device 916 as described above, such as, but not limited to, network virtual machines providing one or more aspects of information storage or processing. Typically, the connection between computer 902 and the cloud storage system can be established, for example, via adapter 958 or modem 960 through LAN 954 or WAN 956. When computer 902 is connected to the associated cloud storage system, external storage interface 926 can manage the storage provided by the cloud storage system by means of adapter 958 and / or modem 960, just like other types of external storage. For example, external storage interface 926 can be configured to provide access to cloud storage sources as if those sources were physically connected to computer 902.

[0135] Computer 902 may be operable to communicate with any wireless device or entity operably arranged in wireless communication, such as a printer, scanner, desktop and / or laptop computer, portable data assistant, communications satellite, any device or location associated with a wirelessly detectable tag (e.g., self-service terminal, newsstand, store shelf, etc.) and / or telephone. This may include Wi-Fi and Wireless technology. Therefore, communication can be a predefined structure like a traditional network, or simply self-organizing communication between at least two devices.

[0136] See now Figure 10 The illustration depicts a cloud computing environment 1050. As shown, the cloud computing environment 1050 includes one or more cloud computing nodes 1010 to which local computing devices used by cloud consumers can communicate. These local computing devices include, for example, personal digital assistants (PDAs) or cellular phones 1054A, desktop computers 1054B, laptop computers 1054C, and / or automotive computer systems 1054N. Although... Figure 10 As not shown, cloud computing node 1010 may also include a quantum platform (e.g., a quantum computer, quantum hardware, quantum software, etc.), with which the local computing devices used by cloud consumers can communicate. Cloud computing nodes 1010 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows cloud computing environment 1050 to provide infrastructure, platform, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 10The types of computing devices 1054A-N shown are intended to be illustrative only, and cloud computing node 1010 and cloud computing environment 1050 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0137] See now Figure 11 This demonstrates features such as those provided by the cloud computing environment 1050 ( Figure 10 This provides a set of functional abstractions. It should be understood beforehand. Figure 11 The components, layers, and functions shown are intended to be illustrative only, and the embodiments described herein are not limited thereto. As described, the following layers and corresponding functions are provided:

[0138] The hardware and software layer 1160 may include hardware and software components. Examples of hardware components include: a host 1161; a server 1162 based on a RISC (Reduced Instruction Set Computer) architecture; a server 1163; a blade server 1164; a storage device 1165; and a network and network components 1166. In some embodiments, software components may include network application server software 1167, quantum platform routing software 1168, and / or quantum software (…). Figure 11 (Not shown in the image).

[0139] The virtualization layer 1170 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 1171; virtual storage 1172; virtual network 1173, including virtual private network; virtual application and / or operating system 1174 and / or virtual client 1175.

[0140] In one example, management layer 1180 can provide the following functionalities: Resource Provisioning 1181 can provide dynamic procurement of computing resources and other resources that can be used to perform tasks within the cloud computing environment. Metering and Pricing 1182 can provide cost tracking as resources are utilized within the cloud computing environment and bill or invoice for the consumption of these resources. In one example, these resources may include application software licenses. Security can provide authentication for cloud consumers and tasks, as well as protection for data and other resources. User (or Component) Portal 1183 can provide access to the cloud computing environment for consumers and system administrators. Service Level Management 1184 can provide cloud computing resource allocation and management to meet required service levels. Service Level Agreement (SLA) Planning and Fulfillment 1185 can provide pre-scheduling and procurement of cloud computing resources, anticipating future requirements for those resources according to the SLA.

[0141] Workload layer 1190 can provide examples of functionalities that can leverage a cloud computing environment. Non-limiting examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 1191; software development and lifecycle management 1192; virtual classroom education delivery 1193; data analytics and processing 1094; transaction processing 1195; and / or application transformation software 1196.

[0142] The embodiments described herein may be directed to one or more at any possible level of integration of technical details in a system, method, apparatus, and / or computer program product. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of one or more embodiments described herein. A computer-readable storage medium may be a tangible means capable of retaining and storing instructions for use by an instruction execution apparatus. A computer-readable storage medium may be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, and / or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media may also include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards, or protrusions in recesses having instructions recorded thereon, and / or any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0143] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device. The computer-readable program instructions used to perform the operations of one or more embodiments described herein may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and / or procedural programming languages ​​(such as the "C" programming language or similar programming languages). Computer-readable program instructions may execute entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer, and / or partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing state information of the computer-readable program instructions in order to perform aspects of one or more embodiments described herein.

[0144] This document describes aspects of one or more embodiments of a method, apparatus (system), and computer program product according to one or more embodiments described herein. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, and / or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in the blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions comprises an article of manufacture containing instructions for implementing aspects of the functions / actions specified in the blocks of the flowcharts and / or block diagrams. Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus and / or other equipment to cause a series of operations to be performed on the computer, other programmable apparatus or other equipment to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus and / or other equipment perform the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, computer-implementable methods, and / or computer program products according to the various embodiments described herein. Each block in a flowchart or block diagram may represent a module, segment, and / or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a non-linear order. For example, depending on the functions involved, two consecutively shown blocks may actually execute substantially simultaneously, or these blocks may sometimes execute in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified functions and / or actions or executes combinations of dedicated hardware and / or computer instructions.

[0146] While the subject matter has been described above in the general context of computer-executable instructions running on a computer and / or a computer program product on a computer, those skilled in the art will recognize that one or more embodiments herein can also be implemented in conjunction with other program modules. Typically, program modules include routines, programs, components, data structures, etc., that perform specific tasks and / or implement specific abstract data types. Furthermore, those skilled in the art will recognize that the computer implementation of the methods of the present invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, small computing devices, mainframe computers, and computers, handheld computing devices (e.g., PDAs, telephones), microprocessor-based or programmable consumer or industrial electronics, and / or the like. The aspects shown can also be implemented in a distributed computing environment, where tasks are performed by remote processing devices linked via a communication network. However, some (if not all) aspects of one or more embodiments can be practiced on a standalone computer. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0147] As used herein, the terms “component,” “system,” “platform,” “interface,” etc., may refer to and / or include computer-related entities or entities associated with an operating machine having one or more specific functions. Entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or a computer. As an illustration, both an application running on a server and the server itself can be components. One or more components may reside within a process and / or a thread of execution, and components may reside on one computer and / or be distributed across two or more computers. In another instance, a corresponding component may be executed from a different computer-readable medium having different data structures stored thereon. Components may communicate via local and / or remote processes, such as according to signals having one or more data packets (e.g., data from a component interacting with another component in a local system, a distributed system, and / or via that signal interacting with other systems across a network such as the Internet). As another example, a component may be a device having specific functions provided by mechanical parts operated by electrical or electronic circuitry, which is operated by a software or firmware application executed by a processor. In such a case, the processor can be internal or external to the device and can execute at least a portion of the software or firmware application. As another example, the component can be a device that provides a specific function through electronic components without mechanical parts, wherein the electronic components can include a processor or other means for performing software or firmware that at least partially endows the electronic components with the functions. In one aspect, the component can be emulated via a virtual machine, for example, within a cloud computing system.

[0148] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X adopts A or B" is intended to mean any natural inclusive permutation. That is, if X adopts A; X adopts B; or X adopts both A and B, then "X adopts A or B" is satisfied in any of the foregoing cases. Additionally, the articles "a" and "an" as used in the subject matter specification and figures should generally be interpreted as meaning "one or more," unless otherwise specified or clearly indicated from the context to the singular form. As used herein, the terms "example" and / or "exemplary" are used to indicate that something is used as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. Furthermore, any aspect or design described herein as an "example" and / or "exemplary" is not necessarily to be construed as superior to or better than other aspects or designs, nor does it imply the exclusion of equivalent exemplary structures and techniques known to those skilled in the art.

[0149] As used herein, the term "processor" can refer to substantially any computing processing unit or device, including but not limited to a single-core processor; a single processor with software multithreading capabilities; a multi-core processor; a multi-core processor with software multithreading capabilities; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. Additionally, "processor" can refer to an integrated circuit, application-specific integrated circuit (ASIC), digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic controller (PLC), complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. Furthermore, processors can utilize nanoscale architectures, such as, but not limited to, molecular and quantum dot-based transistors, switches, and gates, to optimize space utilization or enhance the performance of user equipment. Processors can also be implemented as a combination of computing processing units. In this document, terms such as "memory," "storage," "database," "data storage," "database," and substantially any other information storage component, in relation to the operation and function of a component, are used to refer to a "memory component," entity, or component that includes memory. It should be understood that the memory and / or memory components described herein can be volatile or non-volatile memory, or may include both volatile and non-volatile memory. By way of example and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, and / or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). Volatile memory may include, for example, RAM that can act as an external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR SDRAM), and enhanced SDRAM.

[0150] (ESDRAM), Synchlink DRAM (SLDRAM), Direct xx RAM (DRRAM), Direct Rambus Dynamic RAM (DRDRAM), and / or Rambus Dynamic RAM (RDRAM). Furthermore, the memory components of the systems and / or computer-implemented methods disclosed herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0151] The above description includes only examples of systems and computer-implemented methods. Of course, for the purpose of describing one or more embodiments, it is impossible to describe every conceivable combination of components or computer-implemented method; however, those skilled in the art will recognize that many further combinations and substitutions of one or more embodiments are possible. Furthermore, the terms “comprising,” “having,” “possessing,” etc., used in the detailed description, claims, appendices, and drawings are intended to be inclusive in a manner similar to the term “including,” since “including” is interpreted as a transitional word in the claims.

[0152] Various embodiments have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A system for automatic speech recognition, comprising: Memory, which stores computer-executable components; as well as A processor operatively coupled to the memory and executing the computer-executable component stored in the memory, wherein the computer-executable component includes: A recurrent neural network transducer (RNN-T) receives automatic speech recognition (ASR) information. The Voice Activity Detection (VAD) model is trained using the ASR information via multi-task training, and... The RNN-T further includes an encoder and a joint network, wherein one or more outputs of the encoder are integrated with the joint network and one or more outputs of the VAD model, and the one or more outputs of the encoder are integrated with the one or more outputs of the VAD model by projecting the one or more outputs of the VAD model into the same dimension as the one or more outputs of the encoder via a fully connected layer.

2. The system according to claim 1, wherein, The encoder serves as a shared layer for the activities of the VAD model and the RNN-T.

3. The system according to claim 1, wherein, The encoder classifies one or more inputs of the RNN-T into one or more speech activity categories via a pre-trained VAD model.

4. The system according to claim 1, wherein, The VAD model is operated as a subtask of the RNN-T.

5. The system according to claim 1, wherein, The one or more outputs of the encoder are transmitted to the joint network.

6. The system according to claim 1, wherein, An element-wise summation operation is applied between the one or more outputs of the encoder and the one or more outputs of the VAD model to improve robustness to noise.

7. The system according to claim 1, wherein, The system is optimized based on VAD parameters; and the VAD model and RNN-T are jointly optimized.

8. The system according to claim 7, wherein, The system is further optimized by freezing one or more parameters of at least one of the ASR information and the VAD parameters to determine backpropagation.

9. The system according to claim 1, wherein, One or more non-speech segments are added to one or more speech segments of the ASR information to actively train the VAD model and the RNN-T.

10. A computer-based method for automatic speech recognition, comprising: The processor receives Automatic Speech Recognition (ASR) information; The processor uses the ASR information to perform multi-task training on the recurrent neural network transducer RNN-T. as well as The processor integrates the encoder of the RNN-T with the joint network of the RNN-T and one or more outputs of the Voice Activity Detection (VAD) model, and integrates the encoder with the one or more outputs of the VAD model by projecting the one or more outputs of the VAD model into the same dimension as the one or more outputs of the encoder via a fully connected layer, thereby improving robustness to noise.

11. The computer implementation method according to claim 10, wherein, The encoder serves as a shared layer for the activities of the VAD model and the RNN-T.

12. The computer implementation method according to claim 10, further comprising: The processor classifies one or more inputs of the RNN-T into one or more speech activity categories associated with a pre-trained VAD model.

13. The computer implementation method according to claim 10, further comprising: Operating the VAD model is a subtask of operating the RNN-T.

14. The computer implementation method according to claim 10, further comprising: An element-wise summation operation is applied between the one or more outputs of the encoder and the one or more outputs of the VAD model.

15. A computer program product for training a recurrent neural network transducer RNN-T using a voice activity detection (VAD) model, the computer program product comprising program instructions executable by a processor to cause the processor to: Receive Automatic Speech Recognition (ASR) information; The ASR information is used to perform multi-task training on the RNN-T; The encoder of the RNN-T is integrated with the joint network of the RNN-T and one or more outputs of the VAD model, and the encoder is integrated with the one or more outputs of the VAD model by projecting the one or more outputs of the VAD model into the same dimension as the one or more outputs of the encoder via a fully connected layer.

16. The computer program product according to claim 15, wherein, The program instructions further cause the processor to: One or more inputs of the RNN-T are classified into one or more speech activity categories associated with a pre-trained VAD model.

17. The computer program product according to claim 15, wherein, The program instructions further cause the processor to: An element-wise summation operation is applied between the one or more outputs of the encoder and the one or more outputs of the VAD model to improve robustness to noise.

Images