Speech Recognition System
By segmenting the voice input and adjusting the field weight, the problem of insufficient recognition accuracy of the existing speech recognition system is solved, and the accuracy and system efficiency of speech recognition are improved.
Patent Information
- Application Number
- CN202110154554.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2016-01-06
- Filing Date
- 2016-12-23
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2036-12-23
AI Technical Summary
When the existing voice recognition system processes voice input, the recognition accuracy is insufficient, resulting in users needing to repeatedly provide voice input to obtain accurate results, which affects the system efficiency.
By segmenting the voice input and adjusting the field weight based on the segments of the voice input, dynamically adjusting the transcription text to improve recognition performance.
Improve the accuracy of speech recognition, reduce the possibility of users providing voice input repeatedly, and enable the speech recognition system to process other voice input more efficiently.
Smart Images

Figure CN112992146B_ABST
Abstract
Description
[0001] Description of the case
[0002] This application is a divisional application of Chinese invention patent application No. 201611207951.1, filed on December 23, 2016. Technical Field
[0003] The present disclosure relates to speech recognition. Background Art
[0004] Conventional speech recognition systems are designed to convert speech input from a user into text output. The text output can be used for a variety of purposes, including, for example, as a search query, a command, a word processing input, etc. In a typical speech search system, a speech interface receives a user's speech input and provides the speech input to a speech recognition engine. The speech recognition engine converts the speech input into a text search query. The speech search system then submits the text search query to a search engine to obtain one or more search results. Summary of the invention
[0005] In general, an innovative aspect of the subject matter described in this specification can be implemented in the following method, which includes the following steps: receiving data encoding speech input; determining a transcription of the speech input, wherein, for multiple segments of the speech input, determining the transcription of the speech input includes: obtaining a first candidate transcription of a first segment of the speech input; determining one or more contexts associated with the first candidate transcription; adjusting a corresponding weight of each of the one or more contexts; and determining a second candidate transcription of a second segment of the speech input based in part on the adjusted weights; and providing the transcription of the multiple segments of the speech input for output. The method described in this specification can be implemented as a computer-implemented method. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the steps of the method. For a system of one or more computers to be configured to perform a specific operation or action, it means that the system has been installed on software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action in operation. For one or more computer programs to be configured to perform certain operations or actions, it is meant that the one or more programs include instructions that, when executed by a data processing apparatus, cause the apparatus to perform the operations or actions.
[0006] Another innovative aspect of the subject matter described in this specification can be implemented as a computer-readable medium storing software, the software including instructions executable by one or more computers, which when executed cause the one or more computers to perform the following operations, including: receiving data encoding speech input; determining a transcript of the speech input, wherein, for multiple segments of the speech input, determining the transcript of the speech input includes: obtaining a first candidate transcript for a first segment of the speech input; determining one or more contexts associated with the first candidate transcript; adjusting the corresponding weight of each of the one or more contexts; and, determining a second candidate transcript for a second segment of the speech input based in part on the adjusted weights; and providing the transcripts of the multiple segments of the speech input for output.
[0007] Each of the foregoing and other embodiments can optionally include one or more of the following features (alone or in any combination thereof). For example, one embodiment includes a combination of all of the following features. The method includes obtaining a first candidate transcript of a first segment of a speech input: determining that the first segment of the speech input satisfies a stability criterion; and, in response to determining that the first segment of the speech input satisfies the stability criterion, obtaining a first candidate transcript of the first segment of the speech input. The stability criterion includes one or more semantic characteristics of the first segment of the speech input. The stability criterion includes a time delay that occurs after the first segment of the speech input. The second segment of the speech input occurs after the first segment of the speech input. The one or more contexts are received from a user device. The one or more contexts include the following data: the data includes a user's geographic location, a user's search history, a user's interests, or a user's activities. The method includes: storing multiple scores for multiple contexts; and, updating the adjusted scores of the one or more contexts in response to adjusting the corresponding weights of each of the one or more contexts. The method also includes providing the output as a search query, such as to a search engine, which can then provide one or more search results to a user device in response to the search query. The first candidate transcription includes a word, a subword, or a phrase.
[0008] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. Compared to conventional speech recognition systems, the speech recognition system is able to provide more accurate text search queries based on segmentation of speech input. Because the system adjusts the weights of the context based on the segmentation of the speech input and determines the transcripts of subsequent segments of the speech input based in part on the adjusted weights, the system is able to dynamically improve recognition performance. Therefore, the system is able to improve the accuracy of speech recognition. This improved accuracy reduces the likelihood that a user must repeat the process of providing speech input for the speech recognition system to process, thereby allowing the speech recognition system to process additional speech inputs.
[0009] In the drawings and the following description, details of one or more embodiments of the subject matter of the present specification are set forth. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. It should be understood that these aspects and embodiments can be combined with each other, and that features described in the context of one aspect or embodiment can also be implemented in the context of other aspects or embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 is a diagram that provides an exemplary speech recognition system.
[0011] Figure 2 is a diagram showing an exemplary scenario.
[0012] Figure 3 is a diagram illustrating an exemplary process for determining whether stability criteria are met.
[0013] Figure 4 is a flow chart of an exemplary method for providing a transcription of speech input.
[0014] Figure 5 is a flow chart of an exemplary method for determining a transcript of speech input.
[0015] Like reference numerals in the various drawings denote like elements. DETAILED DESCRIPTION
[0016] Figure 1 is a diagram that provides an exemplary speech recognition system 100. The speech recognition system 100 includes one or more computers that are programmed to: receive speech input 110 from a user 10 from a user device 120, determine a transcript of the speech input 110, and provide the transcript of the speech input 110 as an output. Figure 1In the example shown, the output may be a search query 150, which is provided to a search engine 160 to obtain search results 170 in response to the search query 150. One or more search results 170 are then provided to the user device 120. The speech recognition system 100 may be implemented, for example, on one or more computers including a server or on a user device.
[0017] The speech recognition system 100 includes a speech recognition engine 140 that communicates with a user device 120 through one or more networks 180. The one or more networks 180 can be telephone and / or computer networks, including wireless cellular networks, wireless local area networks (WLAN) or Wi-Fi networks, wired Ethernet, other wired networks, or any appropriate combination thereof. The user device 120 can be any appropriate type of computing device, including but not limited to: a mobile phone, a smart phone, a tablet computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, or other handheld or mobile devices including one or more processors and a computer-readable medium. The user device 120 is configured to receive a speech input 110 from a user 10. The user device 120 may include or be connected to, for example, an acoustic-electric transducer or sensor (e.g., a microphone). In response to the user 10 inputting the speech input 110, the speech input may be submitted to the speech recognition engine 140. (In general, this can be accomplished by submitting data representing the speech input or encoding the speech input to the speech recognition engine 140. The speech recognition engine 140 may process the data to extract the speech input from the received data).
[0018] The speech recognition engine 140 can recognize the speech input in sequence, for example, the first part 111 of the speech input 110 can be recognized, and then the second part 112 of the speech input 110 can be recognized. Based on a specific stability criterion, one or more parts of the speech input 110 can be recognized as independent segments of the speech input 110. A portion thereof may include a word, a sub-word, or a phrase. In some embodiments, as described in more detail below, one or more segments of the speech input 110 can provide intermediate recognition results that can be used to adjust one or more contexts.
[0019] Although an example of a search query is used throughout this document for illustration, voice input 110 can represent any type of voice communication, including voice-based instructions, search engine query terms, dictation, a dialog system, or any other input that uses transcribed speech or invokes a software application that uses transcribed speech to perform an action.
[0020] The speech recognition engine 140 may be a software component of the speech recognition system 100 configured to receive and process the speech input 110. Figure 1 As shown, the speech recognition engine 140 converts the speech input 110 into a text search query 150 that is provided to the search engine 160. The speech recognition engine 140 includes a speech decoder 142, a context module 144, and a context adjustment module 146. The speech decoder 142, the context module 144, and the context adjustment module 146 may be software components of the speech recognition system 100.
[0021] When the speech recognition engine 140 receives the speech input 110, the speech decoder 142 determines a transcript of the speech input 110. The speech decoder 142 then provides the transcript of the speech input 110 as an output, such as a search query 150 to be provided to the search engine 160.
[0022] The speech decoder 142 uses a language model to generate candidate transcriptions of the speech input 110. The language model includes probability values associated with words or sequences of words. For example, the language model may be an N-gram model. As the speech decoder 142 processes the speech input, intermediate recognition results may be determined. Each intermediate recognition result corresponds to a stable segment of the transcription of the speech input 110. Figure 3 The stability criteria used to determine stable segments of the transcript are described in more detail.
[0023] The speech decoder 142 provides each stable segment to the context adjustment module 146. The context adjustment module 146 identifies the relevant contexts from the context module 144. Each context identified can be associated with a weight. The basic weight of each context can be initially specified based on various criteria, such as the ubiquity of the context, the temporal proximity of the context (i.e., whether a particular context has been frequently used in a recent time period), or the recent or overall use of the context. The basic weight may generate an initial deviation based on the likelihood that the user's input is associated with a particular context. Once the context adjustment module 146 identifies the relevant context, the context adjustment module 146 adjusts the weight of the context based on one or more stable segments provided by the speech decoder 142. The weight can be adjusted to indicate the degree to which the transcription of the speech input is associated with a particular context.
[0024] The context module 144 stores contexts 148 and weights associated with the contexts 148. The context module 144 may be a software component of the speech recognition engine 140 that is configured to cause the computing device to receive one or more contexts 148 from the user device 120. The speech recognition engine 140 may be configured to store the received contexts 148 in the context module 144. In some cases, the context module 144 may be configured to generate one or more contexts 148 customized for the user 10. The speech recognition engine 140 may be configured to store the generated contexts 148 in the context module 144.
[0025] The context 148 may include, for example: (1) data describing user activities, such as the time interval between multiple repeated voice inputs, gaze tracking information reflecting eye movements from a front-side camera near the screen of the user's device; (2) data describing the situation when the voice input is issued, such as the type of mobile application used, the user's location, the type of device used, or the current time; (3) previous voice search queries submitted to the search engine; (4) data describing the type of voice input submitted to the voice recognition engine, such as commands, requests, or search queries to the search engine, and (5) entities, such as members of a specific category, place names, etc. For example, multiple contexts can be formed based on previous search queries, user information, entity databases, etc.
[0026] Figure 2 is a diagram showing an exemplary context. The speech recognition engine is configured to store a context 210 associated with "Tennis Players" and a context 220 associated with "Basketball Players", for example, in a context module, such as context module 144. Context 210 includes entities corresponding to specific tennis players, such as "Roger Federer", "Rafael Nadal", and "Novak Djokovic". Context 220 includes entities corresponding to specific basketball players, such as "Roger Bederer", "Rafael Madall", and "Novak Jokovich".
[0027] The context module 144 may be configured to store weights of the contexts 210, 220. The weights may represent the degree to which one or more transcriptions of the speech input are associated with the contexts 210, 220. When the context adjustment module 146 identifies the contexts 210, 220, the context adjustment module also identifies the weights associated with the contexts 210, 220.
[0028] When the speech decoder 142 obtains the first candidate transcription "Howmany wins does tennis player" for the first segment 111 of the speech input 110, the speech decoder 142 provides the first candidate transcription of the first segment 111 to the context adjustment module 146. The context adjustment module 146 identifies the contexts 210, 220 as relevant contexts in the context module 144 and identifies weights associated with the contexts 210, 220. The context adjustment module 146 is then configured to adjust the respective weights of the contexts 210, 220 based on the first candidate transcription of the first segment 111 of the speech input 110. In particular, the context adjustment module 146 is capable of adjusting the respective weights of the contexts 210, 220 for use in identifying subsequent segments of the speech input 110.
[0029] The basic weights of the various contexts may initially bias speech recognition toward the context of "basketball" having a higher initial weight, for example, because of the historical popularity of basketball-related speech inputs compared to tennis. However, after adjustment based on the intermediate recognition result, speech recognition may be biased toward the context of "tennis". In this example, the first candidate transcription of speech input 110, "How many wins does tennis player", includes the word "tennis player". Based on the word "tennis player" in the first candidate transcription, the context adjustment module 146 can be configured to adjust the weights of one or more contexts. For example, the context adjustment module 146 can increase the weight of context 210, for example, from "10" to "90", can reduce the weight of context 220, for example, from "90" to "10", or can perform a combination of increasing and reducing weights.
[0030] The speech decoder 142 can be configured to determine a second candidate transcription for the second segment 112 of the speech input 110 based in part on the adjusted weights. In response to the adjustment of the respective weights of the contexts, the speech recognition engine 140 can be configured to update the adjusted weights of the contexts 210, 220 in the context module 144. In the above example, to determine the second candidate transcription for the second segment 112 of the speech input 110, the speech decoder 142 can assign a greater weight to the context 210 than to the context 220 based on the adjusted weights. Based on the weight of the context 210, the speech decoder 142 can determine "Roger Federer" as the second candidate transcription for the second segment 112 of the speech input 110.
[0031] In contrast, if the context adjustment module 146 does not adjust the weights of the contexts 210, 220 based on the first candidate transcription of the first segment 111, the speech decoder 142 may determine the second candidate transcription of the second segment 112 based on the basic weights of the contexts 210, 220 stored in the context module 144. If the weight of the context 220 is greater than the weight of the context 210, the speech decoder may determine the name of a basketball player such as "Roger Bederer" as the second candidate transcription of the second segment 112. Therefore, the speech decoder 142 may provide an incorrect recognition result.
[0032] After the speech decoder 142 obtains the transcript of the entire speech input 110, the speech decoder 142 may provide the transcript of the speech input 110 for output. The output may be provided directly to the user device or used for further processing. Figure 1 In the example, the output recognition is used as a text search query 150. For example, when the speech decoder 142 determines "RogerFederer" as the second candidate transcription of the second segment 112 of the speech input 110, the speech decoder 142 can output the entire transcription "How many wins does tennis player Roger Federer have?" as the search query 150 to the search engine 160.
[0033] The search engine 160 performs a search using the search query 150. The search engine 160 may include a web search engine coupled to the speech recognition system 100. The search engine 160 may determine one or more search results 170 in response to the search query 150. The search engine 160 provides the search results 170 to the user device 120. The user device 120 may have a display interface for presenting the search results 170 to the user 10. In some cases, the user device 120 may have an audio interface to present the search results 170 to the user 10.
[0034] Figure 3 is a diagram illustrating an exemplary process for determining whether a given segment satisfies a stability criterion. The speech decoder 142 is configured to determine that the portion of the speech input 110 satisfies the stability criterion.
[0035] When speech decoder 142 receives portion 311 of speech input 310, speech decoder 142 may be configured to determine whether portion 311 of speech input 310 satisfies a stability criterion. The stability criterion indicates whether the portion is likely to be altered by additional speech recognition.
[0036] The stability criteria may include one or more semantic features. If a portion of a speech input is semantically expected to be followed by one or more words, the speech decoder 142 may determine that the portion does not meet the stability criteria. For example, when the speech decoder 142 receives portion 311 of the speech input 310, the speech decoder 142 may determine that it is semantically expected that the portion 311 will be followed by one or more words. Then, the speech decoder 142 determines that the portion 311 does not meet the stability criteria. In some embodiments, when the speech decoder 142 receives "mine" as part of a speech input, the speech decoder 142 may determine that it is semantically expected that the portion "mine" will not be followed by one or more words. Then, the speech decoder 142 may determine that the portion "mine" meets the stability criteria of the segment. The speech decoder 142 may provide the segment to the context adjustment module 146 to adjust the weight of the context.
[0037] The speech decoder 142 may also determine that a portion does not meet the stability criteria if it is semantically expected that another subword or subwords will follow the portion. For example, when the speech decoder 142 receives "play" as portion 312 of the speech input 310, the speech decoder 142 may determine that it is semantically expected that the portion 312 will be followed by one or more words because, semantically, the portion 312 may be followed by one or more subwords such as "play-er", "play-ground", and "play-off". The speech decoder 142 then determines that the portion 311 does not meet the stability criteria. In some embodiments, when the speech decoder 142 receives "player" as a portion of the speech input, the speech decoder 142 may determine that it is semantically expected that the portion "player" will not be followed by one or more words. The speech decoder 142 may then determine that the portion "player" meets the stability criteria for the segment. The speech decoder 142 may provide the segment to the context adjustment module 146 to adjust the weight of the context.
[0038] In some implementations, the stability criteria may include a time delay that occurs after a portion of the speech input 310. If the length (duration) of the time delay after the portion of the speech input 310 satisfies a threshold delay value, the speech decoder 142 may determine that the portion of the speech input 310 satisfies the stability criteria. When the speech decoder 142 receives the portion of the speech input 310, the speech decoder 142 may measure the time delay from the time the portion is received to the time a subsequent portion of the speech input 310 is received. If the time delay exceeds the threshold delay value, the speech decoder 142 may determine that the portion satisfies the stability criteria.
[0039] Figure 4 4 is a flow chart of an exemplary method 400 for determining a transcript of received speech input. For convenience, the method 400 will be described by a system that performs the method 400.
[0040] The system processes (410) the received speech input in the order in which it is spoken to determine a portion of the speech input as a first segment. The system obtains (420) a first candidate transcript for the first segment of the speech input. In order to obtain the first candidate transcript for the first segment, the system may determine whether the first segment of the speech input meets the stability criteria. If the first segment of the speech input meets the stability criteria, the system may obtain the first candidate transcript for the first segment. If the first segment of the speech input does not meet the stability criteria, the system may not obtain the first candidate transcript. The system may then receive one or more portions of the speech input and identify a new first segment of the speech input to determine whether the new first segment of the speech input meets the stability criteria. As described above Figure 3 As described, the system may use process 300 to determine that a first segment of the speech input satisfies a stability criterion.
[0041] The system determines (430) one or more contexts associated with the first segment from the set of contexts. A specific context associated with the first segment can be determined based on the context provided by the first segment. For example, a specific keyword of the first segment can be identified as being associated with a specific context. Figure 2 , the system can identify contexts associated with "tennis players" and contexts associated with "basketball players". The tennis player context can be associated with keywords such as "RogerFederer", "Rafael Nadal", and "Novak Djokovic". The basketball player context can be associated with keywords such as "Roger Bederer", "Rafael Madall", and "Novak Jocovich". The system can be configured to store the weight of each context. When the system identifies a context, the system can also identify the corresponding weight of the context. The corresponding weight of the context represents the degree to which one or more transcribed texts of the speech input are associated with the context.
[0042] The system adjusts (440) the corresponding weight of each of the one or more contexts. The system can adjust the corresponding weight of each context based on a first candidate transcription of the speech input. For example, the first candidate transcription of the speech input "How many wins does tennis player" includes the word "tennis player". Based on the word "tennis player" in the first candidate transcription, the system can be configured to adjust the weight of the context. For example, the system can increase the weight of the context, such as from "10" to "90", can decrease the weight of the context, such as from "90" to "10", or can be a combination of increasing and decreasing the weight.
[0043] In some embodiments, only the weight of the most relevant context is adjusted (e.g., increased), while all other contexts remain constant. In some other embodiments, all other contexts are de-weighted, while the most relevant context remains constant. In addition, any suitable combination of the two can be generated. For example, the boost value of one relevant context can be different from the de-weight value of another context.
[0044] The system determines (450) a second candidate transcription for a second segment of the speech input based in part on the adjusted weights. In response to adjusting the corresponding weights of the contexts, the system can update the adjusted weights of the contexts. For example, the system can give more weight to a first context that is identified as being more relevant to the first segment than to a second context based on the adjusted weights. Based on the adjusted weights, the speech decoder can determine a second candidate transcription for the second segment of the speech input. The process continues until there are no additional portions of the speech input to be recognized.
[0045] Figure 5 is a flow chart of an exemplary method 500 for providing voice search. For convenience, the method 500 will be described with respect to a system that performs the method 500.
[0046] The system receives (510) speech input. The system may be configured to receive speech input from a user. The system may receive each segment of the speech input in real time as the user speaks.
[0047] When the system receives speech input, the system determines (520) a transcript of the speech input. Figure 4As described, the system determines the transcript. Once the system determines (520) the entire transcript of the speech input, the system provides (530) the transcript of the speech input for output. The system can provide the output as a text search query. The system can use the text search query to perform a search and obtain search results. The system can provide the search results to a user. In some embodiments, the system can provide a display interface to present the search results to the user. In other embodiments, the system can provide an audio interface to present the search results to the user.
[0048] The embodiments of the subject matter and operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs encoded on a computer storage medium, that is, one or more modules of computer program instructions, for being executed by a data processing device or controlling the operation of a data processing device. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as an electrical, optical or electromagnetic signal generated by a machine, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium may also be or be included in one or more separate physical components or media, such as multiple CDs, disks or other storage devices.
[0049] The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.
[0050] The term "data processing device" includes all types of devices, equipment and machines for processing data, such as programmable processing units, computers, systems on chips, personal computer systems, desktop computers, laptop computers, notebook computers, netbook computers, large computer systems, handheld computers, workstations, network computers, application servers, storage devices, consumer electronic devices such as cameras, video cameras, set-top boxes, mobile devices, video game consoles, peripheral devices such as switches, modems, routers, or generally any type of computing or electronic equipment, or a plurality of or a combination of the foregoing. The device may include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include a code for creating an execution environment for the computer program, such as constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine or a combination of one or more of them. The device and execution environment may implement various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.
[0051] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can but need not correspond to a file in a file system. A program can be stored in a portion of a file that stores other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple collaborative files (e.g., a file that stores one or more modules, subroutines, or a portion of a code). A computer program can be deployed to execute on a single computer or on multiple computers located in the same location or distributed in multiple locations and interconnected by a communication network.
[0052] The processes and logic flows described in this specification may be performed by one or more programmable processing units executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and the apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
[0053] Processing units suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors and any one or more processing units of any kind of digital computer. Typically, the processing unit receives instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processing unit for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a magnetic disk, a magneto-optical disk or an optical disk, or operably coupled to the one or more large-capacity storage devices for storing data, to receive data from it or to transmit data to it. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a palmtop computer (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, a network routing device, or a portable storage device, such as a universal serial bus (USB) flash drive, which is just an example. Suitable devices for storing computer program instructions and data include any form of non-volatile memory, media and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM disks and DVD-ROM disks. The processing unit and memory may be supplemented by, or incorporated in, special purpose logic circuitry.
[0054] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, speech, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.
[0055] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or a routing device (e.g., a network router), or a combination of one or more such back-end, middleware, or front-end components. The multiple components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnections (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).
[0056] The computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is established by a computer program executed on a corresponding computer and having a client-server relationship. In some embodiments, the server sends data (e.g., an HTML page) to the client device (e.g., for displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., a result of user interaction) can be received from the client device at the server.
[0057] A system of one or more computers may be configured to perform specific actions by software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform actions in operation. One or more computer programs may be configured to perform specific actions by including instructions that, when executed by a data processing device, cause the device to perform actions.
[0058] Although this specification contains many specific implementation details, it should not be interpreted as a limitation on any invention or the scope of protection that can be claimed, but should be interpreted as a description of the specific features of a specific embodiment of a specific invention. Certain features described in the context of a separate embodiment in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even as originally claimed, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.
[0059] Similarly, although the operations are depicted in a particular order in the accompanying drawings, it should not be understood that these operations are required to be performed in the particular order shown or in a sequential order, or that all the operations shown are performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, in the above-mentioned embodiments, the separation of various system components should not be understood as requiring such separation in all embodiments, but should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.
[0060] Therefore, specific embodiments of the subject matter are described. Other embodiments are also within the scope of the appended claims. In some cases, the actions described in the claims can be performed in different orders and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous. Therefore, other embodiments are also within the scope of the appended claims.
Claims
1. A speech recognition method, comprising: At an automatic speech recognition (ASR) system implemented on a user device, receiving a user's voice input captured by a microphone of the user device, the voice input being spoken by a user of the user device to perform an action; determining, by the ASR system, a specific context associated with the speech input, the specific context being customized for the user and comprising a named entity list corresponding to the specific context, the named entity list being stored on the user device; as well as processing, by the ASR system, the speech input using an N-gram language model including likelihood values associated with words or sequences of words to generate a transcription of the speech input, the N-gram language model biasing the transcription of the speech input to include one of the named entities in the named entity list stored on the user device corresponding to the specific context by adjusting weights associated with the specific context; as well as The transcription biased by the N-gram language model is provided by the ASR system for output from the ASR system to invoke a software application to perform the action.
2. The method according to claim 1, wherein: The voice input from the user is configured to invoke the software application to perform the action using the transcription of the voice input.
3. The method according to claim 1, wherein: The specific context associated with the speech input includes a corresponding weight indicating a likelihood that the speech input is associated with a specific context.
4. The method according to claim 1, wherein: Determining the specific context associated with the voice input includes determining the specific context based on a type of the software application invoked by the voice input to perform the action.
5. The method according to claim 1, wherein: Determining the specific context associated with the speech input includes determining the specific context based on data describing a type of the speech input received at the ASR system.
6. The method according to claim 1, wherein: The microphone is configured to provide captured speech input to the ASR system.
7. The method according to claim 1, wherein: Determining the specific context associated with the voice input includes determining the specific context based on a type of the user device.
8. A user equipment, comprising: microphone; Data processing hardware; as well as Memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including: At an automatic speech recognition (ASR) system implemented on the user device, receiving a user's voice input captured by a microphone of the user device, the voice input being spoken by a user of the user device to perform an action; determining a specific context associated with the speech input, the specific context being customized for the user and comprising a named entity list corresponding to the specific context, the named entity list being stored on the user device; as well as processing, by the ASR system, the speech input using an N-gram language model including likelihood values associated with words or sequences of words to generate a transcription of the speech input, the N-gram language model biasing the transcription of the speech input to include one of the named entities in the named entity list stored on the user device corresponding to the specific context by adjusting weights associated with the specific context; as well as The transcription biased by the N-gram language model is provided by the ASR system for output from the ASR system to invoke a software application to perform the action.
9. The user equipment according to claim 8, wherein: The voice input from the user is configured to invoke the software application to perform the action using the transcription of the voice input.
10. The user equipment according to claim 8, wherein: The specific context associated with the speech input includes a corresponding weight indicating a likelihood that the speech input is associated with a specific context.
11. The user equipment according to claim 8, wherein: Determining the specific context associated with the voice input includes determining the specific context based on a type of the software application invoked by the voice input to perform the action.
12. The user equipment according to claim 8, wherein: Determining the specific context associated with the speech input includes determining the specific context based on data describing a type of the speech input received at the ASR system.
13. The user equipment according to claim 8, wherein: The user device includes a microphone configured to capture the speech input spoken by the user and provide the speech input to the ASR system.
14. The user equipment according to claim 8, wherein: Determining the specific context associated with the voice input includes determining the specific context based on a type of the user device.
15. A speech recognition method, comprising: At an automatic speech recognition (ASR) system, receiving a current speech input from a user, the current speech input being associated with at least two contexts, each of the at least two contexts having a corresponding weight, the corresponding weight indicating a likelihood that the speech input is associated with the corresponding context; generating, by the ASR system, an intermediate recognition result of the current speech input from the user; adjusting, by the ASR system, the respective weights of the at least two contexts based on the intermediate recognition result; as well as The current speech input is transcribed by the ASR system using a language model, the language model biasing the transcription of the speech input toward one of the at least two contexts based on the adjusted weights.
16. The method according to claim 15, wherein: The language model includes an N-gram model.
17. The method according to claim 15, wherein: Adjusting the corresponding weights of the at least two contexts associated with the current speech input includes increasing a corresponding basic weight of at least one context of the at least two contexts.
18. The method according to claim 15, wherein: Adjusting the respective weights of the at least two contexts based on the intermediate recognition result includes: Determining a most relevant one of the at least two contexts by identifying a specific keyword in the intermediate recognition result; and The corresponding weight of the most relevant one of the at least two contexts is increased.
19. The method according to claim 15, wherein: The current voice input from the user is configured to invoke a software application to perform an action using the transcription of the current voice input.
20. The method of claim 15, further comprising providing the transcription of the current speech input to a dialog system interacting with the user.
21. The method according to claim 15, wherein: At least one of the at least two contexts includes data indicating a current time when the current speech input is received at the ASR system.
22. The method according to claim 15, wherein: At least one context of the at least two contexts associated with the current voice input is based on one or more previous voice inputs from the user within a past time period of the current voice input.
23. The method according to claim 15, wherein: At least one of the at least two contexts includes a named entity associated with a particular category.
24. The method according to claim 15, wherein: The ASR system resides on a server in communication with a computing device associated with the user, the computing device being configured to capture the current speech input spoken by the user and transmit the captured speech input to the ASR system.
25. An automatic speech recognition (ASR) system, comprising: Data processing hardware; as well as Memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including: receiving a current voice input from a user, the current voice input being associated with at least two contexts, each of the at least two contexts having a corresponding weight, the corresponding weight indicating a likelihood that the voice input is associated with the corresponding context; generating an intermediate recognition result of the current speech input from the user; adjusting the respective weights of the at least two contexts based on the intermediate recognition result; as well as The current speech input is transcribed using a language model, the language model biasing the transcription of the speech input toward one of the at least two contexts based on the adjusted weights.
26. The ASR system of claim 25, wherein: The language model includes an N-gram model.
27. The ASR system of claim 25, wherein: Adjusting the corresponding weights of the at least two contexts associated with the current speech input includes increasing a corresponding base weight of at least one context of the at least two contexts.
28. The ASR system of claim 25, wherein: Adjusting the respective weights of the at least two contexts based on the intermediate recognition result includes: Determining a most relevant one of the at least two contexts by identifying a specific keyword in the intermediate recognition result; and The corresponding weight of the most relevant one of the at least two contexts is increased.
29. The ASR system of claim 25, wherein: The current voice input from the user is configured to invoke a software application to perform an action using the transcription of the current voice input.
30. The ASR system of claim 25, wherein: The operations also include providing the transcription of the current speech input to a dialog system interacting with the user.
31. The ASR system of claim 25, wherein: At least one of the at least two contexts includes data indicating a current time when the current speech input is received at the ASR system.
32. The ASR system of claim 25, wherein: At least one context of the at least two contexts associated with the current voice input is based on one or more previous voice inputs from the user within a past time period of the current voice input.
33. The ASR system of claim 25, wherein: At least one of the at least two contexts includes a named entity associated with a particular category.
34. The ASR system of claim 25, wherein: The data processing hardware and the memory hardware reside on a server in communication with a computing device associated with the user, the computing device being configured to capture the current speech input spoken by the user and transmit the current speech input to the ASR system.
35. A computer-implemented method comprising: receiving audio data corresponding to a voice input of a user, the voice input comprising a first segment and a second segment; providing for output a transcription of the speech input of the user, the transcription comprising a first transcription corresponding to the first segment and a second transcription corresponding to the second segment, wherein: The first transcription of the first segment is associated with one or more contexts, the one or more contexts being respectively associated with one or more base weights; as well as The second transcription of the second segment is determined based on adjusting a corresponding base weight of the one or more base weights for each of the one or more contexts based on the first transcription.
36. The method of claim 35, wherein: The one or more contexts include data including a user's geographic location, a user's search history, a user's interests, or a user's activities.
37. The method of claim 35, further comprising maintaining data representative of the one or more contexts.
38. The method of claim 35, further comprising: receiving one or more search results responsive to the transcription; as well as The one or more search results are provided to the user.
39. The method of claim 35, further comprising: determining that the first segment of the speech input satisfies a stability criterion; as well as In response to determining that the first segment of the speech input satisfies the stability criterion, the first transcription of the first segment is obtained.
40. The method of claim 39, wherein: The stability criteria include one or more semantic features of the first segment of the speech input.
41. The method of claim 39, wherein: The stability criterion includes a time delay occurring after the first segment of the speech input.
42. A speech recognition system, the system comprising: One or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, are operable to cause the one or more computers to perform operations comprising: receiving audio data corresponding to a voice input of a user, the voice input comprising a first segment and a second segment; and providing for output a transcription of the speech input of the user, the transcription comprising a first transcription corresponding to the first segment and a second transcription corresponding to the second segment, wherein: The first transcription of the first segment is associated with one or more contexts, the one or more contexts being respectively associated with one or more base weights; and The second transcription of the second segment is determined based on adjusting a corresponding base weight of the one or more base weights for each of the one or more contexts based on the first transcription.
43. The system of claim 42, wherein: The one or more contexts include data including a user's geographic location, a user's search history, a user's interests, or a user's activities.
44. The system of claim 42, wherein: The operations also include maintaining data representative of the one or more contexts.
45. The system of claim 42, wherein: The operations also include: receiving one or more search results responsive to the transcription; and The one or more search results are provided to the user.
46. The system of claim 42, wherein: The operations also include: determining that the first segment of the speech input satisfies a stability criterion; and In response to determining that the first segment of the speech input satisfies the stability criterion, the first transcription of the first segment is obtained.
47. The system of claim 46, wherein: The stability criteria include one or more semantic features of the first segment of the speech input.
48. The system of claim 46, wherein: The stability criterion includes a time delay occurring after the first segment of the speech input.
49. A non-transitory computer-readable medium storing software, the software comprising instructions executable by one or more computers, the instructions, when so executed, causing the one or more computers to perform operations comprising: receiving audio data corresponding to a voice input of a user, the voice input comprising a first segment and a second segment; as well as providing for output a transcription of the speech input of the user, the transcription comprising a first transcription corresponding to the first segment and a second transcription corresponding to the second segment, wherein: The first transcription of the first segment is associated with one or more contexts, the one or more contexts being respectively associated with one or more base weights; as well as The second transcription of the second segment is determined based on adjusting a corresponding base weight of the one or more base weights for each of the one or more contexts based on the first transcription.
50. The non-transitory computer readable medium of claim 49, wherein: The operations also include maintaining data representative of the one or more contexts.
51. The non-transitory computer readable medium of claim 49, wherein: The operations also include: receiving one or more search results responsive to the transcription; and The one or more search results are provided to the user.
52. The non-transitory computer readable medium of claim 49, wherein: The operations also include: determining that the first segment of the speech input satisfies a stability criterion; and In response to determining that the first segment of the speech input satisfies the stability criterion, the first transcription of the first segment is obtained.
53. The non-transitory computer readable medium of claim 52, wherein: The stability criteria include one or more semantic features of the first segment of the speech input.
54. The non-transitory computer readable medium of claim 52, wherein: The stability criterion includes a time delay occurring after the first segment of the speech input.
Citation Information
Patent Citations
Device and method for reducing recognition error via context relation in dialog bouts
CN101266793A
Enhanced stability prediction for incrementally generated speech recognition hypotheses
CN103918026A
Enhanced stability prediction for incrementally generated speech recognition hypotheses
US20130110492A1
Adjusting language models
US8352245B1
Dynamic language model
WO2013192218A2
