Multi-participant voice ordering

Abstract

A voice interface recognizes spoken utterances from a plurality of users. The voice interface responds to these utterances by modifying item instance attributes, etc. The voice interface computes a voice vector for each utterance and associates it with the modified item instance. For subsequent utterances with highly matched speech vectors, the speech interface will modify the same instance; for subsequent utterances for which the speech vector does not match the speech vector stored for any item instance, the speech interface will modify different item instances.

Description

[0001] Cross-references

[0002] This application claims the benefit of U.S. patent application No. 18 / 391,886, filed on December 21, 2023, which claims priority to U.S. Provisional Patent Application No. 63 / 476,928, filed on December 22, 2022, which are incorporated herein by reference. Background Art

[0003] Computerized speech recognition systems are currently used in various scenarios to receive voice input from multiple users, but with limited success. One example is using speech recognition systems to receive food orders. One challenge is distinguishing between different speakers. A second difficulty is recognizing the speech of multiple users. Given these difficulties, speech recognition systems have not yet been successfully applied in such scenarios. Summary of the Invention

[0004] The following description describes systems and methods for recognizing spoken utterances from multiple speakers and distinguishing between speakers based on their speech. Such systems then modify one of multiple items of the same type, where the modified item corresponds to the user being recognized.

[0005] This summary is provided to introduce a series of concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all of the disadvantages identified in the background. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 The segmentation of audio into request utterances is shown.

[0007] Figure 2 Processing of utterances to compute speech vectors and recognize spoken requests is shown.

[0008] Figure 3 A fast food order data structure is shown as it changes over an utterance sequence.

[0009] Figure 4 A flow chart illustrating a process for modifying one or another instance of a type of item based on speech is shown.

[0010] Figure 5 It shows a user ordering food by voice at a self-service terminal in a fast food restaurant.

[0011] Figure 6AA flash RAM chip is shown.

[0012] Figure 6B A system on a chip is shown.

[0013] Figure 6C A functional diagram of the system on a chip is shown. DETAILED DESCRIPTION

[0014] Various devices, systems of connected devices, API-controlled cloud services, and other things that present a computerized voice interface are capable of receiving audio, detecting speech contained in the audio, inferring a transcription of the speech, and understanding the transcription as a query or command. Such voice interfaces are then capable of acting on the understood query or command by performing an action, retrieving information, or determining that the action cannot be performed, and then responding accordingly in the form of information that may be useful to the user.

[0015] Some voice interfaces receive audio directly from a microphone or from digital samples of the air pressure waves driving the microphone. Some voice interfaces receive digital audio from a remote device as direct digital samples, frames of a frequency domain transform of such sampled digital signals, or compressed representations thereof. Examples of formats for audio representation include WAV, MP3, and Speex.

[0016] Voice interfaces on devices such as mobile phones output information directly on a display, through a speaker using synthesized speech, through a haptic vibrator, or using other actuator functions of the phone. Some voice interfaces (e.g., APIs hosted by cloud servers) output information as a response message to a request message. The output information may include textual or spoken answers to questions, confirmation that an action or other invoked function has been initiated or completed, or the status of the interface, device, system, server, or data stored on any of them.

[0017] One example is a voice interface for ordering food from a restaurant. Such interfaces operate in conversations that end with payment and begin with a subsequent user interaction. In some cases, a user interaction begins when the interface detects that a person has spoken a specific wake-up phrase or senses that a person has manually interacted with the device. In some cases, the voice interface continuously performs speech recognition and, for words recognized with a sufficiently high confidence, matches them to patterns that correspond to an understanding of the speaker's intent in saying those words.

[0018] In order to infer the understanding of the words spoken in a continuous sequence of audio, it is necessary to segment the audio. This can be achieved in several ways. One way is to run a voice activity detection function on the audio and determine the start of a segment as the time when voice activity is detected until the time when voice activity is no longer detected (if no further voice activity is detected for a certain period of time).

[0019] Another way to segment audio is to identify semantically complete word sequences. This can be done by comparing the most recent word sequence in the buffer to a pattern. It is possible to handle situations where a semantically complete pattern is a prefix of another pattern by implementing a delay in the range of approximately 1-10 seconds after the match and discarding the match if a match to a longer pattern occurs within the delay.

[0020] To avoid erroneous matching of a pattern where the end of an earlier sequence and the start of an unrelated subsequent sequence would match the pattern, it is possible to reset the word sequence after a period of time (e.g. 5-30 seconds) in which no new words are added to the buffer. Thus, an item will only be modified by commands if they are received within a period of less than this many seconds.

[0021] For semantic segmentation, it may be helpful to label each word with the approximate wall-clock time at which recognition of each word began from the input audio and / or the approximate wall-clock time at which speech recognition ended for that word. The approximate time at which recognition of the first word in a semantically complete sequence of words begins is the start time of a segment. The approximate time at which recognition of the last word in the semantically complete sequence ends is the end time of a segment.

[0022] Figure 1 A diagrammatic view of audio segmentation is shown. The segmentation function operates on the audio stream. This can occur continuously in real time, incrementally, or offline for non-real time analysis. The segmentation function calculates the start and end times of the segments in the audio stream and outputs the separated audio segments. Figure 1 In [1], each audio segment comprises a request to the voice interface.

[0023] Speech Vector

[0024] One way to achieve multi-participant voice ordering is to identify the voices by numerically representing them. Calculating a value for a voice along a single dimension can achieve gender-based identification, but may not be sufficient to distinguish between people with similar sounding voices. A vector with multiple numbers representing the voice's sound along each of the different dimensions provides greater accuracy in voice characterization and identification. In many cases, it can even distinguish the voices of identical twins.

[0025] Choosing the right dimensions improves accuracy. Choosing specific dimensions (e.g., estimating gender, age, or even regional accent) is feasible. But using machine learning on large datasets with highly diverse speech to learn a multidimensional space using training that maximizes the dispersion of the calculated speech vectors in the space can be even more accurate.

[0026] Using an appropriate multidimensional vector space, it is possible to represent speech sounds in speech audio as points represented by vectors. One approach to achieving this representation is to continuously compute d-vectors for each frame of audio or a relatively small number of samples. The paper "Deep Neural Networks for Small Footprint Text-Dependent Speaker Verification" by Ehsan Variani, Xin Lei, Erik McDermott, Ignacio Lopez Moreno, and Javier Gonzalez-Dominguez describes a method for computing d-vectors using a deep neural network (DNN).

[0027] One way to calculate the speech vector for an entire segment is to aggregate the d-vectors calculated for each audio frame from the start time to the end time of the segment. Aggregation can be achieved in various ways, for example, by computing the inter-frame average dimension by dimension. In some cases, it may also be helpful to exclude d-vectors calculated for frames where the energy is dispersed in the spectrum (such as is common during the pronunciation of the phonemes 's' and 'sh'). The D-vectors calculated for such frames may sometimes have noise added to them, which reduces the accuracy of the aggregated speech vector calculation. The advantage of a continuous frame-by-frame approach for calculating speech vectors during a segment is that the CPU cycle requirements are relatively constant and are not affected by the length of the speech segment.

[0028] Another way to calculate the speech vector for the entire segment is to calculate the x-vector for the entire segmented utterance once the end of the segment is detected. The paper "X-VECTORS: ROBUST DNN EMBEDDINGS FOR SPEAKER RECOGNITION" by David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur describes a method for calculating x-vectors using DNNs. It is possible to calculate the x-vector once for each segment after it is fully recognized. In some cases, the following operation is more energy-efficient: buffer the audio while performing the segmentation, and then wake up the faster, high-performance CPU only a short time to calculate the x-vector for the buffered audio data for the full length of the segment.

[0029] Computing speech vectors can be used instead of or in addition to other segmentation functions. One approach is to compute relatively short-term d-vectors and longer, aggregated d-vectors. As long as the spoken speech is the same, they will be similar. When speech changes in the audio, they will diverge. The sum of the per-dimension differences between the short-term and long-term averaged d-vectors indicates the segment transition at or shortly before detectable divergence begins.

[0030] Multiple voices

[0031] To implement multi-participant voice ordering, it may be useful to segment the utterances, calculate voice vectors for the segments, and then apply pattern matching or other forms of natural language understanding to act on the voice requests. The action taken in response to the voice request may then depend on which of the multiple voices made the request. For example, if items that are members of a list are associated with separate voices, a request for information or commands specific to the items on the list can be performed specifically on one or more items associated with the corresponding voice (and not on other items).

[0032] Some voice interfaces don't know in advance how many voices will be using the interface simultaneously. In some cases, it might be a single voice. In other cases, it might be several voices. For such voice interfaces, it might be helpful to be able to: (a) identify voices that have interacted with the interface during a conversation, and (b) infer that speech segments were produced during a conversation by voices that had not previously interacted with the interface. In the latter case, the interface can add the new voice vector to the list of voices known during the conversation.

[0033] One way to achieve both identification of recognized speech and inference of new speech is to store an aggregate vector for each speech. For example, it could be the aggregate of the d-vectors computed over all frames of the most recent speech segment attributed to the speech. It could also be an aggregate across multiple segments attributed to the same speech.

[0034] A speech vector is then calculated for each new segment. If the new speech vector is within a threshold distance of any other known speech vector for the conversation, the speech is recognized. If the new speech vector is within a threshold distance of multiple known speech vectors for the conversation, it is recognized as the known speech closest to the newly calculated speech vector.

[0035] If the newly calculated speech vector is not within a threshold distance of any speech vector associated with speech already present in the conversation, the speech interface can infer that the segment is from new speech and, in response, instantiate another speech with the calculated speech vector within the speech already known in the conversation.

[0036] In some implementations of a voice interface, when a voice vector calculated for a segment is within a threshold distance of more than one known voice vector, if the understood request would have different effects depending on which of the multiple voices spoke it, the voice interface can perform a disambiguation function, such as outputting a message requesting the user to try again or specifically asking which possible result is correct, rather than selecting the closest known voice vector as the correct one. Such a message might be a request, "Did you mean the first or the second?" Then, if the next voice segment clearly indicates the first or the second, the interface would store the request in memory and respond accordingly based on the next voice segment.

[0037] Some implementations can handle multiple voices speaking to each other. This is achieved by classifying the recognized text into one of three types. Text can be recognized and relevant, for example, by matching word patterns. If the speech recognition has a high confidence score, but the text of the segment does not match the pattern, the text is recognized but not relevant. If the speech recognition of a segment has a low confidence score, either in whole or in part, the text may be uninterpretable.

[0038] Some voice interfaces, such as those built into personal mobile devices or smart home speakers, have a known set of possible users. They are able to associate speech with a specific user identity. Thus, the voice interface can respond to a match with a known voice vector of the user by speaking directly to the user, even by name. Such voice interfaces can also store and access information such as the user's personal preferences and habits.

[0039] Example scenario

[0040] Figure 2 This scenario shows two users each ordering burgers using a multi-participant voice ordering interface, divided into four steps. As they speak, a segmentation function divides the audio into multiple segments, each with a voice request. In request 0, someone initiates the conversation by saying the phrase "We want two burgers." Automatic speech recognition (ASR) receives the audio and transcribes it into text with the spoken words. In some implementations, other functions besides ASR can be run on the audio.

[0041] Based on request 1, the speech vector calculation function runs on the audio and calculates speech vector 21894786. The ASR runs on the audio and transcribes the words "Put onions on my burger".

[0042] Based on request 2, the speech vector calculation function runs on the audio and calculates speech vector 65516311. The ASR runs on the audio and transcribes the words "I don't want onions on my burger."

[0043] Based on request 3, the speech vector calculation function runs on the audio and calculates speech vector 64507312. The ASR runs on the audio and transcribes the word "I want a tomato".

[0044] Figure 3 The data structure and the changes to the data structure as requests are processed are shown. Each request is matched to a pattern. Some implementations support simple, easily defined patterns, for example, a specific sequence of words and the corresponding function to be performed. Some implementations support patterns with slots so that a sequence of words is matched to any word or a set of specific words in the slot position in the pattern. Some implementations support patterns with complex regular expressions. Some implementations support programmable functions that can match the recognized word sequence. Some implementations take into account the ASR score of each word or complete word sequence recognized from the audio clip.

[0045] In the example scenario, a voice interface is programmed with the patterns needed to recognize a burger order at a restaurant. When a conversation begins, the interface creates a data structure that contains an empty list of items and is able to store, for each item instance, a speech vector and a list of attribute values ​​for that item instance.

[0046] Request 0 has recognized the text "We want two burgers." These words are matched to a pattern that recognizes the words "want" and "burger" and has an optional slot for multiple instances, which is filled with the number 2. In response to understanding the word sequence, the voice interface adds two burger instances to the order data structure.

[0047] Request 1 has speech vector 21894786 and the recognized text "Put onions on my burger." These words are matched to a pattern that has slots for a list of known burger attributes. One such attribute is onion, which can have a Boolean value of "yes" or "no." The word "onion" followed by the phrase "on my burger" causes the speech interface to add the attribute onion to the data structure associated with instance burger 0 and assign it the value "yes." The interface stores speech vector 21894786 associated with burger 0.

[0048] Request 2 has speech vector 65516311 and the recognized text "I don't want onions on my burger." These words are matched to a pattern that has the word "no" followed by a slot for a list of known burger attributes, including onions like request 1.

[0049] The interface searches the list of burgers for an instance with an associated speech vector whose cosine distance is within a threshold distance of the speech vector for request 2. The interface finds only one burger with a speech vector: 21894786. This is a large cosine distance in speech vector space from the speech vector for request 2. Therefore, the speech interface infers that request 2 corresponds to a different burger than any other burger in the list. Because the speech vector for request 2 is greater than the threshold distance from the speech vector associated with any item in the list, the speech interface can also infer that the speech for request 2 comes from a different user than the user who spoke previously in the ordering session.

[0050] Because the word from request 2 matches the pattern, the speech interface adds the onion attribute to the data structure associated with hamburgers, and no speech vector is yet associated with hamburger 1, the speech interface assigns the onion attribute a value of "no regarding hamburger 1." The speech interface also stores speech vector 65516311 associated with hamburger 1 from request 2.

[0051] Request 3 has speech vector 64507312 and the recognized text "I want tomatoes." These words are matched to a pattern that has the word "want" followed by a slot for a list of known burger attributes, including tomato as an attribute.

[0052] The interface searches the burger list for an instance with an associated speech vector whose cosine distance is within a threshold distance of the speech vector for request 3. The interface has two burgers. Speech vector 65516311 is stored associated with burger 1. The cosine distance between the speech vector for burger 1 and the speech vector for request 3 is within the threshold. Therefore, the speech interface infers that request 3 comes from the same person who made the request associated with burger 1. Therefore, the speech interface adds the tomato attribute to burger 1 and assigns it a value of "yes."

[0053] Although two request segments rarely have identical voice vectors computed due to random variations and differences in the analyzed phonemes between them, by identifying that the requested voice vector is close to the voice vector stored for the burger instance, the voice interface is actually able to specifically configure the properties of the same burger as requested by the same speaker across different requests. Conversely, by identifying large distances between voice vectors, the voice interface is able to tailor the properties of the burger to different users.

[0054] Figure 4A flow chart of a method for recognizing multi-participant voice orders is shown. The method begins at the start of a voice ordering session and instantiates two items of a particular type 40. In a next step, the method receives a first spoken request 41 to modify an item. In response, the method modifies the first item instance 42. It also calculates and stores a first speech vector associated with the first item 43. The first speech vector is stored in computer memory 44. Next, the method receives a second spoken request 45 to modify the item. The method then calculates a second speech vector 46. It proceeds to compare the second speech vector to the first speech vector 47. If the speech vectors match because they are within a threshold distance of each other, the method proceeds to modify the first item instance 48. If the speech vectors do not match, the method proceeds to modify the second item instance.

[0055] Figure 5 Two people with different voices are shown using a self-service kiosk with a voice interface to order two burgers, one with onions and the other with tomatoes. Figure 4 The method described in Figure 2 and Figure 3 The device of the described example scenario. The Hamburg self-service terminal implements these methods by utilizing a computer processor in a system-on-chip to execute software stored in a memory device.

[0056] Device Implementation

[0057] Figure 6A A flash memory chip 69 is shown. This is an example of a non-transitory computer readable medium capable of storing code that, if executed by a computer processor, causes the computer processor to perform a method for multi-participant voice ordering. Figure 6B A system on chip 60 is shown. It is packaged in a ball grid array package for surface mounting to a printed circuit board.

[0058] Figure 6CA functional diagram of a system on chip 60 is shown. It includes an array of multiple computer processor cores (CPUs) 61 and graphics processor cores (GPUs) 62, which are connected to a dynamic random access memory (DRAM) interface 64 and a flash memory interface 65 via an on-chip network 63. The DRAM interface 64 is used to store information such as item attribute values ​​and speech vectors, and the flash memory interface 65 is used to store and read software instructions for the CPU and GPU. The on-chip network also connects the functional blocks to a display interface 66, which can output a display for a voice ordering kiosk, for example. The on-chip network also connects the functional blocks to an I / O interface 67 for connecting to microphones and other types of devices (such as speakers, touch screens, cameras, and tactile vibrators) for interacting with the user. The on-chip network also connects the functional blocks to a network interface 68, which allows the processor and its software to execute API calls to cloud servers or other connected devices.

[0059] Summarize

[0060] A system is provided for recognizing spoken commands, queries, or other types of utterances from multiple users and identifying the user who uttered the utterance based on characteristics of their voice. The system then modifies one of multiple instances of a type of item, where the modified instance corresponds to the user who was identified. The system includes speech recognition configured to transcribe the spoken commands. The system also includes speech discrimination that characterizes the speech of the utterance and can identify the user from several other users based on the user's voice characteristics.

[0061] In one embodiment, the system is configured to display the modified instance of the item on a display device (e.g., a computer screen or a mobile device). The system can also be configured to generate audio or visual output corresponding to the modified instance of the item, such as synthesized speech that speaks the name of the item or a visual representation of the item.

[0062] In another embodiment, the system is configured to receive input from a user, such as a voice command or touch gesture indicating a preference for a particular instance of an item. The system can then select the particular instance of the item and modify it based on the recognized voice characteristics of the user's spoken command.

[0063] The system can also be configured to learn from past usage and adjust the selection and modification of items based on individual user preferences or the context of the spoken command. For example, if a user frequently selects a certain instance of an item, the system can automatically select that instance in future instances of the user's spoken command.

[0064] In yet another embodiment, the system can be configured to incorporate additional information (such as context or personal preferences) into the selection and modification of items for each user. For example, the system can take into account the user's location or time of day when selecting and modifying item instances for each user.

[0065] The system provides a convenient and intuitive way to interact with spoken commands and modify item instances based on the speaker's voice characteristics, thereby providing a personalized experience for multiple users. The system can be implemented in various settings, such as in educational or entertainment applications, or in voice-controlled personal assistants.

Claims

1. A computer-implemented method comprising: receiving a first spoken utterance specifying a type of item to be modified; calculating a first speech feature vector based on the first spoken utterance; responsive to the first spoken utterance, modifying a first item of a specified type; storing the first speech feature vector in association with the first item; receiving a second spoken utterance for modifying an item of the specified type; calculating a second speech feature vector based on the second spoken utterance; In response to determining that the second speech feature vector differs from the first speech feature vector by more than a threshold, modifying a second item of the specified type; and An indication of the modified status of the first item and the modified status of the second item is output.

2. The method according to claim 1, wherein The speech feature vector is calculated by the following operations: Identify the onset of speech activity in audio; performing automatic speech recognition on the audio to identify words; detecting utterance completion by matching recognized words to word patterns; as well as The speech feature vector is calculated as a vector of aggregated speech features in the audio between the onset of speech activity and the completion of the utterance.

3. The method according to claim 1, wherein Modifying the second item is performed in response to the second spoken utterance being received within a period of time after receiving the first spoken utterance, the period of time being less than thirty seconds.

4. The method according to claim 1, wherein The first item and the second item are members of a list.

5. The method according to claim 1, wherein Determining that a difference between the second speech feature vector and the first speech feature vector is greater than a threshold includes: Computing distances between points represented by vectors in multidimensional space; and It is determined that a distance between the second speech feature vector and the first speech feature vector in the vector space is greater than a threshold.

6. A computer-implemented method comprising: receiving a first spoken utterance specifying a type of item to be ordered or modified; calculating a first speech feature signature based on the first spoken utterance; ordering or modifying a first item of a specified type in response to the first spoken utterance; storing the first speech feature signature in association with the first item; receiving a second spoken utterance for ordering or modifying an item of the specified type; calculating a second speech feature signature based on the second spoken utterance; as well as In response to determining that the second voice characteristic signature differs from the first voice characteristic signature by more than a threshold, a second item of the specified type is ordered or modified.

7. The method according to claim 6, further comprising the steps of: An indication of the modified status of the first item and the modified status of the second item is output.

8. The method of claim 6, wherein: The step of calculating a first speech feature signature based on the first spoken utterance includes the step of calculating a first speech feature vector based on the first spoken utterance, and wherein the step of calculating a second speech feature signature based on the second spoken utterance includes the step of calculating a second speech feature vector based on the second spoken utterance.

9. The method of claim 8, wherein: The speech feature vector is calculated by the following operations: Identify the onset of speech activity in audio; performing automatic speech recognition on the audio to identify words; detecting completion of the utterance by matching the recognized words to word patterns; as well as The speech feature vector is calculated as a vector of aggregated speech features in the audio between the onset of speech activity and the completion of the utterance.

10. The method of claim 8, wherein: Determining that a difference between the second speech feature vector and the first speech feature vector is greater than a threshold includes: Computing distances between points represented by vectors in multidimensional space; and It is determined that a distance between the second speech feature vector and the first speech feature vector in the vector space is greater than a threshold.

11. The method according to claim 6, wherein: Modifying the second item is performed in response to the second spoken utterance being received within a period of time after receiving the first spoken utterance, the period of time being less than thirty seconds.

12. The method of claim 6, wherein: The first item and the second item are members of a list.

13. A computer-implemented method comprising: calculating a first speech feature signature based on a received first spoken utterance specifying a first item of a specified type to be ordered or modified; storing the first speech feature signature in association with the first item; calculating a second voice feature signature based on a received second spoken utterance, the second spoken utterance being used to order or modify an item of the specified type; as well as In response to determining that the second voice characteristic signature differs from the first voice characteristic signature by more than a threshold, a second item of the specified type is ordered or modified.

14. The method of claim 13, further comprising the steps of: An indication of the modified status of the first item and the modified status of the second item is output.

15. The method of claim 13, wherein: The step of calculating a first speech feature signature based on the first spoken utterance includes: the step of calculating a first speech feature vector based on the first spoken utterance, and wherein the step of calculating a second speech feature signature based on the second spoken utterance includes: the step of calculating a second speech feature vector based on the second spoken utterance.

16. The method of claim 15, wherein: The speech feature vector is calculated by the following operations: Identify the onset of speech activity in audio; performing automatic speech recognition on the audio to identify words; detecting completion of the utterance by matching the recognized words to word patterns; as well as The speech feature vector is calculated as a vector of aggregated speech features in the audio between the onset of speech activity and the completion of the utterance.

17. The method of claim 15, wherein: Determining that a difference between the second speech feature vector and the first speech feature vector is greater than a threshold includes: Computing distances between points represented by vectors in multidimensional space; and It is determined that a distance between the second speech feature vector and the first speech feature vector in the vector space is greater than a threshold.

18. The method of claim 13, wherein: Modifying the second item is performed in response to the second spoken utterance being received within a period of time after receiving the first spoken utterance, the period of time being less than thirty seconds.

19. The method of claim 13, wherein: The first item and the second item are members of a list.

Images