Multi-person voice ordering

Abstract

The speech interface recognizes spoken utterances from multiple users. The speech interface responds to the utterances in ways such as modifying attributes of item instances. The speech interface calculates a speech vector for each utterance and associates it with the item instance to be modified. For subsequent utterances with an exactly matching speech vector, the speech interface modifies the same instance. For subsequent utterances with a speech vector that does not exactly match the one stored for any item instance, the speech interface modifies a different item instance.

Description

Technical Field

[0001] Priority Claim This application claims the benefit of U.S. patent application Ser. No. 18 / 391,886, filed Dec. 21, 2023, which claims the priority of U.S. Provisional Application No. 63 / 476,928, filed Dec. 22, 2022. These applications are hereby incorporated by reference herein.

Background Art

[0002] Background Currently, computerized speech recognition systems are used in various situations where voice input is received from multiple users, but have met with limited success. One example is using a speech recognition system to take food orders. One problem is distinguishing between different speakers. A second problem is recognizing the voices of multiple users. Given these problems, speech recognition systems have not yet been successful in their use in these scenarios. [[ID=...]]

Summary of the Invention

[0003] Summary In the following specification, systems and methods for recognizing utterances spoken by multiple speakers and distinguishing between speakers by voice are described. Such a system can then change one of a plurality of items of the same type. The item changed here corresponds to which user was identified.

[0004] This summary is provided to introduce, in simplified form, a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to embodiments that solve any or all of the disadvantages shown in the background.

Brief Description of the Drawings

[0005] [Figure 1] This diagram shows the segmentation of audio into request utterances. [Figure 2] This diagram shows the speech processing involved in calculating the speech vector and recognizing the spoken request. [Figure 3] This diagram shows the fast food order data structure, which changes along with the sequence of utterances. [Figure 4] This is a flowchart of the process of changing one or another instance of an item type based on voice input. [Figure 5] This diagram shows a user placing an order by voice at a fast food kiosk. [Figure 6A] This is a diagram showing a flash RAM chip. [Figure 6B] This is a diagram showing a system on a chip. [Figure 6C] This is a functional diagram of a system-on-a-chip. [Modes for carrying out the invention]

[0006] Detailed explanation Various devices presenting computerized voice interfaces, networked device systems, API-controlled cloud services, etc., can receive audio, detect if it contains speech, infer a transcription of the speech, and understand the transcription as a query or command. Such a voice interface can then act on the understood query or command by taking action, retrieving information, or determining that these actions are impossible, and can then respond in a form of information that may be useful to the user.

[0007] Some audio interfaces receive audio directly from a microphone, or from digital sampling of the air pressure waves that operate the microphone. Some audio interfaces receive digital audio from a remote device as direct digital samples, as frames of frequency-domain conversions of such sampled digital signals, or as compressed representations thereof. Examples of audio representation formats include WAV, MP3, and Speex.

[0008] Voice interfaces on devices such as mobile phones output information directly on the display screen, through a speaker using synthesized speech, through a haptic vibrator, or using other actuator functions of the phone. Some voice interfaces, such as APIs hosted by cloud servers, output information as response messages corresponding to request messages. Output information may include text or spoken answers to questions, confirmations that an action or other activated function has started or completed, or the status of data stored in an interface, device, system, server, or any of these.

[0009] One example is a voice interface for ordering food from a restaurant. Such an interface operates in a session that ends with payment and begins with the following user interactions. In some cases, the user interaction begins when the interface detects that a person has uttered a specific wake phrase or senses that a person has manually interacted with the device. In other cases, the voice interface continuously performs speech recognition and, for words recognized with sufficient confidence, matches those words to patterns that correspond to an understanding of the intent of the person speaking the words.

[0010] To infer the understanding of spoken words in a continuous sequence of audio, it is necessary to segment the audio. This can be done in several ways. One method is to perform a speech activity detection function on the audio and determine the start of a segment when no further speech activity is detected for a specified period, considering that speech activity up to the point where no further speech activity is detected has occurred.

[0011] Another method for segmenting audio is to recognize semantically complete word sequences. This can be done by comparing the most recent word sequence in a buffer with a pattern. If the semantically complete pattern is a prefix to another pattern, a delay of approximately 1 to 10 seconds can be implemented after the match, and if a match to a longer pattern occurs during that delay, the first match can be discarded.

[0012] To avoid incorrect matching when the end of an earlier sequence and the beginning of an unrelated later sequence match a pattern, it's possible to reset the word sequence after a period of 5 to 30 seconds, during which no new words are added to the buffer. Therefore, an item will only be modified by a command if it is received within a relatively short period of time.

[0013] For semantic segmentation, it may be useful to tag each word with the approximate real time when it began to be recognized from the input audio, and / or the approximate real time when its pronunciation recognition ended. The approximate start time of recognition of the first word in a semantically complete word sequence is the start time of the segment. The approximate end time of recognition of the last word in a semantically complete sequence is the end time of recognition.

[0014] Figure 1 shows a schematic diagram of audio segmentation. The segmentation function is performed on the audio stream. This can be done continuously in real time, incrementally, or offline for non-real-time analysis. The segmentation function calculates the start and end times of segments in the audio stream and outputs separate audio segments. In Figure 1, each audio segment contains a request to the audio interface.

[0015] Audio vector One method for conducting multi-person voice orders is to characterize voices numerically to distinguish them. Calculating voice values ​​along a single dimension may allow for gender-based discrimination, but this may be insufficient for distinguishing between individuals with similar voices. Multiple numerical vectors representing the sound of a voice along different dimensions provide greater accuracy in voice characterization and discrimination. This can, in many cases, even distinguish between the voices of identical twins.

[0016] Choosing the correct dimensions improves accuracy. Selecting specific dimensions, such as gender, age, and even regional accent estimation, can also be effective. However, it may be more accurate to use machine learning on large datasets with highly diverse speech, training the multidimensional space to maximize the variance of the speech vectors computed in the space.

[0017] With the appropriate multidimensional vector space, it is possible to characterize speech within audio as points represented by vectors. One method for doing this is to continuously compute d-vectors for each frame of the audio or for a relatively small number of samples. One method for computing d-vectors using deep neural networks (DNNs) is described in the paper DEEP NEURAL NETWORKS FOR SMALL FOOTPRINT TEXT-DEPENDENT SPEAKER VERIFICATION by Ehsan Variani, Xin Lei, Erik McDermott, Ignacio Lopez Moreno, and Javier Gonzalez-Dominguez.

[0018] One method for calculating the speech vector for an entire segment is to aggregate the d-vectors calculated for each frame of audio from the beginning to the end of the segment. Aggregation can be done in various ways, such as calculating the average across frames for each dimension. In some cases, it may also be useful to exclude d-vectors calculated for frames where the energy is dispersed across the spectrum, such as during the pronunciation of the phonemes "s" and "sh". D-vectors calculated in such frames may, in some cases, add noise that reduces the accuracy of the aggregated speech vector calculation. A continuous frame-by-frame approach to calculating the speech vector during a segment has the advantage that the CPU cycle demand is relatively constant regardless of the length of the speech segment.

[0019] Another way to compute the audio vector for the entire segment is to compute the x-vector for the entire segmented utterance when the end of the segment is detected. One approach to computing the x-vector using a DNN is described in the paper X-VECTORS, ROBUST DNN EMBEDDINGS FOR SPEAKER RECOGNITION by David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. After the segment has been fully recognized, it is possible to compute the x-vector once for each segment. In some cases, it is more energy-efficient to buffer the audio while performing segmentation and then to start a faster, higher-performance CPU for a short time to compute the x-vector for the entire length of the buffered audio data of the segment.

[0020] The computation of the audio vector can be used instead of or in addition to other segmentation features. One approach is to compute a relatively short-term d-vector and a longer aggregated d-vector. These are similar as long as the same voice is speaking. These diverge when a voice change occurs in the audio. The sum of the dimension-by-dimension differences between the short-term and long-term average d-vectors indicates the segment transition at or slightly before the point where a detectable divergence begins.

[0021] Multiple voices For the purpose of multi - person participation voice ordering, it can be useful to segment the voice, calculate voice vectors for the segments, and then apply pattern matching or other forms of natural language understanding to act according to the voice request. Next, the action in response to the voice request can be conditioned on which of the multiple voices made the request. For example, if items that are members of a list are associated with separate voices, information requests or commands specific to an item on the list are particularly executed for one or more items associated with the corresponding voice and may not be executed for other items.

[0022] Some voice interfaces do not know in advance how many people's voices will use the interface simultaneously. In some scenarios, it can be a single voice. In other scenarios, it can be several voices. For such voice interfaces, it can be useful to be able to (a) distinguish between voices that have interacted with the interface during a session, and (b) infer that an utterance segment is by a voice that has not yet interacted with the interface during the session. In the latter situation, the interface can add a new voice vector to the list of voices known during the session.

[0023] One way to perform discrimination between recognized voices and inference of new voices is to store an aggregated vector for each voice. This can be, for example, an aggregation of d - vectors calculated over all frames of the most recent utterance segments belonging to that voice. This can also be an aggregation over multiple segments belonging to the same voice.

[0024] Next, an audio vector is calculated for each new segment. If the new audio vector is within a threshold distance of any other known audio vector for the session, the audio is identified. If the new audio vector is within a threshold distance of multiple known audio vectors for the session, it is identified as the known audio closest to the newly calculated audio vector.

[0025] If a newly calculated speech vector is not within a threshold distance of any speech vector already associated with a speech in the session, the speech interface can infer that the segment is from a new speech and, in response, use the calculated speech vector to instantiate another speech within the known speeches for the session.

[0026] In some embodiments of the voice interface, when a calculated voice vector for a given segment falls within a threshold distance of two or more known voice vectors, instead of selecting the closest known voice vector as correct, the voice interface may perform a resolution function if the understood request has different effects depending on which of the multiple voices uttered it. This can be done by requesting the user to retry or by outputting a message that specifically asks which of the possible outcomes is correct. Such a message might be a request like, "Did you mean the first or the second?" The interface then stores the request in memory and responds accordingly based on the next voice segment if the next voice segment clearly identifies either the first or the second.

[0027] Some embodiments can handle multiple voices speaking at the same time. This can be done by classifying the recognized text into one of three types. The text may be recognized and become relevant, for example, by matching word patterns. The text may be recognized but become irrelevant if the speech recognition has a high confidence score but the text in the segment does not match any patterns. The text may be uninterpretable if the speech recognition across segments has a low confidence score overall or partially.

[0028] Some voice interfaces, such as those embedded in personal mobile devices or home smart speakers, have a known set of possible users. These interfaces are capable of associating voice with a specific user identity. Thus, a voice interface can even directly address a user by name in response to a match with a known voice vector. Such voice interfaces can also remember and access information such as the user's personal preferences and habits.

[0029] Exemplary Scenario Figure 2 illustrates a four-step scenario in which two users each order a hamburger using a multi-user voice ordering interface. As the users speak, a segmentation function divides the audio into segments, each containing a voice request. Request 0 initiates the session by one person saying the phrase "we want two burgers." Automatic speech recognition (ASR) receives this audio and transcribes it into text using the spoken words. In some embodiments, functions other than ASR may be performed on this audio.

[0030] In request 1, the speech vector calculation function is performed on the audio, and the speech vector 21894786 is calculated. The ASR is then performed on the audio, and the word "put onions on my burger" is transcribed.

[0031] In request 2, the speech vector calculation function is performed on the audio, and the speech vector 65516311 is calculated. The ASR is performed on the audio, and the word "no onions on my burger" is transcribed.

[0032] In Requirement 3, the speech vector calculation function is performed on the audio, and speech vector 64507312 is calculated. ASR is performed on the audio, and the word "I do want tomatoes" is transcribed.

[0033] Figure 3 shows the data structure and its changes during the processing of a request. Each request is matched against a pattern. Some embodiments support simple and easily definable patterns, such as specific word sequences and corresponding functions to perform. Some embodiments support patterns with slots, allowing the pattern to match a word sequence with any word or specific set of words at a slot position within the pattern. Some embodiments support patterns with complex regular expressions. Some embodiments support programmable functions that can match recognized word sequences. Some embodiments consider an ASR score for each word or entire word sequence recognized from an audio segment.

[0034] In an exemplary scenario, the voice interface is programmed with patterns necessary to recognize a hamburger order at a restaurant. When a session begins, the interface generates a data structure with an empty item list and the ability to store, for each item instance, the voice vector and attribute value list of that item instance.

[0035] Request 0 recognized the text "we want two burgers". These words were recognized as "want" and "burger" and matched with a pattern that has an optional slot for the number of instances, which is filled with the number 2. In response to understanding the word sequence, the voice interface adds two hamburger instances to the order data structure.

[0036] Request 1 has the speech vector 21894786 and recognized the text "put onions on my burger". These words are matched with a pattern that has slots in a list of known burger attributes. One such attribute is onions, which can have a boolean value of "yes" or "no". The word "onions" followed by the word "on my burger" causes the speech interface to add the attribute onions to the data structure, associating it with instance burger 0, and assign the value "yes" to the attribute onions. The interface stores the speech vector 21894786 as associated with burger 0.

[0037] Request 2 had the audio vector 65516311 and recognized the text "no onions on my burger". These words are matched to a pattern that has slots in a list of known burger attributes, including "onions" followed by "no", similar to Request 1.

[0038] The interface scans the list of hamburgers to find instances that have an associated voice vector for request 2, where the cosine distance of the voice vector for request 2 is within a threshold distance. The interface has only one hamburger with a voice vector, which is 21894786. This has a large cosine distance in voice vector space from the voice vector for request 2. Therefore, the voice interface infers that request 2 corresponds to a hamburger that is not any hamburger in the list. Since the voice vector for request 2 is also beyond the threshold distance from any voice vector associated with any item in the list, the voice interface can also infer that the voice for request 2 is from a user that is not any user that has spoken in the ordering session so far.

[0039] By matching the words in request 2 to the pattern, the voice interface associates them with hamburgers and adds the attribute `onions` to the data structure. Since no voice vector is yet associated with burger 1, the voice interface assigns the value "no in relation to burger 1" to the attribute `onions`. The voice interface also associates the voice vector 65516311 from request 2 with burger 1 and stores it.

[0040] Request 3 has the speech vector 64507312 and recognized the text "I do want tomatoes". These words are matched to a pattern that has a slot in a list of known hamburger attributes, with "want" followed by "tomatoes" as the attribute.

[0041] The interface scans the list of hamburgers to find instances with associated voice vectors where the cosine distance of the voice vector of request 3 is within a threshold. The interface has two hamburgers. Voice vector 65516311 is stored associated with burger 1. The cosine distance between the voice vector of burger 1 and the voice vector of request 3 is within a threshold. Therefore, the voice interface infers that request 3 is from the same person who made the request associated with burger 1. Therefore, the voice interface adds the attribute "tomatoes" to burger 1 and assigns it the value "yes".

[0042] Due to random variations and differences in the phonemes parsed between request segments, it is rare for two segments to have exactly the same speech vector calculation. However, by recognizing that a request has a speech vector similar to one stored in association with a hamburger instance, a speech interface can, in practice, construct the same hamburger attributes, especially those requested in different requests by the same speaker. Conversely, by recognizing large distances between speech vectors, a speech interface can customize the hamburger attributes separately for different users.

[0043] Figure 4 shows a flowchart of a method for recognizing a multi-person voice order. The method begins when a voice order session starts and instantiates two items of a specific type (40). In the next step, the method receives a first spoken request to change an item (41). In response, the method changes the first item instance (42). The method also calculates a first voice vector and stores it associated with the first item (43). The first voice vector is stored in computer memory (44). Next, the method receives a second spoken request to change an item (45). Next, the method calculates a second voice vector (46). The method proceeds to compare the second voice vector with the first voice vector (47). If the voice vectors match by being within a threshold distance of each other, the method proceeds to change the first item instance (48). If the voice vectors do not match, the method proceeds to change the second item instance.

[0044] Figure 5 shows two people with different voices ordering two hamburgers, one with onions and the other with tomatoes, using a kiosk with a voice interface. The kiosk is a device that implements the methods shown in Figure 4 and performs the example scenarios described in relation to Figures 2 and 3. The hamburger kiosk implements these methods by executing software stored in a memory device using a computer processor on a system-on-chip.

[0045] Device Implementation Figure 6A shows a flash memory chip 69, an example of a non-temporary computer-readable medium that, when executed by a computer processor, can store code that causes the computer processor to execute a method of multi-person voice ordering. Figure 6B shows a system-on-a-chip 60, which is packaged in a ball grid array package for surface mounting to a printed circuit board.

[0046] Figure 6C shows a functional diagram of the system-on-chip 60. This includes an array of multiple computer processor cores (CPUs) 61 and graphics processor cores (GPUs) 62, and is connected by a network-on-chip 63 to a dynamic random access memory (DRAM) interface 64 for storing information such as item attribute values ​​and voice vectors, as well as a flash memory interface 65 for storing and reading software instructions for the CPUs and GPUs. The network-on-chip also connects functional blocks to a display interface 66 that can output displays, for example, for a voice ordering kiosk. The network-on-chip also connects functional blocks to an I / O interface 67 for connecting microphones and other types of devices for user interaction, such as speakers, touchscreens, cameras, and haptic vibrators. The network-on-chip also connects functional blocks to a network interface 68 that allows the processors and their software to make API calls to cloud servers or other connected devices.

[0047] summary A system is provided that recognizes spoken commands, inquiries, or other types of utterances from multiple users and identifies the user who made the utterance based on the characteristics of their voice. The system then modifies one of several instances of an item type corresponding to which the modified instance is identifying the user. The system includes speech recognition configured to transcribe spoken commands. The system also includes speech discrimination that can characterize the voice of an utterance and identify a user from among several users based on their voice characteristics.

[0048] In one embodiment, the system is configured to display an instance of the modified item on a display device such as a computer screen or a mobile device. The system may also be configured to generate audio or visual output corresponding to the modified item instance, such as synthesized speech uttering the item name or a visual representation of the item.

[0049] In another embodiment, the system is configured to receive user input, such as a voice command or touch gesture indicating a preference for a particular item instance. The system can then select the specified item instance and modify it according to the voice characteristics of the identified user's spoken command.

[0050] The system can also be configured to learn from past usage and adjust item selection and modification based on individual user preferences or the context of spoken commands. For example, if a user frequently selects a particular item instance, the system can automatically select that instance in future instances of commands spoken by that user.

[0051] In yet another embodiment, the system can be configured to incorporate additional information, such as context or personal preferences, into item selection and modification for each user. For example, the system can take into account the user's location or time when selecting and modifying item instances for each user.

[0052] The system provides a convenient and intuitive way to interact with spoken commands and modify item instances based on the speaker's voice characteristics, enabling a personalized experience for multiple users. The system can be implemented in a variety of environments, such as educational or entertainment applications, or voice-controlled personal assistants.

Claims

1. Receiving a first spoken utterance that specifies the type of item to be changed, The first speech feature vector is calculated from the first spoken utterance, In response to the first spoken utterance, modify the first item instance containing the first item in the specified type of data structure, The first audio feature vector is stored in association with the first item, Receiving a second spoken utterance, The calculation of a second speech feature vector from the second spoken utterance, In response to determining that the second speech feature vector and the first speech feature vector have a difference exceeding a threshold, the second item instance containing the second item is modified in the specified type of data structure, Outputting indications for the modified first item instance and the modified second item instance, The way a computer performs actions, including [specific actions].

2. The speech feature vector is, Identifying the start of speech activity within audio, The process involves performing automatic speech recognition on the aforementioned audio to recognize words, The completion of the utterance is detected by matching the recognized word with a word pattern, The process involves calculating the speech feature vector as a vector of aggregated speech features in the audio between the start of the speech activity and the completion of the utterance, The method according to claim 1, calculated by...

3. The method according to claim 1, wherein the modification of the second item instance is performed in response to the second spoken utterance being received within a period of reception of the first spoken utterance, the period being less than 30 seconds.

4. The method according to claim 1, wherein the first item and the second item are members of a list.

5. Determining that the second speech feature vector and the first speech feature vector have a difference exceeding a threshold means that Calculating the distance between points represented by the second and first speech feature vectors in a multidimensional space, It is determined that the second speech feature vector and the first speech feature vector have a distance exceeding a threshold in the multidimensional space, The method according to claim 1, including the method described in claim 1.

6. Receiving a first spoken utterance specifying the type of item to be ordered or modified, The first speech feature vector is calculated from the first spoken utterance, In response to the first spoken utterance, ordering or modifying a first item instance containing the first item in the specified type of data structure, The first audio feature vector is stored in association with the first item, Receiving a second spoken utterance, The calculation of a second speech feature vector from the second spoken utterance, In response to determining that the second speech feature vector and the first speech feature vector have a difference exceeding a threshold, the second item instance containing the second item in the specified type of data structure is ordered or modified. The way a computer performs actions, including [specific actions].

7. The method according to claim 6, further comprising the step of outputting indications for the modified first item instance and the modified second item instance.

8. The speech feature vector is, Identifying the start of speech activity within audio, The process involves performing automatic speech recognition on the aforementioned audio to recognize words, The completion of the utterance is detected by matching the recognized word with a word pattern, The process involves calculating the speech feature vector as a vector of aggregated speech features in the audio between the start of the speech activity and the completion of the utterance, The method according to claim 6, calculated by...

9. Determining that the second speech feature vector and the first speech feature vector have a difference exceeding a threshold means that Calculating the distance between points represented by the second and first speech feature vectors in a multidimensional space, It is determined that the second speech feature vector and the first speech feature vector have a distance exceeding a threshold in the multidimensional space, The method according to claim 6, including the method described in claim 6.

10. The method according to claim 6, wherein the modification of the second item instance is performed in response to the second spoken utterance being received within a period of reception of the first spoken utterance, the period being less than 30 seconds.

11. The method according to claim 6, wherein the first item and the second item are members of a list.

12. Calculating a first speech feature vector from a first spoken utterance received, specifying a first item of a specified type to be ordered or modified, The first audio feature vector is stored in association with the first item, To calculate a second speech feature vector from the second received spoken utterance, In response to determining that the second speech feature vector and the first speech feature vector have a difference exceeding a threshold, the second item instance containing the second item in the specified type of data structure is ordered or modified. The way a computer performs actions, including [specific actions].

13. The method according to claim 12, further comprising the step of outputting an indication of the modified second item instance.

14. The speech feature vector is, Identifying the start of speech activity within audio, The process involves performing automatic speech recognition on the aforementioned audio to recognize words, The completion of the utterance is detected by matching the recognized word with a word pattern, The process involves calculating the speech feature vector as a vector of aggregated speech features in the audio between the start of the speech activity and the completion of the utterance, The method according to claim 12, calculated by...

15. Determining that the second speech feature vector and the first speech feature vector have a difference exceeding a threshold means that Calculating the distance between points represented by the second and first speech feature vectors in a multidimensional space, It is determined that the second speech feature vector and the first speech feature vector have a distance exceeding a threshold in the multidimensional space, The method according to claim 12, including the method described in claim 12.

16. The method according to claim 12, wherein the modification of the second item instance is performed in response to the second spoken utterance being received within a period of reception of the first spoken utterance, the period being less than 30 seconds.

17. The method according to claim 12, wherein the first item and the second item are members of a list.

Images